JSON Schema

Validates that the agent output matches a JSON schema. Useful for structured output agents.

Config

Supported Schema Properties

• type — string, number, boolean, object, array, null
• required — required object fields
• properties — nested object schemas
• items — array item schema
• minLength / maxLength — string length
• minimum / maximum — number range
• enum — allowed values

Use Cases

• Structured output validation — Verify that agents using JSON mode or function calling return output that matches the expected shape. Catch missing fields, wrong types, or unexpected values before they hit downstream systems.
• API response checking — If your agent generates API payloads, validate that the payload conforms to the target API's schema before it would be sent.
• Tool call validation — Assert that tool-calling agents produce well-formed tool invocations with the correct function name, required arguments, and properly typed parameters.
• Data extraction pipelines — When agents extract structured data from unstructured text (e.g., parsing invoices or resumes), validate that the extracted fields are present and correctly typed.

Examples

Basic structured answer

{
  "schema": {
    "type": "object",
    "required": ["answer", "confidence"],
    "properties": {
      "answer": { "type": "string", "minLength": 1 },
      "confidence": { "type": "number", "minimum": 0, "maximum": 1 },
      "sources": {
        "type": "array",
        "items": { "type": "string" }
      }
    }
  }
}

Tool call format

// Validate that the agent returns a properly formatted tool call
{
  "schema": {
    "type": "object",
    "required": ["function", "arguments"],
    "properties": {
      "function": {
        "type": "string",
        "enum": ["search", "calculate", "lookup"]
      },
      "arguments": {
        "type": "object",
        "required": ["query"],
        "properties": {
          "query": { "type": "string", "minLength": 1 },
          "limit": { "type": "number", "minimum": 1, "maximum": 100 }
        }
      }
    }
  }
}

Nested object with arrays

// Validate a complex product recommendation response
{
  "schema": {
    "type": "object",
    "required": ["recommendations", "total_count"],
    "properties": {
      "recommendations": {
        "type": "array",
        "items": {
          "type": "object",
          "required": ["name", "price", "category"],
          "properties": {
            "name": { "type": "string", "minLength": 1 },
            "price": { "type": "number", "minimum": 0 },
            "category": {
              "type": "string",
              "enum": ["electronics", "clothing", "home", "books"]
            },
            "in_stock": { "type": "boolean" }
          }
        }
      },
      "total_count": { "type": "number", "minimum": 0 },
      "query": { "type": "string" }
    }
  }
}

What Happens on Parse Failure

If the agent output is not valid JSON (e.g., it contains natural language, truncated JSON, or has syntax errors), the evaluator returns a score of 0.0 with reasoning that explains the parse error. The schema validation step is skipped entirely since there is no valid JSON to validate against.

Common parse failures include:

• Output wrapped in markdown code fences (e.g., ```json ... ```) — strip these before evaluation or instruct your agent not to use them
• Trailing commas in objects or arrays
• Single quotes instead of double quotes
• Agent mixing JSON with natural language explanation

Tips

• Start with loose schemas, tighten over time — Begin by validating just type and required fields. Once you have confidence in your agent's output shape, add enum, minLength, and range constraints to catch edge cases.
• Use required strategically — Only mark fields as required if their absence is a true failure. Optional fields let you validate shape without penalizing agents that sometimes omit non-critical data.
• Combine with Contains or Regex — Use JSON Schema to validate structure and a Contains or Regex evaluator to validate specific field values within the same evaluation suite.

Notes

• Uses a lightweight built-in validator (no external dependency)
• Returns 1.0 (valid) or 0.0 (invalid)
• Runs locally with no API calls — execution time is under 1ms