Getting structured output from LLMs

You're building an agent that needs to extract a decision, a confidence score, and a list of entities from an LLM response. You can't just regex the output. You need it typed, validated, and reproducible. The LLM keeps hallucinating extra fields or returning malformed JSON. Your evals are passing but production is throwing parsing errors at 3am.

This is the structured output problem. It's not hard, but the solution space has real tradeoffs and failure modes that matter at scale.

The core problem

LLMs generate text. You need data. There's a gap between what the model outputs and what your application can consume. That gap is where bugs live.

You could prompt the model to return JSON and parse it. This works sometimes. It also fails silently—the model returns valid JSON that doesn't match your schema, or returns text that isn't JSON at all. You can't rely on it.

The straightforward approach: use a tool that enforces schema. OpenAI's JSON mode, Anthropic's structured output, or open libraries like Instructor or Outlines. These constrain the model's generation to a schema you define upfront. The model can't deviate.

But "can't deviate" is doing work. It means the model's token generation is guided by a parser. It means latency increases. It means you need to know your schema before you call the model. It means some edge cases still fail.

Three strategies in production

Strategy 1: Schema enforcement via API

OpenAI's JSON mode and Anthropic's structured output are the easiest entry point. You define a schema (JSON Schema or a TypeScript type), send it in the request, and the API guarantees valid output matching that schema.

from anthropic import Anthropic

client = Anthropic()

response = client.messages.create(
    model="claude-3-5-sonnet-20241022",
    max_tokens=1024,
    messages=[
        {
            "role": "user",
            "content": "Extract entities from: 'Alice works at Acme Corp in NYC'",
        }
    ],
    response_format={
        "type": "json_schema",
        "json_schema": {
            "name": "entities",
            "schema": {
                "type": "object",
                "properties": {
                    "people": {
                        "type": "array",
                        "items": {"type": "string"},
                    },
                    "organizations": {
                        "type": "array",
                        "items": {"type": "string"},
                    },
                    "locations": {
                        "type": "array",
                        "items": {"type": "string"},
                    },
                },
                "required": ["people", "organizations", "locations"],
            },
        },
    },
)

entities = json.loads(response.content[0].text)

Pros: The model literally cannot output invalid JSON or missing fields. No parsing errors in production. The API handles it.

Cons: You're locked to that provider's schema format. Latency is slightly higher because the API is doing constrained sampling. If your schema is wrong—too strict, missing a case—the model will refuse or hallucinate within the constraints. You can't easily fall back to open-source models. And you're paying for the constraint enforcement.

Strategy 2: Parsing with validation

You prompt the model to return JSON, then parse and validate it yourself. Use a library like Pydantic or Zod to define your schema and validate the response.

from pydantic import BaseModel, ValidationError

class EntityExtraction(BaseModel):
    people: list[str]
    organizations: list[str]
    locations: list[str]

response = client.messages.create(
    model="claude-3-5-sonnet-20241022",
    max_tokens=1024,
    messages=[
        {
            "role": "user",
            "content": "Return JSON with keys: people, organizations, locations. Extract from: 'Alice works at Acme Corp in NYC'",
        }
    ],
)

try:
    data = json.loads(response.content[0].text)
    entities = EntityExtraction(**data)
except (json.JSONDecodeError, ValidationError) as e:
    # Handle retry or fallback
    pass

Pros: Works with any model. You control the validation logic. Easy to add custom logic (e.g., post-processing, filtering). Lower latency than constrained sampling.

Cons: The model can return invalid JSON or missing fields. You need to handle retries. You need to decide: do you retry with a different prompt? Do you return a default? Do you fail? Each choice has different cost and latency implications.

Strategy 3: Constrained generation (local)

Libraries like Instructor (Python) or Outlines (Python) sit between you and the model. They intercept the token stream, constrain it to your schema, and validate on the fly. This works with local models and API models.

import instructor
from pydantic import BaseModel

class EntityExtraction(BaseModel):
    people: list[str]
    organizations: list[str]
    locations: list[str]

client = instructor.from_anthropic(Anthropic())

entities = client.messages.create(
    model="claude-3-5-sonnet-20241022",
    max_tokens=1024,
    messages=[
        {
            "role": "user",
            "content": "Extract entities from: 'Alice works at Acme Corp in NYC'",
        }
    ],
    response_model=EntityExtraction,
)

Pros: Schema enforcement without vendor lock-in. Works with local models. Single interface across providers. Handles retries automatically.

Cons: Adds a dependency and latency overhead. The library needs to parse the token stream, which means it's doing work on every token. For constrained decoding on local models, you're doing the sampling yourself, which is slower than a native API. Debugging failures is harder because the constraint logic is in the library.

The decision tree

flowchart TD
    A["Need structured output?"]
    A -->|Yes| B{"Using closed API?"}
    B -->|OpenAI/Anthropic| C{"Schema stable?"}
    C -->|Yes, simple| D["Use API JSON mode"]
    C -->|No, complex| E["Use Instructor"]
    B -->|Local/OSS| F{"Need constraints?"}
    F -->|Yes| G["Use Outlines/Instructor"]
    F -->|No| H["Parse + validate"]
    D --> I["Low latency, no parsing errors"]
    E --> J["Flexible, handles retries"]
    G --> K["Works offline, flexible"]
    H --> L["Simplest, but needs retry logic"]

What actually breaks

Schema too strict. You define a schema that's theoretically correct but the model can't reliably produce it. Example: you require exactly three fields, but the model sometimes returns four because it's being helpful. The constraint fails or the model hallucinates within the constraint. Solution: make your schema permissive, validate downstream.

Latency creep. Constrained sampling is slower than free generation. If you're generating 500-token responses with tight constraints, you're adding 200-500ms of latency per request. At scale, this matters. Solution: measure it. Use simpler schemas. Consider whether you actually need constraints or just validation.

Retry storms. If your schema is wrong and the model can't satisfy it, your retry logic will loop. You'll burn tokens and time. Solution: add a retry limit. Log failures. Have a fallback that returns partial data or a default. Test your schema against your model before deploying.

Vendor drift. OpenAI and Anthropic have different schema syntax. If you switch models, your structured output code breaks. Solution: use a library like Instructor that abstracts this. Or accept that you'll rewrite this layer if you switch.

Hallucination within constraints. Even with schema enforcement, the model can hallucinate data that's valid JSON but semantically wrong. Example: it returns a valid list of people, but the names are made up. Solution: this isn't a structured output problem, it's an evals problem. You need to validate the data, not just the format.

What you should actually do

Start with API-native structured output if you're on OpenAI or Anthropic and your schema is simple and stable. It's the fastest path and the fewest moving parts.

If your schema is complex or evolving, or you need to support multiple models, use Instructor. The abstraction is worth the overhead.

If you're on a local model or need maximum performance, use Outlines for constrained decoding or just parse and validate.

Always add validation after parsing. Schema enforcement is about format, not correctness. Your evals should check that the data makes sense, not just that it's valid JSON.

Log parsing failures separately from inference failures. They're different problems with different solutions.

Test your schema against your actual model before deploying. What works in theory doesn't always work in practice.