Structured Outputs / Constrained Decoding

Overview

Structured Outputs (also called constrained decoding) force a model to emit text that conforms to a developer-supplied JSON Schema (or grammar / regex). Instead of “please return JSON,” the inference stack masks illegal next tokens so the sampler can only choose tokens that keep the string valid against the schema.

Why it matters in production:

• Deterministic shape for downstream code. Parsers, typed SDKs, and agent tool routers stop dying on missing keys, wrong enums, or trailing commas.
• Fewer retries. JSON mode only guarantees valid JSON; strict schema mode guarantees schema adherence. Retries for format errors become rare.
• Safer agent loops. Tool arguments and extraction payloads become typed contracts; failures shift to content quality, not syntax.
• Programmatic refusals. With OpenAI Structured Outputs, safety refusals are surfaced as a distinct refusal path instead of malformed JSON that crashes your deserializer.

Mental model: JSON mode constrains the language to JSON. Strict Structured Outputs constrains the dialect to your schema. Schema validity is by construction — semantic correctness (truthfulness, business rules, numeric ranges beyond what the grammar encodes) is not guaranteed. You still validate.

OpenAI explicitly positions Structured Outputs as the evolution of JSON mode: both produce parseable JSON; only Structured Outputs guarantees schema adherence. Prefer strict schema whenever the model / provider supports it.

JSON mode vs Structured Outputs

Provider notes (high level)

• OpenAI: response_format / Responses API text.format with json_schema + strict: true; SDKs map Pydantic / Zod → schema.
• Gemini: schema-constrained JSON / response schema APIs (provider-specific schema subset).
• Anthropic: tool-use / structured helpers; treat as a tool schema contract rather than OpenAI-identical CFG masking APIs — validate provider docs for current guarantees.
• Local (Outlines / vLLM / XGrammar): guided decoding backends (json, regex, choice, grammar) apply the same idea at the sampler.

How constrained decoding works (CFG / token masking)

At each decode step the model scores the full vocabulary. Constrained decoding inserts a mask between logits and sampling:

1. Compile the JSON Schema (or EBNF / regex) into a context-free grammar (CFG) / automaton.
2. Track the current parse state given tokens emitted so far.
3. Compute the set of legal next tokens for that state.
4. Zero out (mask) illegal token logits → sample only from the legal set.
5. Advance the automaton with the chosen token; repeat.

OpenAI’s launch write-up describes converting JSON Schema → CFG and using a cached structure so mask updates stay cheap per step. Local stacks (XGrammar, Guidance / llguidance, Outlines) do analogous work inside vLLM / SGLang-style engines.

flowchart TB
  schema["JSON Schema or EBNF"]
  cfg["Compile to CFG / automaton"]
  state["Parse state from tokens so far"]
  logits["Model logits over full vocab"]
  schema -->|JSON Schema or EBNF| cfg
  cfg -->|Compile to CFG / automaton| state

Important consequence: schema validity is by construction, not “usually.” The model can still invent values that type-check. Ground with RAG / tools, prefer nullable unknowns, and run business validators.

Schema rules that trip people

Strict mode typically requires (OpenAI-compatible rules; check your provider):

1. Root must be an object (not a bare array / anyOf root in many APIs).
2. Every property must be listed in required. “Optional” fields are usually modeled as nullable (type: ["string","null"] or a union) while still required. Absence is null, not a missing key.
3. additionalProperties: false on every object. Extra keys are forbidden; forgetting this is a common rejection reason.
4. Keep schemas flat. Deep nesting, huge property counts, and recursive shapes increase compile cost, latency, and unsupported-feature risk. Prefer shallow objects plus arrays of simple items.
5. Supported type subset only. string / number / boolean / integer / object / array / enum / anyOf are common; many pattern, format, minLength, $ref gymnastics are limited or unsupported depending on backend.
6. Key order follows schema order in several implementations — design schemas intentionally if consumers care about order.
7. Parallel tool calls + strict function schemas may need parallel calling disabled depending on provider constraints.

Anti-pattern: stuffing business logic into an ultra-deep schema. Better: flat extract → code validates invariants.

System design: extraction / agent pipeline

Components

1. Ingest — raw text, PDF OCR, tickets, emails, tool traces.
2. Schema registry — versioned Pydantic / Zod models → JSON Schema artifacts (schema_id, semver).
3. Prompt + schema binder — system instructions + few-shots that complement (not replace) the grammar.
4. Constrained decoder — provider Structured Outputs or vLLM guided backend (XGrammar / Guidance / Outlines).
5. Typed parse — SDK .parse() / Zod parse → domain objects.
6. Business validators — ranges, cross-field rules, referential integrity the CFG cannot express.
7. Router / agent — either (a) structured answer objects or (b) structured tool calls; never free-form glue.
8. Observability — log schema_id, finish reason, refusal, token usage, validation failures.

Data flow

Failure modes (design for these)

Tradeoffs vs tool calling

Cost / latency: grammar compile + mask overhead is usually small vs retries. Complex / dynamic schemas favor backends with fast TTFT (for example Guidance) or cached schemas (XGrammar).

Teaching code: Pydantic + OpenAI SDK

SDK helpers compile the Pydantic model to a strict JSON Schema (additionalProperties: false, required fields). Always branch on refusal before using parsed. Re-validate business rules (totals, SKU existence) after parse. This fence talks to the OpenAI API — not runnable in the sandbox.

from typing import Literal, Optional
from pydantic import BaseModel, Field
from openai import OpenAI

class LineItem(BaseModel):
    sku: str
    qty: int = Field(ge=1)
    unit_price_usd: float

class InvoiceExtract(BaseModel):
    vendor: str
    invoice_id: Optional[str]  # still required in schema; null if unknown
    currency: Literal["USD", "EUR", "GBP"]
    line_items: list[LineItem]
    confidence: Literal["high", "medium", "low"]

client = OpenAI()
completion = client.beta.chat.completions.parse(
    model="gpt-4o-2024-08-06",
    messages=[
        {
            "role": "system",
            "content": (
                "Extract invoice fields from the user text. "
                "Use null for unknown optional strings. "
                "Do not invent SKUs."
            ),
        },
        {
            "role": "user",
            "content": "Acme Corp invoice AC-991: 2x WIDGET-1 at $9.50, 1x GADGET-9 at $20.",
        },
    ],
    response_format=InvoiceExtract,
)
msg = completion.choices[0].message
if msg.refusal:
    raise RuntimeError(f"refused: {msg.refusal}")
invoice: InvoiceExtract = msg.parsed
print(invoice.model_dump_json(indent=2))

Notes:

• Optional[str] becomes a required key whose value may be null — that is the strict-mode encoding of “unknown.”
• Field(ge=1) is a business constraint. Some backends will not encode ge into the grammar; always re-check in code.
• Local serving: pass the same schema to vLLM StructuredOutputsParams / guided_json. Cache repeated schemas for throughput; pick backend for TTFT vs long-generation tradeoffs.

Teaching code: Zod + OpenAI SDK

Same invoice contract. zodResponseFormat names the schema (invoice_extract) so the API can cache the compiled grammar.

import OpenAI from "openai";
import { z } from "zod";
import { zodResponseFormat } from "openai/helpers/zod";

const LineItem = z.object({
  sku: z.string(),
  qty: z.number().int().positive(),
  unit_price_usd: z.number(),
});

const InvoiceExtract = z.object({
  vendor: z.string(),
  invoice_id: z.string().nullable(),
  currency: z.enum(["USD", "EUR", "GBP"]),
  line_items: z.array(LineItem),
  confidence: z.enum(["high", "medium", "low"]),
});

const client = new OpenAI();
const completion = await client.beta.chat.completions.parse({
  model: "gpt-4o-2024-08-06",
  messages: [
    {
      role: "system",
      content:
        "Extract invoice fields. Use null when a string field is unknown. Do not invent SKUs.",
    },
    {
      role: "user",
      content:
        "Acme Corp invoice AC-991: 2x WIDGET-1 at $9.50, 1x GADGET-9 at $20.",
    },
  ],
  response_format: zodResponseFormat(InvoiceExtract, "invoice_extract"),
});

if (completion.choices[0].message.refusal) {
  throw new Error(completion.choices[0].message.refusal);
}
const invoice = completion.choices[0].message.parsed!;
console.log(JSON.stringify(invoice, null, 2));

In-memory schema + token mask (run this)

No network, no SDK. First half is the closed-world check production schemas actually need (required + additionalProperties: false + nullable + enum). Second half is a tiny CFG: given a JSON prefix, only a few tokens stay legal — everything else is masked. That is constrained decoding, shrunk to a whiteboard.

Press Run. Snippets must be self-contained — no network, files, or native modules.

Run it and notice: JSON-mode-shaped objects with an extra debug key fail the closed schema; invoice_id: null passes; "maybe" is never a legal next token once the automaton is in the enum state.

Interview Q&A

Pitfalls

Go Deeper

Docs and posts:

• OpenAI — Structured model outputs
• OpenAI — Introducing Structured Outputs in the API
• OpenAI Cookbook — Introduction to Structured Outputs
• vLLM — Structured Outputs
• XGrammar (MLC)
• Outlines
• Pydantic
• Zod

YouTube:

• OpenAI DevDay 2024 — Structured outputs for reliable applications
• Constrained Decoding: How AI Agents Are Forced to Return Valid JSON