What Is Structured Output Prompting and How LLMs Are Made to Return Valid JSON

The first time a developer asks an LLM to “return JSON” and receives a response that begins with three sentences of explanation followed by a JSON block wrapped in markdown code fences — they usually blame the prompt. They rewrite it. They add “return ONLY valid JSON, no explanation.” The model complies for a while, then slips again. The blame moves to the model. Then to the temperature setting.

The actual problem is none of these things. A language model doesn’t produce text by following instructions the way a function follows a type signature. It samples the next token from a probability distribution — and nothing in that distribution is intrinsically shaped like a closing brace.

Understanding that distinction is what separates a patch from an architecture.

The Two Ways to Force a Model Into a Schema

There is a fundamental fork in how structured output systems work, and it determines everything about their failure modes, latency profiles, and accuracy guarantees.

The first approach operates outside the model. You ask for JSON, receive a string, then validate it against a schema — and if validation fails, you retry. The second approach operates inside the sampling loop, modifying which tokens the model is even allowed to consider. These two approaches share a name but almost nothing else.

What is structured output prompting?

Structured Output Prompting is the practice of constraining a language model’s outputs to conform to a predefined Structured Output schema — most commonly JSON Schema, a Pydantic model, or a typed data structure — so that downstream code can parse and use the response without string manipulation or defensive error handling.

The word “prompting” in the name is historically accurate but increasingly misleading. In early implementations, the constraint lived entirely in the prompt: “Return a JSON object with keys name, age, and email.” The model was asked; it sometimes obeyed. Today, the most reliable implementations have moved the constraint from language into the sampling algorithm itself.

How does structured output prompting work?

Two mechanisms, radically different in where they intervene:

Client-side validation with retries is the older approach. A library like Instructor wraps the model call, parses the output as the target type (a Pydantic model, a TypeScript interface), and retries if parsing fails — feeding the validation error back as context. Instructor 1.15.3 (Instructor on PyPI) supports Python, TypeScript, Go, Ruby, Elixir, and Rust, with over 3 million monthly downloads — which signals how often teams reach for this pattern. The ceiling of this approach is the model’s instruction-following ability. When the model drifts, the library catches it. When the model drifts badly enough, the retries exhaust.

Constrained decoding intervenes at the Logits level during inference. Before each token is sampled, the system sets the logit scores of all tokens that would produce invalid output to −∞ (XGrammar Paper, arXiv). The model never “tries” to emit an invalid token and fails — it simply cannot select one. The output is valid by construction, not by correction.

This is not a semantic difference. It is a mechanical one.

What are the main components of an LLM structured output system?

Three layers compose a complete system:

1. Schema definition — the contract specifying what valid output looks like. Usually JSON Schema, a Pydantic model, or a DSL like BAML. BAML (BoundaryML Docs) compiles schema definitions to native language bindings in Python, TypeScript, Ruby, Java, C#, Rust, and Go — treating the schema as source code rather than a runtime annotation.

2. Enforcement mechanism — either client-side validation or constrained decoding, with the trade-offs described above.

3. Grammar representation — the internal format the enforcement mechanism uses to track which outputs are currently valid. This is where FSMs and context-free grammars (CFGs) enter.

Why FSMs Break on Nested JSON — and What Replaces Them

A finite state machine is the natural structure for enforcing a regular language: at each step, you know which tokens keep you on a valid path. For flat structures — a JSON object with fixed top-level keys and scalar values — FSMs work well and run fast.

Nested JSON is not a regular language. An object containing arrays containing objects requires a grammar that can track arbitrarily deep recursion. FSMs cannot represent this without exploding in state count. This is the core technical argument for context-free grammars in constrained decoding, documented in the LMSYS blog’s analysis of compressed FSMs.

Outlines (version 1.3.0, Outlines on PyPI) uses this FSM-based approach for simpler schemas, with its core logic split into a Rust-based outlines-core package for performance (dottxt GitHub). XGrammar takes a different route: it uses an efficient CFG engine capable of handling recursive JSON structures with near-zero overhead in generation speed (XGrammar GitHub). As of June 2026, xgrammar is the default constrained decoding backend in vLLM, SGLang, and TensorRT-LLM, a position it earned by demonstrating that CFG enforcement need not impose meaningful latency.

The published research formalizes this: in “XGrammar: Flexible and Efficient Structured Generation Engine for LLMs” (MLSys ‘25), the authors show that the key bottleneck is not the grammar matching itself but the preprocessing step of computing which tokens are valid at each grammar state. XGrammar precomputes this ahead of inference, making per-token decisions fast enough to vanish into normal generation latency (XGrammar Paper, arXiv).

The grammar caching insight applies at the API level too. Anthropic’s structured outputs compile the grammar on the first request (~added latency) and cache it for 24 hours (Claude Platform Docs). For high-volume applications, this means the latency cost is amortized almost immediately.

What the Mechanism Predicts — and Where It Breaks

Understanding the enforcement layer turns passive knowledge into something useful. Several practical consequences follow directly from the architecture:

If you use client-side validation (the Instructor pattern), your error rate is bounded by the model’s instruction-following fidelity on your specific schema. Simpler schemas fail less often; deeply nested schemas with many optional fields fail more. Adding schema complexity increases retry frequency — a cost that compounds under load.

If you use constrained decoding, invalid outputs become structurally impossible, but a different failure mode emerges: the model can produce valid-structure output with semantically wrong content. A schema-valid JSON object can contain the wrong values — hallucinated strings, missing reasoning, confident nonsense. The grammar enforces syntax; it cannot enforce truth.

If you use a provider’s native structured output feature (OpenAI’s response_format: {type: "json_schema", ...} for GPT-4o-2024-08-06 and later models, or Anthropic’s output_config.format with strict: true), you trade schema flexibility for reliability. Anthropic’s implementation, for instance, does not support recursive schemas, numerical min/max constraints, or complex array constraints (Claude Platform Docs) — constraints that most client-side libraries handle without friction. Gemini 2.5+ models support JSON Schema on all endpoints via response_format with mime_type: application/json (Google AI Docs), though teams using the old Gemini schema API format should note that Google required migration by June 8, 2026.

Rule of thumb: use constrained decoding for structural guarantees; use client-side validation for schema flexibility. The strongest production systems often combine both.

When it breaks: constrained decoding can fail silently when the CFG representation of the schema has ambiguities or when the precomputed token mask is stale relative to a tokenizer update. Client-side validation breaks entirely when the model’s instruction-following degrades under context pressure — which happens predictably when the prompt is long and the schema is complex.

The Quiet Architecture Decision Nobody Talks About

There is a design question baked into every structured output system that most teams never make explicitly: where does the schema live relative to the model?

In the Instructor pattern, the schema is a runtime object — a Pydantic class, a TypeScript type, something that exists in application code and is translated into a prompt or a tool call schema at request time. The schema is coupled to the application layer.

In the BAML approach, the schema is a first-class artifact. BAML is a DSL that you define once and compile to native bindings in your target language. The schema definition is the source of truth, not the application type. When the schema changes, you recompile — and the type system in your target language catches mismatches before runtime.

This distinction matters more than it appears. Teams that treat schemas as prompt annotations tend to drift — a developer updates the Pydantic class but forgets to update the system prompt, or vice versa. Teams that treat schemas as compiled contracts have a single point of change. The BAML model imports this practice from static typing: a change propagates through type-checking, not through hoping the model read the updated instructions.

Not a preference. A different operational risk profile.

The Data Says

Structured output prompting is not a prompt technique in the traditional sense — it is a constraint applied to the sampling distribution, either after the fact (validation + retry) or during it (constrained decoding via FSM or CFG). The choice of enforcement mechanism determines failure modes, latency characteristics, and schema flexibility independently of model quality. As native constrained decoding becomes the default in major inference engines and provider APIs, the remaining engineering question shifts from “how do we get valid JSON” to “how do we represent the schema as a first-class artifact.”

Security & compatibility notes:

• xgrammar DoS (CVE-2026-25048): Deeply nested schemas could trigger uncontrolled recursion in versions below 0.1.32. Fix: use current v0.2.2 or later (SentinelOne CVE DB).
• Gemini API schema migration: Google required migration to the new schema format by June 8, 2026. Code using the old Gemini schema API format may break silently (AI CERTs News).