Asking for JSON Is Not Enforcing It: One Schema, Three Backends, Three Guarantees

On March 3rd, 2026, a planning step in our content pipeline asked the model for a list of entities and got back exactly what it asked for. That was the problem.

The prompt said: return the entities as JSON. The model returned JSON. Perfectly valid JSON. Any parser on earth would have accepted it. What it returned was this:

["prompt-caching", "context-windows", ...]

What the code expected was this:

{ "entities": ["prompt-caching", "context-windows", ...] }

A bare top-level array instead of an envelope object. JSON.parse sailed through without a complaint. The failure surfaced one layer later, in Zod validation, which knew the shape was wrong even though the syntax was fine.

I’m an IT analyst, not a software engineer by training, and this was the moment the distinction between two failure layers stopped being abstract for me. When people say “the LLM broke my JSON,” they usually mean one specific thing. But there are two separate things that can go wrong, and they need separate defenses:

Parse failure: the text is not valid JSON at all. Markdown fences around it, a chatty preamble, a truncated tail.

Schema failure: the JSON is valid but has the wrong shape. Bare array instead of envelope. Missing field. String where a number should be.

The second kind is sneakier, because it passes the first gate. Nothing crashes at the parse step. Your code carries a well-formed object deeper into the pipeline and falls over somewhere that has nothing to do with where the problem was born. “Ask for JSON” only addresses layer one, and only optimistically.

The fix had two halves, and both mattered

The commit that closed the March incident did two things, and I keep coming back to why it needed both.

First half: the exact JSON schema went directly into the prompt. Not “respond in JSON,” not a prose description of the fields — the literal schema, so the model sees the precise expected shape, envelope and all. This is prompt engineering in its least glamorous form: the prompt as a contract document, not a wish.

Second half: a fallback in the parsing code. If the model returns a bare top-level array where an envelope is expected, the code wraps it automatically. The model’s most likely deviation, handled deterministically, in code, where handling is free and repeatable.

Prompt asks precisely; code catches the known failure mode. Neither half replaces the other. The schema in the prompt reduces the error rate; the fallback makes the residual error harmless.

Out of that scar grew a house rule: when generating LLM prompts that expect structured JSON responses, always include the exact schema in the prompt. It is written, word for word, in two layers of our project configuration — once in the global rules that apply to every project, once in the project-level rules. The same sentence twice. That duplication is deliberate emphasis, not sloppiness. Rules born from incidents get written where they cannot be missed.

At the time I thought of it as a pragmatic patch. It took the pipeline growing two more backends to reveal it as something else.

Three backends, three different promises

Our pipeline — an agent orchestration setup that plans, researches, writes, and verifies articles for bestaiweb.ai — runs all its AI calls through a single runner interface. Behind that interface sit three interchangeable backends: the Claude CLI (the default, on a subscription), the Claude Agent SDK (API key, pay-per-token), and codex (GPT-5.5, via a ChatGPT subscription). Same prompts, same phases, swap the backend with a flag.

Here is what surprised me when I actually lined up what each backend guarantees about structured output:

The SDK is the only one of the three that can enforce the shape at the API layer. Our Zod schema converts to JSON Schema and goes into the request as an output format. The model’s response cannot arrive in the wrong shape — enforcement, in the strict sense. The output is constrained before it reaches our code.

The CLI only requests. The prompt carries the schema and asks nicely; the response comes back as free text, and our code cuts the JSON out of it and validates after the fact. Nothing upstream prevents a malformed or misshapen answer. Requesting, not enforcing.

codex actively refuses native enforcement. This was the finding I did not expect. It has an output-schema mode, but that mode rejects schemas with references — and ordinary Zod schemas, converted to JSON Schema, produce exactly those — and it imposes strict-mode constraints that our real schemas violate. So the codex backend fell back to the same technique as the CLI: schema in the prompt, parse from free text. The newest backend, from a different vendor, converging on the oldest defense.

Which means the only protection layer present on all three backends is the schema written into the prompt. Native structured output is a per-vendor, per-interface feature with per-vendor, per-interface limitations. The prompt travels everywhere.

That reframed the house rule for me. It is not a workaround we tolerate until enforcement arrives everywhere. It is the portable layer. On the SDK it is a double safety alongside enforcement; on the CLI and codex it is the only line of defense before validation. Either way, it earns its tokens.

The steelman deserves saying out loud, because it is a good argument: schema-in-prompt is dated advice, providers have native structured output and function calling now, use those and stop burning prompt tokens on schemas. For a single-provider stack with guaranteed structured output, that is often simply correct. Our answer holds under three specific conditions: we run multiple backends and native enforcement is not available everywhere, nor identical where it exists — one backend rejects referenced schemas outright; even where enforcement exists, a text-parse fallback path remains in the code; and the schema in the prompt was never meant to replace enforcement — it is the common denominator across backends, and where enforcement exists on top of it, the two are complementary, not redundant. If all three of those conditions disappear for your stack, take the steelman’s advice.

Cutting JSON out of free text, without “repairing” it

Two of three backends hand us free text that merely contains JSON. So around the parse step grew a deterministic extraction layer, and the design principle behind it matters more than the code: extraction, never repair.

Three stages, in order:

1. Markdown code fences. If the response wraps the JSON in fences, take what’s inside. Models trained on markdown do this constantly.
2. Balanced-brace scan. No fences? Scan the text for a balanced {...} or [...] region — tracking string boundaries and escape characters, so a brace inside a quoted string doesn’t fool the counter.
3. Whole-text fallback. Neither of those? Try parsing the entire response as-is.

If all three stages fail, the call fails, loudly. There is no stage four where we guess at missing commas or auto-close brackets. “JSON repair” felt tempting exactly once; the trouble is that a repaired guess can be wrong in ways that parse cleanly, and then you are back to the sneaky failure class — syntactically fine, semantically corrupted, exploding three phases downstream where nobody thinks to look at the parser.

One more guard earned its place separately: empty output. A timeout or a silent backend failure used to surface as Unexpected end of JSON input — technically true, practically useless, pointing at the parser when the parser was the only component that did nothing wrong. Now empty output is caught before parsing and reported as what it is. Small change; disproportionate debugging time saved.

There is a philosophy underneath both the array fallback and the extraction ladder, and it is the same one we wrote about in 117 ghost errors: fix the class, not the instance, and put the fix in the deterministic layer. It shows up in one more place worth naming — fields the model is forbidden to produce. Mechanical values like slugs, priorities, and type labels are not requested and then validated; the prompt explicitly tells the model not to generate them, and code assigns them deterministically after the fact. The most reliable structured output is the part you never asked the model for. Fallbacks live in code. Contracts live in prompts. Creativity lives only where nothing else can do the job.

The same disease, wearing other formats

Once you have seen the two-layer failure in JSON, you start recognizing its relatives everywhere in an LLM pipeline. Format drift is not a JSON problem; JSON is just where it is easiest to diagnose.

Our validator agent was supposed to end its report with a VERDICT: line. Sometimes it wrote RESULT: instead. Same meaning, different token, broken parser — until the parser learned to accept both.

Glossary terms arrived from generation runs with three different separators across runs. Same data, three shapes.

The worst one ran silently. Our self-correction loop — the mechanism that reads validation findings and fixes articles — expected its input files to contain checkbox lists. The files had drifted to tables. The parser found no checkboxes, concluded there was nothing to correct, and reported success. The loop was de facto switched off, and everything looked green.

That last one belongs to the most insidious sibling class of all: parsers that silently drop invalid input instead of failing loudly. Our home-grown YAML mini-parser did this to us twice. Once it discarded an entire nested block because of an inline comment on the key’s line. Two months later — different bug, same silence — it returned block lists as empty objects and quietly dropped keys containing hyphens. Both times, no error. Both times, downstream code operated confidently on partial data. The loud parse failure of March 3rd was fixed in a single commit the same day. The silent parser bugs cost far more, and none of it was visible while it was happening. If I could enforce only one rule across every parser in the pipeline, it would not be about schemas at all. It would be: never swallow what you cannot parse.

What crossed the vendor line, and what didn’t

The three-backend setup enabled an experiment I keep thinking about. We took an identical, locked writing brief and ran it through the codex backend — a different vendor’s model, GPT-5.5, inside our unchanged pipeline. The resulting article went through the 90+ checks we actually run; for its article type, that meant all 34 deterministic checks. It passed with zero errors. 2,552 words against Claude’s 2,594 — a 1.6% difference on length, full marks on structure, links, scaffolding, required elements.

And it still sounded different. Clause-heavy FAQ questions. The occasional leak of internal scaffolding vocabulary into reader-facing text. Nothing a structural check flags; everything a reader feels.

So: structural validity is portable across vendors. Voice is not. And the deterministic validators, which I trust deeply, measure the floor — the shape of an acceptable article — not the quality of a good one. For anyone designing a multi-backend system, that splits the work cleanly in two: enforce the floor in code, identically, everywhere, because that transfers; treat voice as a separate, per-model concern, because it does not.

What I can’t back with numbers

Some honesty about the edges of this story. The bare-array fallback has no counter on it — we do not measure how often it actually rescues a run, so I cannot tell you whether it fires weekly or has fired twice since March. The claim that structured-output pain is widespread and largely unanswered comes from a small, heuristically clustered sample of developer questions, not a rigorous survey. The cross-vendor comparison is n=1: one article, one locked brief, one pair of models — suggestive, not conclusive.

The pain being widespread, though, is not just my impression. In our sample of developer questions from GitHub and Stack Overflow: a LangChain issue — “Doesn’t honour pydantic model field datatype and randomly throws OutputParserException” — open since April 2026 with 28 comments and counting. “Getting inconsistent structured output from Ollama models with Genkit,” Stack Overflow, September 2025, no answer. “json.loads() fails with JSONDecodeError: Extra data when parsing LLM output,” January 2026 — that one is our extraction ladder’s stage two, unbuilt. “MessageChatMemoryAdvisor stores raw JSON response… causing context loss,” February 2026, no answer. In that sample, 80% of the structured-output questions on Stack Overflow sit unanswered. Small sample, rough clustering — but the shape of it matches what our pipeline lived through: everyone asks for JSON, and asking is the easy half.

The bet I’d make again

Here is the closing thesis, and I mean it as a bet, not a summary: the schema in the prompt is not a stopgap on the road to native structured output. It is a contract layer, and it will outlive any particular backend behind our runner interface. Backends churn — we have already swapped and added them, and one of them refused the “modern” mechanism outright. The prompt contract survived every swap unchanged. API-layer enforcement is a bonus you add where a backend offers it, and you should add it there. But it is not the foundation you build on. The foundation is the thing that travels.