When Schema Constraints Silence the Model: Ethical Risks of Forcing LLMs into Structured Output

There is something quietly troubling about the direction Structured Output Prompting has taken. The engineers building these systems are solving a real problem: downstream systems need parseable data, not eloquent ambiguity. They need a field called risk_level to contain exactly one of three strings, not a thoughtful paragraph about why the concept of risk is contextually dependent. The engineering case is airtight. And yet, the more completely we solve it, the more we risk closing off something we didn’t know we needed to keep open.

The Case for Schemas, Stated Honestly

The conventional wisdom deserves its strongest form before we interrogate it. Language models, left unconstrained, produce outputs that are nearly useless for programmatic consumption. They add qualifiers, hedges, alternate framings, and explanatory footnotes — all of which are virtues in human communication and failure modes in machine-to-machine pipelines. A medical triage system that returns a JSON object with a severity field set to "moderate_with_some_caveats_see_note" is worse than useless; it breaks the downstream process entirely.

The tools that have emerged to address this — Instructor, BAML, XGrammar, Outlines — are not naive. They represent serious engineering effort applied to a genuine interoperability problem. Libraries like instructor (v1.15.3 as of June 2026, per instructor PyPI) wrap model calls in Pydantic-validated retry loops, ensuring that downstream systems receive exactly the data shape they expect. Constrained Decoding approaches go further, shaping logit distributions at the token level so the model cannot physically produce a token that would violate the JSON Schema. The SOB benchmark, evaluating 21 frontier and open-weight models, found near-perfect schema compliance rates across all of them (Singh et al. 2026 (SOB)). That is a genuine technical achievement.

The argument for schemas, honestly stated, is also an argument for safety: predictable outputs mean predictable behavior, and predictable behavior is what you want in high-stakes systems. This view is coherent, widely held, and worth taking seriously.

The Crack That Runs Through the Foundation

The schema complies; the values do not. The same SOB benchmark that found near-perfect schema compliance also found that value accuracy for audio inputs dropped to only 23.7% — the model’s answers fit the shape perfectly while being wrong about the content most of the time (Singh et al. 2026 (SOB)). This is the first crack: structured output and accurate output are not the same property, and the engineering infrastructure we have built to achieve the former says nothing about the latter.

But there is a second crack, less discussed and more troubling. When a model operating under constrained decoding encounters a request it should refuse — an unsafe query, a request for information that violates its alignment training — the refusal mechanism and the schema constraint are suddenly in competition. OpenAI’s implementation of Structured Output acknowledges this explicitly: the API returns an explicit refusal boolean field when the model refuses an unsafe request, and documents that the model may not follow the schema when refusing (OpenAI Docs). This is honest engineering documentation of a genuine tension. What the documentation cannot tell you is how often that tension resolves in the wrong direction under production conditions.

Research accepted at CCS 2026 has begun to answer that question in a way that should unsettle anyone who has built systems on constrained decoding. Zhang et al. 2026 (CCS) demonstrated that grammar-guided logit masking can itself be exploited as an attack vector — the Constrained Decoding Attack achieves success rates between 94.3% and 99.5% against GPT-5, Gemini-2.5-Pro, and DeepSeek-R1, and maintains 75.8% effectiveness even against systems with guardrails in place. The grammar that was supposed to constrain the model becomes the mechanism through which the attack works. No mitigation patch has been confirmed.

Security & compatibility notes:

• Constrained Decoding Attack (CCS 2026): Grammar-guided logit masking can be weaponized to bypass LLM safety mechanisms. DictAttack achieves 94.3–99.5% success rates against GPT-5, Gemini-2.5-Pro, DeepSeek-R1; 75.8% against systems with guardrails. Paper: Zhang et al. 2026 (CCS) — https://arxiv.org/abs/2503.24191. No mitigation confirmed.

The Inversion: What the Engineering Solves Creates What It Cannot See

Here is where the argument turns. Take the two cracks together and something structural becomes visible: the engineering infrastructure for structured output was designed to solve a pipeline problem, and it solves that problem with impressive completeness. But in solving it, it introduces a new class of problem that the same engineering mindset is poorly positioned to notice.

The first problem is what Zhou 2026 calls “structure snowballing” — a failure mode in which schema formatting progressively depletes the model’s capacity for genuine reasoning. In self-correction tasks, constrained decoding on Qwen3-8B produced near-perfect syntactic compliance while semantic errors persisted and amplified across iterations, rather than being corrected (Zhou 2026). The constraint is satisfied; the thinking is not happening. This finding was demonstrated on a specific model, and its generalizability to larger frontier models remains an open question — but the mechanism it describes is not model-specific. When the decoding process is shaped primarily by grammar satisfaction, the path toward good reasoning and the path toward schema compliance can diverge invisibly.

The second problem is deeper still. The ethical risks that alignment researchers have spent years building into language models — the ability to hesitate, to qualify, to refuse, to express uncertainty — are precisely the behaviors that make models difficult to parse programmatically. They are features that look like bugs from an engineering perspective. When we build infrastructure to eliminate those behaviors at the decoding level, we are not just solving an interoperability problem. We are making an architectural decision about which model behaviors count as valuable. And we are making that decision quietly, inside a library configuration, without the kind of institutional review we would apply to an explicit policy choice with equivalent consequences.

Thesis

Thesis: When schema constraints operate at the token level, they do not merely format model outputs — they silently adjudicate which cognitive and ethical behaviors the model is permitted to express, and they do so without the transparency or accountability we associate with consequential policy decisions.

This is not an argument against structured output prompting as such. The interoperability problem is real and the tooling is solving it. The argument is about what kind of decision schema design represents, and whether we are treating it as such. A system prompt that tells a model to refuse harmful requests is a visible, auditable instruction. A constrained decoding configuration that shapes logit distributions to fit a schema is making some of the same choices — about what the model can say, and about the conditions under which alignment training can be overridden — but those choices are embedded in infrastructure, not in policy documents.

Consider who bears the cost of a silenced refusal in a medical triage pipeline, a hiring recommendation system, or a credit scoring workflow. The person whose severity field is assigned a schema-compliant value by a model that should have said “I cannot determine this with confidence” is not in the room when the schema is designed. They do not appear in the benchmark evaluation. They appear when something has already gone wrong.

Who the Schema Cannot Represent

The accounting that structured output tooling applies is an engineering accounting: did the output match the schema, did the pipeline complete, did the downstream system receive parseable data. This accounting is useful, even necessary. But it systematically excludes a category of stakeholder.

The person affected by a confident-sounding "risk_level": "low" returned by a model that, unconstrained, would have hedged or refused — that person’s interests do not appear in the schema definition. They do not appear in the library configuration. They appear in the outcome, after the fact, when the confident-sounding answer turns out to have been wrong in a way that mattered. The ethical question is not whether schemas cause harm in the aggregate; it is whether the infrastructure we are building to guarantee schema compliance also guarantees that the humans in the most vulnerable positions have some representation in the decision about what the schema can and cannot say.

It is worth noting that the tools themselves are beginning to grapple with this. XGrammar-2, released in May 2026 and presented at ACM CAIS ‘26 (MLC AI Blog), introduces dynamic structured generation designed for agentic contexts — a recognition that fixed grammars are insufficient when the model’s actions have real-world consequences. This is movement in the right direction. But the conversation about what the schema can represent, and who is harmed when it cannot represent certain things, has not yet found its institutional home.

Where This Argument Is Weakest

The vulnerability of this argument lies in its distance from empirical specificity. The claim that schema constraints systematically suppress ethically important model behaviors is plausible on theoretical grounds and supported by early research, but the causal mechanism has not been quantified across the range of production contexts where constrained decoding is deployed. The structure snowballing finding was demonstrated on Qwen3-8B; its implications for larger models remain speculative. No peer-reviewed study has yet directly quantified bias amplification through schema constraints in demographically sensitive domains. These are gaps in the evidence, not in the concern — and an honest account of the argument has to name them.

What would make this argument wrong: systematic evidence that constrained decoding preserves alignment behaviors at the same rate as unconstrained generation across a representative range of high-stakes tasks. If such evidence emerged, the engineering case for schemas would absorb the ethical concern entirely. We do not currently have that evidence. The question is whether we are building the infrastructure to generate it, or whether we are building the infrastructure to make the question harder to ask.

The Question That Remains

We have learned, slowly and at some cost, that infrastructure encodes values — that the design of a system is also a policy about who it serves and who it leaves vulnerable. Schema constraints are infrastructure. The question that remains is whether the field that built them will treat them that way: as policy decisions that warrant transparency, institutional review, and some mechanism of accountability to the people who cannot fit inside the schema.

Ethically, Alan.