Trust Boundaries and Why LLMs Cannot Reliably Separate Instructions from Data

An AI customer-service agent reads a support ticket. Buried in paragraph three is a sentence instructing the model to exfiltrate the conversation history to an external URL. The developer wrote explicit confidentiality instructions, positioned them in the system prompt as every tutorial recommended, and watched them get ignored completely. The interesting question is not why the attack worked — it is why the developer expected the defense to hold.

The Privilege Model That LLMs Do Not Have

Classical computer security runs on privilege separation — Saltzer & Schroeder formalized this principle in 1975, and every operating system since has enforced some variant of it. The kernel has one memory region, user processes have another, and crossing that boundary requires explicit mechanisms: syscalls, capability tokens, hardware rings. LLMs have no equivalent. The transformer processes all inputs as a flat sequence of tokens; position in that sequence conveys order, but no runtime mechanism enforces authority.

What are system prompts, trust levels, and the privilege model in LLM applications?

System Prompts — the developer-controlled instructions passed to the model before user input — are typically described as “trusted” in API documentation. This framing is architecturally misleading. In a traditional privilege model, trust is enforced by the runtime: the OS kernel verifies that a user process cannot read kernel memory regardless of what code that user writes. In an LLM application, “trust level” is a convention honored only by the model’s pattern-matching on token position. Conventions are exactly what adversarial prompts target.

The typical implementation of a privilege model places the system prompt at positions 0 to N, user input at position N+1 onward. The model has learned, from training data, to treat early-position content as authoritative instruction. This is a statistical pattern acquired from training distribution — not a hardware guarantee. An instruction at token 5,000 can override the behavioral implications of an instruction at token 5 if the model’s training has exposed it to override patterns that match.

Trust Boundary is the term practitioners use for the conceptual line between trusted and untrusted input. In LLMs, this boundary is semantic, not structural. It exists in the developer’s intention, not in any enforcement mechanism inside the model.

What background do developers need before implementing prompt injection defenses?

Understanding Prompt Injection as a defense problem requires rethinking three assumptions that transfer cleanly from classical security:

Assumption 1: Parsing and interpretation are separate. In web security, the HTTP parser is distinct from the JavaScript interpreter; SQL injection was ultimately defeated by parameterized queries, which enforce the data/code boundary at the database layer. In LLMs, the same module that parses the input also interprets it as instruction or data based on contextual priors. There is no separation layer to protect.

Assumption 2: Sanitization at the boundary is sufficient. In classical systems, input filtering works when an enforcement mechanism is downstream of the filter. In LLMs, there is no enforcement mechanism downstream — only a model that interprets whatever arrives. Filtering reduces attack surface without eliminating the ambiguity.

Assumption 3: Authority is positional. Most prompt architectures give the system prompt de facto higher authority because it appears first and models are trained to treat it as such. But this is a training artifact, not a runtime guarantee. Indirect Prompt Injection exploits exactly this: attack content never needs to appear in the “untrusted” user position. It arrives through a channel the developer classified as trusted — retrieved web content, tool outputs, database records — and the model cannot distinguish it from legitimate data.

That vulnerability was demonstrated against Bing Chat (GPT-4) in 2023, injecting instructions through retrieved search results — a channel the application treated as data (Greshake et al. 2023, arXiv:2302.12173). The conclusion was direct: “effective mitigations of these emerging threats are currently lacking.” That finding predates the proliferation of agentic architectures where models routinely read files, call APIs, and browse the web on behalf of users; the attack surface has only expanded since.

What Structured Output Tools Actually Guarantee

The rise of constrained generation tools has introduced a specific confusion: developers assume that enforcing JSON schema compliance also enforces trust boundary compliance. Both problems sound like “making the model do what you told it.” They are not the same problem.

Why do current prompt injection filters fail and what makes the problem technically hard to solve in 2026?

Three tools define structured LLM output in 2026, each with a different enforcement approach:

XGrammar v2 (released May 4, 2026) is the default Constrained Decoding backend in vLLM, SGLang, TensorRT-LLM, and MLC-LLM. It masks invalid tokens at generation time, guaranteeing 100% schema conformance with near-zero token overhead — 80× faster grammar compilation than xgrammar v1 (MLC Blog). Instructor 1.15.3 (released June 15, 2026) uses Pydantic-based post-hoc validation with retry: it validates and corrects output after generation rather than constraining token-by-token (Instructor Docs). BAML, actively developed as of June 2026 (latest nightly build 0.12.2), uses Schema-Aligned Parsing — tolerating malformed JSON and chain-of-thought wrapping by extracting schema-conforming data from imperfectly formatted responses (BoundaryML Docs).

All three enforce that output is valid JSON Schema against the Draft 2020-12 specification — the current stable standard. The Structured Output Prompting approach they implement guarantees structural validity of the model’s response. None constrain what values appear inside the schema. An attacker can inject "action": "exfiltrate_history" into a field that expects a user action string, and that value passes schema validation without issue. Structure is guaranteed; trust is not.

The reason current prompt injection filters fail is the same reason SQL injection was difficult to eliminate before parameterized queries existed: the language used to convey instructions — natural language tokens — is identical to the language used to convey data. Unlike SQL, where the injection problem was structurally solved at the database layer, no analogous enforcement layer exists for natural language inside a transformer.

Keyword-based filters catch naive attack patterns like “ignore previous instructions.” They fail against paraphrases, semantic equivalents, indirect framings, and multilingual variants. Models fine-tuned to resist known injection patterns remain vulnerable to novel framings because the training distribution of “safe” and “compromised” responses overlaps significantly in the latent space; there is no clean decision boundary between them.

The OWASP LLM Top 10 2025 places prompt injection at LLM01 — the top-ranked risk — defining it as an attack where “manipulating LLMs via crafted inputs can lead to unauthorized access, data breaches, and compromised decision-making” (OWASP LLM Top 10). It holds this position not because detection has failed, but because the architectural conditions enabling the attack remain unchanged.

Not a solvable bug. A property of the design.

What the Architecture Predicts About Defenses

The mechanism makes falsifiable predictions about what defenses will and will not hold:

• If you rely solely on prompt-level instructions to separate trusted and untrusted behavior, every external input channel — user text, retrieved documents, tool outputs — is a potential injection surface. The scope of that surface scales with the capabilities you grant the model.
• If you implement keyword-based filtering, you catch the payloads your filter was trained on and miss variants it was not. The space of possible phrasings for “do something you shouldn’t” is larger than any training distribution.
• If you apply constrained decoding or schema validation, you gain structural guarantees about output format — not behavioral guarantees about whether the model was manipulated before generating that format.
• If you scope the model’s permissible outputs to an explicit allow-list of actions — so that DELETE_RECORDS is not a string the model can output regardless of what any injected instruction says — you reduce the attack surface mechanically, without depending on the model’s resistance.

Two attack classes were formally characterized at the NeurIPS ML Safety Workshop in 2022: goal hijacking — changing what the model does — and prompt leaking — extracting what it was told to do (Perez & Ribeiro 2022). Both remain viable in 2026 against production systems. The taxonomy is stable; the defenses are not.

The defenses that come closest to working treat the model as an untrusted executor rather than a trusted one. Minimum-permission output schemas, explicit action allow-lists, and human approval for consequential actions provide mechanical constraints that do not depend on the model’s learned resistance to injection — a resistance that is, by its nature, imperfectly generalized.

The architecture trades separation for generality — and no filtering layer can undo that tradeoff from the outside.

Rule of thumb: Treat the model’s output as adversarial input to your downstream systems, not as a trusted signal from a trusted executor.

When it breaks: Any defense relying on the model’s trained resistance to injection fails against sufficiently novel framings — learned behaviors generalize imperfectly, and the space of possible injection patterns is systematically larger than the training distribution that built the resistance.

The Data Says

Every input channel an LLM reads is a potential instruction surface. Prompt injection has ranked first in the OWASP LLM Top 10 not because the attacks are sophisticated — they often are not — but because the gap between the developer’s mental model and the model’s actual behavior is structural. The system prompt is not trusted in any mechanical sense. It is positioned first in a sequence that the model has learned to treat as authoritative, until it encounters tokens that override that learned prior. That override is available to anyone who controls text the model reads.