Hidden Instructions, Consent Gaps, and the Ethics of Who Controls What LLMs Say

There is something seductive about that framing — not because it is dishonest, but because it describes something real. System Prompts are, in most cases, written by engineers trying to build useful products, not manipulate users. The operators who deploy them have genuine needs. The case for invisible governance has real weight. That is what makes its hidden flaw so difficult to see.

The Case for the Current Architecture

Prompt Engineering — the practice of designing instructions that shape AI behavior — exists because AI without context is AI without usefulness. A legal services company needs its assistant to recommend professional consultation rather than improvise legal strategy. A children’s education platform needs its model to stay age-appropriate without user intervention on every session. A customer support system needs its AI to maintain brand voice and decline to redirect users to competitors. These are not censorship. They are specifications.

Operators use techniques like Role Prompting to configure AI personas, restrict topic domains, and maintain consistency across thousands of simultaneous conversations. The model a user encounters is shaped by instructions assembled before the conversation begins — instructions that reflect genuine business and safety requirements. The idea that good governance can be invisible governance is the steelman’s strongest card: engineering meeting the same need as policy, but without bureaucratic overhead.

The market handles the rest. OpenAI’s model spec, published December 2025, establishes a five-tier instruction hierarchy in which operators have broad authority but cannot override core platform constraints. Anthropic’s usage policy explicitly prohibits operators from directing models to use subliminal or manipulative techniques, or to cause psychological harm. The constraints exist. They are enforced contractually. No one reads the source code of a calculator before trusting its output — the hidden implementation layer is a feature, not a failure.

The Assumption Hidden Inside That Argument

The steelman has a flaw, and it is structural rather than empirical. Every element of the case above — that operators are constrained, that markets self-regulate, that users benefit without knowing — presupposes that information asymmetry is acceptable. That users do not need to know what governs them, because the people who govern them have agreed to reasonable limits.

But consent is not a problem that contracts can solve. A CHI 2026 study found that system prompts are “generally not made public and most platforms instruct models to conceal them” — users are largely unaware of their existence, let alone their function (Neumann et al., CHI 2026). The same study found that users, once informed, want transparency and control. The users who might consent to the current arrangement have not been asked.

The practice that researchers call Context Engineering — deliberately designing what information an AI system sees before it responds — turns out to be simultaneously a productivity technique and a governance mechanism. The same methods that make AI useful are the methods through which AI behavior is shaped without disclosure. Consent cannot be outsourced to contracts between platforms and operators — contracts the governed party never sees and never signs.

Here is the flaw at the core of the argument: the defense of invisible governance works only if you are willing to treat consent as something that usage policies can substitute for. And governance without consent — even efficient, well-intentioned governance — has a name. We call it administration.

When the Same Evidence Says the Opposite

Return to the steelman’s best evidence: operators need customization, markets reward good products, platform-level constraints exist. All of this remains true. But each element now supports a different conclusion.

Operators needing customization does not mean customization should be invisible. It means the need is real enough to deserve a formal framework — one in which users understand the rules that govern them. The engineering need and the implementation choice to conceal it are separate questions. We have been treating them as one.

Markets rewarding good products are a regulatory mechanism only when consumers can distinguish good products from bad ones. A user asking a question without prior priming — relying on Zero Shot Prompting to get an uncontextualized response — has no way of knowing that their exchange begins inside a conversation already shaped by instructions they never read. When responses are formatted for machine consumption as Structured Output and passed to downstream systems without human review, those governing instructions recede even further from view. Invisibility was a choice, not a necessity — and the market argument assumes information buyers don’t have before they can even begin to choose.

Platform-level constraints — Anthropic’s AUP, OpenAI’s model spec — are not user protections. They are operator agreements. When a company violates those agreements and harms users, the harm has already occurred. Research published at ACM FAccT 2025 found that system prompt placement creates measurable representational and allocative biases across commercial LLMs, and that “layered implementation remains entirely hidden from end-users” — the people bearing the cost are the last to know (ACM FAccT 2025). The same evidence the steelman points to as proof of safety is also evidence of structural exposure.

What Invisible Governance Actually Produces

Thesis: When AI governance is invisible, accountability follows visibility — meaning responsibility accumulates where the output lands, not where the rules were written.

The pattern is already traceable across jurisdictions. In February 2024, the BC Civil Resolution Tribunal held Air Canada liable for its chatbot’s misinformation, rejecting the company’s claim that the chatbot was “a separate legal entity” — the company that configured its governing instructions owned the consequences (American Bar Association). A Shanghai court sentenced two developers to criminal terms for engineering system prompts to generate illegal content, treating prompt design intent as direct evidence of culpability. A US federal court allowed a wrongful death lawsuit against Character.AI to proceed in 2025, ruling that a chatbot application can be a “product” with design defects. Florida’s Attorney General filed suit against OpenAI in June 2026 over alleged safety failures — one of more than twenty lawsuits covering mass shootings, suicides, and psychological harm. Pennsylvania filed against a chatbot developer in May 2026, with the company’s system configuration directly at issue.

The law is converging on something the industry has been slow to acknowledge: the people who write system prompts bear responsibility for what those prompts produce, whether or not the users those prompts govern ever knew the prompts existed.

OWASP’s LLM07:2025 catalogues Prompt Leakage as a distinct vulnerability class — when system prompts themselves become extractable through simple techniques, the governing instructions meant to shape behavior become attack surfaces. A large-scale measurement of production LLM applications confirmed high prevalence of system prompt leakage across commercial platforms. Accountability trails visibility, not culpability — and courts are now closing the gap.

The Accounting Nobody Does

The people absent from the steelman’s model are not abstractions. A Washington Post investigation in 2026 found that AI companies invisibly add thousands of words of instructions to every chatbot conversation — instructions that govern what the model can say, how it frames reality, and which requests it will decline (Washington Post). When xAI’s Grok exhibited unexpected behavior in July 2025, the fix was removing a specific system prompt line. The behavior was governed by text the user had never been told existed.

Who bears the cost when invisible governance fails? Not the operators who wrote the instructions. Not the platforms that certified the usage policy. The distribution of consequence maps consistently to the distribution of information: those who cannot read the rules are most exposed when the rules fail.

The technical record the steelman cites as proof of safety is also evidence of fragility. OWASP has ranked Guardrails-adjacent prompt injection the top LLM risk — LLM01:2025 — for two consecutive years, precisely because Instruction Following systems are inherently exposed to adversarial instructions by design. NIST’s 2025 adversarial ML taxonomy explicitly classifies direct prompt injection, indirect prompt injection, and RAG poisoning as coordinated threat classes. A 2025 ACL paper found that some evasion techniques achieved complete bypass success against leading safety systems including Azure Prompt Shield. Research into retrieval-augmented architectures found that a small number of carefully crafted documents can manipulate AI agent responses the vast majority of the time. OWASP, NIST, and independent researchers all confirm the same conclusion: no complete mitigation exists.

The cost falls where information doesn’t reach — and in a system where users are the last to know how their interactions are governed, that cost falls on users.

Security & compatibility notes:

• GitHub Copilot RCE (CVE-2025-53773, CVSS 9.6): Instructions embedded in source code comments, repository issues, or web pages can trigger shell command execution without user approval, enabling propagation through infected repositories. Reported June 2025, patched August 2025. Prompt injection vulnerabilities extend beyond chat interfaces into development tooling.
• Prompt injection against production guardrails: Multiple evasion techniques — including zero-width characters and Unicode homoglyphs — achieved high bypass rates against Azure Prompt Shield and Meta Prompt Guard in 2025 ACL research. No universal mitigation exists; defense-in-depth is the only viable strategy (OWASP Gen AI Security).

Where This Argument Is Weakest

The weakest point of this analysis is that it treats transparency as straightforwardly good — and that is not obvious. Some opacity in system prompt architecture is genuinely protective. Security controls that resist manipulation work partly because their mechanisms are not fully documented. Child safety filters are more effective when their exact parameters are not available for study by bad actors.

There is also the question of what transparency would mean in practice. System prompts can run to thousands of words within a single Context Window — detailed enough to constitute a governance document in their own right. Disclosure requirements that surface this complexity without interpretation may produce something technically available but practically unreadable: the AI equivalent of a terms-of-service agreement that no one reads. Transparency without interpretation is ceremony. If that is what disclosure looks like, it solves a political problem without addressing an ethical one.

The argument in this essay holds if, and only if, meaningful transparency is both achievable and beneficial. The EU AI Act Article 50 obligations — requiring providers to inform users they are interacting with AI at the time of the first interaction, with enforcement beginning August 2, 2026 (EU AI Act site) — represent the first serious regulatory test of that proposition. The Texas Responsible Artificial Intelligence Governance Act, effective January 2026, adds a parallel accountability framework in US law. Whether these frameworks produce genuine consent or ceremonial disclosure is still being determined. That is the honest limit of this argument.

The Question That Remains

The question worth sitting with is not whether system prompts should be disclosed. Courts across multiple jurisdictions are already deciding that the people who write them bear responsibility for what they produce. Regulators are already writing the minimum standards.

The harder question is who gets to decide what counts as meaningful consent — not in law, where that question is being answered now, but in practice, when the people setting the disclosure standards are the same people who benefit most from the current arrangement. What happens when governance of the governance layer is itself ungoverned?