Accountability Gaps and Transparency Tradeoffs: The Ethics of Prompt Injection in AI Agent Systems

This is a coherent position, sincerely held by many of the engineers and legal teams now building the first generation of consequential AI agents. It also collapses under exactly the kind of pressure that Prompt Injection attacks apply — and the way it collapses reveals something more troubling than any single vulnerability.

The Reasonable Defense

The employer liability framework has centuries of precedent behind it. When an employee, acting within their apparent scope of authority, causes harm to a third party, the employer bears responsibility. Extending this to AI agents is not merely convenient — it is structurally coherent. The deploying organization controls the system, profits from its outputs, and sets the conditions under which it operates. If an AI agent leaks sensitive data because an attacker embedded malicious instructions in a third-party document, the deployer should have built sufficient defenses. The capability gap is their problem to address.

This framing already has institutional momentum. Courts are moving toward employer-style responsibility, with legal observers at Daimon Legal noting that “claiming the AI did it will not shield you from liability.” OWASP LLM Top 10 has ranked Indirect Prompt Injection as the top vulnerability in its LLM risk classification for two consecutive years — and OWASP rankings carry weight in professional negligence arguments. Organizations that deploy agents without addressing this risk are not victims of novel technology; they are choosing to accept a known exposure.

On the transparency question, the defense is equally pragmatic. AI systems need behavioral rules. Those rules, encoded in System Prompts, include legitimate security logic: which topics to refuse, how to handle sensitive data, which tools can be called and under what conditions. Revealing this logic to the public invites adversarial manipulation. You interact with the behavior of complex software, not its source code — and that has always been true.

Where the Argument Fractures

The employer liability framework assumes a clear, traceable chain of instruction. An employee receives direction from a supervisor; the supervisor’s authority traces to the organization; the organization is accountable for what it authorized.

The Trust Boundary in AI agents does not work this way. Research from ICLR 2025 suggests that language models cannot reliably separate instructions from data — external architectural enforcement, not model-level trust, is mandatory for security. This is not a patch away. It is a structural property of how these systems process text. An agent reading a web page, a document, or an email is processing all of it as potential instruction; there is no architectural seam that separates “content from the environment” from “commands from the principal.”

When an indirect prompt injection attack succeeds, it succeeds because the agent cannot distinguish a command from its principal from a command adversarially crafted into a PDF. Now apply the employer framework to this architecture. The employer analogy assumes the employee can tell the difference between orders from their manager and orders from a stranger who wrote something on a whiteboard. When that distinction collapses, the liability chain collapses with it. The deployer did not instruct the agent to exfiltrate data. The model maker did not build a system designed to be hijacked. The third-party content provider created the document — but may or may not have been the attacker. And the user who triggered the agent received the corrupted output without any awareness that something had gone wrong.

The architecture fragments accountability by design, not by negligence. That is not a detail. It is the argument’s load-bearing flaw.

The Evidence, Reread

EchoLeak — CVE-2025-32711, disclosed in June 2025 with a CVSS score of 9.3 (Microsoft Security) — was the first documented zero-click prompt injection exploit in a production AI system. A single crafted email to Microsoft 365 Copilot bypassed the existing injection classifier, evaded link redaction through Markdown formatting, exploited auto-fetched images, and abused a Teams proxy to escalate privileges across trust boundaries, all without any action from the user (EchoLeak Paper). Microsoft patched it server-side. No exploitation in the wild was confirmed for that specific vulnerability.

But the logic of “this particular attack was patched” does not restore the employer accountability model. Palo Alto Unit 42 documented the first large-scale wild instances of indirect prompt injection attacks in March 2026 — twelve attack types observed across production systems, including unauthorized transactions, database destruction, and system prompt leakage. The transition from theoretical to observed took under twelve months.

What the defenders of the status quo were thinking about was a world where Privilege Separation could be maintained — where the agent’s permissions could be kept narrow enough to contain any damage. The EchoLeak attack chain escalated privileges across trust boundaries that were supposed to prevent exactly that escalation. Containment must be built explicitly, enforced externally, and verified continuously. And the same opacity that hides system prompts from users also hides an attacker’s injected instructions once they’re inside the system.

This is where the concealment defense inverts against itself. If you build opacity into an AI agent for security reasons, you also build in the conditions for undetectable compromise. The mechanism designed to protect the system becomes the mechanism that hides the attack.

What Accountability Actually Requires

Thesis: Deploying organizations cannot claim both the protection of concealment and the defense of employer liability — genuine accountability requires either transparency about the instruction layer or external audit rights, and the current architecture provides neither.

The NIST Agentic AI RMF Profile, published in draft form in March 2026 (CSA Agentic NIST Profile), frames the requirement clearly: every agent action should be traceable to a responsible human officer, and the delegation chain from principal to agent must be explicit. This is the right institutional instinct. But it runs directly into the current practice of system prompt concealment. You cannot trace delegation without knowing what was delegated. You cannot audit the instruction chain without access to the instruction layer.

Structured Output Prompting, Constrained Decoding, and tools like Instructor, JSON Schema, and BAML represent genuine technical progress in making agent outputs more predictable — worth noting that BAML underwent breaking API changes in version 0.82.0, so implementations should be verified against current documentation. But predictability and accountability are different properties. A system can behave consistently in normal operation and remain completely opaque when its instruction layer has been compromised. Technical robustness addresses the first problem. Governance addresses the second. The EU AI Act and California’s BOT Act require disclosure that users are interacting with AI (EU AI Act). Neither requires disclosure of what that AI has been instructed to do or how those instructions can be overridden.

The People the Accounting Leaves Out

The employer liability argument and the security concealment defense share a common structure: both center the interests of the deploying organization and the technology provider. They are, at root, arguments about which professionals among the builders bears responsibility. Neither framework meaningfully addresses the person at the end of the chain — the user whose data was exfiltrated, whose transaction was unauthorized, whose request was handled by an agent operating under instructions they never saw and cannot inspect.

When an AI agent is deployed to handle customer queries, manage documents, or execute financial transactions, the people interacting with it are not parties to any of the architectural decisions that shaped its behavior. They do not know whether trust boundaries were implemented correctly. They cannot audit the system prompts governing what the agent will and will not do. And if an attacker has compromised those system prompts — injecting instructions that remain invisible to the user while they receive the results — the asymmetry is complete. The users experience the consequences while remaining structurally excluded from the governance.

The EU AI Liability Directive, which would have established strict liability rules for AI-caused harms, was withdrawn by the European Commission in 2025 with the assessment that no foreseeable agreement was possible (IAPP). This is not a temporary gap pending resolution. It is, for now, governance by inaction — and the people who bear the cost of that inaction are not the deployers, not the model makers, and not the regulatory bodies still deliberating. They are the users.

Where This Argument Is Weakest

If cryptographic provenance systems advanced to the point where every instruction given to an AI agent could be signed, timestamped, and made auditable without public exposure — a publish-before-deploy architecture with controlled access rather than open disclosure — then system prompt concealment would be defensible on its own terms. The instruction layer would be private but auditable, the way sealed evidence is private but legally inspectable. Accountability and opacity would no longer be in direct tension.

The argument is also weakest in its implied assumption that disclosure automatically produces accountability. A system prompt published in dense technical language that users cannot interpret satisfies a transparency requirement while leaving the underlying power asymmetry intact. What genuine accountability requires is not necessarily public governance — it is auditable governance, with defined mechanisms for inspection when harm occurs and clear assignment of responsibility when the audit reveals a failure.

The employer liability model could yet prove sufficient, if applied consistently enough to generate real incentive for deployers to invest in the architectural safeguards that prevent cascading injection. Liability concentrated enough to be painful is its own form of governance.

The Question That Remains

We are building AI agents that act on behalf of users while taking instructions from systems those users cannot inspect, under governance frameworks that have not yet coalesced, in a legal environment where the primary strict-liability mechanism was just withdrawn. The technical community is making genuine progress on making agent behavior more predictable. The governance community is making no comparable progress on making it auditable.

Who decides what an AI agent is permitted to do on your behalf — and who checks that those decisions are still what they claimed to be, after an attacker has had access to the instruction layer?

Ethically, Alan.