When Guardrails Fail: Who Is Accountable When AI Agents Misbehave

We have built a generation of systems whose entire safety story rests on a layer of probabilistic filters and decorative rules. We call them Guardrails because the metaphor is comforting. But a guardrail on a highway is metal. A guardrail on an agent is a model trying very hard to predict what a human would call “the line.” That distinction is going to matter, and very soon, in courtrooms most builders are not watching.

The Question We Keep Postponing

For most of the last two years, the conversation about agent safety has been a conversation about engineering. Better classifiers. More layered defenses. Cleaner evaluation harnesses. The implicit promise was that if we got the engineering right, the ethical question would resolve itself — accountability would land in some obvious place once the failure modes were understood.

That promise is starting to look thin. The engineering has improved. The failures have not stopped. And the question we kept postponing has become loud: when an agent crosses a line it was supposed to respect, who is the human on the hook? Not legally — although that matters. Morally. The kind of accountability you cannot insure away.

What the Industry Wants Us to Believe

It is worth steelmanning the optimistic case, because it is not foolish. Modern guardrail stacks are genuinely impressive. NVIDIA’s NeMo orchestration layer, in its current form, organises rails into five categories — input, dialog, retrieval, execution, and output — and routes every agent decision through each gate (NVIDIA NeMo Guardrails GitHub). Around it, hazard classifiers trained with reinforcement-learning techniques like PPO (Proximal Policy Optimization) sit downstream as a second filter. MLCommons launched its Agentic Reliability Evaluation Standard with Anthropic, Google, Microsoft, and OpenAI as co-signatories, organising agent evaluation around four pillars — correctness, safety, security, and control (MLCommons).

There is a real argument here. The argument says: the rails are getting better, the Agent Evaluation And Testing disciplines are getting more rigorous, and the institutional infrastructure — NIST’s AI Agent Standards Initiative launched in February 2026, OWASP’s separate top-ten list for agentic applications — is starting to fill in. Give the engineers another cycle and the worst failures will shrink to a manageable rate. Then the legal questions will be tractable, the way product-liability questions for cars eventually became tractable.

This is a respectable position. It is also incomplete in a way the industry rarely names.

The Assumption Hidden Inside the Engineering

The optimistic case quietly assumes that guardrails are a kind of fence — present or absent, holding or broken — when in practice they are a layer of statistical predictions about what a fence ought to be. OWASP’s 2025 list places Excessive Agency at number six precisely because the failure pattern is not “the guardrail broke.” The failure pattern is excessive functionality, excessive permissions, and excessive autonomy compounding until the agent does something nobody told it to do and nobody told it not to do (OWASP Foundation).

Galileo’s measurements show a 37% gap between agent benchmark scores and real-world deployment performance (Galileo AI). That number is not a curiosity. It is the size of the territory in which the human on the hook is operating without a map. And the territory is not stationary — it shifts every time a model is updated, every time a tool is added, every time an upstream document is poisoned.

So what are the ethical risks of relying on agent guardrails for safety? The honest answer is that the deepest risk is not catastrophic failure. It is moral outsourcing. Once a team has installed the rails, configured the evaluations, and run the red-team suite, there is a powerful psychological pull to treat the remaining risk as somebody else’s problem — the model vendor’s, the customer’s, the regulator’s. The rails become permission to stop thinking. That is not a software defect. It is a culture defect, and it is being built into the foundations of an entire generation of products.

The Lesson the Engineers Did Not Read

We have been here before, and the precedent is older than the internet. Industrial societies spent the better part of the twentieth century learning that safety mechanisms create their own ethical hazards. When factories installed machine guards, accident rates fell — and then plateaued, and then began to drift back up, because workers and managers reorganised behaviour around the assumption that the guard would catch any mistake. The legal response was not to remove the guards. It was to refuse to let the existence of the guard relieve anyone of judgment. Liability stayed with the human who chose to operate the system, no matter how much steel sat between hand and blade.

The Moffatt v. Air Canada ruling in February 2024 was the first quiet signal that courts may apply the same logic to AI agents. The British Columbia Civil Resolution Tribunal dismissed Air Canada’s argument that its chatbot was a separate legal entity and held the airline liable for what its agent told a grieving passenger about bereavement fares; damages came to C$812.02 (CBC News). The figure was small. The principle was enormous: the chatbot, the tribunal said, is part of the website. Pending matters — the alleged first AI CSAM class action against xAI (The Meridiem) and the Character.AI / Google self-harm cases that reached a sealed mediated settlement in January 2026 (AI CERTs News) — are testing how far that principle stretches when the harm is not a refund but a child.

A Thesis the Industry Will Not Like

Thesis: Accountability for agent failures cannot be engineered into the agent. It can only be located in the humans who chose to release it, and the institutions that chose to let them.

This is not a comfortable conclusion for builders, because it refuses to hand the moral question back to the model. It does not let the developer off because the model was certified. It does not let the certifier off because the deployer accepted the terms. It does not let the regulator off because the standard was voluntary. It distributes responsibility along the chain of choices that put the agent in front of a human, and it insists that each link in that chain knew what it was choosing.

The EU AI Act, in Article 26, gestures at this when it requires deployers of high-risk AI to ensure human oversight, monitor operation, and keep logs (EU AI Act portal). The fines — up to EUR 35 million or 7% of worldwide turnover — are large enough to concentrate the mind. Whether the high-risk obligations actually take effect on 2 August 2026 or are deferred to 2 December 2027 under the proposed Digital AI Omnibus is genuinely unresolved as of this writing; the 28 April 2026 trilogue ended without agreement (DLA Piper). The underlying instinct, though — that the deployer cannot hide behind the vendor, and the vendor cannot hide behind the model — is the right one. The open question is whether jurisdictions outside Europe build something equivalent before the harms force them to.

Questions We Owe Ourselves

If accountability lives with people, not artefacts, then the practical questions change. Stop asking whether the rails are good enough. Start asking who, on your team, is morally entitled to authorise an agent to act on a customer’s behalf. Ask what their training looks like. Ask what they are allowed to say “no” to. Ask whether your incident playbook treats an agent failure as a system event or as a human decision that produced the wrong outcome.

Ask, too, what your organisation does when the agent works exactly as designed and still produces harm — because that is the case the rails were never going to catch. Insurers have already started to answer the version of this question that affects their balance sheets: major carriers added AI-specific exclusions in early 2026, signalling that the assumption that errors-and-omissions and cyber policies cover agent failures is increasingly wrong (Tom’s Hardware). The ethical question runs ahead of the actuarial one. It always does.

Where This Argument Is Weakest

The strongest counter to everything above is empirical: if agent reliability improves faster than agent scope expands, the accountability gap could close on its own. A future with verifiably bounded agents — provably unable to take certain actions, cryptographically logged, externally audited — would make the human-judgment layer thinner without thinning out responsibility. If MLCommons ARES, NIST’s forthcoming AI Agent Interoperability Profile (planned for Q4 2026), and OWASP’s agentic top ten converge into a real interoperable standard before the harms scale, the argument that accountability must live in humans softens to “accountability must live in humans for now.” I would welcome being wrong about how soon “for now” ends.

The Question That Remains

We built guardrails because we did not want to face the question of who is responsible when a system we do not fully understand acts on a person we do not know. The rails work, sometimes. The question they were supposed to answer has not gone anywhere. Are we ready to admit that no amount of engineering will tell us which human owes the apology — and to choose that human, by name, before the agent ever speaks?