When an Agent Fails Without Throwing: Operating for Drift

The agent went into production in April. It did not crash. Six weeks later the numbers that actually matter had moved anyway: the task-resolution rate your team quietly depends on had slipped, and per-task cost had crept up enough that finance asked a pointed question. So you do the thing that has closed every incident of your career — open the deploy history and bisect. Nothing shipped. No commit, no config change, no alert, no on-call page. Then you reach for the tool that always cracks the case: pull the failing runs and watch exactly where they go wrong. There is no replay. The old runs are gone, and the ones you can reconstruct never execute the same way twice.

Nothing failed. Something drifted.

Operating an AI agent means running a system whose failures do not throw. Reliability work shifts from catching exceptions to detecting behavioral drift across many runs — a slow slide in quality or cost that no single request crosses a threshold to announce. The assumption that breaks is the one under every debugging habit you own: that a failure is an event you can bisect, reproduce, and replay. An agent failure is a distribution, and it moves.

Your Postmortem Has No Root-Cause Line

The incident review is where the new world announces itself. You gather the trace, you walk the timeline, and you get to the line every postmortem needs — root cause — and there is nothing to write. No failed request, no thrown exception, no bad deploy. Just a system that started answering worse.

Your instincts here are mostly right, and that is the trap. Agent Observability extends the distributed tracing you already run: the same span tree, the same trace-context propagation across every hop, the same discipline of tagging each step with a user ID and a session ID so you can pull one customer’s path out of the noise. If you have instrumented microservices, you already know how to instrument an agent — the span-design mechanics carry over almost verbatim. That instinct still saves you. Keep it.

What breaks is what the span has to say. That instinct no longer holds once the span must explain a decision rather than clock a call. A microservice span answers a bounded question: how long did this RPC take, did it return 200. An agent span has to answer something stranger — what did the model decide, against what context, and which tool did it pick as a result. And here the plumbing hits a wall the deep-dive on OpenTelemetry’s GenAI limits names directly: agent runs are non-replayable. You can record every step, but you cannot re-execute the trace against the same input and get the same path. The record is a photograph, not a debugger.

Mental Model Map: agent reliability From: A failure is an exception you catch, bisect to a deploy, and replay Shift: Agent failures are distributions that drift with no deploy and no replay To: Reliability means detecting behavioral drift across many runs, not catching one Key insight: You stop asking “which request threw” and start asking “when did the distribution move.”

In practice, this means your trace is for reconstruction, not reproduction. You will diagnose an agent by comparing many runs to find where the population changed, not by re-running one failing case until it confesses.

The Alert You Cannot Write

Sit down to write the alert and the problem gets concrete. Alerting assumes a signal that crosses a line: latency over budget, error rate above one percent, a queue past its depth. Drift has no line. Quality does not fall off a cliff on request 4,096 — it sags a little across thousands of runs, and no single one of them is wrong enough to fire.

The framing you brought from operations still holds. SLOs, error budgets, on-call escalation, runbooks — all of it survives the transition. You still define a target, still spend a budget, still page a human when the budget burns. That scaffolding is correct and you should keep every piece of it.

The signal that trips the scaffolding is what changed. It is no longer a request crossing an error threshold; it is a slow statistical shift you can only see by scoring outcomes over time. Telemetry alone will not surface it — and this is the gap most teams walk into. Across a 2026 survey of agent teams, roughly 89% had shipped observability while only 52% had shipped evaluation (a directional figure, but the shape is real): the traces are being collected, the grader is not. A trace tells you what the agent did. It cannot tell you whether what it did was good. For that you need Agent Evaluation And Testing running as a production signal, not a pre-merge afterthought.

And the grader has to watch the whole path, not the last message. Score only the final answer and you miss the agent that fabricated an ID, hit a permission error, apologized fluently, and recovered to a plausible reply — a clean output over a broken trajectory. Vendor research in 2026 put a number on the blind spot: agents pass roughly 20–40% more test cases under output-only scoring than under trajectory-level scoring. That gap is exactly the space where drift lives unmeasured. It also explains why a benchmark score and a production score diverge — Galileo’s measurements found a 37% gap between agent benchmark performance and real-world deployment.

In practice, this means the alert you cannot write on a threshold becomes a trend you have to grade. The on-call rotation stays; the trigger moves from “a request failed” to “the score has been sliding for a week.” For the full prerequisite map and the common questions, the topic hub is the register — this article stays on the one shift that reorganizes your whole operations practice.

Retries Turn Loud Failures Into Quiet Ones

Here is the misconception that ships the most fragile agents: “we have guardrails and retries, so it’s production-ready.” Both are good engineering. Both also make failure quieter, and quieter is not the same as rarer.

Watch what a retry-and-fallback layer does to your signal. A tool 500s, the retry catches it. The model returns malformed JSON, self-correction re-prompts and fixes it. A step fails validation, the fallback substitutes a degraded answer. Every one of those is the right call in isolation. Together they convert a loud, pageable failure into a silent, graceful one that resolves before anyone looks — and graceful degradation, by design, hides the very event you needed to see.

Your recovery instincts still transfer, and they matter more than ever. Agent Error Handling And Recovery is built from primitives you already own: typed retry contracts with bounded attempts and backoff, idempotency keys so a retried action does not fire twice, durable execution so a crash at step seven resumes instead of restarting. Three independent frameworks converged on that last point — persisting state between every step is now the assumed substrate for any long-running agent. One caveat the classical model does not prepare you for: the idempotency key must not be generated by the model. Draw the line so deterministic code mediates every action with a side effect, and the non-deterministic part can stay non-deterministic without poisoning the ledger.

The same quiet-failure logic applies to safety. Agent Guardrails are layered classifiers around the tool-use loop — useful, and demonstrably partial: they are runtime filters with measurable holes, not fences. Treating them as a binary “safe/unsafe” gate is how a policy violation slips through and never surfaces. And when degradation does need a human, the escalation path is its own design problem. Human In The Loop For Agents only works if the human still has attention left to spend — route too many low-signal approvals to a reviewer and you get rubber-stamping, which is oversight on paper and nothing in production.

In practice, this means every resilience layer you add is also a place a signal can disappear. Instrument the recoveries, not just the crashes — a spike in silent retries is often the first visible edge of a drift you would otherwise find in a support ticket.

The Invoice Becomes Your First Alarm

For many teams the first honest signal that agent behavior changed is not a graph they built — it is the bill. Cost drifts before quality complaints arrive, because a subtly worse agent loops more, retrieves more, and retries more to reach the same answer.

Your capacity instincts need re-pointing here. Agent Cost Optimization does not scale the way request-rate capacity does. Cost is a function of tokens and iterations, not calls, and it degrades with input length rather than traffic — output tokens run several times the price of input, and a single volatile value near the front of a prompt (a timestamp, a session ID) silently breaks the cache you were counting on. A load test that varied request rate but held sequence length constant told you nothing about the shape of this bill. Treat per-step cost as a first-class telemetry signal, wired into the same traces as latency and errors, and the invoice stops being the alarm of last resort.

Shift Diagram: agent reliability operations Classic: request fails → alert fires → bisect to deploy → replay and fix AI: quality drifts → no alert, no deploy to blame → grade trajectories over time → locate the shift

What Transfers and Where It Snaps

Most of your operations toolkit survives the move to agents. The failure is assuming it survives unchanged. The instincts transfer; the trigger conditions do not.

The pattern under the whole table: your instincts about response are intact, and your instincts about detection need rebuilding. You still know what to do once you see the problem. Seeing it is the part that changed.

Before You Call Your Agent Reliable

Run this against your own stack before you tell anyone the agent is production-ready. These are questions for your system, not a checklist to publish.

If more than one of those has no answer, you do not yet have an operable agent — you have one that has not drifted noticeably yet.

The mental model to carry out of here: you are no longer operating a service that fails by throwing, but a system that fails by drifting, and the work moved from catching the exception to grading the trend. Your next concrete step is to put one grader on live traffic — scoring trajectories, not final answers — and watch it for a week before you trust any dashboard that only shows you green.