117 Identical Failures, Zero Bugs: Anatomy of a Flaky AI Agent

Between May 15 and July 2 of this year, the session transcripts of our content pipeline accumulated at least 117 copies of the same error. File does not exist. One error class, 117 occurrences, spread across seven weeks of overnight runs. When I finally sat down and traced it, I found no bug. Not one line of code was doing anything other than what it was written to do.

An error that fires 117 times in a codebase with nothing wrong in it is not a story about sloppy engineering. I think it is a genuinely new class of failure, one that people who have written software for twenty years have mostly never had the chance to meet. This is my attempt to describe it from the inside.

First, the disclaimer I owe you: I am an IT analyst, not a software engineer by training. I run the content pipeline behind bestaiweb.ai together with a colleague who is the actual programmer. My side is orchestration, audits, review. That division of labor turned out to matter, because the thing that cracked this open was not a debugger. It was counting.

Eighteen agents a night

The pipeline generates technical articles overnight. One topic takes roughly 18 agent sessions: research agents, an article writer, a claim verifier, image generation, validators, all coordinated through agent orchestration. Each agent receives a small YAML file the pipeline calls a brief: here is the article you are writing, here is the fact sheet, here is where your output goes. The brief is produced by deterministic code, TypeScript and Python, and consumed by an LLM agent.

That handoff is the exact spot this whole story lives in. On one side of the file, code. On the other side, an interpreter.

Scale matters for what comes later, so, briefly: hundreds of agent sessions a week, running unattended through the night. We have measured the economics of this setup before, in prompt caching measured on our own bill; the short version is that the pipeline is big enough for per-call costs and per-call failures to add up to real money and real hours.

The error that would not hold still

Around the turn of June and July, one line started recurring in the transcripts. Here it is verbatim, because the detail at the end turned out to be the whole plot:

File does not exist. Note: your current working directory is /Users/userxy/code/your-project.

Read that note again. The agent helpfully reports its own working directory, and that helpful note is precisely the trap: it is the base the agent is tempted to resolve a relative path against, whether or not that base is the right one.

The maddening part was the pattern, or rather the lack of one. The same pipeline phase, on the same kind of input, would pass on Tuesday and fail on Wednesday. The pipeline has retries, and the retries mostly absorbed it: a failed read got retried, the agent tried another path, found the file, moved on. Articles kept arriving in the morning, so nothing looked broken. That is retry masking doing exactly what retry masking does: converting failures into costs. Now and then a cousin showed up, an EISDIR error, the agent opening a directory as if it were a file, the same disease with a different symptom.

There is a tell in how the rest of the world reads this symptom. Go looking for the causes of an AI agent intermittent file not found error and you land squarely in the classical world: race conditions, temporary files deleted too early, a directory that did not exist yet. Good answers, wrong disease. None of them describe an executor that reads the same contract twice and resolves it differently.

Here is the part I keep trying to explain to developer friends. If classical, deterministic code had this bug, it would be boring. Mixed path conventions in ordinary software fail consistently: first run, same stack trace every time, found in a minute, fixed before coffee. Deterministic code fails deterministically. What we had was different: identical code, identical input file, different outcome per run. That is the signature of nondeterministic AI agent failures: a nondeterministic failure of a deterministic-looking input. Nothing in a classical software career prepares you for a bug that comes and goes while nothing changes.

Counting, because watching had failed

A confession before the numbers: our monitoring never saw any of this. The pipeline writes run-reports after every run, and those reports captured this error exactly zero times. The failures lived one level lower, inside the session transcripts, where a retried error leaves a trace but no alarm. An error that a retry survives is invisible in production. If you only watch outcomes, it does not exist. The only way to find a class like this is to count it across runs. Monitoring flaky AI agents is a counting problem, not a dashboard problem.

So on July 2 I did the unglamorous thing: a transcript audit. Mining 1,811 session transcripts from May 15 to July 2 and tallying error shapes. The path class came out at 117 occurrences, at minimum. At minimum, because transcripts truncate long messages, and whatever scrolled out of a truncated message was never counted. The path errors were not alone in there either; the same audit surfaced three sibling classes, which I will get to, because they all ended up in the same commit.

Why chase 117 errors the retries already absorbed?

You could reasonably push back at this point. 117 hits across 1,811 transcripts over seven weeks, with retries that worked and content that shipped, sounds like noise. Why chase it? Three reasons, and I want to state them plainly rather than dramatically. One: 117 is a floor, not a count. Two: every one of those retries was paid tokens and paid minutes, on a pipeline that runs every night. Three, the real one: triage. A deterministic error you triage once and file away. A nondeterministic error cannot be triaged at all, because every occurrence looks new; it burns a little investigation every time it appears, indefinitely. Against all that, the fix turned out to be one commit. When fixing costs one commit and not fixing costs an open-ended tax on attention, “was it worth it” stops being an interesting question. The honest boundary of this argument: the arithmetic holds for a multi-agent pipeline with hundreds of runs. A hobby script that makes five LLM calls will never meet this class often enough to know it exists.

Two kinds of paths in one contract

The diagnosis took less time than the counting. I opened the briefs the failing agents had received. Inside a single YAML file: some paths written out absolute, starting /Users/userxy/code/your-project/..., and a few fields below them, the article path written repo-relative — site-generation/workspace/..., no leading slash, no declared base. Two conventions in one contract. And nothing in the file saying which base a relative path resolves against. The agent’s working directory? The repo root? The folder the brief sits in? A code library would have a documented convention for this. The agent had nothing but a guess.

And the guess is made fresh, per run. This is the first thing I would want a classical developer to take away, so let me put it as plainly as I can: with an interpreting executor, ambiguity does not compile down to one consistent wrong behavior. Each run resolves the ambiguity independently, so the same underspecified input becomes a probability distribution over behaviors. Ambiguity in, probability out. On nights when the guess matched the real layout, everything passed. On the other nights: File does not exist, with that working-directory note sitting in the message like a friendly wrong hint.

I want to be fair to the agent here. Given the contract it received, none of its guesses were unreasonable. It was not being stupid; it was being an interpreter. It filled the gap the contract left open, the way it fills every gap, which is the entire reason we employ it. Paths just happened to be a place where we wanted no filling at all.

The fix we refused, and the one we shipped

The tempting shortcut would have been prompt-patching: add a rule to the agent prompts, “when a path is relative, resolve it against the repo root”, and hope. That temptation is worth refusing, and the refusal matters more than the fix. Patching prompts to handle a path convention means fighting probability with more probability. You add words to a nondeterministic layer and the miss rate maybe drops; you can never show it reached zero. Prompt engineering is the right tool for shaping judgment, voice, and reasoning. It is the wrong tool for facts that a deterministic layer already owns. Which is the whole question when you are fixing AI agent errors: prompt vs code is not a matter of taste, it is a matter of which layer owns the decision. The prompt is not decoration around the work; it is part of the execution contract. The layer that manufactured the ambiguity was the brief generator. So that is where the fix belongs.

This is the shape of it, from the generator code; wt is the absolute worktree root:

- f"{args.phases_base}/06-article-generation/…"
+ f"{wt}/{args.phases_base}/06-article-generation/…"

One prefixed base. After that line, the generator is not able to emit a relative path. Not instructed not to. Not able to. That is the difference between patching a prompt and fixing the deterministic layer: the error class is not discouraged, it is unrepresentable. The same normalization went into every path field a brief hands to an agent.

But a contract has two ends, and fixing one end is half a fix. Change only the generator, and the agents keep their learned habit of helpfully joining paths against the working directory whenever something looks off. Change only the agent side, and the next generator someone writes reintroduces relative paths that the agents now trust as absolute. So the convention went into all five agent contracts in the same commit, stated flat: paths in briefs are absolute, use them as-is, never join them with your working directory. Producer and consumer, both ends, one commit, the same day as the audit.

That commit’s own accounting is the artifact I would frame and hang on a wall:

One commit, 20 files. Four error classes closed at once:

• 117× path resolution failures
• 172× writes to files never read first
• ~20× edits anchored to stale file content
• ~19 sessions where an LLM fixer was dispatched to repair a formatting error a script could fix deterministically

I show the list for what its items have in common: every class on it is a place where a deterministic layer left a decision to a probabilistic one. Path resolution was merely the loudest of the four.

If I compress everything above into the two rules I now actually follow, they are these. Where to fix: in the deterministic layer that produces the ambiguity, never in the prompts that struggle with it; kill the class, not the occurrence. And how to write an agent contract: stricter than a contract for code. With a library you leave path bases, orderings, and defaults to convention and documentation. With an agent, everything you would normally leave to convention has to be explicit, and the contract has to be enforced on both sides, in the code that produces the input and in the contract of the agent that consumes it.

789 briefs, zero relative paths

A claim like “this one line kills the class” deserves a check against reality, not only against tests. The verification note from that day, quoted as recorded: “real generate-briefs.py verify run = 789 briefs with zero non-absolute paths”. A real run of the generator, 789 briefs out, not a single relative path in any of them. The TypeScript and Python test suites were green on top of that, but the number I trusted was the 789, because it came from actual pipeline artifacts rather than fixtures.

Since then the rule has lived in the project’s written conventions and in the agent contracts, not in anyone’s memory of a bad week. A new generator or a new agent inherits it by default. That is the part I am quietly proudest of: not that an error got fixed, but that a class of error stopped being expressible.

What we still don’t know

Some honesty before the closing argument. We never re-mined the transcripts after the fix, so I cannot claim the error never happened again; I have no observation to point to. What I can claim is shaped differently: the generator can no longer produce a relative path, so the proof moved from observation to construction. Also: 117 is a lower bound, not a measurement. We do not know the per-call failure rate: the denominator was transcripts, not tool calls, and we never established how many path resolutions the agents attempted in total. And our run-reports captured none of this, ever; everything in this article exists because someone went into the transcripts and counted. Skip that step, and this class would still be firing every night at some unknown rate, fully paid for and fully invisible.

Stricter than code, on purpose

Here is the bet I have landed on, and I am writing it down as a bet, not as a moral.

Contracts for agents must be stricter than contracts for code. Not because agents are dumb, but because they interpret where code executes. Hand deterministic code an ambiguous contract and you get one consistent wrong behavior, discovered on day one, fixed by lunch. Hand an interpreter the same contract and you get a distribution of behaviors, delivered one improbable failure at a time, smeared across weeks of retries where no single failure looks worth investigating.

Which means every ambiguity sitting in your prompts and your agent inputs right now is not a slack spot in the spec. It is a probabilistic error that has not happened yet. Ours had a number, 117, and that number was a floor. If you run agents at any scale and your pipeline retries, yours has a number too. You just have not counted it.

You do not need a transcript audit to start. Take any boundary where your deterministic code hands something to an agent, and ask three questions:

Does the agent have to infer anything deterministic? A path, an ID, an ordering, a format, a destination. Anything your code already knew and merely failed to say out loud.

Can the same contract be read two reasonable ways? The question is not whether it is wrong. It is whether it is underspecified. Two plausible readings mean two behaviors, and each run picks one.

Can retries hide the resulting failures from your run-level monitoring? If they can, then quiet dashboards are not evidence of anything.

Any yes, and that boundary deserves an audit before it deserves a better prompt. Ours cost one commit to fix, and 1,811 transcripts to find.