The Resume Flag That Lied to Me for Three Months

On 16 March 2026 the batch runner for our content pipeline landed in a single commit. That commit already contained a resume flag, a state log with per-item status, error classification, a quota pause, and a lock file. Resumability was not bolted on after it hurt. It was there on day one, and it still did not work for another three months.

That gap is the article.

When people write about resumable AI systems, they usually mean an agent remembering a conversation across turns: threads, checkpointers, replaying a graph from a saved node. This is the other thing. A batch pipeline that has to know which work is already finished, across separate runs and across two different machines.

The order in which we built it is not flattering, so I will put it first. Eleven days before the runner existed, we had a circuit breaker (stop after three consecutive quota failures) and a shutdown path that kills child processes when the parent is interrupted instead of orphaning them. That is survival of one call, not resumability. Four days after the runner, we shipped a retry that continued a half-written file instead of regenerating it, which is the classic confusion between the two: retry patterns are about surviving a call that failed, resume is about not paying again for a call that already succeeded. We built them in that order, and for a while I assumed the second came free with the first.

What the runner actually tracks

Two axes, kept deliberately separate.

An entity (for us: one topic that owns a set of articles) is in exactly one of five states:

A failure additionally carries one of six categories: timeout, quota, validation, phase_error, planning_error, unknown.

Keeping the category off the state axis is what lets paused and failed be different verbs at all. Quota exhaustion is not a defect. It is the system being told to come back later, which is exactly the distinction that separates pausing from failing in any batch that talks to a metered API. A validation error is the opposite: waiting a thousand years will not change a deterministic verdict, so it stops the run for a human.

One detail that only shows up once a run has actually been killed: it leaves its entities frozen in running, and nothing will ever move them on its own. So a resumed run sweeps stale running back to pending before it schedules anything. Note where that reclaim stops: it covers entity state, not the lock file the dead run also left behind. Those needed two different owners, and we found that out the hard way.

22 June: four holes in the same mechanism, all in one day

An audit had turned up four independent ways the resumable runner could be silently wrong, and all four fixes landed in one day. Not one bug with four symptoms. Four bugs.

(a) A quota error came back as an empty result instead of raising. The parallel aggregator treated it as a value, so the batch reported success, missing content was recorded as produced, and the run kept going. This one gets its own section below.

(b) Three separate places in the stack each recognised rate-limit errors their own way, with their own string matching. What one layer flagged, another missed, and which one you hit depended on where the error happened to surface. The fix was not to add the missing phrases in three places. It was to delete two of the matchers and have everything ask the same one, which is the same move as killing a whole class of error rather than the instance in front of you. We had learned the general shape of this before, when a single output contract had to hold identically across three different model backends: if a rule can drift between implementations, it eventually will.

(c) A corrupted state file was read as “fresh start”. Not as an error. As zero progress, cheerfully, and then we paid for everything again.

(d) The lock had a check-then-write window between “is anyone holding this” and “I am holding this”, wide enough for two runners to both walk through.

Where failures go to become data

Hole (a) is the one worth stealing, because the shape is everywhere.

Our generation step launches many agents in parallel for one topic and waits for all of them, collecting outcomes instead of letting the first failure escape. Every language has this primitive, and using it is correct here: one agent dying should not orphan its seventeen-odd siblings mid-flight.

But our runners, on permanent quota exhaustion, returned an empty result rather than raising. An empty result is a value. Values pass through a gather-all aggregator as data, not as alarms.

before:
  run(prompt) -> Result          # exhausted quota yields Result(text="")
  batch = gather_all(runs)       # every entry is fulfilled
  => batch reports success
  => caller records each item as produced
  => run continues into the next topic, against a closed window

after:
  run(prompt) raises QuotaExhausted
  phase re-raises it past the aggregator
  per-topic loop recognises the signature
  => entity moves to paused, not failed
  => the whole run stops and notifies

The bug was not “we forgot to check for errors”. We checked. The check ran against a result object that had been handed a successful-looking shape by a layer that had already given up.

If a failure has to travel through an aggregator, decide explicitly how it survives the trip. Raise, or wrap it in something the aggregator cannot flatten into a value. Deciding by accident means deciding “it becomes data”.

Failure mode, category, reaction

The table we now actually operate by:

That last row deserves honesty. File-based locking on one machine is not distributed coordination, and I am not going to pretend otherwise. What the next section buys us is narrower than it may sound: sequential runs on two machines agree on what is already done, as long as each one pulls the committed output before it starts. Two runs started at the same time are not coordinated at all, and nothing in this design stops them from generating the same work twice.

24 June: “done” stopped being something we stored

The real fix was not a better flag. It was deleting the authority of the state file.

Before: “is this topic finished?” was answered by reading a local, untracked progress file. After: it is derived from two signals that are committed to the repository.

done(topic):
    planned = the slugs the content plan promises for this topic
    return planned is non-empty
           and every slug in planned has a file at content_dir/<slug>.md

The state file still exists. It was demoted to a cache, rebuilt from that derivation, and no longer trusted when the two disagree.

Two consequences follow, and only the second one was obvious to us at the time.

The same committed inputs produce the same answer on any machine. I work on two computers. Machine B pulls, derives, and skips exactly what machine A generated, with no state synchronisation, no shared database, and no reliance on A’s local files. Note the word pulls: this buys agreement between runs that happen one after another, not mutual exclusion between runs that happen at once.

And the worst pre-fix failure mode was never corruption. It was waste. On a fresh clone, everything looked undone, so a run would burn an entire quota window re-researching roughly a hundred already-finished topics before it ever touched new work. Nothing crashed. Nothing warned. The run just spent the night rebuilding the past.

The honest limitation, stated immediately: this checks existence only. No hash, no size, no content inspection. An empty file passes. That is deliberate, because it is the cheapest test that behaves identically on every machine and requires no extra state, but it is a real limit, and the section after next is about where it bites.

26 June: it recovered without me

The unattended overnight loop hit a closed quota window, waited, and came back on its own. Article count went from 640 to 646 while I was asleep. No human touched it.

It reads no usage gauge, because that number is not available programmatically. So the strategy is deliberately dumb: generate until the wall, pause, probe, continue.

It had to learn two things the hard way. First, a closed window frequently surfaces as a plain exit code 1 with no quota signature anywhere in it, which is why unknown and timeout route to back-off rather than to a stop.

Second, blindly retrying into a closed window is expensive in a way that is invisible until you count. On 27 June one topic re-launched its full batch of around eighteen agents about twelve times, in the order of 150 process launches, every one of them dying on the wall. The fix was one cheap probe call before each round whose only output is an exit code: window open, quota exhausted, inconclusive. That shape matters more than it looks. Because the answer is an exit code, the shell wrapper that schedules the rounds never needs to parse an error message, and therefore cannot grow a second, subtly different copy of the quota-recognition logic we had just finished unifying.

4 August: the rule left the pipeline

By August this stopped being pipeline architecture and became a standing rule for every throwaway helper script I write: one file per paid call, and on startup skip whatever already exists on disk.

It earned that promotion. A 360-call collection was interrupted (the run record puts it around call 186), restarted at 187, and finished without paying again for the calls it had already made. One existence check, one file per call.

Where this is still wrong

The general theory of failure modes, idempotency and durable execution is well covered by people who know it far better than I do. What follows is specifically where our instance of it does not hold up, both found when a reviewer pushed on an earlier version of this article.

Output files are not written atomically. Our two run state files are: temp file, then rename. The generated articles are not. They go through a plain write. So a crash mid-write leaves a partial file, and the existence check accepts it as finished, forever, without complaint. This is not the same half-written file the retry knows how to continue: that one sits in the working area, where being unfinished is the expected state. This one has already crossed into the published tree, where existence is the whole proof. The boundary between “in progress” and “done” is that crossing, and the crossing is not atomic. We replaced one kind of false “done” with a rarer kind. The fix is the trick we already use one layer down: write to a temp file, then rename, so the file either exists complete or does not exist at all. There is even a central writer class that every generated file could pass through, which makes this a one-place change rather than a hunt.

“One file per call” is checkpointing, not idempotence. Our identity for a piece of work is the slug plus a hash of the content plan. Nothing else. Change the template, change the prompt, change the model, and nothing invalidates: the old file causes a skip, and the run reports success while serving work produced by a system that no longer exists. There is partial invalidation at the article level (the plan hash, a 14-day freshness window on research, re-verification when article or research content changed), but none of it covers prompt or template versions. If you build this, put the version of everything that shapes the output into the identity, or you will confidently serve stale work.

Worth separating one thing out: resumability answers “don’t redo finished work”. A completely different set of rules answers “when you do redo it, don’t destroy what appeared in the meantime”. We got that second part wrong too, in its own way, and it is its own article.

When none of this is worth building

The strongest objection is that resumability is overengineering, and a stateless rerun is enough. For a short, cheap, single-machine job that is simply correct, and checkpointing infrastructure is a tax on a problem you do not have.

Three conditions flip it. Paid calls make “rerun is free” false, and the price is exact and boring: our hundred re-researched topics. Multi-hour quota windows turn stopping into an operating mode rather than a fault, so “just run it again” is not a thing the operator can do at will. And the cheapest working form is not a framework, a queue, or a workflow engine. It is one existence check and one file per call. If the cheap version is off the table, the objection is usually arguing against something nobody proposed.

What I do not know

The existence check accepts an empty file, and I have not fixed it yet. I have no counter for how often the derived skip actually saves a run, so I cannot tell you the value in hours or euros. The interrupted-collection numbers come from a run record, not from a log I can re-read today. The overnight self-recovery is n=1. And the crash paths are only half tested. There are tests for a corrupted state file and for the atomicity of the state write, which is how those two ended up trustworthy. There is nothing for a kill during a content write, nothing for two runs starting at once, nothing for a plan that changed under a finished topic. Some of what is described here we found by reading the code, some by a reviewer pushing on this article, and the rest the least efficient way available, by it happening to us. The pattern is not subtle: the paths with tests stopped surprising us. That is the honest next step.

Here is the bet I would defend. Resumability is not a property of a run. It is a property of where the truth about finished work lives, and that truth has to be durable, shared, and independently verifiable. For us the authoritative store happens to be the repository, only because our output is files that get committed anyway. For you it may be a database, an object store, an event log, or a workflow engine. The store is an implementation detail. The three adjectives are not.