The LLM Evaluation Metrics We Actually Run: What 90+ Checks per Article Taught Us

I run a content pipeline where AI writes every article — and where AI is, on principle, not trusted. Before any piece ships to our site, it survives more than ninety separate verifications: research checks, fact cross-referencing, a deterministic validator with dozens of rules, integration guards. We never sat down and said “let’s build an LLM evaluation harness.” We sat down and said “let’s not publish embarrassing text,” and then kept adding checks every time something embarrassing nearly got through.

It took me surprisingly long to notice that what we’d accumulated is an evaluation harness — the same category of thing the eval frameworks and judge models are selling, just grown organically around one production system. This is a tour of what those metrics actually are, in the order they run — and of the day the whole stack taught me its own limit.

The Shape of the Stack: Code First, Judgment Last

The ninety-plus number sounds inflated until you itemize it, so let me itemize it. Before a single sentence of an article exists, the research that feeds it passes about twenty checks of its own — is the fact base structurally sound, are the sources real and reachable, is anything suspicious hiding in fetched web content. After the article is written, every factual claim in it gets cross-referenced against the researched fact sheet — around twenty more verifications, this time performed by an AI agent. Then a deterministic validator runs roughly thirty-nine atomic rules over the finished text. Finally, eight guards fire during integration into the site build. Add the self-correction re-checks and you’re past ninety verifications for one article. A full subject — six articles, ten glossary entries, one hub page — crosses a thousand.

The architecture behind that pile is one sentence long: anything code can verify, code verifies; the LLM judges only what code cannot. It’s the single most load-bearing decision in the pipeline, and we made it early, mostly out of cheapness — which is also the honest answer to anyone asking about the ROI of LLM evaluation. Deterministic checks cost nothing, run in seconds, and give the same answer every time. LLM judgment costs money, takes minutes, and drifts. So the expensive judge should only ever see questions the cheap rules can’t answer.

Layer One: What a Regex Can Catch, a Regex Should Catch

The deterministic layer is unglamorous and does most of the work. Structure: required sections present, headings unique, frontmatter complete. Ranges: word counts inside bounds, image alt-texts within limits. Forbidden patterns: the vocabulary that makes text smell machine-written — we call the category “AI-tell words” — and something nastier we call “scaffolding leaks,” where a fragment of the internal writing instructions bleeds into the published prose (“as mentioned in the brief…”). Link integrity, metadata lengths, image references. Thirty check functions in the main validator; the variant for our developer-orientation articles runs thirty-four checks.

None of this measures whether an article is good. That’s the point. This layer defines a floor — the set of failures that should never require human attention again, because a machine catches them identically every single time. If you’re building any LLM evaluation setup and your judge model is checking whether required sections exist, you’ve put the expensive instrument on the cheap problem.

Layer Two: A Judge With the Narrowest Possible Job Description

There is exactly one place in the pipeline where LLM judgment gates content, and its job description is deliberately tiny: claim verification. The agent takes each factual claim in the finished article and asks one closed question — is this claim supported by the fact sheet we researched?

The answer is one of five labels, and the three beyond the obvious pair are where the value hides. A claim can match the research. It can distort it — “$24” written as “$25”, “34%” written as “nearly 40%”. It can be invented, with no basis in the fact sheet at all. It can ignore a caution: the research explicitly noted “enterprise pricing not publicly available” and the article states a price anyway. Or it can be missing attribution — a real number, correctly copied, presented as if it were common knowledge. Anything scored high or medium severity fails the article into a bounded self-correction loop; what can’t be fixed gets surfaced for a human.

That fourth label is the one I’d recommend to anyone building this. Invented facts are the failure everyone designs for. Quietly promoting a known unknown into a stated fact is the one that actually ships.

The narrowness is the design. The research on LLM-as-a-judge is fairly brutal about broad rubrics: ask a model to “rate this article 1–10” and you inherit position bias, self-preference, and scores that drift between sessions — the judge is fast, not objective. But shrink the question to a closed world — this claim, this reference document, supported or not — and the same unreliable grader becomes a usable instrument. We never ask our judge for taste. We ask it for lookups that happen to require reading comprehension.

Worth saying plainly, because the eval-tooling market implies otherwise: there is no general quality judge anywhere in our pipeline. We designed one on paper once, an elaborate multi-dimension rubric, and never built it. Everything that gates production is either deterministic or a closed factual lookup. That absence is a decision, not a gap we haven’t gotten to.

Layer Three: Metrics Over Time, Not Just Per Article

A check that passes today tells you nothing about whether the pipeline is quietly getting worse. So after every generation run, an automated comparison diffs the run’s metrics against the last successful baseline: validation errors, claims flagged for fixing, article and glossary counts, token usage, cost. Regressions get a severity marker; the report separates “act” items (quality got worse) from “watch” items (cost got worse).

One detail from that report I’ve become disproportionately proud of: duration is never flagged. Runtime is the noisiest metric we track, and a noisy metric that triggers alarms trains the operator to ignore alarms. It’s shown for reference, unmarked. Deciding which metrics are not allowed to page you turned out to be as important as choosing the metrics themselves — alert fatigue is an evaluation failure mode, not just an ops one.

The Day the Metrics Maxed Out

Then I got curious about something bigger than a version bump: could the whole pipeline run on a different vendor’s model? Not a swap of one Claude for a newer Claude — a different company’s model, reached through a different backend, driving the same agents, the same brief, the same validators. If the architecture was as model-agnostic as I believed, it should barely notice.

So I ran the cleanest experiment I could design: the same locked brief — same facts, same structure requirements, same voice specification — generated once by Sonnet 4.6, which the pipeline ships on, and once by GPT-5.5 through the Codex backend. Then judged blind.

Both articles passed all thirty-four automated checks. Zero errors, twice. One came in at 2,594 words, the other at 2,552 — a rounding error apart. On paper, my entire evaluation stack declared the two outputs indistinguishable.

They were not indistinguishable. Reading them side by side, the voices were plainly different — one hit the register we tuned for, the other was competent in a way that didn’t sound like us. Afterwards I measured the thing my eye had noticed: average sentence length, 15 words against 11.8, a 21% difference. Punchy, magazine-shaped, perfectly good English. Not ours. And not a single one of my ninety-plus verifications was watching sentence rhythm, because nobody had ever thought to make it a rule.

That’s one sample, not a benchmark. But it demonstrated the boundary crisply: deterministic metrics measure the floor, and both candidates were standing on it. The floor had stopped being the question.

What the Human Read Caught — and What Happened Next

Here’s the part I’d have skipped a year ago, because it makes us look worse before it makes us look better.

Reading those two articles by hand turned up two differences the harness had missed, not one. The voice was the obvious one. The other was small and specific: the second article occasionally referred to its own scaffolding — a sentence pointing at “the fact sheet” as if the reader could see it, an internal artifact leaking into published prose. Exactly the failure category we already had a name for. Our validator just couldn’t see this instance of it, so the article passed thirty-four checks with zero errors while quietly doing the thing we’d forbidden.

Four days later that leak was a deterministic check.

Not a rubric, not a judge prompt — it shipped as six contextual regexes in a shared module, wired into both validators, severity ERROR. The interesting part is the word contextual. You cannot flag the phrase “the brief,” because “the brief” is ordinary English and appears in perfectly good sentences. What you can flag is an internal artifact being referenced as a source of truth the reader is assumed to share. Writing that check meant thinking harder about the failure than the failure deserved, which is roughly the definition of making a floor.

So the real shape of that week isn’t “metrics saturated and the harness was useless.” It’s this: the harness saturated, a human read found two things, and one of the two got promoted into the floor within four days. A floor is not a fixed height. It’s a ratchet.

And I know it’s a ratchet rather than a nice story, because six weeks later the same thing happened again — this time across a hundred articles at once, and this time the culprit was us.

An audit of a different question entirely turned up instruction words sitting in the prose of 103 articles, 53 of them already live and the rest waiting their turn to publish. Not a model failure — the template required the offending marker. We had written the instruction into the label the article was supposed to print, and the pipeline had done exactly as it was told, faithfully, one hundred and three times. The fix swapped the label corpus-wide, corrected both templates, and added a seventh pattern to that same module, so both validators catch it now.

Two promotions, then. Neither was found by the harness — one by reading, one by an audit looking for something else. That is the uncomfortable part and I’d rather write it down than round it off: the checks catch what we already know to look for, and the only thing that finds a genuinely new failure is a person paying attention to something else. The harness’s job isn’t discovery. It’s making sure a discovery only has to happen once.

The other one, the voice, never became a check. It can’t. And what does see it is the oldest trick in model evaluation: blind pairwise comparison. Not “score this 1–10” — absolute scores drift and hallucinate precision — but “here are A and B, which is better on this specific quality, quote the line that proves it.” Relative judgment, evidence required, run by a judge that doesn’t know which output came from which model. That ritual lives entirely outside the pipeline and never gates production. It answers the one question the harness can’t: when both candidates clear the floor, which one is actually better?

What Transfers (or: Common Mistakes in LLM Evaluation, Inverted)

If you’re evaluating LLM output in any production system, the shape of what we learned travels better than the specifics — each of these is a common mistake in LLM evaluation, written down the way we’d want to have read it:

• Deterministic before subjective. Every check you can express as code is a check your judge model never wastes attention on.
• Give the LLM judge the narrowest job you can write. Closed world, closed question. Judgment quality scales inversely with rubric breadth.
• Track deltas between runs, not just per-output passes. Slow degradation is invisible to per-article checks.
• Decide which metrics may not alarm. A noisy alert channel is worse than no alert channel.
• Treat the floor as a ratchet. Every failure a human catches for the second time is a check you owe yourself. Count how often that promotion actually happens — it’s the honest health metric for an eval stack, and it’s the one nobody publishes.
• Watch your own instructions, not just the model’s output. One of our two worst leaks was mandated by our own template. A harness pointed exclusively at what the model produces cannot see a defect in what the model was asked for.
• When metrics saturate, switch instruments. Passing everything is not the same as being good; past the floor, only blind relative comparison discriminates.

What We Don’t Know Yet

Honesty section, because a worklog without one is marketing. We don’t know whether our deterministic floor correlates with reader-perceived quality — the site is young, most of our topical queries sit somewhere on page four or deeper, and we don’t have engagement data at the granularity that would answer it. We don’t know what our claim verifier misses: we count what it flags, and nobody has run a ground-truth audit of what sailed through. The cross-vendor result is n=1 by construction — one article, one subject, one frozen brief; a signal about one content shape, not a scoreboard.

And the ratchet rate I just told you to measure — I can give you two promotions with dates and commits, but I can’t give you a rate, because I only recognized the pattern while writing this piece and went looking backwards. Two is what I can prove, not what happened.

The harness keeps us from shipping broken things. Whether it’s nudging us toward genuinely better things is a claim I’m not yet entitled to make.