RAG Regresses Without a Deploy: From Green Build to Live Eval

Your Retrieval Augmented Generation feature passed every gate in March. Green unit tests, green integration suite, a staging sign-off, a clean deploy. In June, support started forwarding answers that were subtly, confidently wrong. You opened git blame on the retrieval service. Nothing. No commit near the regression, no deploy, no config PR. The code that produced the wrong answer in June is byte-for-byte the code that produced the right answer in March.

The thing that changed was not in your repository — and that is the part your test suite was never built to see.

When you own a RAG feature, testing stops being an event that fires on commit and becomes a discipline that runs against production. The system can regress with no deploy, because the inputs that set answer quality — the corpus, the embedding model, the reranker, the judge that scores them — live outside your repository. This is not a guide to building the harness; it is a map of which testing instincts survive the move.

Why Nobody Pushed and Quality Still Slid

The regression is real, the git history is empty, and both facts are true at once. CI fires on commits. RAG answer quality is a function of state that changes without one.

The corpus grew: an ingestion job added ten thousand new documents overnight, and now the right chunk sits three positions lower in the candidate set. A vendor shipped a new Embedding model version and routed your existing API calls to it behind the same URL. The Reranking service you call swapped its scoring model without a version bump. None of that touches your repository. All of it moves your answers.

You reached for git bisect because a regression means a bad commit sits somewhere in the history. Here there is no bad commit. The bad input arrived through a data pipeline you do not gate and a vendor changelog you do not read. The fix is not in your VCS. It is a scheduled evaluation that runs on a cadence and on every corpus or dependency change — so drift trips an alarm the same way a failing test does, instead of arriving as a support ticket three weeks late.

That vendor swap is not hypothetical. Open-weight embedding models now match proprietary APIs on benchmarks, and the market is churning fast enough that a managed provider can change the model under your endpoint between quarters — DAN’s read on the embedding-market shift tracks who is moving and why. The reranker layer is worse, because the leaders trade places on the leaderboard and the billing model changes with them; DAN’s reranker-race breakdown shows how fast the floor moves.

Mental Model Map: Testing a RAG feature From: Tests fire on commit — a green build means the feature is safe until someone pushes Shift: Answer quality is set by the corpus and external models, which change with no commit To: Evaluation runs continuously against production — green build certifies code, not tomorrow’s answers Key insight: The trigger for a RAG regression is data or dependency drift, not a pull request — so the gate has to watch production, not the diff.

Green Build, Ungreen Answers

A passing test suite tells you the code does what it did yesterday. It tells you nothing about whether today’s answers are still right.

Most of your testing instincts survive the move, and that is the good news. Golden datasets transfer. Regression baselines transfer. Staging-before-prod transfers. The CI gate itself transfers. What changes is what each one does at the boundary. A unit test asserts equality: expected equals actual, true or false, done. A RAG Evaluation metric does not assert — it scores against a threshold, and the score comes from an LLM judge that is not deterministic. Run the same pipeline on the same inputs twice and the two numbers can differ.

That single difference rewires the gate. A green dashboard reads like a green build, so it inherits the same trust — and that trust is the thing that costs you the postmortem. A 0.95 on the dashboard looks like 95% correct. It is not. It is a joint property of your pipeline and the judge model that produced it, and the judge carries a documented taxonomy of biases — verbosity, position, self-preference, and more. MONA’s teardown of LLM-as-judge limits catalogs why any single score is softer than it looks. Swap the judge in a routine model upgrade and the number moves while your pipeline stands perfectly still.

The engineering move is to stop gating on an absolute threshold and start gating on direction. Pin the judge model and the prompt template, record their identity next to every score, and block a deploy on a delta against last week’s baseline rather than on a fixed number. The judge is now part of your metric definition; if you do not version it, you are measuring two things at once and calling them one. A working version of this contract lives in MAX’s evaluation-harness guide, which re-scores on every PR against a pinned stack.

Five Green Lights, One Wrong Answer

Every component can pass its own check while the end-to-end answer is wrong, because the failure lives in the composition, not in any single unit.

Your habit of writing one probe per failure surface — unit tests for the units, integration tests for the seams — transfers cleanly. It just splits along a new fault line. Retrieval earns its own metrics: Hybrid Search recall, context precision, context recall. Generation earns its own: faithfulness, answer relevancy. The two surfaces fail independently and demand different probes. Retrieval can score high — the right chunk is sitting in the candidate set — and the generator can still under-read it because of where it landed in the context window, or override it with what it memorized during training. Correct retrieval is necessary and not sufficient, and MONA’s structural-failures explainer walks the exact seams where a green retriever still produces a wrong answer.

Here is the trap that hides the composition failure: collapsing the per-subsystem vector into one averaged score. The four Ragas metrics — faithfulness, answer relevancy, context precision, context recall — are a diagnostic vector that points at the broken subsystem. Read them as a single average and you hide which side regressed; you reintroduce the black box you were trying to open. MONA’s evaluation-metrics breakdown makes the case that the split is the signal, and her prerequisites piece on recall, MRR, and faithfulness shows why you need at least one metric per surface before any number means anything.

The grounding layer sits on top as a third kind of probe. RAG Guardrails And Grounding checks — citation validity, groundedness, abstention rate — catch a class of failure the retrieval and generation scores both miss: an answer that cites a real passage and still contradicts it. Put the assertion where the failure lives. When the end-to-end answer is wrong but every component is green, the missing test is the one that scores the join.

Shift Diagram: Testing a RAG feature Classic: Write tests → Run on commit → Pass or fail → Ship on green AI: Score retriever + generator separately → Run continuously in prod → Compare delta to pinned baseline → Ship on direction

The Config Change That Moved Numbers

The clearest proof that quality drifts without a deploy is a change you can measure: swap the indexing recipe, hold the corpus and the queries fixed, and the failure rate moves.

Anthropic’s contextual-retrieval evaluation drove the top-20-chunk retrieval-failure rate from 5.7% to 1.9% — roughly a two-thirds cut — by prepending context to chunks and layering BM25 plus reranking, with the calling application untouched. MONA’s contextual-retrieval explainer has the full stack-by-stack numbers. Read that improvement in reverse and you have the shape of every silent regression: a component you did not author changes, and the numbers move underneath a codebase that never did.

Reranker churn makes it concrete. On the early-2026 Agentset leaderboard an open-weight model edged ahead of Cohere’s flagship, and Cohere migrated Rerank from per-search to per-token billing when it shipped Rerank 4 — same job, different scoring behavior, different bill. MAX’s reranking guide treats the reranker as a swappable interface for exactly this reason. The layer you call today is not the layer you called last quarter, and nobody sends you a PR when it changes.

The fix is the one you already apply to every other dependency. Pin the embedding model, the reranker, and the judge as versioned dependencies with an explicit lockfile, and re-baseline the evaluation whenever one of them changes. An unpinned model behind a managed endpoint is an unversioned dependency in your critical path — you would never accept that for a database driver, and it is more dangerous here because the failure is a quiet quality slide, not a crash.

Before You Call the Feature Tested

The Close

The updated model: a RAG deploy ships code, but answer quality is a moving target set by data and models you do not version yet. Your green build is a snapshot, not a guarantee. The next action is one page, written before you wire a single tool — name what you score, which pinned judge scores it, and what runs the evaluation when no one pushes a commit.