Why High SWE-bench Scores Mislead: Contamination, Test Leakage, and Flawed Tasks

Take a frontier coding model, run it on SWE Bench Verified, and watch it resolve roughly four out of five issues. Run the same model, the same week, on a contamination-resistant variant built from the same kind of GitHub bugs, and the success rate can be cut roughly in half. Nothing about the model changed between those two runs. What changed was how much of the benchmark had already leaked into what the model learned. That gap — between the headline score and the held-out score — is the whole story.

What a SWE-bench Score Is Supposed to Mean

The promise is simple, which is exactly why it travels so well on a launch slide. SWE-bench began as 2,294 real GitHub issues pulled from 12 widely used Python projects — Django, Flask, and scikit-learn among them — each paired with the human fix that actually closed it (SWE-bench paper). A model reads the issue, edits the repository, and its patch runs against the project's own test suite. Pass the tests, resolve the issue.

The intuitive reading of a 90% score is “this model fixes nine of every ten real bugs.”

That reading is wrong, and the reasons are mechanical, not philosophical.

To clean up the noisiest tasks, OpenAI released SWE-bench Verified in June 2024: 500 issues that 93 professional developers screened, three reviewers per task, filtered down from 1,699 candidates (OpenAI). Verified became the number labs quote. It is also the number that has quietly stopped measuring what people think it measures.

A Score That Stopped Predicting Capability

A benchmark earns its keep by separating stronger systems from weaker ones. The moment most models cluster near the ceiling, that discriminating power collapses — and SWE-bench Verified is most of the way there.

Why are SWE-bench scores considered unreliable in 2026?

As of mid-2026, the Verified leaderboard averages around 63%, with top reported scores between 85% and 94% (CodeAnt). On its own, saturation would just mean the task got easier. The problem surfaces when you move those same models to a harder, cleaner test. Frontier systems sitting near 80% on Verified drop to somewhere between 23% and 46% on the contamination-resistant SWE-bench Pro (SWE-bench Pro paper). One reported model fell from about 81% to about 46% — a collapse no genuine loss of skill can explain, because it is the same model on the same kind of task.

OpenAI stopped reporting Verified scores around February 2026. Their stated reason was blunt: gains “increasingly reflect how much the model was exposed to the benchmark at training time” rather than real improvement (OpenAI). When the people who popularized a benchmark walk away from it, the score is no longer reading capability — it is reading exposure.

How Memorization Wears the Mask of Reasoning

Two different leaks inflate the number, and they fail in different ways. The first lives in the training data. The second lives in the grader. Start with the data.

What is data contamination and why does it matter for coding benchmarks?

Benchmark Contamination is what happens when the evaluation data, or its answers, end up inside the model’s training set. For coding benchmarks the exposure is almost structural: SWE-bench is built from public GitHub repositories, so the issues, the discussion threads, and the merged fix commits — the Ground Truth patches themselves — were crawled into pretraining corpora long before any model was tested on them. This is Data Leakage of the most direct kind: the answer key shipped with the textbook.

It matters because a passing patch then has two possible explanations, and the score cannot tell them apart. Either the model reasoned its way to a fix, or it reproduced a solution it had already seen. Memorization and reasoning look identical at the output layer. A benchmark that cannot distinguish recall from problem-solving is not measuring problem-solving.

Not capability. Recall.

How does benchmark contamination inflate SWE-bench results?

The fingerprints are measurable. Models can locate the buggy file from the issue text alone up to 76% of the time on Verified, but only 53% of the time on repositories that were never part of SWE-bench (SWE-Bench Illusion paper). That 23-point gap is not reasoning — a model that genuinely understood bugs would not lose the skill the moment the repository becomes unfamiliar.

The generated fixes carry the same tell. Patches share up to 35% consecutive five-gram overlap with the gold-standard human fixes on Verified, against 18% on other benchmarks (SWE-Bench Illusion paper). Roughly a third of successful patches show signs of direct solution leakage. The model is not deriving the fix. It is reciting one.

When the Grader Itself Is Broken

Contamination explains scores that are too high for the wrong reason. The grading explains scores that are simply wrong. A SWE-bench “pass” means a patch satisfied the repository’s existing tests — and those tests were never written to catch every wrong answer.

Researchers behind UTBoost rebuilt the test suites and measured the gap directly: augmented tests caught 15.7% more incorrect patches that had sailed through the original checks, and that added scrutiny reshuffled leaderboard rankings for 24% of agents (UTBoost). A quarter of the standings moved once the grader got stricter.

It gets worse on exactly the tasks that should matter most. OpenAI’s own audit found that 59.4% of the hardest unsolved Verified problems come with flawed test cases, and separately that nearly 60% of failed problems had broken tests (OpenAI). When a model “fails” a broken task, the failure is noise; when it “passes” a weak one, the pass is noise too. A green test suite is evidence of a passing patch, not a correct one.

What the Gap Predicts

Once you see the two leaks, leaderboard behavior becomes predictable rather than surprising.

• If a model’s score on a public, static benchmark towers over its score on a held-out or contamination-resistant one, expect exposure, not understanding.
• If someone strengthens the tests, expect rankings to move — the models most reliant on weak grading fall the furthest.
• If a benchmark is public and frozen, expect its signal to decay over time, because every new model trains on more of it.

For anyone procuring or comparing models, the practical consequence is that a single Verified number is close to uninformative on its own. What carries information is the gap between a contaminated benchmark and a clean one, and how a model performs on tasks published after its training cutoff.

Rule of thumb: trust the held-out score, treat the public-benchmark score as an upper bound, and read any number near the ceiling as saturation until proven otherwise.

When it breaks: the deeper limitation is that no public, static benchmark stays clean. Contamination-resistant successors like SWE-bench Pro, SWE-Rebench, and SWE-MERA discriminate well today precisely because models have not trained on them yet — the moment they become popular and public, the same decay begins. The durable fix is not a better frozen benchmark; it is continuous evaluation on tasks the model has never seen.

The Data Says

A SWE-bench score measures two things at once — genuine coding ability and prior exposure to the benchmark — and offers no way to separate them. The sharp drop from Verified to contamination-resistant tests, the verbatim overlap with gold patches, and the rankings that shift the instant tests get stricter all point the same direction. Treat a high score as a hypothesis about capability, not a measurement of it.