What Is SWE-bench and How Its Patch-and-Test Harness Scores AI Coding Agents

In 2023, the strongest model on SWE-bench resolved less than two percent of the tasks it was handed. By mid-2026, the best systems clear more than ninety percent of the human-validated subset. Benchmarks rarely travel that far that fast, which makes the score worth interrogating: what does it actually measure, and does a ninety today mean what a two once did? The surprising part is the grading: no panel of engineers rating elegance, no second model as referee.

A Benchmark Built From Bugs That Already Happened

Most coding benchmarks ask a model to write a small function from a prompt, then check the output against a handful of examples. SWE-bench starts somewhere harder and more honest: with problems that real maintainers already solved, in code that was never written to be graded.

What is SWE-bench?

SWE-bench is a benchmark that measures whether an AI can resolve real software issues drawn from open-source projects. It was introduced in the paper “SWE-bench: Can Language Models Resolve Real-World GitHub Issues?” at ICLR 2024, and the full set contains 2,294 task instances mined from 12 popular Python repositories, according to Jimenez et al. Each instance is a closed GitHub issue paired with the commit that actually fixed it.

That sourcing is the whole point. Earlier coding tests like HumanEval use hand-written puzzles — clean, but artificial. SWE-bench draws on the messy reality of Agentic Coding: multi-file repositories, real dependencies, and issue reports written by humans who assumed you already understood the codebase. When the benchmark launched, the best model resolved only 1.96% of instances, per Jimenez et al. — a number that told everyone the task was nowhere near solved.

The model is given two things: a snapshot of the codebase at the moment before the fix, and the natural-language issue text. It does not get the human solution, and it does not get the tests it will be judged against. It has to produce a patch, a code diff, the way a developer would.

Inside One Task: The Anatomy of a SWE-bench Instance

A SWE-bench task is not just “a bug.” It is a structured record, and the structure is what makes objective grading possible. Most of the fields exist to pin down one thing: the exact, reproducible conditions under which a fix counts as correct.

What is included in a single SWE-bench task instance?

Each task instance bundles the problem, the hidden answer key, and the machinery to check a fix. An instance carries these fields, per the SWE-bench Docs:

The split between the two test lists is the quiet engineering insight. FAIL_TO_PASS confirms the bug was fixed; PASS_TO_PASS confirms the fix did not silently break something else. The gold patch is the Ground Truth record of how a human actually solved it — but, and this matters, it is not what the agent’s work is compared against. The agent is never scored on how closely its diff resembles the human one.

Why not? Because two correct patches can look nothing alike.

The Harness That Refuses to Have an Opinion

Here is where SWE-bench parts ways with most of modern AI evaluation. Much of the field has drifted toward subjective grading — LLM as a Judge setups where one model rates another, or ELO Rating for LLMs systems that use Human Evaluation for AI through preference votes to rank outputs. That path requires annotation infrastructure like Label Studio, produces ratings that need inter-annotator agreement checks via Cohen's Kappa, and still depends on whether the evaluators agree on what “good code” means. Those methods exist because most model outputs have no objectively right answer. Code is the rare domain that does, and SWE-bench leans on it entirely.

How does SWE-bench test if an AI agent actually fixed a GitHub issue?

The grading runs inside an Evaluation Harness that behaves almost exactly like a Continuous Integration pipeline. For each task, the harness spins up a per-repository Docker environment, applies the model’s predicted patch to the codebase, then applies the hidden test_patch on top, and runs the test suite. An instance counts as resolved only if every FAIL_TO_PASS test now passes and every PASS_TO_PASS test still passes, according to the SWE-bench Docs. Anything short of that is a fail.

Not a judgment. A measurement.

The headline score is “% Resolved” — the share of task instances whose patch survives the harness. There is no partial credit, no Rubric Design to interpret, no second model weighing in. A patch that fixes the bug but trips one pre-existing test scores exactly the same as a patch that does nothing: zero. The reward is reproducibility. Run the same patch through the same harness twice and you get the same verdict, because the verdict is just the exit code of a test run.

This is also why the gold patch can be ignored at scoring time. The benchmark does not ask “did the model match the human?” It asks “do the tests pass?” Any patch that satisfies the hidden tests is accepted, however unusual its approach — which is the right call, because real engineering admits many valid solutions to one bug.

What the Pass/Fail Line Actually Tells You

A pass/fail harness feels like solid ground — until you remember that the ground is a set of tests someone chose to write, run through a scaffold someone chose to build. The score is real, but it describes a system, not a model in isolation.

That distinction has teeth once you start reading scores in the wild:

• If a system’s number on the easy subset jumps while its number on a harder variant stays flat, you are probably watching contamination or scaffolding gains, not new capability.
• If you compare two agents whose numbers came from different harnesses, the gap may be an artifact of the setup. The choice of test harness configuration alone can swing reported scores by roughly twelve percentage points, according to the Contamination Study.
• If a patch passes, you know the hidden tests went green. You do not know the fix is what a maintainer would have accepted.

That last gap is not hypothetical. An empirical study found solution leakage in a substantial share of patches the harness marked successful, with models recalling correct file paths from training data at high rates, according to the Contamination Study. OpenAI’s internal audit of SWE-bench Verified found that for many flagged tasks, the issue was not the model — it was the test: assertions that assumed a specific implementation detail rather than checking observable behavior (OpenAI Deprecation). Headline percentages, then, are harness- and scaffold-dependent, and Benchmark Contamination is a live concern, not a settled footnote.

The benchmark answered this pressure by branching. SWE-bench Lite trims the set to a 300-task test subset for fast iteration. SWE-bench Verified, released in August 2024 by OpenAI’s Preparedness team with the original authors, is a 500-instance set drawn from 1,699 problems, each reviewed independently by three expert software engineers for solvability and fair tests, per OpenAI. SWE-bench Multimodal turns to JavaScript and web repositories whose issues include screenshots and UI assets, while SWE-bench Multilingual spreads across nine programming languages. As the original and Verified subsets approached saturation, Scale AI introduced the harder SWE-bench Pro in 2025, with a public set of roughly 731 problems plus a held-out commercial set designed to resist contamination, according to Scale AI; top scores there sit far lower, in the rough range of fifty to sixty percent as of mid-2026.

The clearest signal of saturation arrived in early 2026, when OpenAI publicly stopped reporting SWE-bench Verified, arguing it no longer measures frontier coding capability once scores cluster near the ceiling. That is OpenAI’s stated position, and it captures the benchmark’s strange success: it became too easy to stay interesting, which is what happens when a hard problem genuinely gets closer to solved.

Rule of thumb: read a SWE-bench percentage as a property of model-plus-scaffold-plus-subset, never of the model alone.

When it breaks: the harness only confirms that a patch turns the hidden tests green without breaking the rest. That is necessary but not sufficient for a correct, maintainable fix — a patch can satisfy the assertions while solving the wrong problem or hard-coding around the test, and the score will still count it as resolved.

The Data Says

SWE-bench measures something narrow and verifiable: whether a model’s patch makes a real repository’s hidden tests pass. That objectivity is both its strength and its ceiling. The percentage is trustworthy as an execution result and misleading as a grade of engineering quality, so the useful question is never “what did it score” but “on which subset, through which harness, and against tests written by whom.”