SWE-bench

What SWE-bench is

While metrics like perplexity, BLEU, and ROUGE measure how well a model handles language at the text level, they don’t answer a more practical question: can this model actually fix a bug in my codebase? SWE-bench fills that gap. It tests whether an LLM can take a real GitHub issue — complete with bug reports, feature requests, and failing tests — and produce a working code patch that resolves it.

Think of it like the difference between testing a mechanic with a written exam versus handing them a broken engine. Perplexity asks the mechanic to predict what comes next in a repair manual. SWE-bench hands them the broken engine and checks whether it runs afterward.

According to arXiv, the benchmark was created by Carlos Jimenez, John Yang, and colleagues at Princeton University, and published at ICLR 2024. According to SWE-bench, the full dataset contains 2,294 task instances drawn from 12 popular Python repositories, including projects like Django, Flask, and scikit-learn. Each task pairs a GitHub issue with the pull request that resolved it, providing a ground-truth patch for evaluation.

The evaluation process works like this: a model receives an issue description and access to the repository code. It must read the issue, explore the codebase, identify the affected files, and generate a patch. That patch is then run against the repository’s existing test suite. If the tests pass, the task counts as resolved.

According to Epoch AI, a curated variant called SWE-bench Verified narrows the dataset to 484 human-validated samples, removing problems with infrastructure ambiguities or unclear requirements. This Verified subset has become the standard variant that most teams and leaderboards report scores on. According to Scale Labs, a harder private variant called SWE-bench Pro exists for separating top-performing models from each other, with notably lower pass rates.

What makes SWE-bench distinct from other evaluation approaches is its end-to-end nature. Metrics like ELO rank models through human preference votes on open-ended responses. BLEU and ROUGE compare generated text against reference outputs. SWE-bench skips subjective judgment entirely — either the test suite passes or it doesn’t. This binary pass/fail structure makes it one of the more reproducible benchmarks in the LLM evaluation space.

How SWE-bench is used in practice

When companies evaluate AI coding assistants — tools like Cursor, Claude Code, or GitHub Copilot — SWE-bench scores are one of the first benchmarks they look at. The score tells you how often a model can go from reading an issue description to generating a patch that passes the project’s test suite. That’s close to what production coding agents actually do day-to-day.

The benchmark doubles as a public accountability tool. Because the task set and test suites are fixed, independent organizations can re-run evaluations and verify vendor claims. According to Epoch AI, the v2.0.0 update in February 2026 changed scaffolding, environments, and token limits, which shifted many reported scores. This update made older results difficult to compare directly with newer ones.

When comparing SWE-bench scores between models, always check the evaluation version and scaffolding setup. A model scored under the original framework may look weaker than one scored after the v2.0.0 update — not because it performed worse, but because the test conditions changed.

How to read the SWE-bench ladder: mechanism, then splits, then the fine print

Start with how SWE-bench’s patch-and-test harness scores AI coding agents to get the mechanism straight — an agent is graded on whether its patch makes a real project’s own hidden tests pass, and on nothing else. From there, choosing the right benchmark split matters immediately: Verified, Lite, Full, and Multimodal test different things, and a quoted score without its split attached is barely a number. Before trusting either, read why high SWE-bench scores mislead — contamination, test leakage, and flawed tasks can all inflate the same resolution rate you just learned to read.

When you are ready to generate a number yourself rather than quote someone else’s, the harness guide walks the setup, the run, and the honest way to report a resolution rate. For where the leaderboard stands right now, Claude Fable 5 topping SWE-bench Verified at 95% covers both the new ceiling and why OpenAI walked away from the split. Close with the ethics of optimizing models for SWE-bench — if your team is choosing what to fine-tune against, read it before the benchmark becomes the target instead of the yardstick.

How SWE-bench differs from classification metrics and contamination audits

• A resolution rate is not an accuracy score. Precision, recall, and F1 score break a classifier’s mistakes into false positives and false negatives; SWE-bench has no such breakdown. Each task is graded once — the patch passes the hidden tests or it doesn’t — and the resolution rate is simply the share of tasks that cleared that bar. There is no false-positive cell to inspect when an agent scores 70%, only the list of which tasks failed.
• Passing hidden tests is not the same audit as a contamination check. Benchmark contamination is the general problem of test data leaking into training data across any benchmark; SWE-bench’s specific defense is curation and human verification of its task set. Even that defense has failed in practice — the same 2026 audit that questioned Verified’s task quality is why trusting any single split as clean still requires its own check, not a generic contamination scan.

When SWE-bench fits — and when it doesn't

Common questions about SWE-bench

Q: Does a high SWE-bench Verified score guarantee a coding agent will perform well on my own codebase? A: Not by itself. Verified tests a fixed set of popular open-source repositories, and a 2026 internal audit found many of those tasks unsolvable as written, which is why OpenAI dropped the split. Treat a high score as a ceiling on capability, and test the agent on your own repository before deciding — see where the 2026 leaderboard stands.

Q: Why do two coding agents within a few points of each other on SWE-bench Verified behave so differently on real tasks? A: Because Verified’s top band has become crowded — a dozen models now cluster within a narrow range — so a few points no longer separate capability the way they did at lower scores. The gap that matters has shifted to edge cases the fixed task set doesn’t sample; the harness guide covers cross-checking on SWE-bench Pro instead of trusting the Verified gap alone.

Q: How is a SWE-bench resolution rate different from a classification accuracy score? A: A resolution rate is the percentage of independent coding tasks whose patch passed a project’s hidden tests — one pass/fail outcome per task, with none of the breakdown that precision, recall, and F1 score give a classifier. There’s no false-positive cell to inspect, only which tasks failed.

Q: Should I treat a model fine-tuned specifically to raise its SWE-bench score the same as a general-purpose coding model? A: Not without checking what it was tuned against. A model optimized to raise one benchmark’s number can look stronger on paper without a matching gain in general coding ability — the exact tension the debate over optimizing for SWE-bench lays out from both sides.

Part of the model evaluation stack · closest neighbour: benchmark contamination. New to model evaluation from a software background? Start with the story: Model Evaluation for Developers: What Maps and What Misleads.

Sources

• arXiv: SWE-bench: Can Language Models Resolve Real-World GitHub Issues? - Original paper by Jimenez et al., published at ICLR 2024
• Epoch AI: SWE-bench Verified — Independent Evaluations - Independent leaderboard tracking human-validated subset results