How to Run the SWE-bench Harness and Read Agent Resolution Rates in 2026

Your team ships a coding agent. Someone runs it against SWE-bench, posts a number in the high seventies in Slack, and the room celebrates. Then a customer asks which split, which harness version, which machine it ran on. Silence. The number was real. The claim was meaningless. That gap is the whole subject of this guide.

Before You Start

You’ll need:

• A machine that can run Docker on x86_64, or an sb-cli account for cloud runs
• A working grasp of how an Evaluation Harness executes tests, and what SWE Bench actually measures
• Your coding agent’s output as patches, or a public predictions file to practice on

This guide teaches you: how to treat a benchmark run as a specification problem. Pin the environment, define the predictions contract, sequence the run, and read the resolution rate as a claim with conditions attached, not a trophy.

The Score That Didn’t Survive One Follow-Up Question

Most people quote a SWE-bench number the way they’d quote a stock price: one figure, no context. The problem is that one figure hides at least four decisions, and changing any of them changes the result. Skip the spec and you get a rumor with a percent sign.

It passed on Friday’s run. Monday’s rerun on a clean machine came back materially lower, because the Docker layer cache had been rebuilt and a handful of environments now behaved differently. Same agent, same prompt, different number. A score without its spec is a rumor, and a rumor doesn’t belong in a benchmark report.

Step 1: Break the Run Into Four Moving Parts

Before you run anything, name the components. A SWE-bench evaluation is not one thing. It is four, and each one can fail on its own without touching the others.

Your system has these parts:

• The dataset split — the set of real GitHub issues your agent attempts. The splits are not interchangeable: Full holds 2,294 tasks, Lite holds 300, Verified holds 500, and Multilingual holds 300 across nine languages and forty-two repositories, per the SWE-bench Verified dataset listing. A score on one says nothing about another.
• The predictions file — your agent’s output, normalized. One JSONL line per task, each carrying instance_id, model_name_or_path, and model_patch (a unified diff string), as the SWE-bench Docs specify. This is the only thing the harness reads from your side.
• The harness — the swebench package. It applies each patch inside an isolated Docker container, runs the repository’s own test suite, and records the outcome.
• The result — the resolution rate. “% resolved” is the share of submitted instances whose patch passes the task’s hidden FAIL_TO_PASS and PASS_TO_PASS tests, written to evaluation_results/, again per the SWE-bench Docs.

Notice what is missing. There is no ELO Rating for LLMs here, no arena vote, no LLM as a Judge scoring a transcript, and no Human Evaluation for AI panel reading diffs. A patch either makes the hidden tests pass or it does not. That binary outcome is the ground truth, and it is why these Benchmark Datasets carried so much weight. Each part can fail independently, so each one needs its own specification.

The Architect’s Rule: If you can’t name the four parts and which split you’re on, you can’t defend the number. The leaderboard can’t either.

Step 2: Pin the Environment Before the First Patch Applies

The harness builds Docker images and runs real test suites. That makes it sensitive to the machine underneath it. The reason Friday and Monday disagreed is almost always here: the environment was never pinned, so it drifted.

Context checklist:

• swebench 4.1.0 installed on Python 3.10 or newer, via pip install swebench (swebench on PyPI)
• Docker running, on an x86_64 host — not ARM64
• At least 120GB free storage, 16GB RAM, and 8 CPU cores, as the SWE-bench Docs require
• Dataset split chosen and recorded — Lite for debugging, Verified for headline claims
• Predictions JSONL validated: every instance_id belongs to the chosen split, every model_patch is a real unified diff
• A run_id chosen up front — it namespaces the run’s logs and images so a later run can’t quietly overwrite this one

The Spec Test: If your context doesn’t pin the split, a Lite 300 score ends up quoted next to a Verified 500 leaderboard, and the comparison is fiction. Same harness, different universe.

Environment & compatibility notes:

• swebench package: 4.1.0 (released September 2025) runs on Python 3.10+; no known CVE in the harness itself (swebench on PyPI).
• Architecture: Run on x86_64. ARM64 support is experimental, and Docker images can fail to build there (SWE-bench Docs).
• Dataset org naming: Official docs and commands use princeton-nlp/SWE-bench_*, while GitHub and Hugging Face now host the same data under the SWE-bench/* org (SWE-bench GitHub). Both resolve. Do not “fix” one to match the other.

Step 3: Run It in an Order That Survives a Rerun

Order matters because a failure in the wrong place gets silently miscounted as an agent failure. Build the run so that each stage either succeeds cleanly or fails loudly.

Run order:

1. Validate the predictions file first — because a malformed diff never applies, and the harness records that as “unresolved.” Your failure count inflates for a reason that has nothing to do with the agent.
2. Run the harness with explicit flags — python -m swebench.harness.run_evaluation --dataset_name princeton-nlp/SWE-bench_Lite --predictions_path <your.jsonl> --max_workers 8 --run_id <unique-id>, the canonical invocation in the SWE-bench Docs. Set --cache_level deliberately: none, base, env, or instance trades disk space for speed, and the wrong choice is what made your runtime swing.
3. Read evaluation_results/ last — because the score only means something once every instance has landed in exactly one bucket: resolved, failed, or errored.

For each part, your run config must specify:

• What it receives — the split name and the validated JSONL path
• What it returns — per-instance logs plus the aggregate resolution rate
• What it must NOT do — share a run_id with a previous run, or count container errors as agent failures
• How it handles failure — retry environment build errors, but never silently fold them into the unresolved pile

No x86_64 box with 120GB free? The sb-cli tool runs the same evaluation remotely: pip install sb-cli, then sb login and sb submit, with Modal supported as well (sb-cli GitHub). It is also the only way to score the Multimodal split, whose test set is held private to resist contamination. Errored is not the same as failed, and whichever path you take, your config has to keep those two apart.

Step 4: Read the Resolution Rate Without Fooling Yourself

You ran it. You have a number. Now comes the step almost everyone skips: deciding what that number is allowed to claim.

Validation checklist:

• Split match — failure looks like: a Lite 300 result quoted against a Verified 500 leaderboard, as if they measure the same thing.
• Errored vs failed — failure looks like: container build errors, out-of-memory kills, or missing images counted as the agent’s unresolved tasks.
• The validity ceiling — failure looks like: treating a Verified score as ground truth in 2026. OpenAI stopped reporting SWE-bench Verified in February 2026 after auditing 138 of the tasks, about 27.6% of the set, and finding 59.4% of the failed tests were themselves flawed, with every frontier model showing signs of contamination (OpenAI).
• Contamination — failure looks like: a model that memorized the fixing commit during pretraining. When that happens, a passing test measures recall, not problem-solving. This is the Benchmark Contamination trap, and a famous leaderboard is the most exposed to it.

The size of the gap is the tell. The same frontier models that score in the seventies and eighties on Verified land materially lower on SWE-bench Pro’s harder, multilingual task set — which is why OpenAI now points to Pro for fresh comparisons. As of mid-2026 the top of the public Verified board is compressed: Claude Fable 5 at 95.0% (suspended since June over export controls, with a return expected around early July), Claude Mythos in preview at 93.9%, and Claude Opus 4.8 at 88.6%. When the leaders sit within a few points of each other and the test set is known to be flawed, the ranking is noise dressed as precision. Treat Verified as a ceiling, not ground truth.

SWE-bench Verified was validated by 93 software engineers who triple-annotated each task. Even with that rigor, an estimated 5–10% error rate remains — tasks where annotators disagreed enough that a Cohen's Kappa calculation would flag the label as ambiguous. That error band is not a flaw to fix; it is an honest admission built into the dataset. It means a perfect score is not theoretically achievable, and a model scoring in that ambiguous zone is not definitively better or worse. Keep that margin in mind when you build a Rubric Design process for your own evaluation pipeline.

Common Pitfalls

Pro Tip

A benchmark score is a claim with a spec attached, and the spec is the part people throw away. Record the split, the harness version, the host architecture, the run_id, and the date next to every number you publish. That habit is the whole of Reproducibility in Agent Evaluation And Testing: a number you can’t reproduce isn’t evidence, it’s a screenshot. The teams whose results survive scrutiny are not the ones with the highest scores. They’re the ones who can hand you the exact conditions and watch you get the same answer.

One more thing. SWE-bench measures coding capability on a closed task set. It doesn’t measure everything. If you also need to evaluate subjective qualities — output clarity, architectural soundness, explanation quality — you’ll need a human annotation layer alongside the harness. Tools like Label Studio handle that side of the evaluation. Keep both layers and keep them separate.

Frequently Asked Questions

Q: How do you set up and run the SWE-bench evaluation harness step by step? A: Install swebench 4.1.0 on Python 3.10 or newer, start Docker on an x86_64 host, then call the run_evaluation module with your dataset split, predictions path, worker count, and a unique run_id. The detail most walkthroughs skip: set –cache_level deliberately, since the default rebuilds images and swings your runtime.

Q: How can you use SWE-bench to evaluate your own coding agent? A: Have your agent produce a unified-diff patch per task, write them to a JSONL predictions file keyed by instance_id, and score them with the harness. The catch teams miss: debug on the Lite 300 split first. It is cheaper than a full Verified run before your pipeline is stable.

Q: How do you correctly read a SWE-bench Verified leaderboard score? A: Treat it as a ceiling, not ground truth. Check the split matches what you are comparing against, confirm errored tasks were not counted as failures, and remember OpenAI stopped reporting Verified in 2026 over flawed tests and contamination. Cross-check the same model on SWE-bench Pro, where scores drop sharply.

Your Spec Artifact

By the end of this guide, you should have:

• An evaluation spec — split, harness version, host architecture, run flags, and run_id, all recorded in one place
• A predictions-file contract — the JSONL schema (instance_id, model_name_or_path, model_patch) you validate before every run
• A score-reading checklist — split-match, errored-vs-failed, validity ceiling, and contamination, applied before any number leaves your team

Your Implementation Prompt

Drop this into Claude Code, Cursor, or Codex when you set up an evaluation run. It encodes the four parts from Step 1 and every constraint from Step 2 as bracketed placeholders. Fill each bracket with your own values before you run anything.

You are helping me set up a reproducible SWE-bench evaluation run.
Do not invent results. Help me build and validate the run configuration.

ENVIRONMENT (verify before anything else):
- Python: [3.10 or newer]
- swebench package: [4.1.0]
- Host architecture: [x86_64]   # ARM64 is experimental; flag it if I am on ARM
- Docker: running, with [120]GB free disk, [16]GB RAM, [8] CPU cores

DATASET:
- Split: [Lite 300 | Verified 500 | Full 2294 | Multilingual 300]  # state which and why
- Record this split with the result; splits are not interchangeable

PREDICTIONS FILE (validate before running):
- Format: JSONL, one object per task
- Required fields: instance_id, model_name_or_path, model_patch (unified diff)
- Check: every instance_id exists in the chosen split; every model_patch is a valid diff

RUN:
- Command shape: python -m swebench.harness.run_evaluation
    --dataset_name [dataset] --predictions_path [path]
    --max_workers [8] --run_id [unique-id]
- cache_level: [none | base | env | instance]  # pick deliberately; note the disk/speed trade
- Namespace all logs and images under run_id: [unique-id]

VALIDATE THE SCORE:
- Read % resolved from evaluation_results/
- Separate resolved / failed / errored; errored tasks are NOT agent failures
- Report it as: [N]% on [split], swebench [version], [arch], run_id [id], [date]
- Cross-check the same model on SWE-bench Pro before calling it ground truth

Ship It

You now have a mental model that separates the run from the number it produces. You can decompose a SWE-bench evaluation into four specifiable parts, pin the environment so a rerun agrees with itself, and read a resolution rate as a conditional claim instead of a verdict. The next time someone hands you a leaderboard figure, you’ll know exactly which question to ask first.

Deploy safe, Max.