Coding · Agentic
SWE-bench Verified
SWE-bench Verified — human-validated 500-instance subset
The 500 SWE-bench issues humans confirmed are solvable and fairly tested.
- Released
- 2024
- Built by
- OpenAI Preparedness, with the SWE-bench authors
- Size
- 500 tasks
- Status
- Near ceiling
- Reported by
- Google DeepMind, Anthropic, Moonshot AI, Mistral
- Judge
- 100
- Graded by the project's own tests. Deterministic, inspectable.
- Headroom
- 40
- The top models crowd the ceiling; gaps are shrinking.
- Resolution
- 82
- 500 instances, so one item moves the score by 0.200 points.
- Fine print
- 45
- 5 documented caveats, the heaviest being possible training-set contamination.
- Adoption
- 74
- Reported by 4 labs; 8 published scores collected here.
Weighted from facts already on the record — not an editorial rating. A low score means a number from this benchmark needs more context to read, not that the benchmark is bad.
How the evaluation works
Every benchmark on this site is broken onto the same five stages. Select a stage to see what it actually involves.
Stage 3 · What the model can do while measured
Three environments, one benchmark
Verified shipped with the containerised harness but still imposes no scaffold, turn limit, or network policy, so in practice three very different environments produce the numbers people quote. The official bash-only configuration runs mini-SWE-agent in Docker at /testbed with bash as the only tool, a 250-step limit, a $3.00 cost limit, a 60-second command timeout, and — crucially — each command run through a fresh subshell so shell state does not persist between turns. Epoch AI's independent setup adds a text editor and an apply-patch tool via native tool-calling, disables network access, deletes all git history after the issue date, and runs 484 of the 500 tasks. Vendors run their own: Moonshot describes an in-house six-tool framework at 262,144-token context, averaged over 10 runs.
The model gets a shell. Scaffold quality and step budget move the score independently of the model.
- Budget
- 250 steps and $3.00 in the official bash-only configuration
- Tools exposed
- bash
Where the tasks came from
500 instances, no dev split. Django contributes 231 of them; seaborn contributes 2 and flask 1.
A human-screened subsample of the 2,294-instance SWE-bench test set, filtered for solvability and fair testing.
Every task is public, so there is no held-out split to check contamination against.
What the number does not tell you
The most useful part of any benchmark is the fine print. These are the reasons a published score can mislead.
The SWE-Bench Illusion found models pick the correct buggy file from the issue text alone at up to 76% on Verified against under 53% on outside repositories, with gold-function 5-gram overlap of 34.9% on Verified versus 18.2% on external tasks — and the hierarchy Verified > Full > external is exactly what memorisation predicts. Separately, a December 2025 paper found Claude models performed 3x better on Verified and were 6x better at locating the edited files than on held-out sets. Anthropic's own Opus 4.7 footnotes state their memorisation screens flag a subset of problems in these evals.
Source
Published scores
Collected from model cards and independent evaluators. The configuration column is the part that decides whether two numbers can be compared at all.
| Model | Score | Configuration | Source |
|---|---|---|---|
| ModelClaude Opus 5Anthropic | Score96% | ConfigurationStated in prose in the Opus 5 system card rather than in Table 8.1.A, and absent from the Opus 5 launch post entirely. Harness, effort setting and trial count are not given. | SourceLab-reported2026-07-24 · Claude Opus 5 system card |
| ModelClaude Opus 4.8Anthropic | Score88.6% | ConfigurationAnthropic renders its benchmark tables as images, so the decimals were read off a secondary tabulation rather than machine-readable text on anthropic.com. | SourceLab-reported2026-05-28 · Claude Opus 4.8 launch materials |
| ModelClaude Opus 4.7Anthropic | Score87.6% | ConfigurationSame image-table caveat as the 4.8 figure. The Opus 4.7 footnotes add that Anthropic's memorisation screens flag a subset of problems in this eval. | SourceLab-reported2026-04-16 |
| ModelClaude Opus 4.7 (max effort)Anthropic | Score83.47% | ConfigurationEpoch AI, n=484 of 500, scaffold v2.0.3, network disabled, ±1.69. | SourceThird-party2026-04-20 |
| ModelGemini 3.1 ProGoogle DeepMind | Score80.6% | Configurationsingle attempt | SourceLab-reported2026-02-01 |
| Modellive-SWE-agent + Claude 4.5 Opus (medium)Anthropic | Score79.2% | ConfigurationOfficial leaderboard, arbitrary scaffold. Tied with Sonar Foundation Agent + Claude 4.5 Opus, also 79.2%, dated 2025-12-05. | SourceCommunity2025-12-15 |
| ModelGLM-5.2 (max)Z.ai (Zhipu) | Score78.7% | ConfigurationEpoch AI's most recent SWE-bench Verified run of any model — Opus 4.8, Opus 5, Fable 5 and GPT-5.6 have no entry at all. | SourceThird-party2026-06-25 |
| ModelClaude 4.5 Opus (high)Anthropic | Score76.8% | ConfigurationBash Only leaderboard: every model run in the same mini-SWE-agent environment, bash as the only tool, 250-step limit, $3.00 cost limit. | SourceThird-party2026-02-17 |
Ordering is not a ranking. Rows with different configurations are not measuring the same thing — a number produced with tools, extended thinking, or majority voting is not comparable to one produced without.
Near ceiling — The top models are close enough to the ceiling to crowd.