Coding · Agentic
SWE-bench Pro
SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?
Bigger fixes in unfamiliar codebases: about 107 lines across four files.
- Released
- 2025
- Built by
- Scale AI
- Size
- 1,865 tasks
- Status
- Active
- Reported by
- Google DeepMind, OpenAI, Anthropic, Moonshot AI, Z.ai, Alibaba (Qwen)
- Judge
- 100
- Graded by the project's own tests. Deterministic, inspectable.
- Headroom
- 82
- Still separates the frontier from everything below it.
- Resolution
- 92
- 1,865 instances, so one item moves the score by 0.054 points.
- Fine print
- 45
- 4 documented caveats, the heaviest being possible training-set contamination.
- Adoption
- 97
- Reported by 6 labs; 7 published scores collected here.
Weighted from facts already on the record — not an editorial rating. A low score means a number from this benchmark needs more context to read, not that the benchmark is bad.
How the evaluation works
Every benchmark on this site is broken onto the same five stages. Select a stage to see what it actually involves.
Stage 3 · What the model can do while measured
One pinned image per task
Each instance has its own pre-built Docker image published on Docker Hub, with the tag carried in the dataset row, so the environment is pinned rather than rebuilt. Every environment was validated twice — automatically, by running the gold tests several times to catch flakes, and by a human checking that each test is actually relevant to the task. The paper ran SWE-agent capped at 50 turns; the live leaderboard runs 250 turns with uncapped cost for most entries and marks the older 50-turn cost-capped runs separately, while asterisked rows use mini-SWE-agent instead. Agentless was tried and rejected because it struggles with multi-file editing.
The model gets a shell. Scaffold quality and step budget move the score independently of the model.
- Budget
- 50 turns in the paper; 250 turns and uncapped cost on the current leaderboard
- Tools exposed
- bashfile editor (SWE-agent ACI)
Where the tasks came from
1,865 problems across 41 repositories: 731 public, 858 held out to monitor overfitting, and 276 from private commercial repositories. Gold patches average 107.4 lines across 4.1 files.
Actively maintained GPL-licensed repositories plus 18 proprietary startup codebases, with every problem statement rewritten by professional engineers.
A held-out or private split exists, which makes contamination easier to detect than on a fully public set.
What the number does not tell you
The most useful part of any benchmark is the fine print. These are the reasons a published score can mislead.
Scale's own public leaderboard has claude-opus-4-6 (thinking) at 51.90% ±3.61. Moonshot's Kimi K2.6 card puts Claude Opus 4.6 at 53.4. Z.ai's GLM-5.2 card reports Claude Opus 4.8 at 69.2, having run the suite with OpenHands using a tailored instruction prompt at temperature=1, top_p=1, max_new_tokens=32k and a 400K context window. And Alibaba's Qwen3.8 card reports Claude Fable 5 at 80.0 under a footnote reading: 'Evaluated with the Claude Code harness, temp=1.0, top_p=0.95, and a 256K context window. Problematic tasks corrected and all baselines evaluated on the refined benchmark.' That last clause means a vendor edited the benchmark and re-ran every baseline on its own modified version, publishing the result under the original name. Anthropic's own Opus 5 system card then reports 79.2 — nearly eighteen points above the top of Scale's public board, on a harness it does not describe.
Source
Published scores
Collected from model cards and independent evaluators. The configuration column is the part that decides whether two numbers can be compared at all.
| Model | Score | Configuration | Source |
|---|---|---|---|
| ModelClaude Opus 5Anthropic | Score79.2% | ConfigurationReported in the system card while the Opus 5 launch post shows no SWE-bench number at all. Compare the 61.50% top of Scale's own public leaderboard. | SourceLab-reported2026-07-24 · Claude Opus 5 system card, Table 8.1.A |
| ModelClaude Opus 4.8Anthropic | Score69.2% | ConfigurationThe same 69.2 figure is echoed for this model by the GLM-5.2 and Qwen3.8 cards, on two different harnesses. | SourceLab-reported2026-05-28 · Claude Opus 4.8 system card, Table 8.1.A |
| ModelMuse Spark 1.1Meta | Score61.5% | ConfigurationPublic set, ±3.10, run with the mini-swe-agent harness. Leaderboard note: 'Models and results that are grayed out were run with a capped cost limit and turn limit of 50. All other models on this page were run with an uncapped cost and with a turn limit of 250.' | SourceThird-party2026-08-16 |
| Modelgpt-5.4 (xHigh)OpenAI | Score59.1% | ConfigurationPublic set, ±3.56, mini-swe-agent harness. | SourceThird-party2026-08-16 |
| ModelGPT-5.5OpenAI | Score58.6% | ConfigurationReported without any SWE-bench Verified figure. Secondary-sourced: openai.com returns HTTP 403 to automated fetching. | SourceLab-reported2026-04-23 |
| ModelGemini 3.1 ProGoogle DeepMind | Score54.2% | ConfigurationPublic set, single attempt. | SourceLab-reported2026-02-01 |
| Modelclaude-opus-4-6 (thinking)Anthropic | Score51.9% | ConfigurationPublic set, ±3.61, mini-swe-agent harness. Compare Moonshot's card at 53.4 and Z.ai's at 69.2 for the same model class. | SourceThird-party2026-08-16 |
Ordering is not a ranking. Rows with different configurations are not measuring the same thing — a number produced with tools, extended thinking, or majority voting is not comparable to one produced without.
Active — Still spreads the field.