Coding · Agentic
SWE-rebench
SWE-rebench: An Automated Pipeline for Task Collection and Decontaminated Evaluation of Software Engineering Agents
A rolling set of fresh issues, with one company running every model itself.
- Released
- 2025
- Built by
- Nebius
- Size
- 21,336 tasks
- Status
- Active
- Judge
- 100
- Graded by the project's own tests. Deterministic, inspectable.
- Headroom
- 82
- Still separates the frontier from everything below it.
- Resolution
- 100
- 21,336 instances, so one item moves the score by 0.005 points.
- Fine print
- 45
- 4 documented caveats, the heaviest being possible training-set contamination.
- Adoption
- 11
- No lab reports it in a model card, so it rarely appears in a comparison.
Weighted from facts already on the record — not an editorial rating. A low score means a number from this benchmark needs more context to read, not that the benchmark is bad.
How the evaluation works
Every benchmark on this site is broken onto the same five stages. Select a stage to see what it actually involves.
Stage 3 · What the model can do while measured
Fixed scaffold, run five times
A container per instance with dependencies frozen at validation time, tests offline, and a real shell. The scaffold is deliberately minimal and identical for every model: a ReAct loop where the model reads terminal output and writes textual commands, standardised at 128K context. The paper's protocol uses no function calling even for models that support it, specifically so tool-calling ability does not confound coding ability; the live board has since added a tools mode and tags entries accordingly. Nebius runs all 117 entries itself, five times each, and publishes mean, standard error, pass@5, tokens and dollars per problem — Fable 5 at high effort burns 2.52M tokens and $4.40 per problem, GPT-5.6 Sol at medium does it in 605K tokens and $0.85.
The model gets a shell. Scaffold quality and step budget move the score independently of the model.
- Budget
- not published; each model is run five times per slice
- Tools exposed
- bash
Where the tasks came from
21,336 execution-validated tasks from 3,468 repositories in the pool, but each leaderboard window uses only 30-110 of them — the current window is 111 problems from 65 repositories.
Fully automated: crawled from GitHub Archive, environments reconstructed by an LLM-written install recipe, validated by execution, and quality-scored by a fine-tuned annotator model.
Every task is public, so there is no held-out split to check contamination against.
What the number does not tell you
The most useful part of any benchmark is the fine print. These are the reasons a published score can mislead.
Nothing clears 65% on the 15 May – 1 July 2026 slice — including Claude Opus 5, at 63.4%, in the same month its own system card put SWE-bench Verified at 96.0. The paper's original demonstration is cleaner still: DeepSeek-V3-0324 scored 39.7% on SWE-bench Verified and 21.3% on a fresh March-April 2025 slice, and LLaMA-3.3-70B-Instruct went 18.1% to 11.2%. That gap is the memorisation discount, made visible.
Source
Published scores
Collected from model cards and independent evaluators. The configuration column is the part that decides whether two numbers can be compared at all.
| Model | Score | Configuration | Source |
|---|---|---|---|
| ModelClaude Fable 5 [high]Anthropic | Score64.5% | ConfigurationWindow 15 May – 1 Jul 2026, 111 problems from 65 repositories. Mean of 5 runs, ±1.41; pass@5 78.4%; $4.40 per problem. | SourceThird-party2026-08-16 |
| ModelGrok 4.5 [high]xAI | Score63.8% | ConfigurationWindow 15 May – 1 Jul 2026. Mean of 5 runs, ±0.60; pass@5 77.5%. | SourceThird-party2026-08-16 |
| ModelClaude Opus 5 [high]Anthropic | Score63.4% | ConfigurationWindow 15 May – 1 Jul 2026. Mean of 5 runs, ±1.35; pass@5 74.8%. | SourceThird-party2026-08-16 |
| ModelGPT-5.6 Sol [medium]OpenAI | Score62.3% | ConfigurationWindow 15 May – 1 Jul 2026. Mean of 5 runs, ±1.83; pass@5 79.3%; $0.85 per problem. | SourceThird-party2026-08-16 |
Ordering is not a ranking. Rows with different configurations are not measuring the same thing — a number produced with tools, extended thinking, or majority voting is not comparable to one produced without.
Active — Still spreads the field.