Coding · Agentic
SWE-bench
SWE-bench (Full test split) — "Can Language Models Resolve Real-World GitHub Issues?"
Can a model fix a real GitHub issue so the project's own tests pass?
- Released
- 2023
- Built by
- Princeton-led academic team (the SWE-bench Team)
- Size
- 2,294 tasks
- Status
- Active
- Judge
- 100
- Graded by the project's own tests. Deterministic, inspectable.
- Headroom
- 82
- Still separates the frontier from everything below it.
- Resolution
- 100
- 2,294 instances, so one item moves the score by 0.044 points.
- Fine print
- 45
- 4 documented caveats, the heaviest being possible training-set contamination.
- Adoption
- 3
- No lab reports it in a model card, so it rarely appears in a comparison.
Weighted from facts already on the record — not an editorial rating. A low score means a number from this benchmark needs more context to read, not that the benchmark is bad.
How the evaluation works
Every benchmark on this site is broken onto the same five stages. Select a stage to see what it actually involves.
Stage 3 · What the model can do while measured
Docker, but the scaffold is yours
Evaluation is a three-layer Docker stack — a base OS image, an environment image caching a repo's dependencies, and a per-instance image that runs the tests — with official guidance of an x86_64 machine, 120GB of storage, 16GB of RAM and 8 cores. What the benchmark does not specify is the agent: in 2023 the setting was single-shot retrieval-augmented generation with no tools at all, SWE-agent added a file viewer, editor, search and bash in 2024, and modern submissions use arbitrary scaffolds with tens to hundreds of turns. That unspecified layer is the single largest source of cross-report incomparability in the whole family. Third-party graders tighten it: Epoch AI runs in a barebones container with network access disabled and all git history after the issue date deleted, so the model cannot read the human fix out of a later commit.
The model gets a shell. Scaffold quality and step budget move the score independently of the model.
- Budget
- unspecified by the benchmark — single-shot in 2023, hundreds of turns now
Where the tasks came from
2,294 test instances plus a 225-instance dev split. Only 12 repositories, and Django alone is roughly a fifth of the set.
GitHub pull requests from 12 popular Python repositories that both close an issue and touch a test file.
Every task is public, so there is no held-out split to check contamination against.
What the number does not tell you
The most useful part of any benchmark is the fine print. These are the reasons a published score can mislead.
SWE-bench+ manually screened every instance SWE-agent with GPT-4 solved and found 32.67% of successful patches were cheating because the fix was spelled out in the issue report or its comments, and a further 31.08% passed only because the tests were too weak to separate correct from incorrect. Removing both classes dropped that system from 12.47% to 3.97%. The same paper found over 94% of issues predate the knowledge cutoffs of the models evaluated. Note the provenance carefully: this statistic is about successful patches on SWE-bench Full and is routinely misquoted as a statement about Verified.
Source
Published scores
Collected from model cards and independent evaluators. The configuration column is the part that decides whether two numbers can be compared at all.
| Model | Score | Configuration | Source |
|---|---|---|---|
| ModelSonar Foundation Agent + Claude 4.5 OpusSonar | Score52.62% | ConfigurationOfficial Full-split leaderboard, self-submitted scaffold. Only 24 submissions have ever been made to this split. | SourceCommunity2025-12-19 |
Ordering is not a ranking. Rows with different configurations are not measuring the same thing — a number produced with tools, extended thinking, or majority voting is not comparable to one produced without.
Active — Still spreads the field.