Coding · Agentic
SWE-bench Multilingual
SWE-bench Multilingual — 300 SWE-bench-style tasks across 9 programming languages
The same job in nine languages, with no Python anywhere in it.
- Released
- 2025
- Built by
- The SWE-bench team
- Size
- 300 tasks
- Status
- Active
- Reported by
- Anthropic, Moonshot AI, Mistral
- Judge
- 100
- Graded by the project's own tests. Deterministic, inspectable.
- Headroom
- 82
- Still separates the frontier from everything below it.
- Resolution
- 68
- 300 instances, so one item moves the score by 0.333 points.
- Fine print
- 45
- 4 documented caveats, the heaviest being possible training-set contamination.
- Adoption
- 47
- Reported by 3 labs; 3 published scores collected here.
Weighted from facts already on the record — not an editorial rating. A low score means a number from this benchmark needs more context to read, not that the benchmark is bad.
How the evaluation works
Every benchmark on this site is broken onto the same five stages. Select a stage to see what it actually involves.
Stage 3 · What the model can do while measured
Two Docker layers, hand-written commands
Core SWE-bench uses three image layers; Multilingual drops the dependency-cache layer, because 300 tasks across 42 repositories share almost no dependencies and the caching layer would have cost more curation effort than it saved. Install and test commands are specified by hand per instance. Unlike Verified, the official leaderboard is entirely single-scaffold: every entry was run by the SWE-bench team under mini-SWE-agent in the bash-only configuration, 250 steps and a $3.00 cost limit, so it never mixed scaffolds the way Verified did.
The model gets a shell. Scaffold quality and step budget move the score independently of the model.
- Budget
- 250 steps and a $3.00 cost limit on all official runs
- Tools exposed
- bash
Where the tasks came from
300 instances across 42 repositories, roughly 42-44 per language. Gold patches have a median of 10 lines.
Issue-linked PRs from the most-starred repositories in nine non-Python languages, each instance manually validated end to end.
Every task is public, so there is no held-out split to check contamination against.
What the number does not tell you
The most useful part of any benchmark is the fine print. These are the reasons a published score can mislead.
Roughly ±7 points of sampling noise per language means claims like 'this model is better at Rust' are weak. The authors also flag that the language gap may not be a difficulty gap: they checked and found Rust solutions modify more lines on average, yet Rust had the highest resolution rate.
Source
Published scores
Collected from model cards and independent evaluators. The configuration column is the part that decides whether two numbers can be compared at all.
| Model | Score | Configuration | Source |
|---|---|---|---|
| ModelClaude Opus 5Anthropic | Score89.5% | ConfigurationReported in the system card while the Opus 5 launch post shows no SWE-bench number at all. Seventeen points above the official single-scaffold leaderboard, on an undisclosed harness. | SourceLab-reported2026-07-24 · Claude Opus 5 system card, Table 8.1.A |
| ModelGemini 3 FlashGoogle DeepMind | Score72.7% | ConfigurationRun by the SWE-bench team under mini-SWE-agent, bash only. Total cost $105.55. | SourceThird-party2026-02-13 |
| ModelClaude 4.6 OpusAnthropic | Score72% | ConfigurationRun by the SWE-bench team under mini-SWE-agent, bash only. | SourceThird-party2026-02-13 |
Ordering is not a ranking. Rows with different configurations are not measuring the same thing — a number produced with tools, extended thinking, or majority voting is not comparable to one produced without.
Active — Still spreads the field.