Coding · Agentic
Multi-SWE-bench
Multi-SWE-bench — 1,632 issue-resolving instances across seven non-Python languages
SWE-bench's recipe rerun in seven other languages, with human difficulty labels.
- Released
- 2025
- Built by
- ByteDance Seed
- Size
- 1,632 tasks
- Status
- Active
- Judge
- 100
- Graded by the project's own tests. Deterministic, inspectable.
- Headroom
- 82
- Still separates the frontier from everything below it.
- Resolution
- 92
- 1,632 instances, so one item moves the score by 0.061 points.
- Fine print
- 45
- 5 documented caveats, the heaviest being possible training-set contamination.
- Adoption
- 22
- No lab reports it in a model card, so it rarely appears in a comparison.
Weighted from facts already on the record — not an editorial rating. A low score means a number from this benchmark needs more context to read, not that the benchmark is bad.
How the evaluation works
Every benchmark on this site is broken onto the same five stages. Select a stage to see what it actually involves.
Stage 3 · What the model can do while measured
One hand-built Docker image per pull request
There is no generic installer for seven language ecosystems, so each PR gets its own image: CI workflow files, READMEs and trial runs are read by hand to separate repo-common from PR-specific dependencies, a Dockerfile is generated and built, and any PR whose image cannot be built or launched at its base commit is discarded. Unlike SWE-bench Multilingual, the leaderboard imposes no scaffold, tool set or step budget — submissions arrive per language from whatever system the submitter runs, and the three reference methods in the paper (MagentLess, MSWE-agent, MopenHands) are ports of Agentless, SWE-agent and OpenHands.
The model gets a shell. Scaffold quality and step budget move the score independently of the model.
Where the tasks came from
1,632 instances over 39 repositories: Go 428, JavaScript 356, Rust 239, TypeScript 224, C++ 129, Java 128, C 128. The leaderboard's Python column is SWE-bench Verified's 500 instances, not part of the 1,632. A mini split of 400 (50 per language across 8 languages) and a 300-instance flash split were released in 2025.
Issue-linked merged pull requests mined from high-star GitHub repositories in seven languages, execution-filtered on test status transitions and then human-verified. Published under Apache-2.0 with the annotation results and the verification questionnaire in the repository. A separate Multi-SWE-RL pool of 4,723 instances exists for reinforcement-learning use and is not the benchmark. The Hugging Face repository also ships eight Kotlin dataset files totalling 105 instances, a language the paper, the dataset card and the leaderboard all omit.
Every task is public, so there is no held-out split to check contamination against.
What the number does not tell you
The most useful part of any benchmark is the fine print. These are the reasons a published score can mislead.
The two benchmarks do the same thing to the same kind of source material. Multi-SWE-bench's seven languages are a strict subset of SWE-bench Multilingual's nine, which add PHP and Ruby; both mine issue-linked merged PRs from high-star repositories, both grade with fail-to-pass and pass-to-pass tests in per-instance Docker, and ten repositories appear in both — axios, ripgrep, fmt, gson, jq, nlohmann/json, nushell, bat, tokio and vuejs/core. Comparing the released instance lists, nineteen of SWE-bench Multilingual's 300 instances are literally the same pull requests as Multi-SWE-bench instances, including eight fmtlib/fmt PRs and four jq PRs. The differences that remain are real but are about packaging, not construct: Multi-SWE-bench is five times larger, carries human time-to-fix difficulty labels that Multilingual does not, and accepts arbitrary submissions where Multilingual's official board is single-scaffold.
Source
Published scores
Collected from model cards and independent evaluators. The configuration column is the part that decides whether two numbers can be compared at all.
| Model | Score | Configuration | Source |
|---|---|---|---|
| ModelMopenHands + Qwen3-Coder-480B-A35BAlibaba Qwen | Score25.75% | ConfigurationThe 400-instance mini split, 50 per language across 8 languages — the only board that pools languages into one figure. Top entry, and the last submission to it. | SourceCommunity2025-07-25 |
| Model | Score | Configuration | Source |
|---|---|---|---|
| ModelCodeArts Agent + CodeArts-GLM-5.1Zhipu AI | Score53.91% | ConfigurationHighest C entry by 43 points over the next row. Not marked checked, not open source; the harness is Huawei's CodeArts and the model is a CodeArts build of GLM-5.1. | SourceCommunity2026-06-24 |
| Model | Score | Configuration | Source |
|---|---|---|---|
| ModelCodeArts Agent + CodeArts-GLM-5.1Zhipu AI | Score47.29% | ConfigurationSame submitter and same caveat as the C row: not checked, not open source. The next entry is InfCode with GPT-5 (Sep) at 25.58%. | SourceCommunity2026-07-14 |
| Model | Score | Configuration | Source |
|---|---|---|---|
| ModelCodeArts Agent + CodeArts-MiniMax-M2.5MiniMax | Score43.75% | Configuration128 Java instances, so 56 resolved. Easy 85.2%, medium 46.2%, hard 8.3% on a 36-instance hard tier. Not marked checked. | SourceCommunity2026-03-27 |
| Model | Score | Configuration | Source |
|---|---|---|---|
| ModelRepoRepair + Claude 3.5 Sonnet (Oct)Anthropic | Score27.23% | ConfigurationHighest TypeScript entry and the last submission to that column, dated September 2025. Checked and open source. | SourceCommunity2025-09-10 |
| Model | Score | Configuration | Source |
|---|---|---|---|
| ModelMopenHands + Claude 3.7 SonnetAnthropic | Score15.9% | ConfigurationStill the top Rust entry. The Rust column has taken no submission since 26 April 2025. | SourceCommunity2025-03-29 |
| Model | Score | Configuration | Source |
|---|---|---|---|
| ModelMopenHands + Gemini 2.5 ProGoogle DeepMind | Score12.62% | ConfigurationTop Go entry on 428 instances, the largest language in the set, and the last Go submission of any kind. | SourceCommunity2025-04-26 |
| Model | Score | Configuration | Source |
|---|---|---|---|
| ModelRepoRepair + Claude 4.5 SonnetAnthropic | Score17.13% | Configuration356 JavaScript instances, 241 of them in the hard tier — the resolve rate there is 10.4%. Not marked checked. | SourceCommunity2026-02-02 |
| Model | Score | Configuration | Source |
|---|---|---|---|
| ModelMopenHands + Claude 3.7 SonnetAnthropic | Score52.2% | ConfigurationThe Python column is SWE-bench Verified's 500 instances, not Multi-SWE-bench data. Easy 71.6%, medium 44.8%, hard 11.1%. Checked by the Multi-SWE-bench team. | SourceCommunity2025-03-29 |
Each table above measures a different quantity, so rows are only ever ordered within their own table. A model can appear more than once, and its numbers are not meant to be added or averaged.
Ordering is not a ranking. Rows with different configurations are not measuring the same thing — a number produced with tools, extended thinking, or majority voting is not comparable to one produced without.
Active — Still spreads the field.