Coding
Aider Polyglot
Aider polyglot coding benchmark
225 hard practice exercises, graded on the code and on the edit format.
- Released
- 2024
- Built by
- Paul Gauthier (Aider)
- Size
- 225 tasks
- Status
- Saturated
- Reported by
- Google DeepMind, DeepSeek, Moonshot AI
- Judge
- 100
- Graded by the project's own tests. Deterministic, inspectable.
- Headroom
- 8
- Bunched at the ceiling. A win here no longer means much.
- Resolution
- 68
- 225 instances, so one item moves the score by 0.444 points.
- Fine print
- 45
- 5 documented caveats, the heaviest being possible training-set contamination.
- Adoption
- 47
- Reported by 3 labs; 3 published scores collected here.
Weighted from facts already on the record — not an editorial rating. A low score means a number from this benchmark needs more context to read, not that the benchmark is bad.
How the evaluation works
Every benchmark on this site is broken onto the same five stages. Select a stage to see what it actually involves.
Stage 3 · What the model can do while measured
Two turns, zero tools
Docker is required only because the harness executes unreviewed model code; the model itself gets no shell and no tools. File discovery is hard-disabled in the harness, so there is no repo map and nothing in context but the stub. The loop is exactly two turns: one attempt, then if the tests fail, the full stderr comes back with a fixed retry instruction and the model gets exactly one more try. Test runs time out at 180 seconds and a timeout counts as a failure.
Single turn, no tools. The most reproducible setup there is.
- Budget
- two turns — one attempt plus exactly one retry
See the testing errors above. The tests are correct, don't try and change them. Fix the code in {file_list} to resolve the errors.Where the tasks came from
225 exercises — Java 47, JavaScript 49, Go 39, Python 34, Rust 30, C++ 26. One exercise is 0.44 points and the binomial standard error near 80% is about 2.7 points, so the top few rows are inside noise.
Exercism practice exercises in six language tracks, filtered to the 225 that three or fewer of seven 2024-era models could solve. Frozen since December 2024.
Every task is public, so there is no held-out split to check contamination against.
What the number does not tell you
The most useful part of any benchmark is the fine print. These are the reasons a published score can mislead.
Every exercise directory carries a .meta/example file containing a complete working solution, on public GitHub since December 2024, and the upstream Exercism repositories carry canonical plus thousands of community solutions. Aider's own methodology page concedes of the predecessor benchmark that all of this code was probably part of its original training data. The set has never been refreshed, so contamination only worsens with time. Moonshot's Kimi K2 report is the only lab we found that addressed it, reporting 60.0% while employing rigorous decontamination procedures.
Source
Published scores
Collected from model cards and independent evaluators. The configuration column is the part that decides whether two numbers can be compared at all.
| Model | Score | Configuration | Source |
|---|---|---|---|
| ModelGPT-5 (high)OpenAI | Score88% | Configurationdiff edit format, 91.6% correct edit format, single run, total cost $29.08. | SourceCommunity2025-08-23 |
| ModelGemini 2.5 ProGoogle DeepMind | Score82.2% | ConfigurationAider Polyglot scores are the pass rate average of 3 trials; diff/diff-fenced formats. | SourceLab-reported2025-07-01 |
| ModelKimi K2Moonshot AI | Score60% | Configurationwhile employing rigorous decontamination procedures | SourceLab-reported2025-07-01 |
Ordering is not a ranking. Rows with different configurations are not measuring the same thing — a number produced with tools, extended thinking, or majority voting is not comparable to one produced without.
Saturated — Scores are bunched at the ceiling; it no longer separates models.