Reasoning · Coding · Math
LiveBench
LiveBench — a challenging, contamination-free LLM benchmark
Rotating question sets, objective ground truth, and no LLM judge anywhere.
- Released
- 2024
- Built by
- Abacus.AI with NYU, Nvidia, UMD and USC
- Size
- 1,270 tasks
- Status
- Active
- Judge
- 58
- Mixed grading, so part of the score inherits the weakest judge.
- Headroom
- 82
- Still separates the frontier from everything below it.
- Resolution
- 92
- 1,270 instances, so one item moves the score by 0.079 points.
- Fine print
- 45
- 5 documented caveats, the heaviest being possible training-set contamination.
- Adoption
- 22
- No lab reports it in a model card, so it rarely appears in a comparison.
Weighted from facts already on the record — not an editorial rating. A low score means a number from this benchmark needs more context to read, not that the benchmark is bad.
How the evaluation works
Every benchmark on this site is broken onto the same five stages. Select a stage to see what it actually involves.
Stage 3 · What the model can do while measured
No tools for six categories, Docker for the seventh
1,198 of the 1,270 questions are single-turn API calls with no tools, no retrieval and no scaffold — the model is given the prompt and its text is graded. The 72 agentic-coding questions are a different measurement entirely: they run mini-SWE-agent against real GitHub repositories in per-task Docker images with a 250-step limit, scored through the Multi-SWE-Bench harness. The repo warns that storing those images can take up to 150GB. That seventh of the benchmark is the only place a scaffold exists, and it is folded into the same headline average as the zebra puzzles.
Single turn, no tools. The most reproducible setup there is.
- Budget
- Single turn everywhere except agentic coding, which allows 250 mini-SWE-agent steps
- Tools exposed
- bash (agentic coding only)
Where the tasks came from
1,270 questions in the 2026-06-25 release, but the categories are wildly uneven: Mathematics has 379, Instruction Following 200, Reasoning 202, Data Analysis and Language 150 each, Coding 117 and Agentic Coding 72. Because every category counts for one seventh, an agentic-coding question carries roughly 5.3 times the weight of a maths question.
Written by the LiveBench team from post-cutoff material and released in eleven dated batches since June 2024; older batches remain selectable but are not refreshed.
A held-out or private split exists, which makes contamination easier to detect than on a fully public set.
What the number does not tell you
The most useful part of any benchmark is the fine print. These are the reasons a published score can mislead.
The public HuggingFace datasets stop at the 2024-11-25 release. Nothing from any 2025 or 2026 release is published, the agentic-coding tasks have no public dataset at all, and the repo's own README tells you to pass --livebench-release-option 2024-11-25 if you want questions you can actually download. So the leaderboard everyone quotes is computed on 1,270 questions no outsider has seen. The cadence has also drifted: the paper and README still say questions are released monthly, the site's front page now says 'refreshed every six months', and the actual record is eleven releases in twenty-six months with gaps as long as six months. The 2026-06-25 release, the one the leaderboard defaults to, has no changelog entry at all.
Source
Published scores
Collected from model cards and independent evaluators. The configuration column is the part that decides whether two numbers can be compared at all.
| Model | Score | Configuration | Source |
|---|---|---|---|
| ModelClaude Fable 5 Max EffortAnthropic | Score82.97% | ConfigurationLiveBench-2026-06-25. Categories: Reasoning 89.7, Coding 86.0, Agentic Coding 62.2, Mathematics 96.0, Data Analysis 80.5, Language 90.7, Instruction Following 75.8. Most expensive model on the board at $1.48 per successful task. | SourceThird-party2026-06-25 |
| ModelGPT-5.6 Sol Max EffortOpenAI | Score81.05% | ConfigurationLiveBench-2026-06-25. Leads Reasoning at 91.7 and Mathematics at 96.2 but trails on Agentic Coding at 56.2. | SourceThird-party2026-06-25 |
| ModelGPT-5.5 Thinking xHigh EffortOpenAI | Score80.19% | ConfigurationLiveBench-2026-06-25. Agentic Coding 54.0. | SourceThird-party2026-06-25 |
| ModelClaude 5 Opus Thinking Max EffortAnthropic | Score80.08% | ConfigurationLiveBench-2026-06-25. Best Agentic Coding score on the board at 65.2, but the weakest Instruction Following of the top ten at 63.8 — the two categories cancel out. | SourceThird-party2026-06-25 |
| ModelSmaug-AgenticAbacus.AI | Score79.51% | ConfigurationLiveBench-2026-06-25. Open-weight fine-tune of Kimi K3 by Abacus.AI, which sponsors LiveBench. Ranks above the Kimi K3 base model it was built from. | SourceThird-party2026-06-25 |
| ModelKimi K3Moonshot AI | Score79.19% | ConfigurationLiveBench-2026-06-25. Highest-ranked unmodified open-weight model. Coding 81.4, Agentic Coding 62.2. | SourceThird-party2026-06-25 |
| ModelGemini 3.7 Flash HighGoogle | Score78.83% | ConfigurationLiveBench-2026-06-25. Tops Instruction Following at 79.9 and costs $0.157 per successful task against $1.48 for the leader. | SourceThird-party2026-06-25 |
| ModelGrok 4.6xAI | Score78.04% | ConfigurationLiveBench-2026-06-25. Agentic Coding 57.0. | SourceThird-party2026-06-25 |
| ModelDeepSeek V4 Pro 0813DeepSeek | Score77.44% | ConfigurationLiveBench-2026-06-25. GA build on DeepSeek's first-party endpoint. | SourceThird-party2026-06-25 |
| ModelGemini 3.1 Pro Preview HighGoogle | Score76.95% | ConfigurationLiveBench-2026-06-25. Agentic Coding 44.1 — the largest gap between a model's strongest and weakest category in the top fifteen. | SourceThird-party2026-06-25 |
Ordering is not a ranking. Rows with different configurations are not measuring the same thing — a number produced with tools, extended thinking, or majority voting is not comparable to one produced without.
Active — Still spreads the field.