Coding
LiveCodeBench
LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code
Programming-contest problems stamped with a date, so you can score only fresh ones.
- Released
- 2024
- Built by
- UC Berkeley, MIT, Cornell
- Size
- 1,055 tasks
- Status
- Near ceiling
- Reported by
- OpenAI, Google DeepMind, Anthropic, DeepSeek, Alibaba (Qwen), Mistral
- Judge
- 100
- Graded by the project's own tests. Deterministic, inspectable.
- Headroom
- 40
- The top models crowd the ceiling; gaps are shrinking.
- Resolution
- 92
- 1,055 instances, so one item moves the score by 0.095 points.
- Fine print
- 45
- 4 documented caveats, the heaviest being possible training-set contamination.
- Adoption
- 86
- Reported by 6 labs; 3 published scores collected here.
Weighted from facts already on the record — not an editorial rating. A low score means a number from this benchmark needs more context to read, not that the benchmark is bad.
How the evaluation works
Every benchmark on this site is broken onto the same five stages. Select a stage to see what it actually involves.
Stage 3 · What the model can do while measured
One shot, no interpreter
Code generation is single-shot with no shell, no interpreter, no internet and no retrieval — the model writes one program from the statement. Self-repair is the one exception and the only place in this benchmark with a feedback loop: the model gets its failing program plus the exception text or failing test and revises once. The output-prediction and execution scenarios forbid execution outright, so the model has to simulate in its head. Default sampling is n=10 at temperature 0.2, top_p 0.95.
Single turn, no tools. The most reproducible setup there is.
- Budget
- single turn (the self-repair scenario adds exactly one more)
Where the tasks came from
1,055 problems in v6 (May 2023 – Apr 2025). v1 had 400. Papers routinely cite 'LiveCodeBench' with no version, which makes those numbers non-comparable.
Scraped continuously from LeetCode, AtCoder and Codeforces contests, each problem stamped with its release date.
Every task is public, so there is no held-out split to check contamination against.
What the number does not tell you
The most useful part of any benchmark is the fine print. These are the reasons a published score can mislead.
v1 through v6 are cumulative windows over a growing scrape: 400, 511, 612, 713, 880 and 1,055 problems. A `code_generation_lite` subset is the default for speed and is smaller again. Because the difficulty mix shifts as new contests land, two papers citing 'LiveCodeBench pass@1' can differ by more than the model gap between them.
Published scores
Collected from model cards and independent evaluators. The configuration column is the part that decides whether two numbers can be compared at all.
| Model | Score | Configuration | Source |
|---|---|---|---|
| ModelClaude Fable 5Anthropic | Score89.78% | Configurationv6. Third-party evaluator; the official leaderboard renders client-side and returned no data, and no lab model card corroborates these figures. | SourceThird-party2026-08-15 |
| ModelClaude Opus 5Anthropic | Score89.03% | Configurationv6. Same third-party evaluator, uncorroborated. | SourceThird-party2026-08-15 |
| ModelGPT-5.2 CodexOpenAI | Score87.99% | Configurationv6. Same third-party evaluator, uncorroborated. | SourceThird-party2026-08-15 |
Ordering is not a ranking. Rows with different configurations are not measuring the same thing — a number produced with tools, extended thinking, or majority voting is not comparable to one produced without.
Near ceiling — The top models are close enough to the ceiling to crowd.