Coding · Reasoning
LiveCodeBench Pro
LiveCodeBench Pro: How Do Olympiad Medalists Judge LLMs in Competitive Programming?
Elite contest problems judged by Olympiad medalists; the hard tier stayed at zero.
- Released
- 2025
- Built by
- NYU, Princeton, UC San Diego and collaborators
- Size
- 584 tasks
- Status
- Frontier
- Reported by
- Google DeepMind
- Judge
- 100
- Graded by the project's own tests. Deterministic, inspectable.
- Headroom
- 100
- Nothing is close to solving it, so the spread is real.
- Resolution
- 82
- 584 instances, so one item moves the score by 0.171 points.
- Fine print
- 65
- 3 documented caveats, the heaviest being construct validity.
- Adoption
- 24
- Reported by 1 lab; 4 published scores collected here.
Weighted from facts already on the record — not an editorial rating. A low score means a number from this benchmark needs more context to read, not that the benchmark is bad.
How the evaluation works
Every benchmark on this site is broken onto the same five stages. Select a stage to see what it actually involves.
Stage 3 · What the model can do while measured
Single submission, human rules
The model writes one program under the same rules as human contestants: no tools, no execution, no feedback, evaluated on the full hidden test set under the original time and memory limits. The paper separately ablates tool augmentation — terminal access and web search for o4-mini-high — and finds that high apparent performance in the wild is driven largely by implementation precision and tool augmentation rather than superior reasoning. Tools do not move the hard tier at all.
Single turn, no tools. The most reproducible setup there is.
- Budget
- single submission
Where the tasks came from
584 problems, mean difficulty 1827 ±822 Elo. Hard-tier problems elude more than 99.9% of human participants and sometimes go unsolved in live contests.
Codeforces, ICPC and IOI problems captured as contests run, before editorials or accepted solutions appear. LeetCode deliberately excluded.
Every task is public, so there is no held-out split to check contamination against.
What the number does not tell you
The most useful part of any benchmark is the fine print. These are the reasons a published score can mislead.
The expert error analysis found that compared to human experts, conceptual errors dominate model failures whereas implementation is a relative strength. Models do well on knowledge-heavy and structured logic-heavy problems and significantly worse on observation-heavy ones demanding creativity, they generate confidently incorrect justifications, and they frequently fail on the sample inputs they were given. A high aggregate score can therefore hide a total absence of the thing the benchmark is named for.
Source
Published scores
Collected from model cards and independent evaluators. The configuration column is the part that decides whether two numbers can be compared at all.
| Model | Score | Configuration | Source |
|---|---|---|---|
| ModelGemini 3.1 ProGoogle DeepMind | Score2887 | ConfigurationNot stated | SourceLab-reported2026-02-01 |
| ModelGemini 3 ProGoogle DeepMind | Score2439 | ConfigurationResults as of November 2025. | SourceLab-reported2025-11-01 |
| ModelGPT-5.1OpenAI | Score2243 | ConfigurationComparison figure published by Google on the Gemini 3 Pro card, results as of November 2025. | SourceLab-reported2025-11-01 |
| Model | Score | Configuration | Source |
|---|---|---|---|
| Modelo4-mini-highOpenAI | Score0% | ConfigurationHard tier (above 3000 Elo). The same model scored 53.5% on medium and 83.1% on easy, at Elo 2116 — the 1.5th percentile among human contestants. | SourceThird-party2025-06-01 |
Each table above measures a different quantity, so rows are only ever ordered within their own table. A model can appear more than once, and its numbers are not meant to be added or averaged.
Ordering is not a ranking. Rows with different configurations are not measuring the same thing — a number produced with tools, extended thinking, or majority voting is not comparable to one produced without.
Frontier — Nothing is close to solving it.