Coding
HumanEval
HumanEval (from "Evaluating Large Language Models Trained on Code")
164 hand-written Python puzzles: write the function body from its docstring.
- Released
- 2021
- Built by
- OpenAI
- Size
- 164 tasks
- Status
- Saturated
- Reported by
- OpenAI, Anthropic, Google DeepMind, Meta, DeepSeek, Mistral, Alibaba (Qwen)
- Judge
- 100
- Graded by the project's own tests. Deterministic, inspectable.
- Headroom
- 8
- Bunched at the ceiling. A win here no longer means much.
- Resolution
- 52
- 164 instances, so one item moves the score by 0.610 points.
- Fine print
- 46
- 4 documented caveats, the heaviest being possible training-set contamination.
- Adoption
- 81
- Reported by 7 labs; 1 published score collected here.
Weighted from facts already on the record — not an editorial rating. A low score means a number from this benchmark needs more context to read, not that the benchmark is bad.
How the evaluation works
Every benchmark on this site is broken onto the same five stages. Select a stage to see what it actually involves.
Stage 3 · What the model can do while measured
One completion, no tools
There is no environment in the modern sense. The model receives the prompt string and emits a continuation: no shell, no filesystem, no interpreter, no test visibility, no second turn. Execution happens afterwards and only on the grader's side, inside a gVisor container with eBPF firewall rules blocking all inbound and outbound traffic. The public harness ships with execution disabled by default and warns that it exists to run untrusted model-generated code.
Single turn, no tools. The most reproducible setup there is.
- Budget
- single turn
Where the tasks came from
164 problems, so one problem is worth 0.6% — and EvalPlus found 18 of the 164 reference solutions were themselves wrong.
Written from scratch by OpenAI staff in 2021 specifically so the problems would not already exist on GitHub, then published on GitHub.
Every task is public, so there is no held-out split to check contamination against.
What the number does not tell you
The most useful part of any benchmark is the fine print. These are the reasons a published score can mislead.
Frontier models reached 90%+ during 2023-2024, and the ceiling is bounded anyway by the 11% of reference solutions that are defective. The LiveCodeBench Pro paper states flatly that modern models achieve near-perfect accuracy on HumanEval. The Gemini 3 Pro model card does not report it at all. No frontier lab publishes a current figure, which is why there is no verified SOTA here.
Source
Published scores
Collected from model cards and independent evaluators. The configuration column is the part that decides whether two numbers can be compared at all.
| Model | Score | Configuration | Source |
|---|---|---|---|
| ModelClaude 3.5 SonnetAnthropic | Score92% | ConfigurationCited in the LiveCodeBench Pro paper as evidence that HumanEval is finished, not as a fresh measurement. | SourceThird-party2025-06-01 |
Ordering is not a ranking. Rows with different configurations are not measuring the same thing — a number produced with tools, extended thinking, or majority voting is not comparable to one produced without.
Saturated — Scores are bunched at the ceiling; it no longer separates models.