Coding
BigCodeBench
BigCodeBench — code generation with diverse function calls and complex instructions
1,140 Python functions to write, using 139 libraries, graded by unit tests.
- Released
- 2024
- Built by
- BigCode Project
- Size
- 1,140 tasks
- Status
- Active
- Reported by
- DeepSeek, Alibaba (Qwen)
- Judge
- 100
- Graded by the project's own tests. Deterministic, inspectable.
- Headroom
- 82
- Still separates the frontier from everything below it.
- Resolution
- 92
- 1,140 instances, so one item moves the score by 0.088 points.
- Fine print
- 45
- 5 documented caveats, the heaviest being possible training-set contamination.
- Adoption
- 48
- Reported by 2 labs; 10 published scores collected here.
Weighted from facts already on the record — not an editorial rating. A low score means a number from this benchmark needs more context to read, not that the benchmark is bad.
How the evaluation works
Every benchmark on this site is broken onto the same five stages. Select a stage to see what it actually involves.
Stage 3 · What the model can do while measured
One shot, no tools — the sandbox belongs to the grader
The model gets a single prompt and returns a code block. It cannot execute anything, read a traceback, or iterate. Docker, E2B and the hosted Gradio evaluator appear only after generation, to run the tests on the submitted code. This is the deliberate contrast with SWE-bench: the dataset card states that BigCodeBench is a benchmark for LLMs rather than LLM agents, and that repo-level and agent-centric benchmarks are unsuitable for the job it wants to do.
Single turn, no tools. The most reproducible setup there is.
- Budget
- single turn, greedy decoding
Where the tasks came from
1,140 tasks in the Full set, so one task is 0.088%. BigCodeBench-Hard is a 148-task subset — one task there is worth 0.68%, which is why Hard scores move in visible steps.
Synthesised by GPT-4 from ODEX Stack Overflow seeds in 2024, then curated by 27 human experts. Fully public; five dataset versions exist (v0.1.0 through v0.1.4) and all five contain the same 1,140 tasks.
Every task is public, so there is no held-out split to check contamination against.
What the number does not tell you
The most useful part of any benchmark is the fine print. These are the reasons a published score can mislead.
Calibrated pass@1 exists because instruction-tuned models drop imports and constants when the prompt is long, which fails the tests for reasons the authors judged uninteresting. The harness therefore prepends the missing setup and then runs the suite. That is a defensible choice, but it means the number measures whether the function body is correct, not whether the model emitted a runnable program — and an uncalibrated pass@1 on the same generations is a different, lower number. The repo produces both files (sanitized and sanitized_calibrated) and reports only the second.
Source
Published scores
Collected from model cards and independent evaluators. The configuration column is the part that decides whether two numbers can be compared at all.
| Model | Score | Configuration | Source |
|---|---|---|---|
| ModelGemini-Exp-1206Google | Score62.4% | ConfigurationCalibrated pass@1, greedy decoding, no response prefilling. Top of the Full-set Complete leaderboard as of its final update. | SourceThird-party2024-12-06 |
| ModelDeepSeek-V3DeepSeek | Score62.2% | ConfigurationCalibrated pass@1, greedy decoding. Scores 50.0 on Instruct and 40.5 on Hard Complete. | SourceThird-party2024-12-26 |
| ModelLlama-4-MaverickMeta | Score61.4% | ConfigurationCalibrated pass@1. 49.7 on Instruct, but only 29.1 on Hard Complete — one of the largest Full-to-Hard drops on the board. | SourceThird-party2025-04-05 |
| ModelGPT-4o-2024-05-13OpenAI | Score61.1% | ConfigurationThe paper's headline figure. Calibrated pass@1 with response prefilling. | SourceThird-party2024-05-13 |
| Model | Score | Configuration | Source |
|---|---|---|---|
| ModelGPT-4o-2024-05-13OpenAI | Score51.1% | ConfigurationThe paper's second headline figure — ten points below the same model on Complete, which is the whole argument for the Instruct split. | SourceThird-party2024-05-13 |
| Model | Score | Configuration | Source |
|---|---|---|---|
| ModelClaude-3.7-Sonnet-20250219 (temperature=1, length=12800, reasoning=3200)Anthropic | Score39.2% | ConfigurationExtended thinking with a 3,200-token reasoning budget and 12,800-token length. Hard set only; no Full-set entry exists for this model. | SourceThird-party2025-02-19 |
| ModelGPT-4.1-2025-04-14OpenAI | Score33.8% | ConfigurationThe most recent entry of any kind on the leaderboard. 31.8 on Instruct. No Full-set numbers were ever published for it. | SourceThird-party2025-04-14 |
| Model | Score | Configuration | Source |
|---|---|---|---|
| Modelo3-mini-2025-01-31 (temperature=1, reasoning=medium)OpenAI | Score33.1% | ConfigurationBest Instruct score on the Hard set. Medium reasoning beats high (32.4) and low (31.1) on Instruct, while all three tie at 37.8 on Complete. | SourceThird-party2025-01-31 |
| Model | Score | Configuration | Source |
|---|---|---|---|
| ModelDeepSeek-V4-Pro-BaseDeepSeek | Score59.2% | Configuration3-shot base-model evaluation in DeepSeek's own comparison table, which does not say which split it used. DeepSeek-V3.2-Base scores 63.9 on the same row and DeepSeek-V4-Flash-Base 56.8. Not comparable with the leaderboard's zero-shot calibrated figures. | SourceLab-reported2026-04-24 |
| Model | Score | Configuration | Source |
|---|---|---|---|
| ModelQwen2.5-Coder-32BAlibaba (Qwen) | Score53.6% | Configuration3-shot base-model evaluation from the Qwen2.5-Coder technical report, Table 5. The same table gives 26.4 on the Hard subset. | SourceLab-reported2024-09-18 |
Each table above measures a different quantity, so rows are only ever ordered within their own table. A model can appear more than once, and its numbers are not meant to be added or averaged.
Ordering is not a ranking. Rows with different configurations are not measuring the same thing — a number produced with tools, extended thinking, or majority voting is not comparable to one produced without.
Active — Still spreads the field.