Knowledge
MMLU
Massive Multitask Language Understanding
Fifty-seven-subject multiple-choice exam, from elementary math to professional law.
- Released
- 2020
- Built by
- UC Berkeley, Columbia, UChicago, UIUC
- Size
- 14,042 tasks
- Status
- Saturated
- Reported by
- OpenAI, Google DeepMind, Anthropic, Meta, Mistral, Alibaba (Qwen), DeepSeek
- Judge
- 80
- Deterministic, but it can only grade a final answer and not the work.
- Headroom
- 8
- Bunched at the ceiling. A win here no longer means much.
- Resolution
- 100
- 14,042 instances, so one item moves the score by 0.007 points.
- Fine print
- 47
- 4 documented caveats, the heaviest being possible training-set contamination.
- Adoption
- 78
- Reported by 7 labs; 0 published scores collected here.
Weighted from facts already on the record — not an editorial rating. A low score means a number from this benchmark needs more context to read, not that the benchmark is bad.
How the evaluation works
Every benchmark on this site is broken onto the same five stages. Select a stage to see what it actually involves.
Stage 3 · What the model can do while measured
Closed book, no tools
The paper's protocol is 5-shot using the dev exemplars. Two incompatible implementations then diverged: log-probability scoring, where the model never generates and you take the argmax over P("A")..P("D"), which is what the original harness and lm-evaluation-harness defaults do; and 0-shot generative chain-of-thought with regex letter extraction, which is what every 2024-2026 reasoning model uses. The same model can differ by many points between the two, and neither is the MMLU number.
Single turn, no tools. The most reproducible setup there is.
- Budget
- Single turn
Where the tasks came from
14,042 test items, so sampling noise is tiny. The uncertainty here comes from the answer key, not from n.
Hand-collected from public practice exams and textbook question banks; 57 subject configs; no private split.
Every task is public, so there is no held-out split to check contamination against.
What the number does not tell you
The most useful part of any benchmark is the fine print. These are the reasons a published score can mislead.
The MMLU-Redux audit found 6.49% of sampled questions have a wrong ground-truth label or are otherwise broken, with the Virology subset at 57% error. Some items reference a figure or passage that was never included. That puts the real ceiling somewhere around 93%, which means a 92% and a 94% claim are not distinguishable from label luck.
Source
Published scores
Collected from model cards and independent evaluators. The configuration column is the part that decides whether two numbers can be compared at all.
No published scores collected for this benchmark yet.
Saturated — Scores are bunched at the ceiling; it no longer separates models.