CodePress

Reference

AI model benchmarks, explained.

Labs publish tables of benchmark scores with almost no explanation of what was measured. This is a plain-English breakdown of 78 of them: the tasks, the environment the model runs in, who grades the answer, and the fine print that decides whether two numbers can be compared at all.

benchmarks documented
78benchmarks documented
labs whose reporting is tracked
44labs whose reporting is tracked
that put the model in a real environment
31that put the model in a real environment

Every eval breaks into the same five stages

Once you see the stages, the differences between benchmarks stop being a wall of acronyms. A knowledge quiz and an agentic coding eval differ mostly in two places — stage three and stage four.

  1. 01 · Source

    Where the tasks came from

    Scraped GitHub issues, exam papers, or questions commissioned from PhDs. This decides what contamination is even possible.

  2. 02 · Task

    What one instance looks like

    A multiple-choice question is a different kind of measurement from a repository at a base commit plus a bug report.

  3. 03 · Environment

    What the model can do while measured

    No tools at all, a Docker container with a shell, a simulated API with another model playing the customer, or the live internet.

  4. 04 · Judge

    Who decides the answer is right

    String comparison, the project's own unit tests, a rubric-following LLM, or paid human experts. Changing the judge changes the score.

  5. 05 · Metric

    What gets published

    Percent resolved, pass@1, pass^8, Elo, dollars earned, or the task length a model clears half the time.

Three reasons a headline score misleads

01

The instrument changes underneath the trend line

MathArena graded USAMO 2025 with four human olympiad judges, and the best model scored 30.06%. It graded USAMO 2026 with a jury of three LLMs, and the best model scored 98.21%. Those two numbers are routinely plotted as one curve. Separately, FrontierMath v2 corrected errors in 42% of its problems — before and after are not the same benchmark.

02

Most benchmarks are too small to separate frontier models

GPQA Diamond is 198 questions, so one question is worth half a point and the top models sit inside the margin of error. AIME is 30 questions, with several models tied one question apart. MathArena Apex is 12 problems, where a single problem moves the score by 8.33 points.

03

The configuration matters more than the model

The same benchmark run with tools, extended thinking, or majority voting is a different measurement from the one run without. On agentic evals a step budget of 15 versus 100 appears in the same comparison table. Two labs have published different scores for the same competitor on the same benchmark version.

How labs actually report

None of the following is hypothetical. Each is drawn from a published model card, system card, or independent audit.

  1. 01

    The scaffold moves the score more than the model does

    Anthropic reported Claude Opus 4.8 at 78.9 on Terminal-Bench under one harness and 82.7 under another. Cognition measured the same model at 86.9 on its own framework. That is an eight-point swing from scaffolding alone — larger than most claimed generational gains.

  2. 02

    Maximum effort is often not the best result

    Labs run a model at several reasoning efforts and publish the row that won. Claude Opus 4.6 scored 62.7 on MCP-Atlas at high effort but 59.5 at max. One Opus 5 table drew its best three results from three different effort settings.

  3. 03

    A benchmark vanishing from a card is itself a result

    It is never the one that improved. Kimi K3 dropped eleven benchmarks its predecessor reported, including SWE-bench Verified at 80.2. GLM-5.2 dropped SWE-bench Verified too — and GLM-5.3 later retro-published GLM-5.2's withheld CyberGym score, which shows the number existed all along.

  4. 04

    The same model can score 15.8% and 92.5% on one benchmark

    On Harvey LAB, both numbers come from the same Grok 4.6 run. One is a task-resolution rate where every rubric criterion must pass; the other is the share of individual criteria passed. A 77-point gap that is entirely metric definition, not capability.

  5. 05

    Passing the test is not always solving the problem

    Cursor audited 731 SWE-bench Pro trajectories and found 63% of one model's successful resolutions retrieved the fix rather than derived it. Under a strict harness that model fell from 87.1 to 73.0 — and the effect is family-specific, so it distorts rankings rather than just absolute scores.

  6. 06

    Almost nobody publishes a table you can copy

    Anthropic, Meta, Mistral, Microsoft, Cohere, MiniMax and Z.ai publish headline numbers as images. Machine-readable tables exist only in system cards or evaluation PDFs, where they exist at all. This is why so many scores circulate through third-party transcription.

Browse every benchmark

Grouped by what the benchmark measures, and ordered within each group by signal — how much a score on it actually tells you, worked out from the judge, the size of the set, how much headroom is left, the documented fine print, and how many labs report it. Open a row to see that breakdown, the five stages and the current top scores without leaving the page.

78 benchmarks

Sort
Category
Environment

Best first — by how much a score on it tells you. Open a row for the breakdown.

Coding17

Write or repair code. The judge is almost always a test suite, which makes these the most objective scores on the site.

Agentic16

Long multi-step runs where the model drives tools itself. The scaffold moves the score as much as the model does.

Reasoning7

Hard problems with a single checkable answer. Small sets, so the error bars are wider than the gaps.

Math6

Competition and research mathematics, graded on the final answer or on the proof.

Knowledge6

What the model knows and whether it admits when it does not. Contamination is the standing risk.

Multimodal9

Images, video or audio in the input, not just text.

Long context3

Retrieval and reasoning over inputs far larger than a normal prompt.

Tool use2

Calling APIs and functions correctly, in the right order.

Safety8

Refusals, honesty and attack resistance. Several of these are scored so that a lower number is better.

Human preference2

Humans pick a winner. Captures taste, and inherits the biases of whoever voted.

Aggregate index2

Composite indices built from other benchmarks. Convenient, and one level further from anything measured.