Coding · Reasoning · Knowledge
SciCode
SciCode: A Research Coding Benchmark Curated by Scientists
Eighty coding problems written by working scientists from scripts they actually run.
- Released
- 2024
- Built by
- UIUC, University of Washington and collaborators
- Size
- 80 tasks
- Status
- Active
- Reported by
- Anthropic, OpenAI, Google DeepMind, Moonshot AI
- Judge
- 88
- Symbolic equivalence, so formatting differences do not change the score.
- Headroom
- 82
- Still separates the frontier from everything below it.
- Resolution
- 36
- 80 instances, so one item moves the score by 1.25 points.
- Fine print
- 56
- 4 documented caveats, the heaviest being construct validity.
- Adoption
- 58
- Reported by 4 labs; 2 published scores collected here.
Weighted from facts already on the record — not an editorial rating. A low score means a number from this benchmark needs more context to read, not that the benchmark is bad.
How the evaluation works
Every benchmark on this site is broken onto the same five stages. Select a stage to see what it actually involves.
Stage 3 · What the model can do while measured
No interpreter, own errors carried forward
Single-shot per subproblem with no tools, no shell, no interpreter access and no error feedback. That matters more here than on most benchmarks because numerical methods are unusually hard to get right without running them. The only thing that accumulates is context, and it is the model's own possibly-wrong output — deliberate, so the benchmark measures error propagation the way a real scripting session would suffer from it.
Single turn, no tools. The most reproducible setup there is.
- Budget
- one turn per subproblem, no retries
Where the tasks came from
80 main problems decomposed into 338 subproblems, split into a 15-problem dev set and a 65-problem test set. At n=80 the confidence intervals are wide.
Written by working scientists from research scripts across mathematics, physics, chemistry, biology and materials science.
Every task is public, so there is no held-out split to check contamination against.
What the number does not tell you
The most useful part of any benchmark is the fine print. These are the reasons a published score can mislead.
Main-problem and subproblem rates differ by about 10x, and supplying the scientist-written background raises scores substantially by converting a science problem into an instruction-following problem. The 2026 figures circulating around 59-60 almost certainly refer to subproblem rate under a with-background configuration, not to the leaderboard's main-problem metric — which is why they are not recorded as scores here.
Published scores
Collected from model cards and independent evaluators. The configuration column is the part that decides whether two numbers can be compared at all.
| Model | Score | Configuration | Source |
|---|---|---|---|
| ModelClaude 3.5 SonnetAnthropic | Score4.6% | ConfigurationMain problem resolve rate at release; the same run scored 26.0% on subproblems. | SourceThird-party2024-07-01 |
| ModelGPT-4oOpenAI | Score1.5% | ConfigurationMain problem resolve rate at release; 25.0% on subproblems. | SourceThird-party2024-07-01 |
Ordering is not a ranking. Rows with different configurations are not measuring the same thing — a number produced with tools, extended thinking, or majority voting is not comparable to one produced without.
Active — Still spreads the field.