Agentic · Coding
MLE-bench
MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering
Can an agent win a Kaggle medal, unaided, from scratch?
- Released
- 2024
- Built by
- OpenAI
- Size
- 75 tasks
- Status
- Active
- Reported by
- OpenAI
- Judge
- 94
- Graded on the final state of the environment, not on prose.
- Headroom
- 82
- Still separates the frontier from everything below it.
- Resolution
- 36
- 75 instances, so one item moves the score by 1.33 points.
- Fine print
- 57
- 3 documented caveats, the heaviest being possible training-set contamination.
- Adoption
- 16
- Reported by 1 lab; 1 published score collected here.
Weighted from facts already on the record — not an editorial rating. A low score means a number from this benchmark needs more context to read, not that the benchmark is bad.
How the evaluation works
Every benchmark on this site is broken onto the same five stages. Select a stage to see what it actually involves.
Stage 3 · What the model can do while measured
GPU container with a shell
A containerised Linux workspace holding the competition data, a GPU, Python and the usual ML stack, plus a shell. The agent writes and runs code across hours of wall-clock, normally under a scaffold such as AIDE, MLAB or OpenHands — and the scaffold choice matters as much as the model does. There is no access to the live Kaggle leaderboard during the run.
The model gets a shell. Scaffold quality and step budget move the score independently of the model.
- Budget
- Hours of wall-clock per competition under a fixed compute budget.
- Tools exposed
- ShellPython with the standard ML stackGPU computeLocal competition dataset
Where the tasks came from
75 competitions, so a single competition is worth about 1.3% of the medal rate.
Curated from real Kaggle competitions that have a usable local evaluation.
A held-out or private split exists, which makes contamination easier to detect than on a fully public set.
What the number does not tell you
The most useful part of any benchmark is the fine print. These are the reasons a published score can mislead.
Competitions and their public write-ups have been on the open web for years, so a model may be recalling a winning approach rather than deriving one. The authors discuss the risk and attempt to measure it, but it cannot be eliminated for a benchmark built out of public competitions.
Source
Published scores
Collected from model cards and independent evaluators. The configuration column is the part that decides whether two numbers can be compared at all.
| Model | Score | Configuration | Source |
|---|---|---|---|
| Modelo1-previewOpenAI | Score16.9% | ConfigurationAt least a bronze medal, with AIDE scaffolding; headline result at release. | SourceLab-reported2024-10 |
Ordering is not a ranking. Rows with different configurations are not measuring the same thing — a number produced with tools, extended thinking, or majority voting is not comparable to one produced without.
Active — Still spreads the field.