Multimodal · Reasoning
MathVista
MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts
Math questions over charts, diagrams and geometry figures, mostly recycled datasets.
- Released
- 2023
- Built by
- UCLA, University of Washington and Microsoft Research
- Size
- 6,141 tasks
- Status
- Saturated
- Judge
- 58
- Mixed grading, so part of the score inherits the weakest judge.
- Headroom
- 8
- Bunched at the ceiling. A win here no longer means much.
- Resolution
- 100
- 6,141 instances, so one item moves the score by 0.016 points.
- Fine print
- 45
- 5 documented caveats, the heaviest being possible training-set contamination.
- Adoption
- 8
- No lab reports it in a model card, so it rarely appears in a comparison.
Weighted from facts already on the record — not an editorial rating. A low score means a number from this benchmark needs more context to read, not that the benchmark is bad.
How the evaluation works
Every benchmark on this site is broken onto the same five stages. Select a stage to see what it actually involves.
Stage 3 · What the model can do while measured
Zero-shot CoT, one image
Zero-shot chain of thought, single turn, one image, no tools in the reference setup. Generation and answer extraction are deliberately separate stages: the model writes free-form CoT prose, and a second pass pulls the final value out of it. The repo also ships a preprocessing path that generates image captions and EasyOCR text so text-only LLMs can be run as a control, which is how the text-only baselines below were measured.
Single turn, no tools. The most reproducible setup there is.
- Budget
- single turn
Where the tasks came from
6,141 examples: testmini 1,000 with public answers, test 5,141 with private labels. Effectively all reporting is on the 1,000-item testmini.
28 existing multimodal datasets plus three new ones (IQTest, FunctionQA, PaperQA).
A held-out or private split exists, which makes contamination easier to detect than on a fully public set.
What the number does not tell you
The most useful part of any benchmark is the fine print. These are the reasons a published score can mislead.
Answer extraction calls an OpenAI model, so two labs running the same model can get different MathVista numbers purely from extraction differences, and the number moves when the extractor model is deprecated or updated. The Levenshtein fallback compounds it: a refusal or a wrong free-text answer can be silently snapped onto a valid option letter and scored as a real attempt.
Source
Published scores
Collected from model cards and independent evaluators. The configuration column is the part that decides whether two numbers can be compared at all.
| Model | Score | Configuration | Source |
|---|---|---|---|
| ModelDreamPRM (o4-mini)Academic | Score85.2% | Configurationtestmini (1,000 items). Top of the official leaderboard, whose most recent dated entry anywhere is 2025-06-04. | SourceCommunity2025-06-04 |
| ModelBlueLM-2.6-3Bvivo | Score82.3% | Configurationtestmini, same OpenVLM snapshot — a 3B model above GPT-5 on the same board | SourceThird-party2025-09-17 |
| ModelGPT-5-20250807OpenAI | Score81.9% | Configurationtestmini, OpenCompass OpenVLM leaderboard data file, snapshot 20250917132916 | SourceThird-party2025-09-17 |
Ordering is not a ranking. Rows with different configurations are not measuring the same thing — a number produced with tools, extended thinking, or majority voting is not comparable to one produced without.
Saturated — Scores are bunched at the ceiling; it no longer separates models.