Multimodal
OCRBench
OCRBench: On the Hidden Mystery of OCR in Large Multimodal Models
A 1000-question OCR stress test scored one point per question.
- Released
- 2023
- Built by
- Huazhong University of Science and Technology and collaborators
- Size
- 1,000 tasks
- Status
- Near ceiling
- Reported by
- Alibaba (Qwen), Shanghai AI Lab
- Judge
- 80
- Deterministic, but it can only grade a final answer and not the work.
- Headroom
- 40
- The top models crowd the ceiling; gaps are shrinking.
- Resolution
- 92
- 1,000 instances, so one item moves the score by 0.100 points.
- Fine print
- 50
- 4 documented caveats, the heaviest being possible training-set contamination.
- Adoption
- 37
- Reported by 2 labs; 4 published scores collected here.
Weighted from facts already on the record — not an editorial rating. A low score means a number from this benchmark needs more context to read, not that the benchmark is bad.
How the evaluation works
Every benchmark on this site is broken onto the same five stages. Select a stage to see what it actually involves.
Stage 3 · What the model can do while measured
Image only, no OCR fed in
Zero-shot and image-only; supplying OCR text would defeat the purpose. Each of the 1,000 items is prompted independently with its short fixed instruction, and everything is scored locally by the released script with no submission server. That makes the benchmark cheap and reproducible, and also makes it easy to run with a different prompt than the official one without anybody noticing.
Single turn, no tools. The most reproducible setup there is.
- Budget
- single turn
Where the tasks came from
Exactly 1,000 items, no train or validation split. One item is worth 1 point of the 1,000.
Sampled from 29 existing OCR datasets, manually filtered and human-corrected; answers shorter than four symbols excluded by construction.
Every task is public, so there is no held-out split to check contamination against.
What the number does not tell you
The most useful part of any benchmark is the fine print. These are the reasons a published score can mislead.
A verbose model that hedges with several candidate readings can hit the gold string by accident, and the only guard is the crude four-symbol minimum-answer filter. In the other direction, a semantically correct answer expressed differently — a different LaTeX rendering, a number written with a comma, a paraphrased field value — scores zero. The metric has no notion of equivalence at all.
Source
Published scores
Collected from model cards and independent evaluators. The configuration column is the part that decides whether two numbers can be compared at all.
| Model | Score | Configuration | Source |
|---|---|---|---|
| ModelInternVL3.5-241B-A28BShanghai AI Lab | Score90.7% | Configurationreported as 907/1000 in the InternVL3.5 paper, Table 4; equals 90.7% accuracy | SourceLab-reported2025-08 |
| ModelGPT-5-20250807OpenAI | Score90.6% | Configurationreported as 906/1000, tabulated third-party in the InternVL3.5 paper; equals 90.6% accuracy | SourceThird-party2025-08 |
| ModelQwen2.5-VL-72BAlibaba Qwen | Score88.5% | Configurationreported as 885/1000 in both the Qwen2.5-VL report and the InternVL3.5 comparison; equals 88.5% accuracy | SourceLab-reported2025-02 |
| ModelClaude 3.5 SonnetAnthropic | Score78.8% | Configurationreported as 788/1000, tabulated third-party in the InternVL3.5 paper; Anthropic does not report OCRBench itself | SourceThird-party2025-08 |
Ordering is not a ranking. Rows with different configurations are not measuring the same thing — a number produced with tools, extended thinking, or majority voting is not comparable to one produced without.
Near ceiling — The top models are close enough to the ceiling to crowd.