Multimodal
DocVQA
DocVQA: A Dataset for VQA on Document Images
Reading answers off scanned business documents: forms, tables, letters, reports.
- Released
- 2020
- Built by
- Computer Vision Center, Universitat Autònoma de Barcelona (host)
- Size
- 50,000 tasks
- Status
- Saturated
- Reported by
- Alibaba (Qwen), Shanghai AI Lab
- Judge
- 88
- Symbolic equivalence, so formatting differences do not change the score.
- Headroom
- 8
- Bunched at the ceiling. A win here no longer means much.
- Resolution
- 100
- 50,000 instances, so one item moves the score by 0.002 points.
- Fine print
- 52
- 4 documented caveats, the heaviest being construct validity.
- Adoption
- 37
- Reported by 2 labs; 4 published scores collected here.
Weighted from facts already on the record — not an editorial rating. A low score means a number from this benchmark needs more context to read, not that the benchmark is bad.
How the evaluation works
Every benchmark on this site is broken onto the same five stages. Select a stage to see what it actually involves.
Stage 3 · What the model can do while measured
Pixels only, no OCR
The original challenge was OCR-assisted: baselines consumed OCR transcriptions and token bounding boxes alongside the image. Modern LMM evaluation is zero-shot and image-only, with the answer produced end-to-end from pixels, which makes old and new numbers on the same benchmark not directly comparable. Test answers are held out and scored by submission to the RRC server, though many papers self-report using the public evaluation script instead.
Single turn, no tools. The most reproducible setup there is.
- Budget
- single turn
Where the tasks came from
50,000 questions over 12,767 images: train 39,463 / 10,194, validation 5,349 / 1,286, test 5,188 / 1,287.
UCSF Industry Documents Library scans, 1900-2018, five industries; questions crowd-annotated and typed.
A held-out or private split exists, which makes contamination easier to detect than on a fully public set.
What the number does not tell you
The most useful part of any benchmark is the fine print. These are the reasons a published score can mislead.
A prediction just under the threshold can score around 0.6 while a marginally worse one scores exactly 0. Edit distance also rewards string overlap rather than semantic correctness: a wrong number one digit off scores high, while a correct answer phrased differently — 'March 13, 1997' against a gold '13 March 1997' — is penalised. The max-over-gold-answers term means scores partly reflect how many answer variants annotators happened to record.
Source
Published scores
Collected from model cards and independent evaluators. The configuration column is the part that decides whether two numbers can be compared at all.
| Model | Score | Configuration | Source |
|---|---|---|---|
| ModelQwen2.5-VL-72BAlibaba Qwen | Score96.4% | ConfigurationDocVQA test split, ANLS; Qwen2.5-VL Technical Report | SourceLab-reported2025-02 |
| ModelGPT-5-20250807OpenAI | Score95.4% | ConfigurationDocVQA test split, ANLS; tabulated third-party in the InternVL3.5 paper, not self-reported by OpenAI | SourceThird-party2025-08 |
| ModelClaude 3.5 SonnetAnthropic | Score95.2% | ConfigurationDocVQA (test, ANLS score) | SourceLab-reported2024-06 |
| ModelInternVL3.5-241B-A28BShanghai AI Lab | Score94.9% | ConfigurationDocVQA test split, ANLS; InternVL3.5 paper, Table 4 | SourceLab-reported2025-08 |
Ordering is not a ranking. Rows with different configurations are not measuring the same thing — a number produced with tools, extended thinking, or majority voting is not comparable to one produced without.
Saturated — Scores are bunched at the ceiling; it no longer separates models.