Long context · Reasoning
RULER
RULER — What's the Real Context Size of Your Long-Context Language Models?
Synthetic long-context tasks that report the length where a model stops working.
- Released
- 2024
- Built by
- NVIDIA
- Size
- Not published
- Status
- Saturated
- Reported by
- NVIDIA, AI21 Labs, Alibaba (Qwen), LG AI Research
- Judge
- 80
- Deterministic, but it can only grade a final answer and not the work.
- Headroom
- 8
- Bunched at the ceiling. A win here no longer means much.
- Resolution
- 50
- Instance count is not published, so per-item weight is unknown.
- Fine print
- 45
- 5 documented caveats, the heaviest being construct validity.
- Adoption
- 74
- Reported by 4 labs; 9 published scores collected here.
Weighted from facts already on the record — not an editorial rating. A low score means a number from this benchmark needs more context to read, not that the benchmark is bad.
How the evaluation works
Every benchmark on this site is broken onto the same five stages. Select a stage to see what it actually involves.
Stage 3 · What the model can do while measured
One prompt, greedy decoding
A single completion per example, no tools and no agent loop, run through vLLM or TensorRT-LLM in BFloat16 with greedy decoding on 8 A100s in the original evaluation. The output budget is deliberately tiny — 128 tokens for retrieval, 30 for variable tracking — and every prompt ends with an answer prefix the model continues, which stops it refusing or padding with explanation. The one substantive knob is the tokenizer: data is generated per model so that 128K tokens means 128K of that model's tokens, which means no two models see literally the same haystack.
Single turn, no tools. The most reproducible setup there is.
- Budget
- single turn; inputs of 4K, 8K, 16K, 32K, 64K and 128K tokens
Where the tasks came from
Nothing is fixed on disk. 500 examples per task per length across 13 tasks and six lengths — 39,000 items — regenerated from scratch for each model's tokenizer.
Fully synthetic, generated by scripts in the repository under Apache-2.0; the QA category draws its passages from SQuAD and HotpotQA.
Every task is public, so there is no held-out split to check contamination against.
What the number does not tell you
The most useful part of any benchmark is the fine print. These are the reasons a published score can mislead.
85.6% is what Llama-2-7B-chat averaged at 4K tokens. There is no claim it marks the boundary of usefulness for any downstream task; the paper calls it a fixed reference point, and the value is fixed in the sense that it never changes, not in the sense that it was derived from anything. Because it is applied to a ladder of six tested lengths, an effective length is also quantised: "64K" means passed at 64K and failed at 128K, and the true crossing point could be anywhere between. Two models can share an effective length and differ by 15 points at 128K — Llama3.1-70B and GPT-4 both read 64K, at 66.6 and 81.2 respectively.
Source
Published scores
Collected from model cards and independent evaluators. The configuration column is the part that decides whether two numbers can be compared at all.
| Model | Score | Configuration | Source |
|---|---|---|---|
| ModelJamba-1.5-Large (94B/398B)AI21 Labs | Score96% | ConfigurationClaimed 256K, effective >128K, 95.1 at 128K. Reported by AI21 in their own technical report, not run by NVIDIA; carried into the repository table under the asterisk convention. | SourceLab-reported2024-08-22 |
| ModelGemini-1.5-ProGoogle DeepMind | Score95.8% | ConfigurationClaimed 1M, effective >128K. 94.4 at 128K. Run by NVIDIA through the Gemini API; the only closed model besides GPT-4 in the table, and the 1M claim is untested above 128K. | SourceThird-party2024-08-06 |
| ModelQwen2.5-14B-Instruct-1MAlibaba (Qwen) | Score95.7% | ConfigurationClaimed 1M, effective >128K, 92.2 at 128K. Self-reported in the Qwen2.5-1M technical report; wAvg columns are TBD in the repository table because NVIDIA never recomputed them. | SourceLab-reported2025-01-26 |
| ModelQwen3-235B-A22BAlibaba (Qwen) | Score95% | ConfigurationClaimed 128K, effective >128K, 90.6 at 128K. Self-reported in the Qwen3 technical report. The most recent frontier-scale entry in the table. | SourceLab-reported2025-05-14 |
| ModelGPT-4 (gpt-4-1106-preview)OpenAI | Score91.6% | ConfigurationClaimed 128K, effective 64K. 96.6 at 4K falling to 81.2 at 128K — the least degradation of any model in the original paper, and still a 15.4-point drop across the window it advertises. | SourceThird-party2024-08-06 |
| ModelGLM-4-9B-chat-1MZhipu AI | Score89.9% | ConfigurationClaimed 1M, effective 64K, 83.1 at 128K. Run by NVIDIA. Ranked 3rd on wAvg (inc) at the time, well ahead of far larger models — the paper's point that training context window size does not determine long-context quality. | SourceThird-party2024-08-06 |
| ModelLlama3.1-70B-InstructMeta | Score89.6% | ConfigurationClaimed 128K, effective 64K. 94.8 at 32K, 88.4 at 64K, 66.6 at 128K. Run by NVIDIA. | SourceThird-party2024-08-06 |
| ModelLWM-Text-Chat-1M (7B)LargeWorldModel | Score72.8% | ConfigurationClaimed 1M, effective <4K. It never clears the threshold at any tested length, starting at 82.3 at 4K. The widest claimed-to-effective gap in the table and the single clearest argument for the benchmark existing. | SourceThird-party2024-08-06 |
| Model | Score | Configuration | Source |
|---|---|---|---|
| ModelLlama-2-7B-chatMeta | Score85.6% | ConfigurationNot a leaderboard entry — this is the reference value. The threshold defining every effective context length in the table is this model's average at 4K, its own claimed limit. | SourceThird-party2024-08-06 |
Each table above measures a different quantity, so rows are only ever ordered within their own table. A model can appear more than once, and its numbers are not meant to be added or averaged.
Ordering is not a ranking. Rows with different configurations are not measuring the same thing — a number produced with tools, extended thinking, or majority voting is not comparable to one produced without.
Saturated — Scores are bunched at the ceiling; it no longer separates models.