Coding · Agentic · Tool use
Terminal-Bench 2.0 / 2.1
Terminal-Bench 2.0 and 2.1, running on the Harbor framework
Harder audited terminal jobs, run through a rewritten cloud harness called Harbor.
- Released
- 2025
- Built by
- Laude Institute, Stanford University and Snorkel AI
- Size
- 89 tasks
- Status
- Near ceiling
- Reported by
- Anthropic, OpenAI, Google DeepMind, Z.ai, Alibaba (Qwen), Moonshot AI, xAI, Meta, DeepSeek
- Judge
- 94
- Graded on the final state of the environment, not on prose.
- Headroom
- 40
- The top models crowd the ceiling; gaps are shrinking.
- Resolution
- 36
- 89 instances, so one item moves the score by 1.12 points.
- Fine print
- 55
- 4 documented caveats, the heaviest being harness sensitivity.
- Adoption
- 92
- Reported by 9 labs; 5 published scores collected here.
Weighted from facts already on the record — not an editorial rating. A low score means a number from this benchmark needs more context to read, not that the benchmark is bad.
How the evaluation works
Every benchmark on this site is broken onto the same five stages. Select a stage to see what it actually involves.
Stage 3 · What the model can do while measured
One tmux pane, many possible agents
Harbor replaced the bespoke CLI. Terminus 2, the reference agent, still has exactly one tool — an interactive tmux session driven by keystrokes — but Harbor abstracts the sandbox so the same task runs on local Docker, Modal, Daytona, e2b, Blaxel, LangSmith or Novita, and the agent adapter is pluggable. That makes a leaderboard row a model-by-adapter-by-sandbox triple rather than a model score. Real configurations vary enormously and are load-bearing: Z.ai documents Terminus-2 runs at 4-hour timeouts, 48k max output tokens, 500 max episodes and 4 CPUs, while its Claude Code runs proxy around the CLI's 64k output cap and remove wall-clock limits entirely. Internet access is permitted, which caused nine of the 2.1 bug fixes. Most trials finish in under 20 minutes; extreme cases ran two hours and burned nearly 100M tokens on one task.
The model gets a shell. Scaffold quality and step budget move the score independently of the model.
- Budget
- adapter-dependent — up to 500 episodes and multi-hour wall clocks
- Tools exposed
- tmux terminal (Terminus 2)Claude CodeCodex CLIGemini CLICursor CLImini-SWE-agentOpenHands
uv tool install 'harbor[modal]'
uv run harbor run -d terminal-bench/terminal-bench@<tag> -k 5 --agent oracle --n-concurrent 500 --env modal
# and for a model rather than the oracle:
# --agent claude-code --model anthropic/claude-fable-5 --ak reasoning_effort=maxWhere the tasks came from
89 pinned tasks in both 2.0 and 2.1. The rolling `@latest` tag is a different dataset — 74 tasks as of 7 Aug 2026 — so the repo's own headline command does not reproduce the published leaderboard.
229 tasks contributed by 93 authors across 44+ institutions, narrowed to 89 by review, licensed CC BY 4.0.
Every task is public, so there is no held-out split to check contamination against.
What the number does not tell you
The most useful part of any benchmark is the fine print. These are the reasons a published score can mislead.
Terminal-Bench is now distributed as tagged Harbor Hub releases, and `terminal-bench/terminal-bench@latest` held 74 continuously updated tasks as of 7 August 2026 — not the pinned 89-task 2.1 set that the leaderboard and the paper report. Running the command in the repo's own quickstart therefore does not reproduce the published numbers. Any score quoted without a pinned tag is uninterpretable.
Source
Published scores
Collected from model cards and independent evaluators. The configuration column is the part that decides whether two numbers can be compared at all.
| Model | Score | Configuration | Source |
|---|---|---|---|
| ModelGPT-5.6 Sol (xhigh)OpenAI | Score89.5% | ConfigurationTerminus 2 in an e2b sandbox, pass@1 averaged over 3 repeats per task, on the fixed 89-task v2.1 set. | SourceThird-party2026-08-16 |
| ModelClaude Opus 5 (Adaptive Reasoning, Max Effort)Anthropic | Score89.1% | ConfigurationTerminus 2 in an e2b sandbox, pass@1 averaged over 3 repeats per task, fixed 89-task v2.1 set. | SourceThird-party2026-08-16 |
| ModelClaude Code + Fable 5Anthropic | Score83.8% | ConfigurationOfficial tbench 2.1 leaderboard, ±1.2. Model-plus-adapter pair, not a bare model score. | SourceCommunity2026-06-07 |
| ModelCodex + GPT-5.5OpenAI | Score83.1% | ConfigurationOfficial tbench 2.1 leaderboard. | SourceCommunity2026-05-01 |
| ModelTerminus 2 + Fable 5Anthropic | Score80.4% | ConfigurationOfficial tbench 2.1 leaderboard. Same model as the 83.8% row above, different adapter. | SourceCommunity2026-06-05 |
Ordering is not a ranking. Rows with different configurations are not measuring the same thing — a number produced with tools, extended thinking, or majority voting is not comparable to one produced without.
Near ceiling — The top models are close enough to the ceiling to crowd.