Agentic · Tool use
tau-bench
τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains
Can an agent follow company policy while a simulated customer keeps talking?
- Released
- 2024
- Built by
- Sierra AI (with Princeton)
- Size
- 165 tasks
- Status
- Active
- Reported by
- Sierra AI, OpenAI, Anthropic, Google DeepMind
- Judge
- 94
- Graded on the final state of the environment, not on prose.
- Headroom
- 82
- Still separates the frontier from everything below it.
- Resolution
- 52
- 165 instances, so one item moves the score by 0.606 points.
- Fine print
- 48
- 4 documented caveats, the heaviest being construct validity.
- Adoption
- 52
- Reported by 4 labs; 0 published scores collected here.
Weighted from facts already on the record — not an editorial rating. A low score means a number from this benchmark needs more context to read, not that the benchmark is bad.
How the evaluation works
Every benchmark on this site is broken onto the same five stages. Select a stage to see what it actually involves.
Stage 3 · What the model can do while measured
Mock company backend
There is no internet and no GUI. A JSON/SQL-like mock database of users, orders, reservations and products sits behind a fixed set of Python API functions. The customer is a second LLM prompted with a persona the agent never sees, replying in natural language turn by turn. Read tools and write tools are both exposed, so the agent can take irreversible wrong actions — cancelling the wrong order is a real, scored outcome rather than a hypothetical.
Another model plays the user, so its behaviour is part of the measurement.
- Budget
- Roughly 30 agent steps by default; an episode also ends when the agent stops or the user says a stop phrase.
- Tools exposed
- get_user_detailsget_ordercancel_orderbook_reservationtransfer_to_human
Where the tasks came from
115 τ-retail tasks (4 databases, 15 tools) and 50 τ-airline tasks (3 databases, 13 tools).
Hand-authored by the Sierra team; policy documents adapted from real public airline and retail policy pages.
Every task is public, so there is no held-out split to check contamination against.
What the number does not tell you
The most useful part of any benchmark is the fine print. These are the reasons a published score can mislead.
The user simulator can be inconsistent or unrealistically compliant, and later work shows that swapping the simulator model moves scores by several points. Two labs reporting τ-bench with different simulator models are not measuring the same thing, and the simulator is almost never stated alongside the number.
Source
Published scores
Collected from model cards and independent evaluators. The configuration column is the part that decides whether two numbers can be compared at all.
No published scores collected for this benchmark yet.
Active — Still spreads the field.