Safety · Tool use · Agentic
AgentDojo
AgentDojo — a dynamic environment for prompt-injection attacks and defences
Realistic tool-calling tasks with attacker instructions hidden in the data.
- Released
- 2024
- Built by
- ETH Zurich (SPY Lab)
- Size
- 629 tasks
- Status
- Near ceiling
- Reported by
- Meta
- Judge
- 94
- Graded on the final state of the environment, not on prose.
- Headroom
- 40
- The top models crowd the ceiling; gaps are shrinking.
- Resolution
- 82
- 629 instances, so one item moves the score by 0.159 points.
- Fine print
- 45
- 5 documented caveats, the heaviest being construct validity.
- Adoption
- 35
- Reported by 1 lab; 16 published scores collected here.
Weighted from facts already on the record — not an editorial rating. A low score means a number from this benchmark needs more context to read, not that the benchmark is bad.
How the evaluation works
Every benchmark on this site is broken onto the same five stages. Select a stage to see what it actually involves.
Stage 3 · What the model can do while measured
74 mock tools over stateful Python objects
No container and no network: the tools are Python functions that read and write in-memory environment state, with outputs serialised to YAML and fed back to the model. The paper's body counts 74 tools while its own Table 1 says 70. The attack surface is a set of named injection endpoints inside that state, so an attack is a string substitution rather than anything the model can detect out of band. Because the environment is code, the same run can be replayed under a different attack or a different defence with everything else held fixed — that is the whole design.
Another model plays the user, so its behaviour is part of the measurement.
- Budget
- No fixed step budget; the loop runs until the model stops calling tools. Meta ran it at a 16K context window.
- Tools exposed
- emailcalendarcloud driveSlack messagingweb page readerbanking transactionstravel booking
Where the tasks came from
629 security test cases built from 97 user tasks and 27 injection targets; a single case is worth 0.16%. The suites are unequal — Workspace alone is 40 of the 97 user tasks, Banking contributes 9 of the 27 injection targets.
Entirely hand-authored by the ETH Zurich SPY Lab; synthetic environments with fictional users, no real data.
Every task is public, so there is no held-out split to check contamination against.
What the number does not tell you
The most useful part of any benchmark is the fine print. These are the reasons a published score can mislead.
The results page carries the disclaimer: 'Please note that this is not a leaderboard because we did not attack all models with all attacks and defenses, so it would be too difficult to have a fair comparison across models/defenses/attacks.' On top of that the task suites are versioned — v1, v1.1, v1.1.1, v1.1.2, v1.2, v1.2.1 and v1.2.2 all ship in the package, each fixing or replacing tasks — and the published rows do not record which version produced them. Two AgentDojo numbers are comparable only if the model, the attack, the defence and the suite version all match.
Source
Published scores
Collected from model cards and independent evaluators. The configuration column is the part that decides whether two numbers can be compared at all.
| Model | Score | Configuration | Source |
|---|---|---|---|
| ModelGPT-5 (high reasoning)OpenAI | Score0.2% | ConfigurationMeasured by Meta for the Meta SecAlign paper, 16K context window. Not undefended: the paper states it runs every AgentDojo evaluation with the benchmark's provided repeat_user_prompt (sandwich) defence. Its Table 10 puts the genuinely undefended figure at 0.2% ASR on 83.5% utility. OpenAI has never published an AgentDojo figure of its own. | SourceThird-party2026-02-06 |
| ModelMeta-SecAlign-70BMeta AI | Score1.9% | ConfigurationMeta's own defended model, trained with a separate untrusted-data message role and never shown injected prompts during training. The Llama-3.3-70B-Instruct baseline under the same sandwich defence is 14.7%; undefended, the paper's Table 10 puts that baseline at 23.0% and this model at 2.3%. | SourceLab-reported2026-02-06 |
| ModelGemini 3 ProGoogle DeepMind | Score2.3% | ConfigurationMeta SecAlign paper, v3 revision, 16K context window, run with the benchmark's repeat_user_prompt (sandwich) defence like every other AgentDojo number in that paper; Table 10 gives 3.8% undefended. Google's own Gemini 3 materials claim increased resistance to prompt injection without publishing a number. | SourceThird-party2026-02-06 |
| Modelgpt-4o-2024-05-13OpenAI | Score3.66% | Configurationdirect attack, no defence. Same model, same day, same suite as the 47.69% row — only the attack string differs. | SourceThird-party2024-06-05 |
| Modelgpt-4o-2024-05-13 + tool_filterOpenAI | Score6.84% | Configurationimportant_instructions attack with the tool-filter defence, which raised benign utility to 72.16% as well. The most effective defence in the paper. | SourceThird-party2024-06-05 |
| Modelclaude-3-7-sonnet-20250219Anthropic | Score7.31% | Configurationimportant_instructions attack, no defence. | SourceThird-party2025-02-24 |
| ModelGrok 4.20xAI | Score40% | ConfigurationSame audit. The most injection-resistant of the twelve models measured, and simultaneously the most harmful on AgentHarm at 0.74 — the ranking reversal the audit was built to surface (Kendall's W = 0.10, p = 0.94 across four benchmarks). | SourceThird-party2026-04-11 |
| Modelgpt-4o-2024-05-13 + spotlighting_with_delimitingOpenAI | Score41.65% | Configurationimportant_instructions attack. A published defence that removes only six points of attack success. | SourceThird-party2024-06-05 |
| Modelgpt-4o-2024-05-13OpenAI | Score47.69% | Configurationimportant_instructions attack, no defence. The paper's headline attack figure. | SourceThird-party2024-06-05 |
| ModelGPT-5.4OpenAI | Score100% | ConfigurationThird-party audit, Inspect wrapper over OpenRouter, 50 injection samples from each of five workspaces macro-averaged, n=250, 95% Wilson interval [.98, 1.00]. Irreconcilable with Meta's 0.2% for GPT-5 and with the 7.31% the official harness recorded for Claude 3.7 Sonnet; included because it is a published measurement, not because it is credible in isolation. | SourceThird-party2026-04-11 |
| ModelClaude Opus 4.6Anthropic | Score100% | ConfigurationSame audit and same configuration as the GPT-5.4 row. The audit's own reading is that this reflects genuine vulnerability rather than a measurement ceiling; the same model is the least misaligned of the twelve on ODCV-Bench. | SourceThird-party2026-04-11 |
| Model | Score | Configuration | Source |
|---|---|---|---|
| ModelGemini 3 ProGoogle DeepMind | Score92.8% | ConfigurationHighest benign utility recorded on AgentDojo in any source checked. Utility under attack 90.6%. | SourceThird-party2026-02-06 |
| Modelclaude-3-7-sonnet-20250219Anthropic | Score88.66% | ConfigurationOfficial results table, no defence. The newest entry the benchmark's own harness has recorded. | SourceThird-party2025-02-24 |
| ModelMeta-SecAlign-70BMeta AI | Score84.5% | ConfigurationAbove the Llama-3.3-70B-Instruct baseline of 59.8% measured the same way, which is the unusual part — most AgentDojo defences buy security by giving up utility. | SourceLab-reported2026-02-06 |
| ModelGPT-5 (high reasoning)OpenAI | Score80.3% | ConfigurationSame run as the 0.2% ASR figure. Utility under attack is 79.7% — a 0.6-point drop, against the 10-25 point drops the original paper reported for 2024 models. | SourceThird-party2026-02-06 |
| Model | Score | Configuration | Source |
|---|---|---|---|
| Modelclaude-3-7-sonnet-20250219Anthropic | Score77.27% | Configurationimportant_instructions attack, no defence. An 11.4-point drop from benign utility. | SourceThird-party2025-02-24 |
Each table above measures a different quantity, so rows are only ever ordered within their own table. A model can appear more than once, and its numbers are not meant to be added or averaged.
At least one table here is lower is better — a harm score, lying rate, or attack success rate. The best result is at the top of every table, which means the smallest number leads where the direction is inverted.
Ordering is not a ranking. Rows with different configurations are not measuring the same thing — a number produced with tools, extended thinking, or majority voting is not comparable to one produced without.
Near ceiling — The top models are close enough to the ceiling to crowd.