Safety · Agentic · Coding
CyberGym
CyberGym — 1,507 OSS-Fuzz vulnerability reproduction tasks at scale
Given an unpatched repository and a bug description, write an input that crashes it.
- Released
- 2025
- Built by
- UC Berkeley (Sunblaze group / Berkeley RDI, Dawn Song)
- Size
- 1,507 tasks
- Status
- Near ceiling
- Reported by
- Anthropic, OpenAI, Z.ai, DeepSeek, Meta, Moonshot AI, Microsoft
- Judge
- 94
- Graded on the final state of the environment, not on prose.
- Headroom
- 40
- The top models crowd the ceiling; gaps are shrinking.
- Resolution
- 92
- 1,507 instances, so one item moves the score by 0.066 points.
- Fine print
- 45
- 5 documented caveats, the heaviest being possible training-set contamination.
- Adoption
- 100
- Reported by 7 labs; 21 published scores collected here.
Weighted from facts already on the record — not an editorial rating. A low score means a number from this benchmark needs more context to read, not that the benchmark is bad.
How the evaluation works
Every benchmark on this site is broken onto the same five stages. Select a stage to see what it actually involves.
Stage 3 · What the model can do while measured
Local Docker, and the patched build is hidden
Agents run in containers against a local submission server that holds both the vulnerable and the fixed images; the agent is given only repo-vul.tar.gz and a submit.sh endpoint, and the -fix image is never reachable from the agent. The harness ships a Squid domain-allowlist firewall because the FAQ treats network access as a live reward-hacking vector — an agent that can reach the project's issue tracker, changelog or commit history can read the answer instead of deriving it. Teams that hand the agent a runnable vulnerable image for dynamic analysis are told to delete /src/**/.git and /tmp/poc first, since those contain the git history and the reference PoC. The full server data is roughly 10TB, or 130GB in binary-only mode.
The model gets a shell. Scaffold quality and step budget move the score independently of the model.
- Budget
- 100 agent iterations in the paper's OpenHands configuration; roughly $2.00 per task in the agent-comparison experiment
- Tools exposed
- bashpythonfile read/writePoC submission endpoint
Where the tasks came from
1,507 instances across 188 projects; a randomly selected 300-instance subset (~20%) is published for cheaper runs. 65.7% of instances have a ground-truth PoC longer than 100 bytes, and agents score around 10% on that majority.
Historical vulnerabilities found by Google's OSS-Fuzz, imported via the ARVO Docker corpus plus 139 later bugs collected directly, filtered by a quality-assurance pipeline.
Every task is public, so there is no held-out split to check contamination against.
What the number does not tell you
The most useful part of any benchmark is the fine print. These are the reasons a published score can mislead.
DeepSeek-V4-Flash sits on the official leaderboard three times: 76.7% under DeepSeek's own agent, 91.2% under Fudan's Whitzard, and 93.17% under Sangfor AI. GLM-5.2 appears at 77.2% in Z.ai's own documentation and at 90.84% under DARKNAVY's DoGNAVY agent. The leaderboard tags rows 'dynamic' when the agent is handed a sanitised vulnerable Docker image and 'test-time mem.' when it carries a knowledge base across instances, and states that these 'denote different evaluation strategies, rather than a reduction in task difficulty' — but they are not held constant, so a CyberGym number without its scaffold, its network policy and whether it had a runnable binary is close to meaningless.
Source
Published scores
Collected from model cards and independent evaluators. The configuration column is the part that decides whether two numbers can be compared at all.
| Model | Score | Configuration | Source |
|---|---|---|---|
| ModelSangfor AI agent + DeepSeek-V4-FlashSangfor | Score93.17% | ConfigurationLevel 1, 1 trial, orchestration and multi-stage agent, dynamic: run against a sanitised vulnerable Docker image. Top of the official leaderboard. | SourceCommunity2026-08-08 |
| ModelWhitzard agent + DeepSeek-V4-FlashFudan Whitzard | Score91.2% | ConfigurationLevel 1, 1 trial, dynamic. The same backbone model scores 76.7% under DeepSeek's own agent. | SourceCommunity2026-08-07 |
| ModelMDASH (GPT-5.4 + Claude Opus 4.6 + Claude Sonnet 4.6)Microsoft | Score90.97% | ConfigurationLevel 1, 1 trial, multi-model orchestration. Microsoft's earlier submission on the same stack was 88.45% on 2026-05-12. | SourceLab-reported2026-06-17 |
| ModelDoGNAVY + GLM-5.2DARKNAVY | Score90.84% | ConfigurationLevel 1, 1 trial, multi-agent with memory, dynamic. Z.ai's own figure for GLM-5.2 is 77.2%. | SourceCommunity2026-08-03 |
| ModelGPT-5.5-CyberOpenAI | Score85.6% | ConfigurationLevel 1, 1 trial, OpenAI's own agent. Reported alongside the restricted-release Daybreak cyber model announcement. | SourceLab-reported2026-06-22 |
| ModelGLM-5.3 (max)Z.ai (Zhipu) | Score84.5% | ConfigurationZ.ai's own documentation, which calls this 'the best result on the benchmark'. Scaffold, trials and network policy are not stated. Not on the official leaderboard at the time of writing. | SourceLab-reported2026-08-14 |
| ModelClaude Mythos 5Anthropic | Score83.8% | ConfigurationReported in Z.ai's GLM-5.3 comparison table, and independently by Anthropic — the Claude Fable 5 and Mythos 5 system card, section 3.2.3, states that Mythos 5 reproduced the same 83.8%. One of the few cross-lab measurements on this board that both parties agree on. | SourceThird-party2026-08-14 |
| ModelGPT-5.6 SolOpenAI | Score83.6% | ConfigurationAlso from Z.ai's GLM-5.3 table. OpenAI has published no CyberGym figure for this model. | SourceThird-party2026-08-14 |
| ModelClaude Mythos PreviewAnthropic | Score83.1% | ConfigurationLevel 1, 1 trial, Anthropic's internal agent, from the Claude Mythos Preview system card. | SourceLab-reported2026-04-07 |
| ModelGPT-5.5OpenAI | Score81.8% | ConfigurationLevel 1, 1 trial, OpenAI's own agent. | SourceLab-reported2026-04-23 |
| ModelGPT-5.4OpenAI | Score79% | ConfigurationOpenAI's own agent, 1 trial. Z.ai measured the same model at 66.3% through Codex CLI — see the row below. | SourceLab-reported2026-04-23 |
| ModelGLM-5.2Z.ai (Zhipu) | Score77.2% | ConfigurationPublished for the first time in the GLM-5.3 documentation, months after GLM-5.2 shipped; the GLM-5.2 model page itself carries no CyberGym figure and Z.ai made no leaderboard submission for it. | SourceLab-reported2026-08-14 |
| ModelGLM-5.1 (Claude Code)Z.ai (Zhipu) | Score68.7% | ConfigurationZ.ai's own submission, run through Claude Code as the scaffold. Level 1, 1 trial. | SourceLab-reported2026-04-07 |
| ModelClaude Sonnet 4.5 (30 trials)Anthropic | Score66.7% | ConfigurationThirty trials per instance, solved if any trial succeeds. The same system card also reports 28.9% at one trial — both rows appear separately on the official leaderboard. | SourceLab-reported2025-09-29 |
| ModelClaude Opus 4.6Anthropic | Score66.6% | ConfigurationLevel 1, 1 trial, from the Claude Opus 4.6 system card. Claude Sonnet 4.6 is reported at 65.2%. | SourceLab-reported2026-02-05 |
| ModelGPT-5.4 (Codex CLI)OpenAI | Score66.3% | ConfigurationMeasured by Z.ai as a baseline for GLM-5.1, running Codex CLI. 12.7 points below OpenAI's own figure for the same model. | SourceThird-party2026-04-07 |
| ModelOpenHands + GPT-5OpenAI | Score39.4% | ConfigurationFull 1,507 instances at level 1, 1 trial, run by the CyberGym team. Their own last posted run of the reference OpenHands scaffold. | SourceThird-party2025-12-05 |
| ModelGemini 3.1 Pro (Gemini CLI)Google DeepMind | Score38.8% | ConfigurationMeasured by Z.ai through Gemini CLI as a GLM-5.1 baseline. Google has published no CyberGym figure of its own. | SourceThird-party2026-04-07 |
| ModelClaude Sonnet 4.5 (1 trial)Anthropic | Score28.9% | ConfigurationOne trial per instance, same model and same system card as the 66.7% figure above. | SourceLab-reported2025-09-29 |
| ModelOpenHands + Claude-Sonnet-4 (non-thinking)Anthropic | Score17.9% | ConfigurationThe original paper's best result: OpenHands, 100 iterations, thinking disabled, full 1,507 instances at level 1. Included as the 2025 baseline the current 90%+ entries should be read against. | SourceThird-party2025-05-23 |
| Model | Score | Configuration | Source |
|---|---|---|---|
| ModelOpenHands + GPT-5 (high reasoning)OpenAI | Score22% | ConfigurationThinking-mode ablation on the randomly selected 300-instance subset, not the full benchmark. The same model at minimal reasoning scores 7.7% — a 14-point swing from the effort setting alone. Dated to the arXiv v3 revision that carries the figure. | SourceThird-party2026-03-24 |
Each table above measures a different quantity, so rows are only ever ordered within their own table. A model can appear more than once, and its numbers are not meant to be added or averaged.
Ordering is not a ranking. Rows with different configurations are not measuring the same thing — a number produced with tools, extended thinking, or majority voting is not comparable to one produced without.
Near ceiling — The top models are close enough to the ceiling to crowd.