Coding · Agentic
DeepSWE
DeepSWE — 113 original long-horizon engineering tasks, authored rather than mined
Tasks written from scratch, never merged upstream, graded by hand-written verifiers.
- Released
- 2026
- Built by
- Datacurve
- Size
- 113 tasks
- Status
- Active
- Judge
- 100
- Graded by the project's own tests. Deterministic, inspectable.
- Headroom
- 82
- Still separates the frontier from everything below it.
- Resolution
- 52
- 113 instances, so one item moves the score by 0.885 points.
- Fine print
- 45
- 5 documented caveats, the heaviest being possible training-set contamination.
- Adoption
- 22
- No lab reports it in a model card, so it rarely appears in a comparison.
Weighted from facts already on the record — not an editorial rating. A low score means a number from this benchmark needs more context to read, not that the benchmark is bad.
How the evaluation works
Every benchmark on this site is broken onto the same five stages. Select a stage to see what it actually involves.
Stage 3 · What the model can do while measured
mini-swe-agent, pinned, offline
Every configuration runs under mini-swe-agent at a pinned commit with its stock system prompt and agent loop — one bash tool, one shared prompt, no per-vendor editing primitives such as GPT's apply_patch or Claude's str_replace_based_edit_tool. The container holds a shallow clone at the base commit only, so unlike SWE-Bench Pro's containers there is no .git history in the workspace from which the gold fix could be recovered. The only limit is a 9,000-second wall clock set as a sanity bound; there is no step cap and no cost cap, and only 67 of 7,174 scored rollouts reached the timeout. Shipped task files declare their own per-task budgets and set the agent and the verifier to no-network.
The model gets a shell. Scaffold quality and step budget move the score independently of the model.
- Budget
- 9,000 seconds of wall clock, no step cap and no cost cap
- Tools exposed
- bash
[metadata]
task_id = "happy-dom-abort-pending-body-reads"
display_title = "Abort pending body reads on shutdown"
category = "bugfix"
language = "typescript"
repository_url = "https://github.com/capricorn86/happy-dom"
base_commit_hash = "82a0888cb2c87a6123e05424b528f8e8c9b3e426"
[verifier]
network_mode = "no-network"
environment_mode = "separate"
timeout_sec = 1800.0
[agent]
network_mode = "no-network"
timeout_sec = 5400.0
[environment]
docker_image = "public.ecr.aws/d3j8x8q7/swe-bench-202605:kh7c2re7cvbseq7xz6samd1xr182y1dc-v1.1"
cpus = 2
memory_mb = 8192Where the tasks came from
113 tasks over 91 repositories, so one task is worth about 0.9 points and the median repository contributes a single task. Five languages: TypeScript, Go, Python, JavaScript, Rust.
Written from scratch by Datacurve authors against repositories with 500+ stars and permissive licences, each pinned to an immutable commit. Prompts, verifiers, reference solutions and the full trajectory record are released; the LLM-judge prompt used for the auxiliary audit is the one component held back.
Every task is public, so there is no held-out split to check contamination against.
What the number does not tell you
The most useful part of any benchmark is the fine print. These are the reasons a published score can mislead.
Agentica and Together AI released DeepSWE-Preview in 2025, a 32B coding agent RL-trained on Qwen3-32B that reports 42.2% pass@1 and 59% with test-time scaling on SWE-bench Verified. That is a model, not a benchmark, and its numbers are measurements of SWE-bench rather than of anything in this record. Separately, the benchmark's own numbers moved between versions: the paper and the launch blog report GPT-5.5 at 70% from May 2026 runs, while the v1.1 leaderboard dated 13 August 2026 — which moved grading into separate verifier environments — lists gpt-5.5 at 67%. The live board also prints no reasoning-effort column, so a row there cannot be matched to a paper row with confidence.
Source
Published scores
Collected from model cards and independent evaluators. The configuration column is the part that decides whether two numbers can be compared at all.
| Model | Score | Configuration | Source |
|---|---|---|---|
| ModelClaude Opus 5Anthropic | Score74% | Configurationv1.1 leaderboard, ±4. Median $11.84 per trial, 118k output tokens, 99 steps. Reasoning effort is not printed on the leaderboard. | SourceThird-party2026-08-13 |
| ModelGPT-5.6 SolOpenAI | Score73% | Configurationv1.1 leaderboard, ±3. Median $8.39 per trial, 60k output tokens, 61 steps — a third of the tokens and two thirds of the steps of the model above it. | SourceThird-party2026-08-13 |
| ModelClaude Fable 5Anthropic | Score70% | Configurationv1.1 leaderboard, ±4. Median $21.63 per trial, second only to Claude Sonnet 5 at $26.40. | SourceThird-party2026-08-13 |
| ModelGPT-5.5 (xhigh)OpenAI | Score70% | ConfigurationPaper leaderboard, runs collected May 2026, 95% run-to-run interval [67.2, 72.9]; the Wilson interval on pooled attempts is [65.6, 74.1]. The v1.1 board lists gpt-5.5 at 67%. | SourceThird-party2026-07-08 |
| ModelKimi K3Moonshot AI | Score69% | Configurationv1.1 leaderboard, ±5. Median $4.65 per trial, 98 steps. | SourceThird-party2026-08-13 |
| ModelGPT-5.6 LunaOpenAI | Score67% | Configurationv1.1 leaderboard, ±4. Median $0.61 per trial against $11.84 for the top entry, which is the paper's point that cost does not track pass rate. | SourceThird-party2026-08-13 |
| ModelGemini 3.7 FlashGoogle DeepMind | Score65% | Configurationv1.1 leaderboard, ±2. No Gemini Pro configuration appears among the 17 rows the leaderboard shows. | SourceThird-party2026-08-13 |
| ModelDeepSeek-V4 ProDeepSeek | Score63% | Configurationv1.1 leaderboard, ±6. Median $0.24 per trial, 155 steps. | SourceThird-party2026-08-13 |
| ModelClaude Opus 4.8Anthropic | Score59% | Configurationv1.1 leaderboard, ±2. Median $13.22 per trial. | SourceThird-party2026-08-13 |
| ModelGPT-5.4 (xhigh)OpenAI | Score55.5% | ConfigurationPaper leaderboard, [53.4, 57.7]. Overlaps Claude Opus 4.7 and is therefore not separated from it. | SourceThird-party2026-07-08 |
| ModelClaude Opus 4.7 (max)Anthropic | Score54.2% | ConfigurationPaper leaderboard, [49.5, 58.9]. In the audited sample this family missed enumerated requirements more often than any other — typically implementing one branch of a 'support both X and Y' prompt and not mirroring it. | SourceThird-party2026-07-08 |
Ordering is not a ranking. Rows with different configurations are not measuring the same thing — a number produced with tools, extended thinking, or majority voting is not comparable to one produced without.
Active — Still spreads the field.