Agentic · Computer use · Multimodal
OSWorld
OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments
A real Ubuntu desktop where the agent controls the mouse and keyboard.
- Released
- 2024
- Built by
- XLANG Lab (HKU), CMU, Salesforce Research, University of Waterloo
- Size
- 369 tasks
- Status
- Active
- Reported by
- XLANG Lab, Anthropic, OpenAI, Google DeepMind, ByteDance
- Judge
- 94
- Graded on the final state of the environment, not on prose.
- Headroom
- 82
- Still separates the frontier from everything below it.
- Resolution
- 68
- 369 instances, so one item moves the score by 0.271 points.
- Fine print
- 47
- 4 documented caveats, the heaviest being construct validity.
- Adoption
- 65
- Reported by 5 labs; 0 published scores collected here.
Weighted from facts already on the record — not an editorial rating. A low score means a number from this benchmark needs more context to read, not that the benchmark is bad.
How the evaluation works
Every benchmark on this site is broken onto the same five stages. Select a stage to see what it actually involves.
Stage 3 · What the model can do while measured
Ubuntu VM with a desktop
A real virtual machine running Ubuntu with a full desktop, originally under VMware or VirtualBox and later Docker and AWS. The agent observes screenshots, optionally with the accessibility tree or Set-of-Mark annotations, and acts through pyautogui primitives plus WAIT, FAIL and DONE. The VM is snapshotted and restored between tasks. The step budget is set by the harness rather than the benchmark, and that single knob moves scores by many points.
Pixel-level control. Slow, flaky, and expensive to run at scale.
- Budget
- Set by the harness, not the benchmark; max_steps of 15, 50 and 100 are all in common use.
- Tools exposed
- pyautogui.click(x, y)pyautogui.typewritepyautogui.hotkeypyautogui.scrollWAITFAILDONE
Where the tasks came from
369 tasks; the full release also carries a Windows subset depending on configuration.
Collected from real user workflows, then hand-instrumented with per-task setup scripts and custom Python verifiers.
Every task is public, so there is no held-out split to check contamination against.
What the number does not tell you
The most useful part of any benchmark is the fine print. These are the reasons a published score can mislead.
OSWorld comparisons routinely print numbers run at max_steps of 15, 50 and 100 in the same table as if they were the same measurement. They are not: raising the step cap gives an agent more chances to recover from a bad click, and the gap between a 15-step and a 100-step run is often larger than the gap between two models. Always check the step budget before comparing two OSWorld figures.
Source
Published scores
Collected from model cards and independent evaluators. The configuration column is the part that decides whether two numbers can be compared at all.
No published scores collected for this benchmark yet.
Active — Still spreads the field.