OSWorld
In plain wordsA test in which AI agents must use real desktop computers — clicking, typing and switching between apps such as spreadsheets, browsers and file managers — to finish everyday office tasks.
A benchmark of 369 open-ended computer tasks carried out in real operating systems running in virtual machines, covering web browsing, office documents, image editing, coding tools, email and file management; many require switching between programs. An agent observes screenshots, the accessibility tree or both, and acts through mouse and keyboard commands; each task has a custom script that checks the resulting system state, so any successful sequence of actions counts. The best multimodal model at release succeeded on 12.24% of tasks, against 72.36% for human testers, with most failures traced to GUI grounding — locating the right element on screen — and to not knowing how particular applications work. OSWorld became the main yardstick for computer usecomputer use, n.: An AI's ability to work a computer through its normal screen and controls: it studies pictures of what is displayed and sends mouse and keyboard actions, so it can use ordinary apps that were built for people. and GUI agentsGUI agent, n.: An AI agent that works through the buttons, menus and screens of ordinary software — on a desktop, phone or website — instead of through behind-the-scenes programming interfaces.; a revised OSWorld-Verified release (2025) repaired broken tasks and standardized the evaluation infrastructure. By late 2025, leading agents reported success rates above 60%.
On OSWorld the agent handles single-app tasks well but loses track in multi-app workflows.
Origin
Introduced in April 2024 by Tianbao Xie and colleagues, led from the University of Hong Kong with collaborators including Salesforce Research and Carnegie Mellon University, and published in the NeurIPS 2024 Datasets and Benchmarks Track.
Usage note
Scores vary with observation type (screenshots only versus accessibility tree), step limits and harness, so comparisons need matched settings.
Related
- computer usecomputer use, n.: An AI's ability to work a computer through its normal screen and controls: it studies pictures of what is displayed and sends mouse and keyboard actions, so it can use ordinary apps that were built for people.
- GUI agentGUI agent, n.: An AI agent that works through the buttons, menus and screens of ordinary software — on a desktop, phone or website — instead of through behind-the-scenes programming interfaces.
- WebArenaWebArena, prop. n.: A test world of realistic, self-hosted websites — a code-hosting site, an online shop, a forum and more — in which AI agents must complete tasks such as finding orders or editing posts.
- AI agentAI agent, n.: An AI system that does not just answer but acts: given a goal, it decides on steps, uses tools such as a browser or code editor, checks the results and keeps going until the job is done.
- visual groundingvisual grounding, n.: Linking words to the exact part of an image they refer to, for example finding 'the man in the red hat on the left' and pointing to him.
- Terminal-BenchTerminal-Bench, prop. n.: A test of AI agents working in a computer's command line — building software, analyzing data, securing or repairing systems — with each task checked automatically inside a sealed-off container.
Sources
- Xie, T. et al. (2024). OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments. NeurIPS 2024 Datasets and Benchmarks Track. arxiv.org/abs/2404.07972
Mentioned in
computer usecomputer use, n.: An AI's ability to work a computer through its normal screen and controls: it studies pictures of what is displayed and sends mouse and keyboard actions, so it can use ordinary apps that were built for people., Terminal-BenchTerminal-Bench, prop. n.: A test of AI agents working in a computer's command line — building software, analyzing data, securing or repairing systems — with each task checked automatically inside a sealed-off container., WebArenaWebArena, prop. n.: A test world of realistic, self-hosted websites — a code-hosting site, an online shop, a forum and more — in which AI agents must complete tasks such as finding orders or editing posts., GUI agentGUI agent, n.: An AI agent that works through the buttons, menus and screens of ordinary software — on a desktop, phone or website — instead of through behind-the-scenes programming interfaces.