computer use
In plain wordsAn AI's ability to work a computer through its normal screen and controls: it studies pictures of what is displayed and sends mouse and keyboard actions, so it can use ordinary apps that were built for people.
The capability of a model to operate a graphical computer environment through its human interface: the model receives screenshots (sometimes supplemented by accessibility data), reasons about what is on screen, and issues low-level actions — move the pointer to pixel coordinates, click, drag, scroll, type, press keys — which a harnessharness, n.: The software around an AI model that gives it instructions and tools, carries out its actions and decides when it stops. The model is the engine; the harness is the rest of the car. executes on a real or virtual machine before returning a fresh screenshot. Because it works with any application a person can use, computer use extends agentsAI agent, n.: An AI system that does not just answer but acts: given a goal, it decides on steps, uses tools such as a browser or code editor, checks the results and keeps going until the job is done. to legacy software, websites and desktop tools that lack APIs; the costs are slowness, brittleness when layouts change and errors in locating elements. Exposure to arbitrary on-screen content makes prompt injectionprompt injection, n.: Tricking an AI system by slipping in text that it mistakes for instructions — for example, a hidden line in an email telling an AI assistant to forward the user's messages to a stranger. a central risk, so providers advise running computer-use agents in isolated virtual machines with minimal privileges and having a human confirm consequential actions such as purchases. Progress is tracked on benchmarks such as OSWorldOSWorld, prop. n.: A test in which AI agents must use real desktop computers — clicking, typing and switching between apps such as spreadsheets, browsers and file managers — to finish everyday office tasks..
We automated the old invoicing app with computer use because it has no API.
Origin
Anthropic released computer use in public beta with an upgraded Claude 3.5 Sonnet on 22 October 2024, the first such offering from a frontier model; it scored 14.9% on OSWorld's screenshot-only setting, where humans score about 72%. OpenAI followed with its Computer-Using Agent, powering the Operator product, on 23 January 2025 (38.1% on OSWorld), and other laboratories released similar capabilities.
Usage note
Unlike traditional robotic process automation, which replays scripted interface steps, computer-use agents decide each action from what they see. Broader than a browser agentbrowser agent, n.: An AI agent that uses a web browser on your behalf — searching, clicking links, filling in forms and reading pages — to complete tasks such as booking, shopping or gathering information.; GUI agentGUI agent, n.: An AI agent that works through the buttons, menus and screens of ordinary software — on a desktop, phone or website — instead of through behind-the-scenes programming interfaces. is the research term covering desktop, web and mobile interfaces.
Related
- GUI agentGUI agent, n.: An AI agent that works through the buttons, menus and screens of ordinary software — on a desktop, phone or website — instead of through behind-the-scenes programming interfaces.
- browser agentbrowser agent, n.: An AI agent that uses a web browser on your behalf — searching, clicking links, filling in forms and reading pages — to complete tasks such as booking, shopping or gathering information.
- OSWorldOSWorld, prop. n.: A test in which AI agents must use real desktop computers — clicking, typing and switching between apps such as spreadsheets, browsers and file managers — to finish everyday office tasks.
- AI agentAI agent, n.: An AI system that does not just answer but acts: given a goal, it decides on steps, uses tools such as a browser or code editor, checks the results and keeps going until the job is done.
- prompt injectionprompt injection, n.: Tricking an AI system by slipping in text that it mistakes for instructions — for example, a hidden line in an email telling an AI assistant to forward the user's messages to a stranger.
- sandboxsandbox, n.: A sealed-off computer space where an AI agent can run programs and change files without being able to harm the real system, reach private data or contact the outside world unless allowed.
- agent–computer interfaceagent–computer interface, n.: The commands, tools and feedback designed for an AI agent to work with a computer — the agent's version of a user interface, shaped around how language models read and act.
Sources
- Anthropic. Computer use tool. Claude API documentation. platform.claude.com/docs/en/agents-and-tools/tool-use/computer-use-tool
- OpenAI (2025). Computer-Using Agent. 23 January 2025. openai.com/index/computer-using-agent
Mentioned in
AI agentAI agent, n.: An AI system that does not just answer but acts: given a goal, it decides on steps, uses tools such as a browser or code editor, checks the results and keeps going until the job is done., actionaction, n.: A choice an AI agent makes at one moment — move left, place a stone, type a command — which changes its situation and may earn it a reward., agentic browseragentic browser, n.: A web browser with an AI assistant built in that can do things for you online — reading pages, clicking, filling in forms, shopping or booking — instead of just showing you websites., browser agentbrowser agent, n.: An AI agent that uses a web browser on your behalf — searching, clicking links, filling in forms and reading pages — to complete tasks such as booking, shopping or gathering information., CAPTCHACAPTCHA, n.: A small test on websites, like typing wavy letters or clicking every picture with a traffic light, meant to be easy for people but hard for computer programs, so as to keep automated bots out., multimodal large language modelmultimodal large language model, n.: An AI language model that can also take in — and sometimes produce — pictures, sound or video, so you can show it a photo or speak to it, not just type., OSWorldOSWorld, prop. n.: A test in which AI agents must use real desktop computers — clicking, typing and switching between apps such as spreadsheets, browsers and file managers — to finish everyday office tasks., robotic process automationrobotic process automation, n.: Software 'robots' that copy the clicks and typing a person would do in office computer programs — such as moving data between forms — to automate repetitive paperwork., WebArenaWebArena, prop. n.: A test world of realistic, self-hosted websites — a code-hosting site, an online shop, a forum and more — in which AI agents must complete tasks such as finding orders or editing posts., agent–computer interfaceagent–computer interface, n.: The commands, tools and feedback designed for an AI agent to work with a computer — the agent's version of a user interface, shaped around how language models read and act., agentic reinforcement learningagentic reinforcement learning, n.: Training an AI assistant by rewarding success on long, multi-step jobs where it must use tools, browse, run code and react to what happens, rather than answer in one go., AI boxAI box, n.: Keeping a powerful AI locked inside a sealed computer environment, with very limited ways to communicate or act, so that it cannot cause harm even if its goals are bad., GUI agentGUI agent, n.: An AI agent that works through the buttons, menus and screens of ordinary software — on a desktop, phone or website — instead of through behind-the-scenes programming interfaces., unhobblingunhobbling, n.: Fixing the simple ways a capable AI model is held back — such as by giving it tools, memory or time to think — so that abilities it already has can actually be used., visual groundingvisual grounding, n.: Linking words to the exact part of an image they refer to, for example finding 'the man in the red hat on the left' and pointing to him.