Claude uses a computer

What Anthropic shipped on 22 October 2024

On 22 October 2024, Anthropic announced an upgraded Claude 3.5 Sonnet, a forthcoming Claude 3.5 Haiku, and, packaged alongside the models, a public beta of a capability it called computer use. In Anthropic's announcement, developers were told they could direct Claude to operate software "the way people do," by looking at a screen, moving a cursor, clicking buttons, and typing text. The narrow, documented claim is precise and modest: a general-purpose model, offered through an API in beta, could be handed screenshots of a graphical interface and would emit low-level input actions, and Anthropic reported measured performance on a public benchmark of real desktop tasks. The company was explicit that the feature was experimental and "at times cumbersome and error-prone."

The broader framing (that models had begun to operate general software) is where care is required. This report separates the two levels of claim. The event was not the first agent to touch a browser, nor a demonstration of reliable autonomy. It was a frontier lab making a screenshot-to-action loop a first-class, documented product surface on a widely used model, and publishing numbers that let others compare. That is a meaningful marker in the history of software agents without being a claim that the problem was solved.

The narrow capability, and why the framing outran it

At the technical core, computer use is unglamorous. The execution environment captures a screenshot and sends it to Claude along with the user's goal. The model, which already had strong vision, returns a structured action: move the cursor to specific pixel coordinates, click, type a string, press a key, take another screenshot. An external loop applies each action to the actual machine and returns the next screenshot. Nothing in that description requires the model to have privileged hooks into applications; it perceives pixels and produces the same inputs a person's mouse and keyboard would.

That design is the source of both the excitement and the caveats. Because the interface is generic (pixels in, input events out), it applies to any software that renders to a screen, including tools that expose no API. Because it is generic, it inherits every fragility of driving an interface blind to its internal state: a button that has not finished rendering, a modal that steals focus, a coordinate that is a few pixels off. The documented capability is a general perception-and-action bridge; the popular reading, that Claude could now "use a computer" in the human sense of dependable, unsupervised operation, ran well ahead of what the beta delivered.

The web-agent lineage the beta inherited

Computer use did not arrive in a vacuum. A research literature on agents that operate interfaces had been accumulating for years, and the beta is best read as a productized descendant of it. ReAct, published by Shunyu Yao and colleagues in 2022, established the pattern of interleaving reasoning traces with actions so a model could plan, act, observe the result, and revise, which is exactly the loop a computer-use agent runs. On the environment side, Mind2Web assembled tasks across many real websites to test whether a model could generalize as a web agent rather than memorize one site, and WebArena built a self-hosted, reproducible web environment so that agent success could be measured by whether a task was actually completed rather than by surface similarity to a reference trajectory.

Two threads made the specifically visual approach viable. Set-of-Mark prompting showed that overlaying labeled marks on regions of an image sharpened a vision-language model's ability to refer to and act on specific elements, addressing the grounding problem of connecting an intention to a pixel location. And WebVoyager, published in early 2024, built an end-to-end web agent driven by a large multimodal model that navigated live sites from screenshots, foreshadowing the screenshot-first design Anthropic later shipped. Earlier commercial efforts, including Adept's action transformer work in 2022, had also aimed at models that take actions in software. Anthropic's contribution was less a new idea than a decision: to train a widely deployed general model to do this competently and to expose it as a standard API tool rather than a bespoke research system.

The constraint it addressed: software without an API

The clearest way to state what the event was for is to name the constraint it targeted. Most automation before it required a machine-readable interface: an API, a webhook, a scriptable object model. Where such an interface exists, calling it is faster, cheaper, and more reliable than pretending to be a user. But an enormous amount of real work lives in software that offers no convenient API at all, or offers one that does not cover the specific action a workflow needs. Legacy line-of-business applications, internal tools, vendor portals, and desktop programs frequently expose only a graphical surface.

Computer use addressed that gap by treating the graphical surface itself as the interface. The grounding for this event puts the point plainly: it targeted workflows where software lacked a convenient dedicated API. This is a genuine widening of scope. It is also why the reliability bar is so demanding. An API call either succeeds or returns an error you can inspect; a mis-aimed click can silently do the wrong thing and leave no structured signal that it failed. The capability removes a hard barrier (no programmatic access) at the cost of introducing a softer, noisier failure mode (acting on an interface it can only see).

How the agent loop actually works

Mechanically, computer use is delivered as a tool the model can call, wired into a loop the developer controls. At launch it was gated behind an explicit beta header, and the reference design ran the target software inside an isolated virtual machine so that Claude's actions were contained. Anthropic's computer use documentation describes the cycle: the application sends the model the goal plus the computer tool; the model responds with one or more actions such as a click at given coordinates, a keystroke, or a request for a fresh screenshot; the caller executes those actions in order and returns the resulting screen and any tool results; the model continues until it judges the task complete. Because later actions in a batch depend on earlier ones, they must be applied strictly in sequence.

Crucially, the computer tool was designed to be combined with other tools rather than used alone. The same release let Claude interleave computer use with a text-editor tool and a bash (shell) tool, so an agent could, for example, run a command, read the terminal output as text, and also click through a graphical dialog when no command existed. Anthropic shipped a runnable reference implementation in its quickstarts repository, a containerized environment plus the loop code, precisely because the model is only half of the system: the surrounding harness that captures screens, scales coordinates to the display resolution, applies actions, and enforces isolation is where much of the real engineering lives. A significant part of the historical claim is architectural: Anthropic standardized that harness pattern and told developers how to build it safely.

The numbers Anthropic reported, and what they measure

The headline evidence came from OSWorld, a 2024 benchmark by Tianbao Xie and colleagues that scores multimodal agents on open-ended tasks in real operating-system environments spanning web and desktop applications, file operations, and multi-application workflows. Anthropic reported that the upgraded Claude 3.5 Sonnet scored 14.9 percent in the screenshot-only category, against 7.8 percent for the next-best system it cited, and reached 22.0 percent when allowed more steps. Those are task-specific completion rates on a hard benchmark of computer tasks, not a measure of general competence and certainly not a percentage of human ability. The right reading is comparative: roughly double the nearest cited system on that benchmark, while still failing most tasks.

The same release reported gains on adjacent agentic evaluations that contextualize why Anthropic thought the model was ready for this. On SWE-bench Verified, which tests whether a model can resolve real GitHub issues, the score rose from 33.4 percent to 49.0 percent. On tau-bench, a benchmark for tool-agent-user interaction, the retail domain improved from 62.6 percent to 69.2 percent and the airline domain from 36.0 percent to 46.0 percent. These are distinct skills from clicking through a GUI, but they share the underlying demand: sustaining a multi-step plan against a stateful environment. The honest summary of the evidence is that Claude had become materially better at agentic work in general, and that on the specific screenshot-driven computer task it led the systems Anthropic compared against while remaining far from reliable.

The upgraded model underneath

Computer use rode on a specific model, and the model matters to the claim. The upgraded Claude 3.5 Sonnet was an iteration on the version Anthropic had introduced in June 2024, which already emphasized strong coding and vision, including chart and diagram interpretation. Strong vision is a precondition for reading a screenshot well enough to act, and strong coding and tool use are preconditions for planning a sequence of actions. The October upgrade concentrated its gains in exactly the agentic and coding areas that computer use exercises. This is why the event is a model release and a capability release at once: the capability is only as good as the model's ability to perceive an interface and reason about what to do next, and Anthropic positioned the two as inseparable. Claude 3.5 Haiku, the smaller and cheaper model announced the same day, was pitched as matching the prior Sonnet on many coding tasks at lower cost, extending the same agentic direction down the price curve.

Competing approaches of its era

Anthropic was early to ship a general computer-use tool inside a mainstream model API, but it was not alone in the direction, and the surrounding field pursued several strategies. One axis is screenshot-only versus accessibility-tree or DOM-based control. Screenshot-only agents, the path Anthropic and WebVoyager took, perceive the rendered pixels and are maximally general but must solve visual grounding. Structured-input agents read the underlying document object model or accessibility tree, as much of the WebArena and Mind2Web work did, which is more reliable where that structure is available but does not extend to arbitrary desktop software.

The other axis is generalist model versus specialized agent. Anthropic bet on teaching a generalist model a broadly transferable skill (counting pixels and acting) rather than training a narrow web-only system. In the months that followed, other major labs shipped their own takes on the same problem, including browser-operating agents launched by OpenAI and by Google in late 2024 and early 2025, most of them also screenshot-driven. That convergence is itself evidence about the era: multiple frontier labs concluded near-simultaneously that a vision-language model driving a real interface was the most general path to software automation, and they largely agreed on the screenshot-plus-action loop, differing mainly in harness, safety posture, and how tightly the agent was scoped to the browser.

The honest limits and the new attack surface

Anthropic was unusually direct about the beta's weaknesses, and those limits are part of the historical record rather than footnotes. The announcement noted that actions people perform effortlessly, such as scrolling, dragging, and zooming, were still difficult for Claude, and that the capability was error-prone. Latency and cost compound the problem: every step requires a screenshot and a model call, so a task that a scripted API integration would finish in milliseconds can take many seconds and many billable tokens through a GUI. Reliability at that low per-step success rate means long tasks fail often, because errors accumulate across steps.

The more consequential caveat is security. A model that can see a screen and take arbitrary input actions introduces a genuinely new attack surface: prompt injection through the content it views. A malicious web page or document can display instructions intended for the agent rather than the user, and an agent that faithfully follows on-screen text can be steered into harmful actions. Anthropic's guidance, echoed in the computer use documentation, was to run computer use in a sandboxed, low-privilege environment, to avoid exposing sensitive credentials or data, and to keep a human in the loop for consequential steps. The grounding for this event states the principle cleanly: a computer-use capability does not authorize unrestricted actions or remove the need for isolation and approval controls. Capability and authorization are different things, and conflating them is the central safety error this class of system invites.

How to place computer use in AI history

The measured way to place this event is as a productization milestone in the longer arc of software agents, not as an invention of the idea or a solution to it. The narrow claim, well supported, is that on 22 October 2024 Anthropic made screenshot-driven computer control a documented, generally available tool on a leading model, published comparable benchmark numbers led by an OSWorld score of 14.9 to 22 percent, and shipped a reference harness and explicit safety guidance. The broad claim, supported but bounded, is that this helped move interface-operating agents from research demonstrations toward something developers could actually build on, and that the industry converged on the approach shortly after.

A useful way to navigate outward from this node is to ask three questions of any later system. Which earlier idea did it depend on, whether the reason-act loop of ReAct or the visual grounding of Set-of-Mark? What did it change, a new harness, a browser-scoped design, tighter safety controls, or a jump in the underlying model? And which limitation still blocked it, reliability, latency, cost, or prompt-injection risk? Read that way, computer use is neither a breakthrough that made autonomy inevitable nor a gimmick. It is the point where a frontier lab committed to a general perception-and-action interface, measured it honestly, and handed the surrounding engineering and its unresolved risks to everyone else to push forward.

References

• Anthropic (2024). Introducing computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku.
• Anthropic (2024). Introducing Claude 3.5 Sonnet.
• Anthropic (2024). Computer use tool documentation.
• Anthropic (2024). Claude quickstarts, including the computer use demo.
• Xie et al. (2024). OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments.
• Yao et al. (2022). ReAct: Synergizing Reasoning and Acting in Language Models.
• Zhou et al. (2023). WebArena: A Realistic Web Environment for Building Autonomous Agents.
• Deng et al. (2023). Mind2Web: Towards a Generalist Agent for the Web.
• Yang et al. (2023). Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V.
• He et al. (2024). WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models.
• Yao et al. (2024). tau-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains.
• Jimenez et al. (2023). SWE-bench: Can Language Models Resolve Real-World GitHub Issues?.