What is Claude's computer use, and how can an AI operate a computer?

Yesterday, October 22, 2024, Anthropic announced three things at once: an upgraded version of its Claude 3.5 Sonnet model, a new small and fast model called Claude 3.5 Haiku, and a capability it calls computer use. The third is the one getting the most attention. With it, Claude can look at a computer screen, move the mouse pointer, click buttons and type, much as you do when you sit at your desk.

Anthropic is careful to call this an early test. It is a public beta, meaning anyone with developer access can try it, but the company itself says it is slow, sometimes awkward and prone to errors. It is released now so that developers can report what works and what doesn't.

What Anthropic released

Here is the day's news in one table:

The upgraded Sonnet posts higher scores on several tests. On SWE-bench Verified, a set of real software bugs the model must fix on its own, it goes from 33.4% to 49.0%. On TAU-bench, which checks how well a model uses tools while helping simulated customers, it rises from 62.6% to 69.2% for a retail store and from 36.0% to 46.0% for an airline. Anthropic says the coming Haiku model scores 40.6% on SWE-bench Verified.

Before release, the US and UK government AI safety institutes tested the new Sonnet together. Several companies, including Asana, Canva, DoorDash and Replit, have already tried computer use on tasks that take dozens or even hundreds of steps.

How computer use works

Until now, if you wanted an AI model to act in software, a developer had to build a specific tool for each job: a function to look up an order, another to send an email. Computer use takes a different path. Instead of a custom tool for every task, Claude learns general skills for using the same screens people use.

The process runs as a loop:

1. A developer gives Claude a goal and access to a computer, usually a separate virtual machine.
2. The software takes a screenshot, a picture of the screen, and sends it to Claude.
3. Claude studies the picture and decides on one action, such as "move the pointer to the Save button and click."
4. To aim, Claude counts how many pixels (the tiny dots that make up a screen) the pointer must travel across and down. Anthropic says teaching Claude to count pixels accurately was essential.
5. The developer's software performs the action, takes a new screenshot and sends it back. The loop repeats until the task is done.

A useful picture is a flipbook. Claude doesn't watch a live video; it sees a series of still frames, one after each action. That explains one of its weaknesses: a message that pops up and disappears between two frames may never be seen.

Anthropic trained this skill on only a few simple programs, like a calculator or a text editor, without letting the model reach the internet during training. The company reports that Claude carried those skills over to many other programs and would even notice a mistake and try again.

How good is it?

On OSWorld, a benchmark (a standard test) of everyday computer tasks, Claude scores 14.9% when it sees only screenshots, against 7.8% for the next-best AI system. With extra steps allowed, it reaches 22.0%. People typically score around 70 to 75%. So Claude leads its category but is far from human ability.

Anthropic lists clear limits. Claude cannot yet drag items or zoom, and scrolling is hard for it. Even while recording its own demos, the team saw slip-ups: Claude once stopped a screen recording by accident, and once left a coding task to browse photos of Yellowstone National Park.

What the risks are

An AI that can click is more powerful than one that only talks, so Anthropic highlights a few dangers. The main one is prompt injection: a web page or document Claude sees on screen could contain hidden instructions that try to pull it away from your request. There is also deliberate misuse, such as spam or fraud.

Anthropic says it built classifiers, small programs that spot when computer use is happening and whether it looks harmful. With the US elections approaching, it is also steering Claude away from posting on social media, registering web domains and using government websites. The company says it does not train its models on the screenshots users send, by default.

Why it matters today

The Verge points out that other assistants, from Microsoft's Copilot Vision to OpenAI's desktop app for ChatGPT and Gemini on Android, can already look at your screen. None has broadly released a tool that clicks and types for you. Anthropic's bet is that it's easier to make the model fit existing software than to rebuild software around the model.

For builders, this opens a door to automating tasks in programs that have no API at all, such as an old accounting tool or a web form. For everyone else, it is an early look at agents, AI systems that carry out a series of steps toward a goal instead of answering one question.

How can developers try computer use safely?

You need an Anthropic API account, or access through Bedrock or Vertex AI. Anthropic provides a reference setup that runs Claude inside a virtual machine or container, an isolated computer inside your computer, so it never touches your real files. Start there rather than giving Claude your everyday desktop.

Choose low-risk tasks first, as Anthropic recommends: filling a test form, opening a file, gathering information from a few pages. Keep sensitive data and logged-in accounts out of the sandbox, and avoid sites where an unexpected click could buy, send or delete something.

Watch each run and read the screenshots and actions Claude reports. If a page shows text that looks like instructions to the AI, treat that as a warning sign. Anthropic is collecting feedback during the beta, so the notes you take on what fails are also useful to send back.