What Is a Computer Use Agent? How AI Sees and Clicks Through Your Browser

Computer use agents are one kind of AI agent: a model that takes actions through tools and picks its next step itself (a workflow fixes the steps in advance).

Other agents use tools made for software, such as running code ( code execution agents) or searching documents ( retrieval-augmented agents). A computer use agent uses a person’s tools: the screen, the mouse and the keyboard. AI agent architecture covers agents in general.

The loop: look, decide, act, look again

Ask an agent for the cheapest book on the first page of a demo bookshop (books.toscrape.com) and it runs this loop:

1. Look: the program running the agent sends the model your request and a screenshot (a picture of the screen).
2. Decide: the model picks an action (a click, typing or a scroll) and says where: with Anthropic’s computer use tool, as a pixel position on the screenshot; tools that read the page’s structure point to an element instead.
3. Act: the program carries it out; the model never touches the computer itself (Anthropic’s documentation says so).
4. Look again: a new screenshot goes back and the loop repeats.

The page has 114 clickable elements (links and buttons); only 42 are in the first screen, so the agent must scroll.

Two ways to see a page: a picture or its structure

A page can reach the model as a screenshot, or as an accessibility tree: a text outline of the page’s links, buttons and fields with their names.

We measured four public pages on 2026-11-11. A screenshot of one screen is 1,334 tokens on every page (calculated, not measured), because every screenshot is the same size. The tree grew with the page: 3.7 times that on the bookshop, 15.9 times on Wikipedia.

A screenshot misses what is off screen; the tree misses how the page looks.

Why it is slow and where it goes wrong

Each step is a model call, and a study of 16 agents (OSWorld-Human) found that even the best took 2.7 to 4.3 times as many steps as needed: ten steps become 27 to 43 (our arithmetic).

Small errors add up: one wrong click opens a page the agent did not plan for. Pop-ups can confuse the model (Google’s documentation), and some sites block automated access (OpenAI’s help page).

What it is safe to let it do

The main risk is prompt injection: text on a page, written to make the AI ignore what you asked. Anthropic says its model sometimes follows such text, and that no browser agent is immune.

The vendors’ advice: ask you before anything hard to undo; use a separate browser, as ChatGPT’s cloud browser does; sign in yourself instead of giving it your password; and watch, because Google’s help page says you are responsible for the agent’s actions, including purchases.

On the bookshop, finding the cheapest book only reads a page and changes nothing; anything you cannot undo is yours to confirm.

Where you meet one today

ChatGPT’s cloud browser (in ChatGPT Work, OpenAI’s setting for longer tasks) runs in its own browser on a separate computer, on paid plans (OpenAI’s help page). OpenAI’s earlier ChatGPT agent and Operator are gone. Claude computer use is a tool for developers. Google offers auto browse in Gemini in Chrome (experimental) and a Gemini computer use model.

If you only wanted to know what a computer use agent is, you can stop here.

The rest is for developers deciding whether one fits a task.

What each way of seeing costs on real pages

The table covers the whole page, beyond the first screen. A fifth page, Hacker News, failed to load in two of three runs and is left out. “Screens” is how many 1280 by 800 screenshots it takes to cover the page.

Screenshot tokens are calculated with Anthropic’s published formula. Text tokens are counted with the tokenizer (the program that cuts text into tokens) of the open Qwen2.5 1.5B model, not Claude’s, so Claude’s counts would differ by an unknown factor.

Neither form is always smaller. All screens as screenshots take more tokens than the tree on three pages and fewer on Wikipedia. On these four pages the tree followed how much was on the page, the screenshots how tall it was.

Anthropic’s browser use documentation says a tree read of a typical page often costs fewer tokens than a screenshot. Our trees covered the whole page and were all larger than one screenshot; a tool that sends only the visible part would send less. We did not measure that, so test your own pages.

The first screen holds only 6.5 % (Wikipedia) to 36.8 % (bookshop) of the clickable elements, so a screenshot-only agent must scroll to reach most of them.

Hybrids, and where researchers disagree

Many tools combine both forms. Anthropic’s browser use tool works through the page’s structure and screenshots, and tells developers to read the tree before taking a screenshot. Microsoft’s Playwright MCP, a tool that lets a model drive a browser, gives the model accessibility-tree snapshots by default and screenshots as an option.

Researchers disagree. The authors of SeeAct, a paper on web agents, found that the best method they built used both structure and visuals. The authors of UGround, a model that finds elements in screenshots, argue for vision only, because text descriptions of the page add noise, gaps and cost. The OSWorld authors found that no form wins everywhere.

What OSWorld shows

OSWorld is a benchmark of 369 real computing tasks on Ubuntu, a version of Linux. Its authors report that humans completed 72.36 % of them, and the best model at the time 12.24 %. Agent S3 later reported 72.6 % (best of several runs, October 2025), above that human rate.

OSWorld 2.0 has 108 long workflows; a typical one (median) takes a person about 1.6 hours. In June 2026, its authors reported that the best model they tested completed 20.6 % of them. They say failures come from losing track of constraints, missing information that arrives mid-task, guessing instead of asking, and skipping verification.

What to check before you build one

• Where does the task live? Anthropic says to use its browser use tool when the task stays inside web pages, and its computer use tool when it needs a whole desktop.
• What would be hard to undo? List those actions first and make the program around the agent ask you before each one. OpenAI also recommends step, time or cost limits.
• What does a step cost? Anthropic puts a screenshot in a long loop at roughly 1,000 to 1,800 input tokens; ours came to 1,334. If a job takes 27 to 43 steps instead of ten, that is about 36,000 to 57,000 tokens of screenshots (our calculation).
• Do you want to write it yourself? Our guide to building a browser agent does it in Python with browser-use, an open-source library for browser agents.

Limits of our test, and when the loop breaks

Our test is five pages, one day, one machine, one browser and a stand-in tokenizer. It shows how page size behaves, not what an agent costs: we ran no agent, compared no success rates and did not measure what any product sends to its model. The benchmark figures come from the authors; we did not run them.

When it breaks: the loop fails when the page changes under the agent (a pop-up, a layout shift), when a site blocks automated access or needs a sign-in the agent cannot do, and over long jobs where small errors add up.