Which tasks are AI agents reliable at, and which still fail?

An AI agent is a system in which a language model chooses actions, such as searching, clicking, running code or calling a company's software, and keeps going until a task is done. This year every major lab has shipped one, and demos make them look capable of almost anything. The research gives a more useful picture: agents are good at some kinds of work, unreliable at others, and the line between the two is fairly predictable.

The single most important idea is the difference between capability (can it ever do this?) and reliability (does it do it every time?). Most headlines report the first. For real work, you need the second.

What the benchmarks show

Three published studies, each from a different angle, point to the same pattern.

Task length matters a lot. The research group METR timed skilled people on software and research tasks, then measured how long a task an agent could finish. In its March 2025 paper, Claude 3.7 Sonnet completed tasks that take an expert about an hour with 50% success. At 80% success, the length dropped to about 15 minutes. METR found the same gap across models: the task length an agent handles reliably is roughly five times shorter than the length it can manage half the time. METR also reports that these horizons have been doubling about every seven months since 2019, so the limits move quickly, but the gap between "sometimes" and "usually" persists.

Consistency is weaker than single scores suggest. The τ-bench (tau-bench) study puts agents in simulated customer-service chats where they must use tools and follow a written policy. It introduced pass^k: the chance an agent succeeds on the same task in all of k attempts. Even GPT-4o, a strong model when the paper came out in 2024, solved fewer than half the tasks, and in the retail setting its chance of succeeding eight times out of eight fell below 25%.

Using a real computer is still hard. When OpenAI introduced its computer-using agent in January 2025, it reported 38.1% success on OSWorld, a test of everyday computer tasks, against 72.4% for people. OpenAI noted that simpler web tasks went much better than harder benchmarks.

Where agents tend to succeed, and where they fail

Combining those findings with METR's analysis of why runs failed gives a practical sorting:

METR's reviewers saw real progress in tool use and in recovering from errors, rather than repeating a failed action. But agents still struggled in "messier" settings without clear feedback, or where they had to seek out information on their own. Its benchmark tasks were also much tidier than most jobs: they averaged 3.2 on a 16-point "messiness" scale, while METR rates a real job like writing a strong research paper at 9 to 15.

A worked pilot plan

Suppose a small accounting office wants an agent to collect monthly bank statements from a shared folder, extract totals and fill a summary spreadsheet. Before trusting it, run a two-week pilot:

1. Define success precisely. "Every total in the spreadsheet matches the statement to the cent, and no file is skipped."
2. Build 20 test cases from past months where you already know the right answers, including awkward ones: a scanned PDF, a statement in a different format, a month with no activity.
3. Run each case three times. Record pass or fail for each run in a simple table.
4. Read the failures. Note whether the agent misread a number, picked the wrong file, or stopped early. Each kind has a different fix.
5. Set a threshold in advance. For example, every case must pass three out of three before it runs without review; anything less keeps a person checking every output.
6. Keep humans on the irreversible step. The agent may draft; a person approves before anything is sent to a client.

An illustrative outcome: 16 of 20 cases pass all three runs, three pass sometimes, and one, the scanned PDF, always fails. That is a useful answer. The agent can handle the standard statements with spot checks, while scanned files stay manual.

How do you test an agent on your own task?

Use your own past work, not a vendor demo. Collect real examples where you know the correct result, including the difficult and unusual ones, because those are where agents break. Twenty is a reasonable start for a pilot; more gives a clearer picture.

Run every example several times and count how many runs succeed, not just whether one did. An agent that passes a case on the first try but fails it on the third is not reliable on that case, which is exactly what pass^k is designed to catch. Check results against the final state, as τ-bench does: is the spreadsheet correct, was the right file saved, rather than does the agent's message sound confident?

Finally, keep a log of every action the agent takes, and review the failures by type. Many can be fixed by narrowing the task, adding a check the agent must pass before finishing, or removing a tool it misuses. When results stay stable over several weeks, widen the agent's responsibility one step at a time.