How do you write evals for an AI agent?

An eval (short for evaluation) is a repeatable test of an AI system: a set of inputs, an expected result, and a way to score what came back. For a single prompt, that can be as simple as comparing an answer with the right one. An agent is harder to test. It takes many steps, calls tools, reads their results and decides what to do next, so two runs on the same task can take different paths. Some of those paths are fine. Some reach the right answer by doing something you never wanted, such as skipping a required check.

So an agent eval needs two kinds of checks. Outcome checks look at the end: is the world in the state it should be? Trajectory checks look at the path: did the agent call the right tools, in an acceptable order, with the right arguments? (A trajectory is the sequence of messages and tool calls an agent produces while it works.)

Check the outcome by looking at the state, not the words

The τ-bench benchmark, published in June 2024, shows a sound way to grade outcomes. A language model plays a customer, the agent has tools and a policy to follow, and at the end the evaluation compares the database with the expected final state. It doesn't grade what the agent said. An agent that writes "Your booking is cancelled" but never cancelled it fails.

You can apply the same idea to your own agent. Here is an illustrative eval set for a booking assistant at an imaginary inn, with four tools: get_policy, check_availability, create_booking and cancel_booking.

Each row has a concrete, checkable end state. Write the check as code: query the test database after the run and compare. Include tasks where the right answer is to do nothing (rows 3 to 5), because agents often fail by being too helpful.

Run each task more than once

τ-bench also introduced a measure called pass^k: the share of tasks an agent completes successfully in all k of k separate tries. In the paper's tests, even strong function-calling agents such as GPT-4o solved fewer than half the tasks, and in the retail domain their pass^8 score fell below 25%. A single successful run tells you the agent can do a task, not that it will.

A practical routine: run every task five times and report two numbers, the average success rate and the share of tasks that passed all five times. If the second is much lower than the first, the agent is unreliable even if it looks good on average.

Do all of this in a sandbox: a test database, fake payment tools, no real emails sent. Anthropic's guidance on building agents points out that their autonomy brings higher costs and the risk of errors compounding, and recommends extensive testing in sandboxed environments with guardrails.

How do you test the steps, not just the result?

Record the trajectory of every run, then compare it with a reference path or a rule. LangChain's open-source AgentEvals package does this for trajectories written as lists of messages in OpenAI's format. Its trajectory match evaluator has four modes: strict (same tool calls, same order), unordered (same calls, any order), superset (the key calls must appear, extras allowed) and subset (no calls beyond the expected ones). For task 2 above, a check might look like this:

import json
from agentevals.trajectory.match import create_trajectory_match_evaluator

reference = [
    {"role": "user", "content": "Please cancel booking 1042."},
    {"role": "assistant", "content": "", "tool_calls": [
        {"function": {"name": "get_policy", "arguments": json.dumps({"topic": "cancellation"})}},
    ]},
    {"role": "tool", "content": "Free cancellation up to 7 days before arrival."},
    {"role": "assistant", "content": "", "tool_calls": [
        {"function": {"name": "cancel_booking", "arguments": json.dumps({"booking_id": 1042})}},
    ]},
    {"role": "tool", "content": "Booking 1042 cancelled."},
    {"role": "assistant", "content": "Your booking 1042 is cancelled."},
]

evaluator = create_trajectory_match_evaluator(trajectory_match_mode="superset")
result = evaluator(outputs=agent_run, reference_outputs=reference)  # agent_run: the recorded messages
print(result["score"])

The superset mode passes if the agent made both key calls, even if it also looked something up along the way. Use strict when order truly matters, as with checking a policy before acting, and subset when calling any extra tool would be a problem. By default, tool calls must have the same arguments to match; the package lets you relax that per tool.

Some step rules are easier to write as plain code: "cancel_booking is never called" is one line over the recorded tool calls. For judgments that don't reduce to rules, such as whether the agent explained a refusal politely, AgentEvals also offers an LLM-as-judge evaluator, which asks a language model to grade the trajectory. Spot-check that judge by hand on a few runs before trusting it.

Use step checks where the path matters for safety, cost or policy, and let outcome checks carry the rest. Requiring one exact path for every task punishes agents for finding a different route that is equally good.