Evaluation

Why an agent is harder to evaluate than one answer

Suppose you ask a model to summarize a bug report. You can grade the summary against a good reference. Now ask Patch to fix the bug. The run takes 20 steps, edits two files, and runs the tests four times. What exactly do you grade?

A run is harder to grade than one answer for four reasons:

• It has many steps. A wrong guess in step 3 can sink the whole run.
• Runs vary. Even with randomness turned down (temperature 0), the same call can return a different reply, and that changes everything after it.
• The result is a change in the world, such as edited files, so you need an environment you can reset.
• The path matters. A run that fixed the bug but pushed to main without approval failed.

You are also testing more than the model, because your code, its tools, and its budgets all move the score. Chapter 18 shows how to tell those causes apart.

Grade the result and the rules

Patch's run ends with the target test passing. But the trace shows that Patch ran git push without approval, or edited the test until it passed. The result looks fine, and the run still failed.

So check two things. First, check the outcome, which is the end state of the world: does the test pass, and is the expense filed correctly? Use code for this wherever you can. Second, check the invariants, which are rules that must hold in every run, such as "never calls open_pr without an approval record".

Do not grade by matching an expected path. A check like "reads auth/session.py in step 2" fails a good run that found the file another way. Anthropic's guide to agent evals agrees: "it's often better to grade what the agent produced, not the path it took."

Nine metrics, and how each one misleads

Patch v2 makes 30% fewer tool calls than v1. Is it better? Not if it saved those calls by skipping the tests. Each metric below is useful, but each one misleads in a predictable way when you read it alone.

Task completion rate

It is the share of tasks that end in verified success, not runs that merely stopped or where the model said "done". Report the step budget with it, because a bigger budget raises the rate.

Tool-use efficiency

It measures tool calls per task, including repeats and failures. Fewer calls can mean skipped tests, so compare only successful runs on tasks that both versions solved. Otherwise a version that also solves a few hard, costly tasks looks wasteful.

Correctness

It asks whether the result is actually right. A run can pass the tests it saw and still be wrong, for example by deleting a failing assertion, so grade with hidden tests or a person's review.

Human intervention rate

It counts how often a person steps in, with planned approvals counted apart from corrections. Fewer interventions can mean skipped approvals, so also count rule violations, with a target of zero.

Latency

It is the time from the start of a task to a checked result. Report the median and the 90th percentile, because an average hides the runs that loop for an hour.

Cost

It is the money spent per task, across every agent, tool, and person. Report the cost per solved task. A budget also hides the worst runs: a run that would have cost $20 shows up as a $3 failure.

Number of model calls

It counts every model call, including subagents, summaries, and graders. Count tokens too, because one call with a huge context can cost more than five small ones.

Recovery from failure

It asks whether the agent gets back on track after a tool error, a timeout, or a crash. Cause these failures on purpose, then count how many runs still succeed and whether any action happened twice.

Robustness

It asks whether success holds when the input changes in ways that should not matter, such as a reworded request or irrelevant files. For a planted instruction in a document, also check that the agent did not obey it.

So read the metrics in pairs, the way an A/B test watches guardrail metrics next to its main metric.

Graders: who decides a run succeeded?

Patch's runs are easy to grade, because a script can rerun the tests. A research report is harder, because no test can say whether it covered the important points. The part of an eval that decides pass or fail is called a grader, and there are three kinds:

• Code checks run tests or inspect the end state. They are cheap and consistent, so use them wherever you can.
• Model graders score the output against a rubric, which is a list of specific checks. They handle open-ended work, but they are noisy.
• People are the reference for judgment calls, but they are slow, and two people can disagree.

Treat a grader like a measuring instrument. Before you trust a model grader, compare it with human labels. Give it evidence, such as the diff and the test output, not only the agent's final message. If the agent uses a grader as its own stop check, grade the eval with a different one, because a grader cannot catch its own blind spots.

Trials, pass@k, and pass^k

Patch fixes bug 412 in two of three attempts. Did it pass? That depends on how Patch will be used. Each attempt is called a trial, and one trial per task tells you little, because the next one may differ.

Write down a reset policy first: every trial starts from the same commit, with empty memory and no leftover files. Otherwise trial 2 can learn from trial 1. Hermes, for example, can save memory and skills after a turn.

Two ways to summarize k trials

• pass@k is the chance that at least one of k trials succeeds. Use it when a person or a check can pick the best attempt.
• pass^k is the chance that all k trials succeed. Use it when every failure costs you, as in customer support.

The two can be far apart. In the τ-bench paper, gpt-4o succeeded on 61% of retail trials, but its pass^8 was below 25%. You can estimate both from n trials of a task with c successes:

from math import comb

# n trials of one task, c passed, k <= n.
# Average the result over tasks.

def pass_at_k(n, c, k):   # at least one succeeds
    return 1 - comb(n - c, k) / comb(n, k)

def pass_hat_k(n, c, k):  # all k succeed
    return comb(c, k) / comb(n, k)

Compare versions on the same tasks

Patch v1 and v2 each run the same 50 bugs, three trials per bug. The numbers here are made up but realistic. v1 passes 62% of its trials, and v2 passes 70%. Is v2 better?

Compare them bug by bug. On 38 bugs, both versions score the same, so those bugs say nothing about the difference. v2 does better on 9 bugs and worse on 3. On average, v2 scores 8 points higher per bug.

That 8 comes from a small sample, so give it an uncertainty range. The range answers one question: if you reran the comparison on other bugs like these, how far could the result move? Here it runs from about 1 to 15 points. So v2 is probably better, but by an uncertain amount, and the evidence rests on 12 bugs. Read those 12 traces before you believe the number.

Two rules keep the range honest:

• Count bugs, not trials. The three trials of one bug are not independent evidence, because a hard bug tends to fail all three times. So treat each bug as one sample: average its trials first, then compare bug by bug.
• Pair the versions. Run both versions on the same bugs in the same hour. Bugs differ a lot in difficulty, and pairing cancels that out. A slow API afternoon also hits both.

Anthropic's eval guide suggests starting with 20 to 50 simple tasks drawn from real failures. Grow the suite as the gains get smaller. Also decide what "better" means before you look. For example, success must rise with a range that stays above zero, and no new rule violations may appear.

Where A/B-test intuition misleads

• Samples are tiny, so only large effects show. In exchange, you can read every trace.
• The test set leaks. Tuning prompts on the suite's failures inflates the gain, so keep a held-out set.
• Budgets cut off the data. A run that hits the cap becomes a cheap failure, so report how often runs hit the budget.

Benchmarks and their limits

Anthropic reported that Claude Opus 4.5 first scored 42% on the CORE-Bench benchmark. After the team fixed bugs in the evaluation code and loosened the agent's setup, the same model scored 95%. So a benchmark score belongs to a model plus the code around it.

Public benchmarks are shared task sets with automatic checks. SWE-bench gives a coding agent 2,294 real GitHub issues from 12 Python repositories. In τ-bench, an agent talks to a simulated customer, and the grader checks the database at the end.

Benchmarks are useful for comparing models on fixed tasks, but they have limits:

• Some tasks are broken. SWE-bench Verified is a subset of 500 tasks that people checked for a clear issue, valid tests, and enough information to solve.
• Tasks decay. Some Terminal-Bench 1.0 tasks broke when outside websites changed, and version 4.0 dropped tasks that were saturated or had published solutions.
• Their tasks are not your tasks, and published numbers differ in trials and budgets.

So use benchmarks to shortlist models, and your own suite to decide.

Evaluation in production

Suppose Patch passes 74% of its suite, but after launch, developers revert one in five of its pull requests. An offline suite is a sample of the past, so you also measure live:

• Run the agent in shadow mode first, where it proposes but does not act, and compare its proposals with what people did.
• Count late outcomes within a window, such as "reverted within a week".
• Have people grade a random sample of runs each week.
• Turn every production failure into a new task for the suite.

Running example: Patch gets an eval suite

Patch's suite is 50 past bugs that a person fixed, each pinned to the commit before the fix, with hidden tests taken from the human fix. Each bug runs three times in a fresh container, and every new version runs on the same 50 bugs.

case = {
    "id": "bug-0412",
    "commit": "a1b2c3d",            # the commit before the human fix
    "task": "Fix tests/test_auth.py::test_login_expired_token",
    "hidden_tests": ["tests/test_auth.py::test_expiry_boundary"],
    "scope": ["auth/", "tests/"],
    "trials": 3,
    "budget": {"steps": 30, "usd": 3.00},   # same limits as production
}

def trial_ok(run, case):
    return (run.target_and_suite_passed
            and run.passed(case["hidden_tests"])
            and not run.unapproved_gated_actions
            and run.diff_within(case["scope"]))

Design checkpoint 5: the expense agent

What goes wrong

You decide, AI builds

Check yourself

Takeaways

• Grade the end state and the rules, not the agent's claim. Read each metric next to the one that keeps it honest.
• Use pass^k when every failure costs you, and pass@k when someone can pick the best attempt.
• Treat the task as the unit. Compare versions task by task, in the same window, and decide what counts as better before you look.

Going deeper

• Demystifying evals for AI agents (Anthropic, January 2026). Tasks, trials, graders, and how bugs in the eval code distort scores.
• τ-bench (Yao and others, June 2024). Tool-using agents graded on the final database state, and the paper that defines pass^k.