Evaluating Agents

Agents break the usual eval playbook

An agent is a model that runs in a loop — deciding its own next step, calling tools, and reacting to what comes back — instead of producing one answer to one input. That loop breaks two assumptions of ordinary evals: the output isn't a single text you can string-match, and there's no single "correct" sequence of steps. Evaluating agents needs its own playbook.

Start tiny: ~20 queries is enough

The most common mistake is delaying evaluation until you've built a big, rigorous suite. Don't. Start with about 20 queries that represent real usage. Early in development, changes are dramatic — a prompt fix can move success rates from 30% to 80% — and effects that large are plainly visible on a tiny sample. You don't need statistical power to see a 50-point jump.

The small suite earns its keep immediately as a smoke eval: a fast check you run on every change to catch obvious breakage. In Anthropic's multi-agent research system, a 20-query smoke eval run on every change to the lead agent's delegation instructions caught a regression where subagents began duplicating each other's searches — a bug no one would have spotted by reading the prompt diff.

Grow the suite later, when changes become subtle and you need finer resolution. The tiny version ships on day one.

Judge end states, not paths

Agents mutate state — they edit files, send messages, update records. And for most real tasks, several different action sequences are valid. An agent asked to fix a bug might patch the function directly, or refactor a helper and then patch — both fine.

If your grader checks the path ("did it call search, then read, then edit?"), it will fail correct runs that took a different route and pass broken runs that happened to make the expected calls. So grade the end state instead: after the agent finishes, is the world in the state the task required?

Path-based check (fragile):
  assert calls == ["search_docs", "read_file", "edit_file"]

End-state check (robust):
  assert test_suite_passes()
  assert bug_reproduction_script_now_fails_to_reproduce()
  assert no_files_changed_outside("src/auth/")

The end-state check doesn't care how the agent got there — only that the destination is right and nothing out of scope was touched.

Humans catch what graders can't

Automated grading has blind spots, and they're exactly the ones you can't anticipate — if you could anticipate them, you'd have written a grader for them. Human testing finds the unknown failure modes.

A real example: human testers of Anthropic's research agent noticed it kept preferring SEO-optimized content farms over authoritative academic PDFs. Every automated metric looked fine — the agent searched, cited, and completed tasks — but the sources were junk. The fix was adding source-quality heuristics to the prompt. No end-state grader would have flagged this, because "task completed with citations" was technically true.

Keep humans in the loop as periodic testers, not as the grading pipeline: automation handles volume, humans hunt for the failure classes automation doesn't know to look for.

Instrument everything

Agents are non-deterministic — the same input can take different paths on different runs — so you cannot debug them from inputs and outputs alone. You need full tracing: a recorded log of every decision, tool call, and result in a run (collected in a privacy-preserving way in production). When an agent fails weirdly, the trace is the only way to answer "what was it thinking at step 7?" Without tracing you're debugging a black box by guesswork.

Evaluating the tools, not just the agent

An agent is only as good as its tools, and tools deserve their own evals. The method:

1. Run realistic multi-step tasks programmatically — not single tool calls in isolation.
2. Collect accuracy plus operational metrics: runtime, tool-call counts, token consumption, error rates.
3. Mine the transcripts for waste: redundant calls, invalid parameters, retries.

The transcripts are where tool problems become obvious. In one case, an agent kept paging through a list_transactions tool call after call; replacing it with a search tool that takes a query and a result limit cut tool calls per task from 12 to 3. The agent's prompt never changed — the tool was the problem, and only the metrics revealed it.

The agent-eval checklist

• ~20 representative queries on day one; grow the suite as changes get subtler.
• Grade the end state the task required, plus "nothing out of scope changed."
• Schedule human testing passes to find failure modes no grader covers.
• Trace every run so non-deterministic failures are debuggable.
• Track tool-level metrics and read transcripts — many "agent" problems are tool problems.