How Do You Test an AI Agent?

How do you test an AI agent?

In three layers: unit-test the deterministic parts (tools, parsing, retrieval, guardrails) with ordinary software tests; run eval suites over a golden dataset scoring both outcomes and trajectories; and continuously evaluate sampled production traffic with the same scorers. No single layer is sufficient.

The reason testing agents feels awkward with conventional tooling is that the usual contract is broken: the same input can legitimately produce different outputs, and 'correct' is often a judgement rather than an equality check. This guide is the deeper companion to the testing section of Agent Evals: A Practical Guide.

Why can't I test AI agents with normal unit tests?

Because the same input can legitimately produce different outputs, many correct answers have no single reference to assert against, failures often occur mid-trajectory while the final output still looks right, and agents regress when a model or tool changes without any code change, so commit-triggered tests never fire.

Non-determinism. Asserting that the output equals an expected string fails on a correct answer phrased differently. You need scoring with tolerance, and pass rates across repeated runs rather than single pass/fail.

Open-ended correctness. Plenty of good outputs have no single reference answer. Some of your scoring has to be criteria-based, which is what LLM-as-a-judge is for, with the accompanying obligation to validate the judge itself.

Failures are mid-path. An agent can reach a right-looking answer through a broken process. Output assertions pass; the process was wrong. This is the failure mode conventional testing is structurally blind to.

Regression without code change. A provider ships a model update, a tool's API shifts, a retrieved document changes, and the agent degrades with your repo untouched. Tests triggered only by commits will never run.

Layer 1: deterministic tests

Ordinary software tests over the parts that genuinely are deterministic, and there are more of them than people assume.

Tools: each tool tested independently, including its error paths. A surprising share of 'agent failures' are ordinary tool bugs. Schema and parsing: structured outputs validate against their schema, and malformed responses are handled. Retrieval: known queries return the expected documents. Guardrails and fallbacks: refusals, timeouts and retries behave as intended.

Fast, cheap, and they run on every commit. Do these first: teams often jump straight to LLM-graded evals while a broken tool sits untested underneath.

Layer 2: eval suites

Run the agent against a golden dataset of representative tasks and score each run, at two levels.

Outcome scoring checks task completion, correctness, faithfulness to retrieved context, format and safety, and catches wrong or unusable answers. Trajectory scoring checks tool-call accuracy, step efficiency, error recovery and goal adherence, and catches the right answer reached through a broken process, plus cost and latency problems.

Trajectory scoring is the layer most teams skip and the one that catches what the others can't. See Agent Evals: A Practical Guide.

Because runs are non-deterministic, run each case several times and report a pass rate. A case that passes four times in five is meaningfully different from one that passes every time, and a single run hides that entirely.

Run this suite on every prompt change, model change, tool change and config change, not only on code changes. See What Is an Eval Harness? for the scaffolding.

Layer 3: continuous evaluation in production

Your test set encodes the failures you've already thought of. Production supplies the rest.

Sample live traffic, score it with the same scorers used offline, and alert on movement. When something scores badly, capture the full trace (see agent tracing) and promote it into the golden dataset so it becomes a permanent regression test. That loop is what stops the same failure shipping twice, and it's the only part of agent testing that compounds.

What should I measure when testing an AI agent?

Start with four: task completion rate, tool-call accuracy, groundedness, and trajectory efficiency. Track cost and latency alongside them. Add more only when a metric will actually change a decision.

Groundedness asks whether the answer is supported by what was actually retrieved; trajectory efficiency compares the steps taken against the sensible path. An agent that gets slower and more expensive while staying accurate is still regressing, which is why cost and latency sit alongside the quality signals.

Resist adding metrics early. A dashboard of fifteen numbers nobody acts on is worse than three that gate a release.

What order should you build agent testing in?

Start with the cheap, deterministic layer and add judgement-based scoring last.

1. Unit-test tools and retrieval. Boring, fast, catches real bugs.

2. Assemble twenty real cases from failures you've actually seen. See Golden Datasets for Agents.

3. Score outcomes on those cases, and wire it into CI on prompt and model changes.

4. Add trajectory scoring, tool-call accuracy first, since it's rule-checkable.

5. Add LLM-as-a-judge only where rules can't reach, and validate the judge against human labels.

6. Sample production with the same scorers, and promote failures back into the dataset.

Most teams get value at step 3 and real leverage at step 6. Prefactor runs all three layers against the same eval definitions, offline as release gates and online against production traffic, with trajectory-level scoring and failing scores linked to the step that caused them.

How many times should I run each test case?

Several. Agent runs are non-deterministic, so a single pass or fail is weak evidence. Run each case multiple times and report a pass rate: a case passing four times in five is a different risk from one that always passes.

When should agent tests run?

On every prompt change, model change, tool change and config change, not only on code commits. Agents degrade when a provider updates a model or a tool's API shifts, with your repository untouched. Scheduled runs catch what commit-triggered runs miss.

What's the difference between testing and evaluating an AI agent?

Testing usually means the deterministic layer: assertions with pass/fail outcomes. Evaluation means scoring open-ended behaviour where correctness is a judgement and results are reported as scores and rates. Agents need both, which is why the practical answer is layered rather than either/or.