What this article covers
If your agent scores well on a single-run benchmark but behaves unreliably in production, the problem is almost certainly compounding step error, not model quality. This article explains the mechanism, shows where instrumentation catches it, and gives you a concrete starting point for per-step evaluation.
The 37% gap is not a rounding error
Benchmark scores and production accuracy diverge by roughly 37% for deployed agents, based on analysis published in June 2026. A single-run accuracy of around 60% falls to approximately 25% across eight consecutive runs. That is not a slow drift; it is a structural property of how errors accumulate across steps.
The arithmetic is straightforward. If each step in a ten-step workflow succeeds with 90% reliability, the probability that all ten steps succeed is 0.9 to the power of 10, which is about 35%. At 85% per-step reliability, the same calculation gives roughly 20%. The agent that "works" in your staging environment, tested once end-to-end, is not the agent your users encounter on the eighth automated retry or the fifteenth daily execution.
This is the core problem with output-only evaluation: you measure the last state, not the path that produced it.
How step-level errors compound
Each step in an agent workflow produces an output that becomes part of the context for the next step. A misclassification at step two does not stay contained; it shapes the retrieval query at step three, the tool call at step four, and the final response at step five. By the time a bad final output appears, the root cause may be three steps back.
Research covering 2025 production deployments found that 17.14% of agent failures are step repetitions, where the agent loops on the same action without making progress, and 13.98% are mismatches between the agent's stated reasoning and its actual tool call. Both failure modes are invisible to final-output checks. An evaluator looking only at whether the response text is correct will miss a reasoning-action mismatch every time.
There are two distinct failure patterns worth separating:
- Cascade failure: an early error produces a plausible-looking intermediate output, so the agent proceeds confidently down a wrong path. The final output looks coherent but is wrong in ways that trace back to a single bad step.
- Repetition lock: the agent detects an obstacle, fails to resolve it, and re-executes the same step repeatedly. Voice agents and customer support agents are particularly exposed here because the conversation context grows with each retry, increasing token cost and degrading response quality simultaneously.
What trajectory evaluation catches that output evaluation misses
Output evaluation asks: is the final answer correct? Trajectory evaluation asks: did the agent take a correct path to get there?
The distinction matters because an agent can produce a correct output via a wrong path, and that wrong path is a reliability liability. It means the agent got lucky, not that it understood the task. In a different context, or on the next run, the same wrong path produces a wrong output.
Trajectory-level evaluation methods record every intermediate state: the tool calls made, the retrieval results used, the reasoning steps logged, and the sequence in which decisions were made. Scoring happens at each node, not just at the terminal state. This makes cascade failures visible before they reach users.
MyOperator, a cloud call-center platform running more than 100 AI voice agents across millions of monthly calls, added SDK instrumentation to measure per-step quality in June 2026. Their hallucination score improved from 2.8 to 9.5 out of 10 after the instrumentation identified specific steps where agents invented product features or answered out-of-scope questions. The improvement came not from changing the model but from knowing which steps to fix.
Discord ran thousands of daily simulations against agents serving millions of users before releases. Their deployment cycle dropped from four weeks to one week by catching step-level issues pre-release rather than in production. That time reduction reflects what it costs to diagnose failures post-deployment versus catching them in structured pre-release evaluation.
What to instrument and where
Instrumentation for step-level accuracy requires recording spans at three levels: the individual tool call or retrieval step, the reasoning-to-action transition, and the full trajectory across a workflow run.
For each span you want to capture:
- The input state entering the step
- The action selected and the parameters passed
- The output returned and whether it matched the expected schema
- The latency and whether the step was retried
Activity schemas let you define what a correct trajectory looks like for a given task type, independently of the specific output content. A schema for a support triage workflow might specify that step two must be a knowledge base lookup before any response is drafted. If the agent skips that step or reorders it, the schema violation is a signal regardless of whether the final response text looks acceptable.
Prefactor records spans at each step and scores them against activity schemas, which means a reasoning-action mismatch or a skipped required step surfaces in the trace rather than hiding behind an acceptable final output. The audit trail records the full trajectory per run, so when a production incident occurs, root cause analysis starts from the actual execution path rather than reconstructed logs.
A 2025 McKinsey survey of 306 practitioners found that 68% of production agents require human intervention within ten steps. That figure is partly a design constraint, but it is also a measurement problem: teams that cannot see which step triggered the intervention cannot reduce the intervention rate systematically.
The production drift problem
Step-level accuracy is not static. Models update, upstream data changes, and tool APIs evolve. An agent that passes trajectory evaluation in August may fail it in October without any deliberate change to the agent itself.
Detecting quality decay in production requires comparing trajectory scores over time against a baseline, not just checking whether today's run passed. Drift monitoring at the step level tells you which specific tool call or reasoning transition degraded, which narrows the remediation scope considerably compared to noticing that aggregate output quality dropped.
Gartner predicted in June 2025 that over 40% of agentic AI projects will be canceled by 2027, partly because organizations cannot systematically evaluate deployed agents. The teams that avoid that outcome are the ones treating per-step accuracy as a first-class metric from the first production deployment, not adding evaluation infrastructure after the first incident.
The pattern that avoids production surprises is: instrument before release, define activity schemas for each workflow type, score trajectories continuously, and alert on step-level degradation before it accumulates into visible output failures. Closing the gap between offline evaluation and production behavior depends on the same trajectory data being available in both environments.
Where to start
Pick one agent workflow, identify its steps, and define what a correct trajectory looks like for that workflow. Instrument the spans, run the workflow across at least eight consecutive executions, and score each step rather than only the final output. The compounding becomes visible immediately.
Start evaluating your agents or read through the docs to see how span recording and activity schema validation are set up.