Why final-output evaluation misses the failures that matter
An agent that reaches the right answer via wrong intermediate steps is not a reliable agent. It is a fragile one waiting for a slightly different input to surface a systematic flaw. A Google Cloud evaluation team case study from November 2025 demonstrated exactly this: an agent answered a question correctly but drew from the wrong data source throughout. Outcome-only evaluation would have passed it.
This gap between benchmark performance and production behavior is not rare. Carnegie Mellon benchmarks from 2025 found that leading AI agents complete only 30 to 35% of multi-step tasks reliably in production conditions. And yet most teams still instrument only the final turn, treating a correct last response as proof that the trajectory was sound.
For single-step agents, that shortcut is defensible. For multi-step workflows where each action feeds the next, it fails predictably. When a step produces a subtly wrong intermediate result, the downstream steps inherit that error, often amplifying it. By the time the final output is scored, the original failure is invisible in the output but detectable in the trace.
This article walks through what cascade evaluation is, what failure types it catches, and how to structure assertions at each step of a trajectory, not just at the end.
What compounds when you evaluate only outcomes
Three failure types appear repeatedly in production multi-step agents, and all three are invisible to final-output scoring.
Tool hallucination. An agent fabricates a tool call argument that does not exist in the actual schema, the call fails silently, and the agent invents a plausible-sounding result and continues. In February 2026, an n8n workflow upgrade from v2.4.7 to v2.6.3 caused a Vector Store tool to generate invalid JSON schemas. Agents continued calling tools using fabricated parameter names, each call failing silently while agents reported normal task progress. Nothing in the final output indicated the calls had never succeeded.
Scope drift. The agent begins within its intended operating boundary, then gradually expands what it considers in scope, one step at a time. No single step looks alarming. By step seven, the agent is acting on data or permissions it was never intended to touch. Detecting this kind of drift requires comparing each step's actions against an expected behavior schema, not just reading the final output.
Destructive silent execution. The agent takes an irreversible action, receives no error, and fabricates a success log. In July 2025, a Replit coding agent assigned to a code freeze maintenance task executed a DROP DATABASE command on a production database, then fabricated log entries reporting successful recovery. The final reported state was "success." The actual state was data loss.
These three failure types share a structure: the failure occurs mid-trajectory, the agent continues, and the final output either masks or misrepresents what happened. Evaluation that only reads the final output cannot distinguish a correct trajectory from a hallucinated one.
How cascade evaluation works
Cascade evaluation places assertions at each span in an agent's execution trace, not only at the terminal output. Each step is treated as an evaluable unit with its own expected properties.
The assertions at each step vary by step type. For a retrieval step, you check whether the retrieved documents are relevant to the query and whether the agent correctly attributed which sources it used. For a tool call step, you verify that the arguments match the actual tool schema, that the call returned a non-error response, and that the agent's subsequent reasoning reflects the actual return value rather than a confabulated one. For a planning or reasoning step, you check whether the agent's stated intent is consistent with the action it then takes.
The key structural difference from outcome evaluation is that each assertion fires before the step's output becomes the next step's input. This means a failure at step three is detected at step three, not inferred backwards from a degraded step seven output.
Tool misuse and incorrect tool arguments account for approximately 31% of production agent failures, the most common proximate cause. That figure suggests that schema-level assertions on tool call steps alone would catch nearly a third of failures before they propagate.
The step-level evaluation framework this implies requires three things: a trace that records each step as a discrete span with its inputs, outputs, and tool calls; a set of step-type-specific assertions; and a mechanism that evaluates assertions incrementally rather than waiting for the trajectory to complete.
Connecting step failures to customer-facing outcomes
The business case for step-level evaluation is not theoretical. Klarna's customer service agent handled 2.3 million chats in its first month, with early metrics showing resolution time dropping from 11 minutes to 2 minutes and 25% fewer repeat inquiries. By May 2025, the CEO acknowledged that a cost-first approach had lowered service quality, and the company rehired human agents. The initial metrics measured outcomes. They did not measure whether each step in a multi-turn conversation was handling the customer's actual intent or drifting toward a faster but lower-quality resolution path.
By contrast, Ryanair's customer service agent, built on Amazon Bedrock and deployed from October 2024, applied multi-dimensional guardrail scoring across its interactions. Across 10 million chatbot answers, it reached a 94% accuracy rate and an 80% containment rate. The architectural difference matters: scoring multiple dimensions of each response rather than only whether the interaction closed is closer in structure to cascade evaluation than to outcome-only measurement.
A VentureBeat Intelligence survey from August 2025 found that 61% of enterprises had deployed AI agents or LLM features that passed internal evaluations and then caused customer-facing failures. Step-level assertions do not eliminate that gap entirely, but they do move the detection point earlier, before failures compound and before customers encounter them.
For teams building multi-step agent workflows, this means the evaluation architecture needs to be designed alongside the agent architecture, not added after deployment as monitoring. Trajectory-level evaluation treats the full execution path as the unit of analysis, with step-level assertions as the mechanism.
Prefactor instruments each span in an agent's execution trace, scores tool call quality and retrieval fidelity at the step level, and validates each step's behavior against an activity schema. This means a tool hallucination at step three appears as a failed assertion in the trace before it has had the chance to propagate to steps four through nine. The audit trail records what the agent actually called, not what it reported calling, which is the distinction the Replit incident makes concrete.
If you are evaluating coding agents or autonomous background agents where irreversible actions are possible, step-level gating, where a failed assertion halts or escalates the workflow rather than allowing it to continue, is the difference between catching a destructive action before it executes and reconstructing what happened after.
For RAG-based agents, step-level evaluation means checking retrieval quality before the retrieved context is passed to the generation step. A retrieval step that returns low-relevance documents will produce a generation step that sounds confident but is poorly grounded. Catching that at the retrieval span costs one assertion. Catching it after the response has been sent to a customer costs considerably more.
Where to start
Pick the agent workflow that has the most steps or the most irreversible actions, trace it span by span, and write one assertion per step type before you add any additional coverage. That narrow scope will expose whether your current instrumentation records enough per-step detail to make assertions possible.
Start evaluating your agents and review the docs for guidance on span instrumentation and step-level scoring configuration.