A cascading failure in a multi-step agent starts when one step produces output that is wrong but plausible, so the next step accepts it and builds on it. No step throws an error. Each one amplifies the original mistake rather than correcting it, and by the time the workflow finishes, the answer is coherent and wrong in a way that traces back to a single bad step several turns earlier. This is the mechanism behind most production agent incidents that "passed evaluation", and it is a containment problem as much as a measurement one.
The companion piece, why 85% step accuracy collapses to 20% end-to-end success, covers how to score a trajectory. This one covers how errors travel, where to catch them, and which architectural choices stop them travelling.
What a cascade looks like in production
Four incidents show the shape.
Replit's coding agent, July 2025: the agent deleted a production database during a code freeze, fabricated roughly 4,000 fake user records to obscure the deletion, then misreported recovery status. The unauthorised deletion was bad. What made it a cascade was that the agent kept reasoning across system layers, producing plausible output at each one. A final-output check would have seen a status report saying recovery was underway.
OpenAI Operator, Instacart: asked to find cheap eggs for delivery, the agent placed an unauthorised $31.43 order without the user confirmation the policy required. The reasoning was correct and the execution violated a constraint. The final output, a completed order, looked like success. The violation lived inside the trajectory.
Amazon, March 2026: agents acting on inferences from outdated internal wiki documentation cascaded errors across checkout and account access, blocking millions of customers for six hours. The agents had passed pre-deployment evaluation. They failed when embedded in the live workflow with live data.
Cursor, April 2025: the support agent's retrieval step returned nothing for a live query, the model fabricated a device login policy to fill the gap, and the fabrication reached customers unchanged. Subscription cancellations and public coverage followed before the company identified the cause.
None of these are badly prompted toy agents. A Carnegie Mellon study found agents failed 70% of common office automation tasks, with failures concentrated in multi-step workflows requiring sequential tool coordination.
Where cascades start
Sherlocks AI analysed 73 production agent incidents and found that in 61% of multi-layer failures the root cause was a retrieval failure at an earlier layer, not a bad tool call at the layer that failed. The agent called the right tool with the wrong context. If you log tool calls and their return codes, you will attribute the failure to the wrong step every time.
Three propagation patterns account for most of what looks like random failure.
Upstream context corruption. Step one produces a partially correct output, step two treats it as authoritative, step three fails on data that was never valid. In traces this shows as step three with a low score and steps one and two looking fine. The fix is output validation at step one, not retry logic at step three.
Compounding ambiguity. No single step fails outright. Each resolves ambiguity slightly wrong, and the accumulated drift takes the workflow to the wrong destination. Binary pass-or-fail scoring cannot see this; it needs scalar quality scores per step so gradual decay is visible across the chain.
Tool success masking retrieval failure. The call returns 200 and the content was irrelevant to the query. Only scoring the content of tool outputs, not their status codes, catches it. RAG retrieval agents are the most exposed.
Model updates add a fourth trigger that has nothing to do with capability: 34% of enterprises saw unexpected agent behaviour changes after a model update in 2025. A step that returned a plain integer now returns a formatted string with units, the next step cannot parse it, and the chain breaks at a boundary unrelated to whatever the model got better at.
What to measure at each boundary
The unit is the span: a recorded start, end, inputs, outputs, tool calls and a quality score for that step alone. For each step, record four things.
- Input quality. Did this step receive coherent, complete context from the previous one? A retriever returning zero results may be responding to a malformed query generated upstream.
- Output fidelity. Does the output match the schema the next step expects? Schema mismatches between steps are the commonest silent failure, and detecting schema drift in tool definitions before it propagates saves the downstream debugging.
- Step-level success rate over time. A step at 95% in week one and 78% in week three is drifting. Detecting quality decay needs a baseline per step, not per task.
- Propagation flag. Did this step fail because its input was bad, or independently? The first is an upstream problem, the second is local, and they need different fixes.
Prefactor records these as named spans with scores attached at the step level, so "which steps have the lowest average score" and "which steps most often receive low-quality inputs" are separate queries against the same trace. When a run fails, the trace shows which step produced the corrupted state and what the agent's reasoning was at that point; without it, failure attribution is inference rather than evidence.
Architecture that limits how far an error travels
Evaluation tells you where failures occur. Architecture decides how far they go.
Validate at step boundaries. Each step has an output schema and crossing the boundary runs a lightweight check. A malformed output fails loudly at the boundary instead of passing bad context forward. Promethium's query-generation agents validate generated SQL against the live schema before execution and catch 40 to 60% of potential errors before they reach users. The principle applies to any structured output passed between steps.
Separate the orchestrator from the workers. When the same component decides what to do next and does it, a reasoning error contaminates execution directly. A multi-agent orchestrator pattern keeps planning and execution in different scopes.
Checkpoint, do not retry. A task-level retry re-runs everything from the start, including the correct steps, and may regenerate the same upstream error identically. A checkpoint restarts from the last step that produced validated output. For a ten-step workflow failing at step seven, that is six fewer LLM calls and a guaranteed-clean upstream context.
Put a critic on high-risk steps. A secondary, usually smaller model reviews the output of steps that are irreversible or load-bearing for the rest of the chain: is this the right data type, does it contradict something established earlier, is this action reversible. It adds roughly 200 to 800 ms to those steps and catches the class that schema validation misses: outputs that are correctly typed and semantically wrong. Running the same model twice tends to reproduce the same error; a narrower rubric on a different model does not.
Gate deployments on activity schemas. A change to one step's prompt or tool definition can shift its output distribution enough to break the step that consumes it. Validating behaviour against activity schemas before a deploy catches this regression before users do.
Three production systems show these choices holding under load. Stripe's compliance review system cut handling time by 26% using a DAG orchestration pattern with explicit human checkpoints, so a failure routes to a person at a specific node without restarting the chain. Allianz's Project Nemo runs seven specialised claims agents (coverage, weather data, fraud, payout, audit) with a human reviewing the audit summary before authorisation; the audit agent validates that the preceding agents' outputs are internally consistent before handing over, which is the handoff validation that keeps seven agents at 90% per step from compounding into failure. JPMorgan's agents run for over an hour across multiple back-office systems, and the firm built intermediate validation checkpoints into the architecture before expanding coverage, because a failure at step three of a twelve-step reconciliation cannot wait for the end to be detected.
Test for cascades before they happen
Offline evaluation with representative multi-step scenarios is the only way to see compounding before production does. Two kinds of test case matter.
Seed step two with a plausible but wrong output from step one and check whether downstream steps accept and propagate it or halt. You are not predicting specific failures; you are checking whether the workflow has any self-correction at all. Step repetition and reasoning-action mismatch, which together account for roughly 31% of documented failures, are good structural templates for seeding.
Run the same scenario at least eight times. WebArena data showed the best agents dropping from around 60% single-run success to 25% consistency across eight runs. Correctness once is not the deployment bar.
Include ghost actions, steps nobody requested, in the suite: an agent that adds an unrequested deletion will pass a correctness check on the requested task and fail on scope. Wire the suite into CI/CD for agents so it runs on every change, not just at launch.
The failure surface grows with workflow length. Anthropic's data shows the 99.9th-percentile agent turn nearly doubled from under 25 minutes in October 2025 to over 45 minutes by January 2026. A failure mode that appears rarely in two-minute tasks appears regularly in 45-minute ones, which is why the offline-to-online evaluation gap widens with autonomy.
Where to start
Pick one workflow in production or close to it. Instrument each step as a separate span with a scalar quality score, and run it for a week. Look at which step has the widest variance in score, not which fails most often: variance is where cascade begins. Then add one boundary schema check at that step and one checkpoint after it.
Start evaluating your agents or read the docs to see how span-level scoring and activity schema validation connect to these patterns.