What good trace data gives you
Most teams can tell you that an agent failed. Fewer can tell you which step failed, why it failed at that step and not an earlier one, and whether the same failure pattern appeared in runs from three days ago without triggering an alert. That gap is what structured trace collection closes.
This article walks through how to instrument agents so that traces carry enough signal to answer those questions, how to classify failures by type rather than just by outcome, and how to build a debugging workflow that routes forensics by risk level. The goal is to get from "something went wrong in production" to a specific, reproducible cause without replaying the agent manually.
Why final-output evals miss the real failure
An agent that returns a plausible-looking answer can still have failed. It may have called the wrong tool, retrieved a stale document, skipped a validation step, or used context from a previous run that was never cleared. Final-output scoring catches the cases where the failure was severe enough to corrupt the visible result. It misses the cases where the failure was partial and the output still looked acceptable.
According to research on 91% of ML models degrading in production, agentic systems in particular tend to degrade silently via context drift, prompt regression, and input distribution shift rather than through a single catastrophic failure. Teams that evaluate thoroughly before launch but skip post-launch monitoring consistently see quality degradation within 30 to 60 days.
This is why evals and observability are different instruments: one tells you whether behavior meets a standard at a point in time, the other tells you whether behavior is drifting between points in time. You need both, connected to the same trace data.
Designing traces that carry diagnostic signal
A span is a unit of work inside an agent run: one LLM call, one tool invocation, one retrieval query. A trace is the ordered set of spans across a full agent run. The diagnostic value of a trace depends entirely on what each span records.
At minimum, each span should capture:
- The input going in and the output coming out, not just the final result
- The tool name, the arguments, and the return status including partial failures
- Timestamps and latency so you can detect which step slowed or timed out
- Any retrieval context used, including the source and the similarity score if available
- The model version and prompt template version, so you can correlate failures with provider updates
Without model version and prompt template version on every span, you cannot distinguish a behavioral regression caused by a provider update from one caused by a prompt change your team made. PwC's deployment of 25,000 agents found that continuous production trace analysis, specifically correlating quality drift with model provider updates, drove a 67% reduction in production incidents compared to periodic evaluation.
OpenTelemetry provides a standard schema for structured span data that works across most agent frameworks. Instrumenting at the span level from the start costs less than retrofitting observability after a failure surfaces.
A failure taxonomy for multi-step agents
Not all agent failures look the same, and routing your forensics correctly depends on knowing which category you are in.
Planning failures occur when the agent constructs a sequence of steps that is logically wrong before any tool is called. The trace shows the plan was generated, all tools executed without error, and the result is still wrong. Figma's agent tracing work in May 2026 specifically distinguished hallucinations in the planning phase from failures in tool execution, because the remediation path is different: planning failures usually point to the system prompt or the model's reasoning about the task.
Tool execution failures are the most common category in multi-step agents. A tool returns an error, a timeout, or a response outside the expected schema. The agent may recover gracefully or may proceed as if the tool succeeded, producing downstream errors that look unrelated. Look at the tool return status on each span, not just whether the final output is correct.
Context drift failures happen when an agent references information that was accurate at an earlier point in a session or in a prior run, but is no longer valid. These are hard to catch from output alone because the agent's reasoning is internally consistent. You detect them by comparing the retrieved context attached to a span against what a current retrieval of the same query would return.
Retrieval failures are a subset often treated separately because they have their own forensic path. A RAG retrieval agent that returns a low-relevance chunk will produce a plausible but unsupported answer. Patronus AI's Lynx framework addresses this specifically by comparing retrieved context against agent output at the trace level, and reduced effective hallucination rates to under 1% for regulated workflows in finance and legal domains.
Step ordering and skip failures occur when an agent executes steps out of the required sequence or skips a step entirely. These matter most in workflows where step order is a compliance requirement, not just a performance preference. Validate these against an activity schema that specifies which steps must occur and in what order.
Routing forensics by risk level
Not every failure warrants the same response. A useful forensic workflow routes by risk before routing by cause.
High-risk failures are those where the agent took an irreversible action, accessed sensitive data, or produced output that reached an external system or a user before the failure was detected. These go to human review first. Ghost actions, where an agent executes steps that were never requested, fall into this category by default.
Medium-risk failures are those where the output was wrong but the action was contained or reversible. Trace forensics applies here: look at which span diverged from expected behavior, check whether the same divergence appears in prior runs, and determine whether the failure correlates with a recent change to the prompt, model version, or tool schema.
Low-risk failures are transient errors, timeouts, and recoverable tool exceptions that the agent handled correctly. These belong in aggregate monitoring rather than individual investigation. Track their rate over time; a rising rate of transient errors often precedes a more serious failure pattern by 48 to 72 hours.
Prefactor records spans at each of these levels and attaches risk scores based on the action type and the data accessed, so teams can triage from a queue rather than reviewing every run manually.
Detecting behavioral decay before users report it
Deloitte's deployment of Zora agents across finance and supply chain workflows used step-level evaluation to catch prompt regressions that would have degraded silently over 30 to 60 days. The detection was possible because the traces carried enough structure to compare behavior across runs, not just to evaluate individual outputs.
The pattern that works across teams is:
- Sample a percentage of production traces for scoring, not just the ones that returned errors
- Track per-step pass rates over time, not just overall task completion rates
- Set change alerts on specific span attributes: tool call frequency, retrieval similarity scores, step count per run
- Correlate any metric change with the model version, prompt version, and input distribution recorded on each span
Detecting agent quality decay and production drift monitoring requires this kind of per-step metric history. Final-output scores can be stable while individual step quality degrades, particularly when the agent compensates through later steps. By the time final-output scores move, the underlying problem has usually been present for days.
For multi-agent orchestrators specifically, the forensic challenge is that a failure in one agent propagates downstream, so the failing span may be two or three agents removed from the observed symptom. Trace correlation across agent boundaries, using a shared trace ID, is the only reliable way to follow a failure to its origin.
Where to start
Instrument one agent with span-level traces that capture model version, prompt version, tool inputs, tool outputs, and retrieval context. Run it in production for a week and look at per-step pass rates rather than task completion rates. That single change will surface patterns that final-output evals cannot see.
Start evaluating your agents and explore the docs for instrumentation guides and schema references.