What this article covers
You will leave here knowing which operational signals predict agent quality failures days before users notice them, how to instrument agents to surface those signals continuously, and how to set alert thresholds that flag real problems without generating noise. The examples below come from teams running agents at production scale in 2025 and 2026.
Why decay is harder to catch than failure
A complete agent failure is visible. The process crashes, the API returns an error, the downstream system times out. Quality decay is different. The agent keeps running. Outputs look plausible. Users get responses. The problem is that those responses are gradually less accurate, less complete, or less aligned with what the task actually requires, and nobody has a dashboard that shows them that.
Scale AI found that 34% of enterprises experienced unexpected agent behavior changes following model updates in 2025, and 12% of those cases produced incidents severe enough to require human intervention. Those numbers are not describing catastrophic failures. They are describing drift that crossed a threshold only after it became a production incident.
The reason decay goes undetected is measurement scope. Gartner found that 84% of CIOs lack a formal process for tracking AI accuracy, which means most teams are monitoring infrastructure health, not behavioral quality. A healthy server running a degraded agent looks identical to a healthy server running a well-functioning one.
There is also a structural gap between how teams build confidence and where problems actually appear. Benchmarks measured in the lab show a 37% performance gap against real-world deployment for enterprise agentic systems. If your evaluation framework stops at pre-deployment testing, you are measuring the wrong environment.
For a deeper look at where that gap opens and why, see offline to online evaluation: why agent production failures surprise teams who tested carefully.
The three operational signals that predict failure
Intervention rate
Anthropic identified human intervention rate as the most predictive leading indicator of agent failure in production. Their analysis showed that an override rate climbing from 5% to 12% over two weeks reliably predicts a system-level quality incident in the following week. That is a seven-day warning window, which is enough time to act if you are measuring it.
Intervention rate measures how often a human steps in to correct, redo, or reject an agent's output. It is a direct signal from the people closest to the task. Tracking it requires recording every step where a human touches an agent-produced result, not just explicit rejections, but edits, escalations, and re-runs triggered by dissatisfaction with the original output.
For teams running customer support agents or internal ops agents, intervention rate is often the fastest-moving signal because human operators are already in the loop. The measurement infrastructure is simpler than it sounds: a span that records whether a task completed without correction is enough to start.
Silent failures and task completion rate
Silent failures are the hardest category. The agent completes a task, no error is recorded, and the output is wrong or incomplete. Common forms include a response that answers a different question than the one asked, a retrieved document that is semantically close but factually off, and a multi-step workflow that finishes but skips a required verification step.
Task completion rate on its own is not sufficient because it counts silent failures as successes. You need a scored completion rate: one that applies a quality check to each completed task, not just a binary success flag. This is where evals in production differ from test-time evals. The scoring has to happen continuously, on live traffic, against a definition of success that reflects what the task was actually for.
Latitude documented this directly. Connecting production failures to pre-deployment tests through an annotation-to-eval loop reduced critical errors by 80% and improved task completion rate from 62% to 78% over eight weeks. The mechanism was closing the feedback loop between what was observed in production and what was measured during testing, so that the definition of success stayed calibrated to real conditions.
Error propagation across agent boundaries
In multi-agent systems, quality problems rarely stay local. An agent that produces a slightly incorrect intermediate result passes that result to the next agent in the chain. The downstream agent cannot distinguish a good input from a subtly bad one, so it operates on degraded data and its output degrades further. By the time the final result reaches a human or a downstream system, the original error has compounded.
Teams who instrument inter-agent communication discover and resolve performance issues three times faster than teams who monitor only individual agents, according to A16z's analysis of multi-agent startups. The instrumentation difference is span coverage: recording what each agent passes to the next, not just what each agent produces in isolation.
Prefactor instruments these handoffs as communication spans, recording the payload, the receiving agent's identity, and the scored quality of the input at the boundary. When an error propagates, the trace shows exactly where it entered the chain. For teams building on A2A protocols or running RAG retrieval agents in a pipeline, this trace structure is often the only way to find the origin of a quality problem that surfaces three steps later.
Setting thresholds that don't cry wolf
The common mistake is setting absolute thresholds: alert when error rate exceeds 5%, alert when latency exceeds two seconds. Absolute thresholds fire on normal variation and train teams to ignore them.
Relative thresholds calibrated against a rolling baseline work better. If your agent's intervention rate averages 6% over thirty days, an alert at 9% (a 50% relative increase sustained over two days) is a meaningful signal. The same alert at 6.5% (an 8% relative increase on a single day) is noise.
A few practical rules for threshold design:
- Measure the baseline over at least two weeks before setting any thresholds. Shorter windows capture transient variation rather than true operational baseline.
- Require a signal to persist across a minimum number of samples before alerting. A single bad hour is different from a bad day.
- Track threshold sensitivity explicitly: log how often each alert fires and how often a human investigation found a real problem. This creates the data to tune thresholds over time rather than guessing.
- Separate monitoring by agent type. A voice agent has different baseline characteristics than a coding agent. Mixing them into a single metric obscures both.
For more on what to measure and how scoring quality signals differ from observability metrics, see evals vs observability: watching your agents is not evaluating them and the silent failure gap.
Prefactor scores each span against configurable quality rubrics and records those scores in the audit trail alongside the raw trace. When you adjust a threshold, the historical scores let you back-test whether the new threshold would have caught past incidents or generated false positives, without waiting for the next production event to find out.
Where to start
Pick one signal, intervention rate is the easiest to instrument if humans are already in your workflow, and measure it against a baseline for two weeks before adding others. Trying to monitor everything at once produces dashboards nobody reads. Start with what is already measurable, establish a baseline, then expand.
Start evaluating your agents or read the instrumentation setup in the docs.