What production degradation actually looks like
You deploy an agent, it passes your test suite, and it handles real traffic without throwing exceptions. Weeks later, users start complaining. When you dig into the logs, the agent has been producing subtly wrong outputs for days, possibly longer. No alert fired because nothing failed in the way your monitoring expected.
This is the failure mode that crashes most production agent programs: not a crash, not a 500, but a slow drift in output quality that your existing infrastructure has no way to catch. Traditional observability catches errors; it does not catch wrong.
The scale of the problem is larger than most teams assume. A Deloitte survey published in August 2026 projects that more than 40% of agentic AI projects will be cancelled by the end of 2027 due to escalating costs, unclear business value, or inadequate risk controls. Poor production quality, discovered too late to fix cheaply, is a direct contributor to that cancellation rate.
The three mechanisms of silent degradation
Prompt behavior drift
A model provider updates their serving infrastructure. Your system prompt stays the same. The model's behavior does not.
This happened in a documented 2026 production scenario where agent routing accuracy dropped from 93% to 71% following a silent LLM model provider update, a 22-percentage-point regression that no test caught and no alert fired on, discovered only after a surge in customer complaints. The system was technically healthy: no errors, normal latency, expected token counts. The problem was invisible to infrastructure monitoring because infrastructure monitoring does not score outputs.
Prompt behavior drift can also come from the other direction. Your team ships a prompt change intended to improve tone. The change also shifts how the agent decides when to escalate versus resolve, and your escalation rate moves by 15% before anyone notices. The gap between what agents do and what you think they do is often widest right after intentional changes.
Tool-call pattern shifts
Agents in production call tools in sequences. Those sequences carry information about reasoning quality. When an agent starts calling a search tool three times per task instead of once, or stops calling a verification tool it previously used on 80% of requests, something in its decision process has changed.
Sherlocks AI analyzed 73 production agent incidents between January and May 2026 and found that tool-call failures were the most common failure type, and that failures stacked across multiple layers rather than clustering at a single point. A single tool-call anomaly often precedes a downstream reasoning failure by several steps.
The n8n case from February 2026 illustrates how quickly tool-call regressions compound: the Vector Store Question Answer Tool began generating invalid JSON schemas for function calling after users upgraded from v2.4.7 to v2.6.3, halting enterprise-licensed production workflows entirely with the only fix being a version rollback. The failure was not gradual, but the detection was: teams only knew something was wrong when workflows stopped producing any output at all. A tool-call pattern monitor would have caught the schema anomaly on the first affected span.
Evaluating coding agents specifically requires measuring tool-call quality, not just final output, but the same principle applies across agent types.
Multi-step failure propagation
Single-step evaluation misses the most damaging class of production failure. An agent that reasons correctly at step one, selects the wrong context at step two, then acts confidently on that wrong context at step three produces an output that looks reasonable in isolation. The error is in the trajectory, not in any individual span.
Step-level evaluation that measures agent actions before failure compounds requires capturing the full execution trace and scoring intermediate decisions, not just the terminal output. This is structurally different from what a logger or a metrics dashboard gives you.
What to instrument
You need three layers of measurement running continuously in production, not just in your test environment.
Output quality scores. For each agent response, run a scorer that checks whether the output meets the criteria your team cares about: factual grounding, instruction following, format correctness, decision appropriateness for the given context. These scores need to be stored per span so you can track them over time and detect when the distribution shifts. Continuous production evaluation requires specific measurement choices.
Tool-call distribution baselines. Record which tools your agent calls, in what order, and at what frequency during a stable period. Then monitor for deviations. A 20% shift in tool-call frequency on a high-volume agent is a signal worth investigating before users feel it. Detecting agent drift through activity schema validation gives you a structured way to define what normal looks like and alert on departures from it.
Trajectory coherence checks. For multi-step agents, score whether the reasoning chain is internally consistent. Did the agent's context-retrieval step actually retrieve content relevant to the task it then executed? Did the verification step occur when the task type warranted it? These checks catch the propagation failures that terminal-output scoring misses entirely.
Datadog's 2026 State of AI Engineering report found that 5% of all LLM call spans in production returned errors in February 2026, with capacity-related failures accounting for 60% of those errors. That 5% is what your infrastructure monitoring already sees. The quality degradation problem lives in the other 95%, the calls that succeeded technically but produced worse outputs than they did three weeks ago.
How Amazon's March 2026 response illustrates the gap
After a cluster of high-severity incidents tied to AI-assisted code deployments, Amazon's internal response was to require additional senior review for AI-generated production changes, grouping incidents under categories like "Gen-AI assisted changes" and "high blast radius". The organizational response was correct: more human review. But the underlying problem was that degraded outputs had already reached production before anyone had enough signal to intervene.
Process controls are a last line of defense. Instrumentation that catches quality changes at the span level, before they accumulate into incidents, gives your human reviewers the right information at the right time rather than after the fact.
Prefactor in this context
Prefactor instruments agents via SDK, records spans, and runs quality and risk scorers on each one. The activity schema validator checks whether the tools called and the decisions made conform to the behavior you defined as acceptable. When scores drift over a rolling window, or when a tool-call sequence falls outside the expected distribution, that surfaces in the audit trail before it surfaces in your support queue. The offline-to-online evaluation gap is what Prefactor is built to close in practice.
Related reading
- Evals vs observability watching your agents is not evaluating them
- Agent evaluation in production what to measure and how to prove it
- Evaluating coding agents requires measuring tool call quality not just
- Evaluation gap offline vs production agent failures
- Offline to online evaluation gap agent production failures
- Openai hugging face incident real time agent evaluation
- Silent agent failures production evaluation gaps cost
Where to start
Pick one agent in production, instrument it to record full execution traces including tool calls and intermediate outputs, then run a quality scorer over two weeks of traffic and plot the score distribution over time. That baseline is what you need before you can detect drift. Start evaluating your agents and read through the docs for scorer configuration and schema validation setup.