What offline evaluation cannot see
You run your agent against a curated hold-out set. It scores well. You ship. Then production traffic arrives with phrasing, context lengths, and tool-call sequences your test set never contained, and failures appear that your evaluator never flagged.
This is not a tooling problem. It is a distributional one. Test sets are snapshots of what you expected users to do. Production traffic is what they actually do, and the two diverge in ways that compound over time as your user base grows, your prompts drift, and the underlying model is updated by the provider.
LangChain's State of Agent Engineering survey (1,340 responses, December 2025) found that 52.4% of organizations run offline evaluations on test sets, but only 37.3% run online evaluations. Among teams that already had agents in production, that figure rose to 44.8%, which suggests teams learn this lesson by experiencing the failure rather than anticipating it.
The gap matters at scale. 73% of enterprise AI agent deployments experience reliability failures within their first year of production, according to Maxim AI research from November 2025. Offline evaluation alone cannot close that gap, because it cannot observe the conditions that cause those failures.
Why the distribution shifts
Three mechanisms drive divergence between test data and production traffic.
First, test sets are assembled by the people building the agent. Those people know the intended use cases and write prompts that fit them. Real users do not know your intent taxonomy, so they phrase requests differently, combine capabilities in unexpected ways, and occasionally ask the agent to do things adjacent to but outside its designed scope.
Second, context accumulates in multi-turn conversations in ways that single-turn test cases do not capture. A user who has already established several facts in a conversation creates a context window that your test harness rarely replicates. Customer-support agents and research analyst agents are particularly exposed here because their conversations run long and carry a high density of user-supplied facts that the agent must track without fabricating.
Third, tool outputs change. If your agent calls an external API and that API changes its response schema or starts returning occasional errors, your offline test suite, which mocked the API, will not catch the downstream failure.
Three named cases: who caught it and who did not
Klarna's customer-service agent now handles queries in 35 languages across 23 markets, reducing average resolution time from 11 minutes to under two minutes and saving the company an estimated $60 million by Q3 2025. That result required sustained investment in production monitoring, not just pre-launch evaluation. Repeat inquiries fell 25%, a signal that the agent's answers were accurate enough that users did not need to return, but that metric is only visible in production traffic, not in any offline test.
GitHub's Copilot code review agent reached 60 million code reviews across 12,000 organisations by March 2026. At that volume, edge cases that appear in one-in-ten-thousand pull requests are encountered thousands of times per day. GitHub's usage metrics API tracks active versus passive engagement, which is a form of online behavioral signal that supplements any offline quality score.
Noveum documented a case where a production customer-service agent had a hallucination score of 2.8 out of 10, a failure rate that was not visible in pre-deployment testing. After adding real-time hallucination detection and correction in production, the score rose to 9.5 and overall success rate moved from 84% to above 95%. The full loop from a failing trace to a verified fix ran in minutes. That loop only exists if you are capturing and scoring live traces.
Morgan Stanley's DevGen.AI agent reviewed 9 million lines of legacy code for 15,000 developers, saving an estimated 280,000 developer hours by January 2025. Code review is a domain where offline evaluation on a sample set is tractable, but the distribution of legacy code patterns across a 15,000-person engineering organisation is broader than any pre-assembled test set. Ongoing measurement of review quality across the actual codebase is what keeps the agent calibrated.
The three-layer framework
Bridging the gap requires layering three complementary practices, not replacing offline evaluation.
Offline evaluation on curated sets remains useful for catching regressions before deployment. Write test cases that cover your intended use cases and run them in CI. The limit is that coverage of unknown-unknowns in production traffic is near zero. Treat offline scores as a floor, not a ceiling. Our post on what to measure and how to prove it in production covers the mechanics of building those test sets.
Shadow testing against live traffic gives you real distribution without real consequences. Route a copy of production requests to a new agent version and score both responses without surfacing the shadow agent's output to users. The scoring happens offline, but the inputs come from production. This is where you find the failure modes your test set never contained.
Continuous online evaluation scores every live trace, or a statistically representative sample, against quality and risk criteria. This is where observability and evaluation diverge: recording spans tells you what happened, but scoring them against defined criteria tells you whether it was acceptable. Prefactor records spans via SDK and scores each trace for quality and risk, which means a failure mode that appears in a Friday afternoon traffic spike is captured and queryable rather than invisible until a user complaint surfaces it Monday morning.
The criterion for moving between layers is confidence, not calendar time. An agent should graduate from shadow testing to production when its online scores on live shadow traffic match or exceed the offline baseline. If they do not match, the offline test set was not representative.
Choosing what to score online
Not every dimension of quality is equally tractable in real time. Factual grounding and tool-call validity can be evaluated automatically against the retrieved context or the API response. Tone and policy adherence can be scored by a fast judge model. Task completion is harder and often requires either user feedback signals or delayed human review.
Activity schemas let you define what a correct sequence of actions looks like and validate live traces against that definition. When a multi-agent orchestrator calls a subagent in the wrong order or skips a required tool call, schema validation flags it immediately rather than after a downstream error propagates. Prefactor validates behaviour against activity schemas on every recorded trace, which means the audit trail reflects both what the agent did and whether that sequence was within spec.
Teams scaling autonomous background agents need this layer more than most. An agent that runs unattended across hundreds of tasks per hour can accumulate a class of systematic errors before any single instance is noticeable.
Connecting evaluation to deployment gates
Online scores are only useful if they feed back into deployment decisions. Define numeric thresholds for the metrics you score online, and block or roll back deployments that fall below them, the same way a test suite blocks a merge on a failing unit test. CI/CD integration for AI agents covers how to wire evaluation results into your pipeline.
The combination of offline baseline, shadow-traffic scoring, and continuous online monitoring reduces the category of failures that reach users without prior signal. It does not eliminate failures, because no evaluation framework can anticipate every future input. What it does is narrow the window between when a failure starts and when you have evidence to act on.
Where to start
Pick one agent that is already in production and instrument it to capture and score live traces before adding shadow testing or expanding coverage. One agent with full trace coverage teaches you more about the distribution gap than a dozen agents with offline-only scores.
Start evaluating your agents or read the docs to see how trace instrumentation and online scoring connect in practice.