What offline evals cannot tell you about production
Most AI agent projects die quietly. 78% of enterprises have AI agent pilots running, but fewer than 15% have reached production scale. The gap between those two numbers is not a resourcing problem or an ambition problem. It is a measurement problem.
The pattern is consistent: an agent scores 90% on your internal test suite, passes a demo, gets approved. Then production load arrives, edge cases multiply, and reliability settles somewhere between 50% and 70%. 91% of benchmark-measured agents drop 15 to 25 percentage points between single-run and four-run reliability. That drop is not random noise. It surfaces the difference between what your eval scaffolding checks and what production actually demands.
Understanding why that gap exists is the precondition for closing it.
The three sources of measurement blindness
Single-shot testing hides variance. A pass@1 score tells you whether an agent succeeded once on a given input. It says nothing about whether it succeeds reliably. Agents that use tool calls, retrieval, or multi-step planning introduce compounding probability at each step. A five-step task where each step succeeds 90% of the time has a 59% end-to-end success rate, not 90%. Most offline eval suites measure individual steps or single runs, so the compounding is invisible until you are in production.
Benchmarks test isolated competence, not system behaviour. OpenAI's Deep Research agent scores 67.36% on the GAIA benchmark's public validation set, a six-fold improvement over raw GPT-4 at roughly 7%. That is a genuine capability signal. But GAIA measures reasoning and retrieval in controlled conditions. It does not measure what happens when the agent is embedded in a workflow with auth timeouts, rate-limited APIs, downstream services that respond slowly, or users who deviate from the expected input format. Production is the benchmark that matters, and most teams are not running it.
Retry logic and multi-agent coordination create failure modes that do not appear in unit evaluation. In multi-agent payment systems, a documented pattern recurs: Agent A requests a payment, Agent B processes it but responds slowly, Agent A times out and retries. The result is double charges to customers despite correct per-agent logic. No single agent failed. The system failed. Evaluating agents in isolation guarantees you will not catch this class of error.
For more on what production evaluation actually requires versus what observability gives you, see evals versus observability: watching your agents is not evaluating them.
What happens when quality gaps go undetected
The Klarna deployment is the most widely cited example of the offline-to-online gap in practice. The agent handled 2.3 million conversations in its first month, automated 67% of customer service chats, and cut resolution time from 11 minutes to 2 minutes. Those numbers are real. By May 2025, Klarna began rehiring human agents as hallucinations and quality failures on complex cases accumulated. The pilot metrics were not wrong. They were measuring the easy cases. The hard cases, disputes, edge-case refund logic, ambiguous policy questions, were not represented in the evaluation distribution.
The cost of catching this late is not only the rehiring budget. It is the customer interactions that went wrong before anyone noticed the pattern.
A more mechanically extreme version of the same problem appeared in a March 2025 fintech incident: a transaction reconciliation agent entered a retry loop with no cost controls or time limits. It ran for 11 days undetected and accumulated $47,000 in costs before someone intervened manually. The agent was doing exactly what it was designed to do. The evaluation process had never checked for the absence of circuit breakers.
Ghost actions, cases where an agent takes consequential steps nobody asked for, are a related failure class. They rarely appear in pre-production testing because test environments are designed to succeed.
What evaluation frameworks that hold up in production actually measure
Teams that close the offline-to-online gap tend to share a few practices.
They test across runs, not just across inputs. Running the same task four times and measuring consistency is a better signal than running 400 different tasks once. The Stanford OSWorld benchmark takes this seriously: it tests agents on live computer interfaces, file creation, spreadsheet editing, terminal operations, and real web forms. Scores reached 66% in 2025, up from 12% early in the year, but that benchmark is designed to surface generalisation failures, not just competence on known inputs. Your internal evals should do the same.
They instrument the agent, not just the output. Scoring a final answer tells you whether the agent got there. Tracing each span tells you why it succeeded or failed, which step introduced latency, and where the model deviated from the expected tool-call sequence. This is the difference between knowing an agent is unreliable and knowing which decision in which context is the source of that unreliability. Prefactor records spans and scores each step against a defined activity schema, so you can query failure patterns across sessions rather than reconstructing them from logs.
They validate behaviour schemas, not just success rates. 88% of AI agent projects fail before reaching production, with 61% of failures attributable to scope creep and data quality. Scope creep at the agent level means the agent is doing things outside its defined remit. A customer service agent that begins offering pricing information it was not given authority to quote is not failing on accuracy. It is failing on boundary compliance. That requires a different kind of check than accuracy scoring.
For autonomous background agents and multi-agent orchestrators, boundary compliance is especially consequential because the blast radius of a boundary violation scales with autonomy. See also access control best practices for AI agents.
They track the cost dimension alongside quality. GitHub's data shows 17 million AI-generated pull requests per month as of March 2026, with 90% of them rejected as noise. Volume without quality filtering is waste. For agents that call APIs, use retrieval, or spawn subagents, measuring token overhead and efficiency alongside task success is what lets you distinguish a capable agent from an expensive one.
Where to start
Pick one agent currently in pilot, instrument it at the span level, and run its ten hardest known cases four times each. Compare the pass@1 and pass@4 rates. If they diverge by more than 15 percentage points, your eval suite is optimistic and you have a concrete gap to close before scale makes it costly.
Start evaluating your agents or read through the docs to see how span recording and behaviour schema validation work in practice.