What offline benchmarks cannot tell you
You run your agent against SWE-bench or WebArena. The numbers look good. Then you ship, and within two weeks support tickets are climbing and costs are drifting in ways nobody planned for. This is not an edge case. Research comparing offline and production agent performance has found that synchronous benchmark success rates of 47% can fall to 11% when the same agents operate on asynchronous, real-world tasks, a gap of roughly 36 percentage points.
The reason is structural. Benchmarks offer clean state, predictable tool responses, and curated prompts. Production offers none of that. Users phrase things badly. External APIs time out. Context windows fill with debris from earlier steps. The agent that aced the benchmark has never met any of this.
Understanding why offline-to-online gaps emerge is the prerequisite for closing them. This article walks through the failure classes that benchmarks miss, the metrics that actually predict production quality, and the instrumentation you need to catch decay before it compounds.
The three failure classes benchmarks do not surface
Step repetition and reasoning-action mismatch
End-to-end pass/fail scores hide what is happening inside a task. A 2025 analysis of multi-agent failures found that 17.14% of failures were step repetitions, cases where an agent loops on the same action without making progress, and 13.98% were mismatches between the agent's stated reasoning and the action it actually took. Neither failure type shows up as a task-level failure until costs have already accumulated or downstream systems have received bad inputs.
Step repetition is particularly expensive in agentic workflows. An agent polling a slow API on a five-second loop while waiting for a lock it already holds can burn tokens and wall-clock time for minutes before any monitoring catches it. Step-level accuracy measurement is the only way to detect this class of failure before it reaches the task boundary.
Silent hallucinations in retrieved context
RAG-based agents retrieve documents, synthesize answers, and take actions based on those answers. When retrieval returns a plausible-but-wrong chunk, the agent often has no internal signal that anything is wrong. The reasoning trace looks coherent. The output looks confident. The downstream system acts on information that was never accurate.
A fintech deployment tracked by Noveum measured this directly. Before continuous hallucination scoring in production, the agent's hallucination score sat at 2.8 out of 10. After instrumenting spans with over 20 scorers including faithfulness checks, the score improved to 9.5 and overall task success rose from 84% to 95%. The improvement did not come from changing the model. It came from measuring what the model was actually doing and feeding that signal back into the fix cycle.
Evaluation-masked context failures
Research from Algolia and Forrester attributes 60% of AI production failures to data quality, context, and governance rather than model limitations. Benchmarks sidestep this entirely because benchmark inputs are clean. A customer-facing agent, however, receives queries with partial information, contradictory prior conversation history, and system prompts that have drifted since the last evaluation run.
This is why detecting quality decay through drift monitoring is not optional for risk-sensitive deployments. The distribution of inputs your agent saw last month is not the distribution it sees this month, and no static benchmark reflects that shift.
Metrics that actually predict production quality
Pass/fail on a task is a lagging indicator. By the time you see it, the failure has already happened. The metrics that give you earlier signal are:
Step-level accuracy. Score each action in the trajectory, not just the final output. An agent with 95% step accuracy on a ten-step task has roughly a 60% chance of completing the task without error. At 80% step accuracy, that falls below 11%. The compounding math means small per-step defects produce large task-level failure rates at scale.
Faithfulness to retrieved context. For any agent that retrieves before it reasons, measure whether the generated claim is supported by the retrieved text. A score above roughly 0.85 on a normalized faithfulness metric correlates with acceptable hallucination rates in the deployments we have instrumented. Below 0.7, expect downstream errors to show up within hours of a distribution shift.
Tool call efficiency. Count the number of tool calls an agent makes per successful task completion and track it over time. Rising tool call counts with flat or falling success rates indicate a reasoning loop forming, often before users notice anything wrong.
Escalation and refusal rates. In customer-facing deployments, the rate at which an agent escalates to a human or refuses a request is a proxy for confidence calibration. Klarna's deployment handled 2.3 million conversations in its first month, and continuous hallucination and faithfulness evaluation was part of how the team maintained quality at that volume rather than discovering problems post-hoc.
How to instrument for production detection
Span-level recording
Every agent action should emit a span: the input state, the tool or model call, the output, and the latency. Spans that are aggregated at the task level lose the step-repetition signal entirely. OpenTelemetry-compatible tracing gives you a vendor-neutral way to collect this data. Prefactor's SDK records span-level traces and attaches quality and risk scores per span, which means a step-repetition loop appears as a repeated span signature rather than a silent timeout.
Activity schema validation
Before you score quality, you need to know whether the agent did what it was supposed to do at all. Validating agent behavior against activity schemas means defining the expected sequence and type of actions for a given task and flagging deviations at runtime. Goldman Sachs embedded engineers for six months to co-develop agents with production-grade evaluation and audit trails for transaction reconciliation, partly because the compliance requirement demanded that every agent action be explainable after the fact.
Continuous scoring, not periodic audits
A weekly evaluation run tells you something went wrong last week. Continuous scoring on live traces tells you something is going wrong now, while you can still interrupt it. Designing evaluation infrastructure around continuous scoring changes the operational question from "what failed?" to "is this trending toward failure?"
The pilot-to-production handoff
Only 15% of AI agent pilots reach production scale, according to research cited by the MIT NANDA initiative. One reason is that teams treat the benchmark score as a readiness gate. A better readiness gate is a week of shadow traffic with span-level scoring running against the live input distribution, before any real action is taken.
JPMorgan Chase's production deployments measure quality through production traces and span-level evaluation across 450 active use cases. The scale makes continuous instrumentation a necessity rather than a preference. For smaller teams, the same principle applies at smaller scope: you cannot safely hand off a pilot to production without a signal source that reflects the actual input distribution.
Understanding the gaps between deployment and production readiness and measuring what agents actually do versus what you expect are the two questions that the benchmark score never answers for you.
Related reading
- Evals vs observability watching your agents is not evaluating them
- Agent evaluation in production what to measure and how to prove it
- Offline to online evaluation gap agent production failures
- Openai hugging face incident real time agent evaluation
- Silent agent failures production evaluation gaps cost
- Silent failure gap agent quality evaluation before production
- Step level accuracy trajectory evaluation production agents
Where to start
Pick one agent that is already in or near production. Add span-level tracing, set up faithfulness and step-accuracy scoring on live traffic, and compare the score distribution against whatever offline evaluation you ran before launch. The gap between those two distributions is your evaluation debt.
Start evaluating your agents and read the docs to see how to instrument your first agent and configure scoring in under an hour.