What benchmark scores cannot tell you
If you have shipped an AI agent and watched its performance degrade quietly over the first 90 days, you already know the problem. Lab benchmarks measure a controlled slice of behavior. Production exposes everything else: edge cases, adversarial inputs, API timeouts, upstream data drift, and the compounding errors that accumulate across a multi-step pipeline.
A 2026 analysis by Kili Technology put a number on this: enterprise agentic systems show a 37% gap between benchmark scores and real-world performance, with up to 50x cost variation for comparable accuracy targets. That gap does not appear because the models are worse than advertised. It appears because the things benchmarks measure, accuracy on a curated dataset, single-turn task completion, are not the same things production stresses.
According to Gartner's 2025-2026 forecasts, over 40% of agentic AI projects are at risk of cancellation by 2027. The cited cause is not insufficient model capability. It is execution failures and governance gaps: the kind of problems that only show up once an agent touches real data, real users, and real downstream systems.
This article covers the evaluation patterns that close the gap, with specific attention to where measurement breaks down and what to instrument instead.
Where single-agent evaluation breaks down
Most teams start by evaluating individual agents in isolation, which is reasonable. The problem is that 61% of multi-agent system failures originate at handoff points between agents, not within any single agent, according to Microsoft Research 2025. An agent that scores well in unit evaluation can still corrupt a pipeline by passing malformed state, silently dropping context, or mis-routing an escalation.
This is not a theoretical concern. Klarna's OpenAI-powered customer service agent handled 1.3 million conversations per month after its February 2024 launch, cutting average resolution time from 11 minutes to 2. By mid-2025, Klarna had rebalanced toward a hybrid model because pure-agent resolution on complex cases showed measurable quality issues. The agent performed well on the tasks it was benchmarked on. The failure mode was a category of case the benchmark did not adequately represent.
Evaluation design needs to answer a prior question: which inputs does your benchmark not cover, and what happens when they arrive in production?
What to instrument instead
Effective production evaluation rests on three practices that work together.
Trace-based evaluation
You cannot score what you cannot see. Trace-based evaluation attaches structured spans to every step an agent takes: tool calls, retrieval queries, LLM calls, branching decisions, handoffs. Each span carries inputs, outputs, latency, token counts, and error state.
This is the model Prefactor uses: the SDK instruments agent runs and records spans that are then scored against quality dimensions and validated against activity schemas. The audit trail is a byproduct of the instrumentation, not a separate logging pass.
Without trace-level data, you are scoring outcomes without understanding the path that produced them. An agent that reaches the correct answer via a hallucinated intermediate step will fail the next time that step is exposed to a different input.
Multi-dimensional scoring
Collapsing agent quality into a single score is how benchmarks mislead. Production agents need scores across at least three dimensions simultaneously: task accuracy, cost efficiency, and risk profile.
Morgan Stanley's DevGen.AI agent, which reviewed 9 million lines of code and saved an estimated 280,000 developer hours between January and June 2025, did not succeed because it optimized a single metric. Code review quality, flag accuracy, and latency were all tracked against baselines derived from senior engineer review.
The cost dimension matters as much as quality. A 95% accurate agent that costs 12x more per task than its target is not a production-ready agent. Separate scoring surfaces these tradeoffs rather than hiding them in an average.
Hybrid human-in-the-loop validation
Automated scoring has hard limits. Naive LLM-as-judge evaluation setups show error rates above 50% due to position bias, length bias, and agreeableness bias. An LLM evaluator will often score a plausible-sounding wrong answer higher than a terse correct one.
The fix is not to abandon automated evaluation. It is to calibrate it. Human reviewers score a stratified sample, automated judges score everything, and the human scores are used to continuously recalibrate the automated judges. This is the pattern behind the insurance claims pipeline described in this case study collection: a 7-agent pipeline with independent monitoring and human escalation at each stage, which achieved a 30% reduction in operational costs without sacrificing compliance auditability.
The staffing cost of full human review does not scale. The cost of a calibration sample does.
Continuous evaluation versus periodic evaluation
Scheduling an evaluation run before each deployment is table stakes. It does not catch the drift that accumulates between deployments: model updates from your LLM provider, schema changes in upstream APIs, seasonal input shifts, and gradual prompt degradation.
Deloitte's 2025 AI Ops Maturity research found that continuous evaluation reduces production incidents by 67% compared to periodic evaluation. The mechanism is early detection: a score that drops from 0.91 to 0.83 over 48 hours is a detectable signal before it becomes a customer complaint.
Detecting quality decay before it causes user-facing failures requires at minimum: per-request scoring on a defined quality rubric, alerting thresholds tied to score bands rather than binary pass/fail, and a clear record of what changed and when. The last point matters for regulated industries where audit trails are not optional.
JPMorgan Chase runs 450 or more AI agent use cases in daily production. Governance and integration patterns, including continuous monitoring, are cited as the mechanism that makes that scale manageable. An audit trail is not only a compliance artifact; it is the primary diagnostic tool when a score drops unexpectedly.
The metrics that predict production success
To summarize the instrumentation pattern that holds across these deployments:
- Step-level accuracy across the full task graph, not just the terminal output
- Handoff success rate at every agent boundary in multi-agent pipelines
- Cost per task against a defined efficiency budget, tracked per agent and per workflow
- Escalation rate and escalation outcome, to measure whether human-in-the-loop triggers are calibrated correctly
- Score trend over time, not just point-in-time snapshots
If your evaluation harness does not produce all five of these, you are flying partially blind. The step-level accuracy article covers the cascading failure dynamics in more depth.
For multi-agent orchestrators specifically, handoff success rate is often the single most predictive metric for production stability. Score the individual agents all you like; if the coordinator is silently dropping context at 3% of handoffs, the pipeline will degrade in ways that are hard to trace after the fact.
Where to start
Pick one production agent, instrument it with span-level tracing, and score its last 500 runs across task accuracy, cost, and at least one risk dimension. Compare the distribution to your pre-deployment benchmark scores. The gap you find is the baseline you actually have to work with.
Start evaluating your agents or read through the docs to see how Prefactor's SDK instrumentation and scoring pipeline map to the patterns described here.