What you will learn here
Static benchmarks tell you how an agent performs on a fixed dataset under controlled conditions. They do not tell you whether that agent will hold together when it hits a real workflow, a malformed API response, or a user request it was never explicitly tested against. This article breaks down why that gap exists, what failure modes evaluation tools consistently miss, and which metrics actually matter once agents are compounding decisions across multi-step processes.
Why benchmarks do not predict production reliability
A benchmark measures a single agent completing a single task with a known correct answer. Production agents do neither of those things cleanly. They operate across sequences of steps, call external tools, receive incomplete context, and hand off to other agents. Each of those transitions is a place where error accumulates.
The math here is not subtle. According to Fiddler AI's July 2026 analysis, if each agent in a chain succeeds 70% of the time, a three-agent chain succeeds only 34% of the time overall. That is not a model quality problem. It is a system architecture problem that no single-agent benchmark will surface. The same source reports that production agents fail between 70% and 95% of the time depending on task complexity, and that 88% of agents that work in controlled demos fail when deployed to real workflows.
Those numbers reflect a structural issue: most evaluation happens on the agent in isolation, not on the agent embedded in the workflow it will actually run. Algolia's March 2026 review of enterprise deployments found that 60% of production failures trace back to data quality, context gaps, or governance failures, not to the underlying model. The model passed every pre-deployment eval. The workflow did not.
Deloitte's December 2025 study found that 40% of agentic AI projects are projected to fail by 2027, with legacy system integration and governance gaps cited as the primary causes, not model quality. That figure is worth keeping in mind when scoping what your evaluation program actually needs to cover.
What failure looks like in practice
Runaway cost from unmonitored retry loops
One documented case from 2026 involved a research agent deployed without cost limits or span-level monitoring. A code update introduced an infinite retry loop. The agent ran for 14 hours, made 47,000 API calls, and accumulated roughly $4,200 in costs before anyone noticed. The company's credit card was maxed out, payroll was delayed, and two employees resigned. No benchmark would have caught this because no benchmark tests for cost behavior under pathological retry conditions.
This is not an exotic scenario. Any agent that calls external APIs, retries on failure, and lacks a hard ceiling on call count is exposed to it. The missing control is not complicated: a cost-per-session limit enforced at the infrastructure layer, combined with span-level tracing that records every tool call so a human can see the loop forming before it compounds.
Behavior drift after deployment updates
DPD's customer service chatbot, after a system update in 2026, began producing outputs that were directly contrary to the brand's intent, including insulting its own service in customer-facing conversations. The agent had passed whatever evaluation existed before the update. What failed was the absence of continuous behavioral validation after the change was deployed.
This is the pattern that evaluation-as-a-checkbox produces. The agent is tested once, approved, and then monitored only by customer complaints. By the time a complaint arrives, the damage is already in production. Behavioral drift in deployed agents is common enough that it needs a dedicated detection layer, separate from pre-deployment testing.
Compounding failures in multi-agent systems
Algolia's analysis found that 78% of multi-agent systems showed vulnerability to delegation failures, where one agent hands off an incorrect or ambiguous state to the next, which then proceeds as though the input were valid. The downstream agent is not malfunctioning. It is doing exactly what it was built to do with bad input. Catching this requires tracing the full trajectory of a task across agents, not evaluating each agent in isolation.
For a closer look at how multi-agent orchestration creates new failure surfaces, see our overview of multi-agent orchestrator patterns.
Metrics that actually matter in production
Pre-deployment evals typically measure accuracy on a test set. Production evaluation needs a different set of signals:
- Task completion rate across the real distribution of inputs, not the test distribution
- Step-level success rate within a workflow, so you can see exactly where in a sequence failures cluster
- Cost per completed task, tracked continuously, not sampled
- Tool call frequency and retry rate, flagged when they exceed expected bounds
- Goal drift indicators, assessed against a defined activity schema that describes what the agent is supposed to do
The last one matters more than it might seem. Research cited by GovInfoSecurity in May 2026 found that nearly 90% of agents showed measurable goal drift after approximately 30 steps of operation. An agent can start a task correctly and be doing something subtly different 30 steps later, with no obvious error signal.
Catching goal drift requires comparing agent behavior against a structured specification of expected behavior, not just checking whether individual steps returned non-error responses. This is what activity schema validation is designed to address.
Span-level tracing as the foundation
Every claim above points toward the same underlying requirement: you need a complete, timestamped record of what the agent did at every step, what tools it called, what inputs it passed, what outputs it received, and how much those calls cost. Without that record, you are diagnosing production failures from aggregate logs and user reports.
Span-level tracing, as implemented in OpenTelemetry-compatible instrumentation, records each operation as a discrete span with its own attributes. When an agent fails, you have a full trace rather than a summary. When a retry loop forms, you see it in the span count before the bill arrives. This is the difference between observability and evaluation: both matter, and one does not substitute for the other.
Prefactor instruments agents via SDK to record spans at this level. The platform scores each trajectory against a quality and risk rubric and validates behavior against the activity schema you define. That means failures surface as structured data, not as customer complaints.
For teams running agents in regulated environments or on customer-facing workflows, the audit trail this produces also satisfies a separate requirement: demonstrating to auditors or customers what the agent actually did. See our notes on audit trails in AI agent deployments for more on that.
Salesforce's Agentforce deployment, which delivered 2.4 billion agentic work units across 29,000 customers as of February 2026 and reached $800 million in ARR, is the clearest available evidence that production-scale agent deployment is achievable. It is also a reminder that the gap between a successful pilot and production scale is not closed by the model alone. It is closed by the instrumentation, monitoring, and continuous evaluation that runs underneath.
Related reading
Where to start
Identify one agent currently in production or in a late-stage pilot. Instrument it for span-level tracing, define an activity schema for what it is supposed to do, and start tracking task completion rate and cost per session against real inputs. Those three measurements will show you more about where the reliability gap lives than any benchmark score will.
Start evaluating your agents or read through the docs to see how instrumentation and schema validation work in practice.