Why does an agent's success rate drop so sharply across multiple runs if it passes single-run evals?

Each step in a multi-step agent introduces its own failure probability. If a 10-step agent has 95% per-step accuracy, the probability of a fully correct end-to-end run is roughly 60%. Across eight independent runs with any variation in context, that compounds further. Single-run benchmarks never expose the cumulative effect.

What is a trajectory metric and how is it different from an outcome metric?

An outcome metric scores the agent's final output against a reference answer. A trajectory metric scores each intermediate step, including which tools were called, in what order, and whether each call produced valid output before passing it downstream. Trajectory metrics catch process failures that produce correct-looking final outputs by luck or coincidence rather than reliable reasoning.

How do I know which steps are worth instrumenting first?

Start with steps that are either irreversible or that feed directly into the agent's next decision. Retrieval calls, external API calls, and any step that writes data are the highest-leverage points. A failure at those steps propagates through everything downstream, so catching them early has a disproportionate effect on overall reliability.

Can existing observability tools replace dedicated agent evaluation?

Standard observability tools record that a call happened and whether it returned an error code. Agent evaluation scores whether the call's output was semantically correct for the task at that step, which requires a different kind of instrumentation. The two complement each other: observability surfaces infrastructure failures; evaluation surfaces quality and behavioral failures.

Why AI agents pass evals but fail in production

What evaluation frameworks are built for, and where they fall short

Most evaluation frameworks measure whether an agent reached the right answer. They record inputs and outputs, compare the output to a reference, and report a score. That works when a task has one correct path. It misses most of what goes wrong in production.

The gap shows up in the numbers. Research from Galileo and Google Cloud Vertex AI found that enterprise agents achieve roughly 60% success on single-run evaluations but only 25% across eight consecutive runs. Standard benchmarks run each task once. Production systems run tasks thousands of times, often in sequence, often in changed conditions. The 35-point reliability gap sits entirely in that difference.

A VentureBeat survey from July 2026 found that only 5% of organizations fully trust automated evaluation, and 29% report that their evaluations align poorly with real-world outcomes. Those numbers reflect a structural problem, not a tooling gap. The structure is this: benchmarks optimize for correctness, but production failures cluster around process, not conclusion.

The silent failure pattern

A silent failure is an agent that produces a correct-looking output through a broken process. Google Cloud's evaluation team documented the pattern in November 2025: an agent answered a question correctly while drawing on the wrong data source. The answer passed every output-level check. The process was unreliable enough that any variation in the data would have produced a wrong answer, and nothing in the evaluation record would have told you why.

This matters because broken processes do not fail uniformly. They fail when context shifts: when a downstream API changes its schema, when a document in the retrieval index gets updated, when a user phrases a request differently from the training distribution. The agent keeps producing plausible outputs right up to the moment it produces a harmful one, with no intermediate signal visible in outcome-only logs.

The offline-to-online evaluation gap is widest for agents that take many steps. A systematic study of 306 practitioners across 26 domains found that 68% of production agents execute at most 10 steps before requiring human intervention. When an agent is stopped that early, the reason is usually not a wrong final answer. It is a step-level failure: a retrieval that returned stale data, a tool call made in the wrong order, a decision node that was skipped.

Where Klarna's numbers pointed

Klarna's customer service agent handled roughly 1.3 million conversations per month by mid-2025, resolved inquiries in 2 minutes versus the previous 11, and cut repeat contacts by 25%. Those are outcome metrics, and they looked good. The underlying picture was more complicated: quality degraded on complex cases, eventually requiring reintroduction of human agents. The outcome metrics did not surface the degradation until it had already affected a meaningful share of interactions.

The lesson is not that the deployment failed. It is that outcome metrics lag. By the time a quality problem shows up in resolution times or repeat contact rates, the failure mode has been accumulating for days or weeks. Detecting quality decay early requires metrics that run closer to the process: step acceptance rates, retrieval precision per query type, tool call success rates broken out by context.

What trajectory evaluation catches instead

Trajectory evaluation scores each step in an agent's execution path, not just the terminal output. It records which tools were called, in what order, with what inputs, and whether each call succeeded on its own terms before passing output to the next step. A trajectory score can tell you that an agent reached the right answer via a detour through three redundant retrieval calls, or that it skipped a validation step 12% of the time without affecting the answer in your evaluation set but in ways that would matter at the tail of your production distribution.

Morgan Stanley's DevGen.AI deployment reviewed 9 million lines of legacy COBOL code and saved developers 280,000 hours since its January 2025 launch. At that scale, even a 1% step-level error rate compounds. If a code translation agent misclassifies a variable scope on one step, the specification it produces is wrong in a way that a downstream developer may not catch until integration testing. Outcome metrics report translation volume. Trajectory metrics report whether the classification step is consistent.

Step-level accuracy and cascading failures are the specific failure mode that trajectory evaluation is designed to surface. The cascade structure means that a step with 95% accuracy in a 10-step pipeline produces a correct end-to-end result only 60% of the time. That is the arithmetic behind the 60%-to-25% reliability gap.

What to instrument

Four categories of signal close most of the gap between benchmark scores and production behavior:

Step-level outcomes. Record whether each tool call, retrieval, or subagent invocation succeeded, with the actual inputs passed and outputs returned, not summaries.
Retrieval quality. For RAG-based agents, score whether the retrieved context was relevant to the query that triggered it. A correct answer drawn from irrelevant context is a silent failure waiting to surface.
Path conformance. Compare the actual execution path to the expected path for that task type. Agents that reach the right answer via unexpected paths are less reliable than their output scores suggest.
Behavioral drift over time. Run the same canonical prompts weekly and track whether step-level scores shift. Agents that degrade gradually rarely announce the change in their final outputs.

AMD's HR agent for 30,000 employees achieved an 80% reduction in resolution time and resolved 50% of queries via self-service. At that employee count, a retrieval failure that routes a benefits question to the wrong policy document has immediate compliance implications. Instrumenting at the retrieval level, not just the resolution level, is the only way to catch that class of failure before it reaches the employee.

Prefactor records spans at each step of an agent's execution, scores them against activity schemas you define, and surfaces deviations in the audit trail. That gives you the trajectory record that outcome-only logging does not produce. The agent evaluation post covers the measurement categories in more detail.

The broader instrument-first approach applies regardless of platform. Validating agent behavior against expected outcomes requires a record of what the agent actually did at each step, not only what it returned. Without that record, you are debugging production failures from the output backward, which is slower and less reliable than catching them at the step where they originate.

Where to start

Pick one agent currently in production and add step-level logging to its three most consequential tool calls. Compare the step success rates you observe against the outcome score you already track. The difference between those two numbers is the failure surface your current evaluation misses. Start evaluating your agents with Prefactor's SDK, or read the docs to see how activity schemas and span scoring map to your existing setup.