Why your pass/fail score is the wrong number to watch
If your agent completes a task, you call it a success. If it does not, you file a bug. That binary view works when a workflow has two or three steps, but most production agents do not work that way. A customer support agent collects account details, validates identity, queries a knowledge base, checks policy, selects a response template, and routes or escalates, often in a single session. Each of those is a step, and each step can fail silently while the final output still looks acceptable.
The compounding math is the reason this matters. At 95% per-step accuracy, a 10-step agent workflow succeeds only about 59% of the time, according to Trantor Inc.'s failure mode analysis from May 2026. That is not a theoretical edge case; it is the expected outcome for any agent running a realistic workflow at a quality level most teams would consider good. Measuring only the final result does not tell you which step is dragging the number down, so you cannot fix it.
This article walks through three practices: step-level span evaluation, trajectory metrics, and scaffolded confidence scoring. Together they give you the instrumentation to catch intermediate failures before they compound into task failure.
What step-level span evaluation actually means
A span is a recorded unit of work inside an agent's execution trace. Your telemetry layer, whether you are using OpenTelemetry-compatible instrumentation or a purpose-built SDK, emits one span per action: a tool call, a retrieval, a model inference, a routing decision. Step-level evaluation means scoring each span against a criterion rather than waiting for the session to close.
The practical consequence is that you score things like: did the tool call receive the correct arguments, did the retrieved chunks match the query's intent, did the routing decision respect the policy in effect at that moment. These are not output quality checks; they are behavioral checks at the point of execution.
Salesforce's internal support deployment illustrates the difference precisely. Processing 1.5 million support requests, the system passed outcome metrics consistently, but step-level review surfaced a policy mis-application: overly restrictive competitor guardrails were causing legitimate refusals. The aggregate outcome score showed zero errors. The step trace showed systematic behavioral drift. Those two pictures cannot both be true at the same time, and the step trace is the one you should trust.
Tool misuse and incorrect tool arguments account for roughly 31% of production failures in 2024 and 2025 deployments, and that class of error is primarily caught through step-level validation rather than outcome review. If your evaluation layer only sees the final answer, roughly a third of your failure surface is invisible to it.
For more on why watching agent logs is not the same as evaluating them, see evals vs observability.
Trajectory metrics: scoring the path, not just the destination
A trajectory is the ordered sequence of steps an agent took to reach its output. Trajectory evaluation asks whether that sequence was appropriate given the task, not just whether the output was correct.
OpenTable's deployment with Agentforce found two problems during live testing that outcome metrics had not surfaced during development. The agent was offering inappropriate escalations outside business hours and drifting in its knowledge base usage. Neither failure produced a wrong final answer in the test harness, but both represented a trajectory that violated the intended operating policy. Fixing them required step-level checks, not output rewrites.
Trajectory metrics you can compute from span data include:
- Step count versus expected count. If a task schema says retrieval should happen once and the trace shows three retrieval calls, that is a signal worth scoring.
- Out-of-order execution. Some workflows have a required sequence: validate identity before accessing account data, for example. A span sequence that inverts this is a policy violation regardless of whether the final answer is correct.
- Dead-end recovery rate. When an agent hits a failed tool call or an empty retrieval, how often does it recover cleanly versus produce a degraded output or hallucinate a workaround?
Comparing these metrics across sessions over time tells you whether your agent's behavior is stable. Detecting agent drift through activity schema validation covers the schema-based approach to formalizing what a valid trajectory looks like.
Scaffolded confidence scoring
Confidence scoring at the step level means attaching a calibrated estimate to each intermediate decision, not just the final output. The term "scaffolded" refers to doing this in layers: the model's own probability signal, a secondary evaluator, and a rule-based gate applied in sequence.
The rule-based gate is the cheapest layer and should run first. If a routing step selects an escalation path when the current time is outside escalation hours, the gate fires before the LLM layer runs any further. 1-800-Accountant's tax season deployment handled more than 1,000 engagements in the first 24 hours with a 70% autonomous resolution rate. That rate is only meaningful if the steps leading to resolution were also valid; a verification step passed incorrectly inflates the resolution rate without reflecting actual correctness.
Stripe's multi-stage LLM support system learned this the hard way. Strong offline performance failed in production because intermediate user rejection was not measured. Outcome metrics looked fine; step-level UX friction was the actual failure mode. Scaffolded confidence scoring at each stage would have flagged the low-confidence intermediate steps before they produced user-facing friction at scale.
At Prefactor, the SDK records individual spans and scores them against configurable criteria per step, producing a per-span risk and quality signal rather than a single session score. That signal feeds into the audit trail so you can inspect which specific step in which session first deviated.
For a broader framing of what to measure and how to prove it, see agent evaluation in production. For the specific problem of failures that only appear after deployment, the offline-to-production evaluation gap covers the mechanisms in detail.
Putting it together: from spans to production stability
Step-level evaluation, trajectory analysis, and confidence scoring are not three separate projects. They share the same underlying data: the span trace your agent emits for every session. The investment is in instrumentation and scoring criteria, not in running three parallel systems.
Continuous evaluation reduces production incidents by 67% compared to periodic evaluation, according to Deloitte's AI Ops Maturity research cited in The Thinking Company, with step-level monitoring identified as the key differentiator. The gap between periodic and continuous narrows as your step-scoring logic becomes more precise, which means the criteria you write for each span type compound in value over time.
Multi-step failure attribution and step-level accuracy and cascading failures are worth reading alongside this article if you are beginning to instrument an existing agent rather than a greenfield deployment.
Where to start
Pick one agent in production and instrument its most consequential step, the one where a wrong decision most directly affects the user or the downstream workflow. Score that step on every live session for two weeks and compare the step-level failure rate against your current outcome metric. The gap between those two numbers is the measurement you have been missing.
Start evaluating your agents and consult the docs for SDK setup, span schema configuration, and scoring criterion examples.