Per-turn evaluation assigns scores to individual steps inside an agent conversation, not just the final output. Final-outcome benchmarks hide the wrong retrievals, bad tool calls, and context drift that accumulate mid-trajectory, which is where 67% of documented production agent failures actually occur.
The gap between benchmark success and production failure
When a team ships an agent, the first instinct is to measure what the agent ultimately produced: did the ticket get resolved, did the code compile, did the order get placed. That instinct is correct for acceptance testing, but it misses most of what goes wrong in production.
A 37% performance gap between controlled lab benchmarks and real-world enterprise deployments is well documented, and part of why that gap exists is structural. Benchmark datasets test end states. Production agents navigate trajectories: they call tools in sequence, carry context across turns, and make decisions whose consequences only surface several steps later. A final-outcome score of "pass" can sit on top of a trajectory full of wrong retrievals, redundant tool calls, and context that drifted two turns before the answer landed.
According to a Deloitte AI Ops Maturity report, continuous evaluation reduces production incidents by 67% compared to periodic evaluation, and teams that stop monitoring after launch experience quality degradation within 30 to 60 days. Those numbers describe exactly what happens when per-turn assessment is absent: the agent looks fine at launch, then quietly degrades as real traffic diverges from the benchmark distribution.
What per-turn evaluation actually measures
Per-turn evaluation assigns a score to each discrete step in an agent's execution, not just the final output. The steps worth scoring include individual LLM calls within a conversation, tool invocations and their outputs, retrieval events in RAG pipelines, and state transitions between stages of a multi-step workflow.
Each step produces a span, a structured record containing the inputs, outputs, latency, and any metadata the instrumentation layer captures. Scoring happens against that span at runtime, not in a post-hoc batch. You can apply several types of scorer in parallel:
- Faithfulness scorers check whether the agent's output in a given turn is grounded in the context it actually received, catching hallucinations that occur mid-trajectory before they compound.
- Tool-call validity scorers verify that parameters passed to external tools are well-formed and within the declared schema, catching argument errors that a final-outcome check would never see if the tool returned a partial result anyway.
- State-consistency scorers compare what the agent believes about the world at turn N against what it knew at turn N-1, surfacing context drift in long conversations.
- Policy scorers check each action against a declared behavior schema, for example verifying that a customer-facing agent never attempts to access account data outside its authorized scope.
This last category matters for autonomous background agents and multi-agent orchestrators especially, because those architectures chain decisions across multiple systems where a single mid-trajectory policy violation can propagate to several downstream agents before anything surfaces in a final output.
Why silent failures cluster in the middle
Klarna's customer service agent handled 2.3 million chats in its first month, equivalent to 700 full-time staff, and reduced average resolution time from 11 minutes to 2 minutes. Those outcome metrics looked strong. The CEO acknowledged in May 2025 that an AI-only approach had caused quality problems on complex cases, problems that final-outcome scoring had not caught in time. The failures were happening inside conversations, not at their ends.
This is a pattern rather than an exception. Shopify's Sidekick evolved from single-action tool calls into open-ended multi-step agentic workflows as of Winter 2026. As agent behavior becomes more open-ended, the number of decision points per session increases, and each one is an opportunity for a silent failure to accumulate before the final state is reached.
BNY Mellon's Eliza platform runs 130 or more digital employees with system credentials and autonomous workflows, and every model in production must pass explainability benchmarks and model-risk review. That requirement exists precisely because autonomous, credentialed agents taking intermediate actions against live systems cannot be evaluated only at the end. The intermediate actions are the risk surface.
DHL's HappyRobot voice agents handle phone and email interactions for appointment scheduling and driver follow-up across warehouse operations. A voice agent navigating an unstructured phone call has no clean final state to score: the conversation itself is the product, and quality lives in the turns.
What cascading failure looks like in practice
A step-level accuracy problem in a multi-turn agent typically begins with a retrieval or context error at turn two or three of a session. The agent proceeds because its local confidence in the next action is still high. Each subsequent turn inherits the error. By turn seven the agent has constructed a response that is internally coherent but factually wrong, because it built on a flawed premise four turns earlier.
Final-outcome scoring marks the session as failed, but gives you no information about where the failure entered. Without per-turn scoring, you cannot tell whether the problem is in the retrieval configuration, the prompt at a specific stage, a tool returning malformed output, or a model routing decision. Detecting quality decay before it becomes a production incident requires the scored intermediate record, not just the endpoint.
Algolia's June 2026 analysis found that 60% of AI production failures trace to data quality, context, or governance problems rather than model limitations. Context problems are, by definition, mid-trajectory problems. They show up in spans, not in final outputs.
Implementing span-based scoring in practice
The instrumentation layer determines what you can score. An OpenTelemetry-based tracing setup lets you capture spans from each agent turn as they execute, attach metadata like model version, tool name, and token count, and route those spans to a scoring layer without modifying your core agent logic.
Each span then gets scored asynchronously against the evaluators you configure. The scores accumulate into a session-level record that shows the quality trajectory of the conversation, not just its endpoint. When a score drops below a threshold at turn three, an alert fires before the session concludes. You can route that session to human review, trigger a fallback, or log it for later analysis depending on your risk tolerance.
Prefactor instruments this layer via SDK, recording each span with its inputs and outputs, then scoring each turn against the criteria you define, including faithfulness, tool-call validity, and your own policy schemas. The audit trail stores every scored span, so you can reconstruct any session's trajectory for review or for compliance reporting in regulated industries.
For teams building RAG retrieval agents, per-turn scoring is where you catch retrieval failures that do not manifest as obviously wrong final answers but do manifest as low-faithfulness scores on the generation turn immediately after the retrieval. That signal is not available without span-level evaluation.
The offline-to-online evaluation gap is real and it widens as agents handle more complex, multi-turn tasks. Closing it requires evaluation that runs on every turn of every live session, not on curated test sets run before deployment.
Where to start
Pick one agent in production, instrument it to emit a span for each turn, and attach a faithfulness scorer to the generation spans. That single addition will surface mid-trajectory failures that your current metrics do not capture. Expand to tool-call and policy scorers once you have baseline data.
Start evaluating your agents and review the docs for SDK setup and scorer configuration.