What you will learn here
Most agent evaluation stops at the output. Did the agent produce the right answer? If yes, mark it passed. That approach misses a category of failure that only shows up when you examine the steps between prompt and result: wrong tool calls, hallucinated intermediate data, unnecessary retries, and reasoning that reaches the correct answer through a path that will break on the next slightly different input. This article explains the three layers of agent evaluation, why final-answer scoring produces a misleading pass rate, and how to implement trajectory-level evaluation without making every production run expensive.
Why task completion scores lie
A task completion score measures one thing: whether the final output matches the expected output. It tells you nothing about how the agent got there.
Consider what happened at an unnamed B2B SaaS company running a research analyst agent for competitive intelligence. Final-output scoring passed 95% of runs, but step-level trajectory evaluation revealed that roughly one in three runs contained fabricated competitor pricing. A single wrong extraction step early in the pipeline was cascading downstream through summarization and formatting stages, which had no way to detect that their input was invented. The final report looked plausible. The data was not.
This is the core problem with final-answer evaluation: it cannot distinguish a reliable result from a lucky one. An agent that took three wrong turns, self-corrected inconsistently, and happened to land on the right answer will score identically to an agent that followed a sound reasoning path. At low scale that is a statistical annoyance. At production scale it is a reliability problem.
The math compounds quickly. At 95% per-step accuracy across a 20-step task, the probability of end-to-end failure is approximately 36%. Across 50 steps, success drops below 8%. A score of 95% sounds like a well-behaved agent. The trajectory tells a different story.
The three layers of agent evaluation
Final-answer scoring checks whether the output is correct. It is the cheapest layer to implement and the one most teams have in place. It is necessary but not sufficient.
Trajectory evaluation checks whether the sequence of steps, tool calls, and intermediate outputs followed a valid path to that answer. This is where silent failures become visible. A trajectory scorer asks: did the agent call the right tools in a reasonable order, did it avoid fabricating intermediate results, and did it recover from errors in a way that a human reviewer would accept?
Per-turn scoring applies quality and risk checks at each individual step. This is the most granular layer and the most expensive to run at full fidelity. It is most useful for multi-agent orchestrators and long-horizon tasks where a failure at step three has a compounding effect on everything that follows.
Prefactor's instrumentation sits at the trajectory and per-turn layers: it records spans for each tool call and reasoning step, scores them against a schema, and stores the full trace so you can audit what actually happened, not just what the agent reported.
What trajectory evaluation catches that final-answer scoring misses
There is a measurable gap. Agents evaluated only on final-output quality pass 20 to 40% more test cases than full trajectory evaluation reveals. That gap represents failures that shipped.
One class of failure is path deviation: the agent reached the correct answer through an invalid or fragile path. A customer support agent that resolves a query by hallucinating a policy detail that happens to be accurate this week will fail when the policy changes, and final-answer scoring will never flag it.
A second class is looping and retry waste. A supply chain company running a forecasting and procurement agent used trajectory-level evaluation to detect looping and retry patterns that were invisible at the output layer. The evaluation feedback loop contributed to 20 to 40% reductions in forecast error and a 31% average inventory reduction. The final outputs before that work had looked acceptable.
A third class is non-determinism across runs. Agents achieving 60% single-run task completion drop to roughly 25% success across eight runs, a figure the pass-at-one metric cannot surface. If your quality gate only runs each test case once, you are measuring best-case performance, not production reliability. This matters most when you are moving from pilots to production and need defensible quality gates.
How to implement trajectory scoring without breaking your budget
Full per-step LLM-as-judge evaluation on every production run is expensive enough to be impractical for most teams. The approaches that work at scale combine coverage and cost control.
Sample strategically. Run full trajectory evaluation on a fixed percentage of production traffic, perhaps 5 to 10%, and flag any run that exceeds a cost or step-count threshold for automatic deep evaluation. Offline-to-online evaluation gaps are real: 52.4% of organizations run only offline evaluations, according to a LangChain survey of 1,340 practitioners. Sampling closes that gap incrementally without requiring you to score everything.
Define activity schemas before you deploy. A schema specifies the expected tool call sequence, the acceptable range of intermediate outputs, and the conditions under which a step is valid. Validating against a schema is cheaper than running an LLM scorer on every step, and it catches structural failures, wrong tool order, missing calls, unexpected retries, that a general-purpose scorer might miss. This is what validating agent behavior against expected outcomes looks like in practice.
Use human review as a calibration signal, not a replacement. IBM's AskHR agent, which handles employee queries across more than 70 business areas and has reached a 94% containment rate across 270,000 employees, combines automated evaluation with targeted human review on edge cases. Human review at scale is a calibration tool: you use it to tune your automated scorers, not to substitute for them.
Track lucky wins separately. A trajectory that reached the right answer through a path that includes a hallucinated step, an unexpected tool call, or a self-correction that depended on information the agent should not have had is a lucky win. Aggregate your lucky win rate and trend it over time. If it is rising, your reliability is degrading even if your task completion score holds. This is closely related to detecting agent quality decay in production.
The metrics that distinguish reliable performance from lucky outcomes
Task completion rate remains useful as a baseline. Pair it with at least three trajectory-level metrics.
Step validity rate: the fraction of steps across all runs that pass schema validation and scorer review. This is the per-step accuracy figure whose compounding effect is described above.
Path efficiency: the ratio of steps taken to the minimum steps required for that task class. An agent that consistently takes twice as many steps as necessary is either looping, self-correcting for errors it introduced, or reasoning inefficiently. All three are cost signals as well as quality signals.
Multi-run consistency: the variance in trajectory structure across repeated runs on the same input. High variance on deterministic tasks is a reliability risk. Salesforce tracked this metric during the Agentforce deployment on help.salesforce.com, which reached an 83% resolution rate and a 1% human escalation rate across 32,000 conversations per week, processing 4.3 million inquiries autonomously. Consistent trajectories at that scale require evaluation infrastructure, not just output scoring.
Escalation accuracy: for agents with human handoff, the rate at which the agent correctly identifies that a case exceeds its capability. An agent that escalates the wrong cases is wasting human review time. An agent that fails to escalate when it should is producing silent failures that compound until something breaks visibly.
Where to start
Pick one task class your agent handles in production, define a schema for its expected trajectory, and run trajectory evaluation against a sample of last week's traces. Compare the trajectory pass rate to your current task completion score. The gap is what you are not seeing.
Start evaluating your agents and read the docs for setup guidance on trajectory instrumentation and schema validation.