What separates a pilot from a production agent
Most agent pilots work. In a controlled setting, with curated inputs and an engineer watching the output, the agent does what it was built to do. The problem surfaces when you remove those conditions: varied inputs, concurrent sessions, downstream systems that behave differently on Tuesdays, and no one watching. That is when the gap opens, and for most teams, it stays open.
A September 2026 analysis put the number at 88% of enterprise agent pilots failing to reach production, despite a 78% pilot success rate. The pilots are not failing because the models are bad. They are failing because the instrumentation that would tell you whether the agent is safe to ship does not exist yet, and the teams building these pilots are not building it.
This article maps the three specific gaps, span recording, quality scoring, and behaviour validation, that account for most of that drop-off.
Gap one: no span recording means no root cause
When an agent fails silently in production, the first question is: what did it actually do? If you recorded only the final output, you cannot answer that. You need a trace: which tools were called, in what order, with what inputs, and what each returned.
A 2026 LangChain survey of more than 1,300 respondents found that 89% of teams with agents in production had adopted some form of observability, but only 52% had evaluation in place. That 37-point gap means most teams can see that something went wrong, but cannot tell you why, or whether it is going wrong right now. Watching spans is not the same as evaluating them.
Span recording at the step level matters because agent failures are rarely single-step. A retrieval step returns a borderline result. The reasoning step treats it as authoritative. The action step acts on the conclusion. By the time the output is wrong, three steps have contributed, and a log of the final message tells you nothing useful about which one to fix. The Prefactor SDK records individual spans across a session, including tool calls and their return values, so the trace you need for multi-step failure attribution is available before you need to explain an incident.
The eight routing and orchestration tools that launched in the week of September 9, 2026 for managing Claude Code and Codex sessions are worth noting here. Each one adds a layer of indirection between the developer and the model. Indirection is useful for throughput, but every additional hop is a place where a span can go unrecorded. If your orchestrator does not emit spans that downstream evaluation can consume, you have observability of the orchestration layer and a blind spot on everything below it. Teams adopting these tools should verify span propagation end to end before treating their traces as complete.
Gap two: quality scoring is not happening at the right time
The second gap is evaluation timing. Teams typically run evals before deployment, on a fixed test set, and then stop. That approach catches regressions against the test distribution. It does not catch quality decay on the real distribution, which shifts as users, inputs, and upstream data change over time.
The pattern shows up in the production numbers. Consider a financial institution that deployed a multi-agent system for financial reporting: an AnalyzerAgent, a NarratorAgent, and a RiskCriticAgent. Reporting time dropped from 15 days to 35 minutes, cost per report from $2,200 to $9, and error rate from 3 to 0.3 per report. Those gains are real, but they are also specific to the task distribution the agents were built and tested against. Any change to the upstream data format, the reporting schema, or the model version beneath one agent can degrade the others. Without quality scoring in production, you learn about that degradation from a user complaint, not a dashboard.
What distinguishes the pilots that scale is that they treat scoring as a continuous process, not a one-time gate. Detecting quality decay in production requires scoring against criteria that reflect what good actually looks like for each agent type, not just whether the output is syntactically valid. A customer support agent that resolves the wrong issue fluently still passes a format check.
The Cleanlab 2025 survey found that 32% of teams cited quality as their top barrier to production. That barrier is not about model capability. It is about not having a scoring signal that gives the team confidence the agent is behaving at an acceptable level on the real input distribution. Intercom's Fin, which reached a 67% autonomous resolution rate across more than 7,000 customers handling over 40 million conversations, got there because resolution rate is a measurable, continuously tracked outcome, not a one-time benchmark.
This is also where the difference between evals and observability becomes load-bearing. Observability tells you what the agent did. Evaluation tells you whether what it did was good. Both are necessary. Neither substitutes for the other.
Gap three: behaviour validation against schemas is missing
The third gap is the one that creates the most risk in production. An agent that records spans and scores well on average can still take individual actions that are outside its intended scope, and in regulated or customer-facing environments, individual out-of-scope actions are where incidents happen.
Behaviour validation means checking each session against a schema of what the agent is permitted to do: which tools it may call, what parameters are valid, which sequences are expected. This is distinct from output quality. An agent can produce a high-quality output through a sequence of actions that no one authorized.
Bell Canada's deployment of ServiceNow AI Agents achieved a 25% improvement in customer response time and 90% positive feedback on AI accuracy. Getting to those numbers in a production telecom environment, where customer data and billing systems are involved, requires that the agents operate within defined boundaries on every call, not just most calls. A 90% accuracy rate means one in ten interactions is not meeting the standard. Without per-session schema validation, you cannot tell whether those failures are benign or systematic.
Activity schemas are a formal way to express what an agent is supposed to do, and validating sessions against them surfaces the cases where behaviour drifted from intent. Detecting drift through activity schema validation is the mechanism that catches the edge cases that aggregate quality scoring misses. Prefactor validates recorded sessions against activity schemas, flags tool calls that fall outside permitted parameters, and stores the result in an audit trail. This is what agent behaviour validation before deployment looks like in practice.
For teams running multi-agent orchestrators or autonomous background agents, this matters more, because the number of possible action sequences grows faster than the test matrix. Schema validation scales with agent complexity in a way that manual review does not.
What the successful pilots have in common
The pilots that reach production share a structure. They instrument from the beginning, treating span recording as a prerequisite rather than a retrospective addition. They score quality on the live distribution, not just the test set. And they validate each session's behaviour against a defined schema before and after deployment.
Evaluating agent readiness for autonomous deployment is not a single gate. It is a continuous process. Teams that treat it as a launch checklist discover the gaps when users do.
The gap between a 78% pilot success rate and 88% of pilots never reaching production is not a model problem. It is an instrumentation and validation problem, and it is solvable with the tooling that exists now.
Where to start
Pick one agent that is currently in pilot and instrument it at the span level. Score its outputs against the criteria you would use to judge a human doing the same task. Then define an activity schema and validate a week of sessions against it.
Start evaluating your agents and the docs cover the SDK setup, schema format, and scoring configuration in detail.