What separates a deployed agent from a production-ready one
Ninety-seven percent of companies have deployed at least one AI agent, but only 11% operate agents at scale in production. That gap is not a confidence problem or a strategy problem. It is an instrumentation problem. Teams that stay stuck between isolated testing and fleet deployment usually share one trait: they cannot tell, with any precision, what their agents are actually doing once they leave the controlled conditions of a sandbox.
This article walks through the three layers of observation infrastructure that move agents from proof-of-concept into production: span instrumentation to establish baseline behavior, schema validation to catch divergence early, and incident scoring to decide which behaviors require remediation first. If you are running agents in a handful of workflows and wondering why scaling feels unsafe, the answer is usually that one or more of these layers is missing.
Why proof-of-concept conditions do not transfer
A proof-of-concept agent succeeds under a narrow set of inputs, reviewed by the team that built it, in an environment where failure has no downstream cost. Production is different on all three dimensions. Inputs are unpredictable, reviewers are not watching every trace, and failures propagate.
The consequences are not hypothetical. Sixty-five percent of organizations experienced at least one agent-related incident in the past year. The incident rate matters less than what it tells you about preparedness: most teams discovered the failure after it had already affected a downstream system, a user, or a record. The observation layer that would have caught it earlier either did not exist or was not recording the right signals.
Consider what happened with early autonomous coding agents. Cognition's Devin resolved 13.86% of real GitHub issues end-to-end on the SWE-bench benchmark, a result that was genuinely useful and widely cited. But benchmark performance tells you what an agent does on a curated, bounded problem set. It does not tell you what the agent does when the repository is larger, the ticket is ambiguous, or a downstream tool returns an unexpected response. Production gaps like these are documented in more detail in our analysis of the 37% quality gap between agent benchmarks and production.
The pattern repeats across domains. A European supermarket operator deployed Global AI's agentic platform for supplier invoice lifecycle management and reached a point where the agent received, validated, and recorded invoices without manual intervention. That outcome required not just a capable model, but a defined set of expected behaviors that the team could verify at each processing stage. Without that schema, "automated" means "unsupervised," which is a different thing.
Layer one: span instrumentation
A span is a timed record of one unit of work inside an agent's execution, a tool call, a retrieval step, an LLM completion, a handoff to a subagent. Spans compose into traces, and traces are what make agent behavior legible after the fact.
The goal at this layer is not to capture everything. It is to capture the signals that let you answer: did the agent take the steps I expected, in the order I expected, with the inputs and outputs those steps should produce? Without spans, you are inferring behavior from final outputs alone, which means cascading failures inside a multi-step workflow can remain invisible until a downstream system surfaces them.
OpenTelemetry provides the most portable instrumentation standard for this. Spans annotated with semantic conventions for agents carry enough structure that a downstream scoring system can evaluate them without bespoke parsing for each agent type. Prefactor's SDK instruments spans at the tool-call level, recording inputs, outputs, latency, and model parameters, which gives the evaluation layer something concrete to score rather than free-form logs.
The baseline you establish here is the reference for everything that follows. If you do not know what a healthy trace looks like for your invoice-processing agent, you cannot detect when a trace is diverging from it. This is covered in more depth in our guide to detecting agent quality decay and production drift monitoring.
Layer two: schema validation
Span data tells you what happened. Schema validation tells you whether what happened matches what was supposed to happen.
An activity schema defines the expected structure of an agent's behavior for a given task type: which tools should be called, in which sequence, what the output of each step should look like, and what conditions should block or halt execution. Validating a trace against that schema turns a stream of spans into a pass or fail result with a specific location for the divergence.
This matters most in multi-step workflows, where a small deviation early in a trace compounds. A Global AI deployment for a European insurance group ran multiple invoice-processing cycles daily, with full auditability as a stated requirement. Auditability at that frequency means the system must detect when a cycle deviated, not just log that it ran. Schema validation is the mechanism that makes "auditable" mean something operationally, not just archivally.
For teams building multi-agent orchestrators or autonomous background agents, schema validation is especially important because the surface area for divergence grows with each subagent in the graph. A subagent that calls a tool it was not expected to call does not always produce a wrong final answer; it may produce a plausible one through an unintended path. Without schema validation, that behavior is invisible.
Our longer treatment of this problem is in validating agent behavior against expected outcomes. The short version: define the schema before you instrument, not after. Schemas derived from observed production behavior tend to encode the anomalies you were trying to detect.
Layer three: incident scoring
Not every schema violation requires the same response. Some deviations are noise: edge cases the schema did not anticipate, benign variations in tool-call order that do not affect output quality. Others are material: an agent writing to a record it should only read, a subagent calling an external endpoint outside its authorized scope, a retrieval step returning content from the wrong index.
Incident scoring assigns a severity to each divergence so that the team can triage rather than audit every trace manually. The inputs to a score are typically the type of divergence (schema violation, policy violation, output quality failure), the blast radius (how many downstream steps or systems were affected), and the reversibility of the action taken.
Ghost actions, where an agent takes steps nobody requested, are one of the harder categories to score without this infrastructure, because the evidence is often indirect. An agent that calls a tool once more than expected may be hallucinating a step or recovering from an error. Scoring tells you which interpretation is more likely given the surrounding trace.
Leena AI reported ticket resolution rates above 70% for HR, IT, and finance workflows at customers including Coca-Cola, which reduced approval sign-off time by 40%. Results like those depend on a system that can distinguish a resolved ticket from a ticket that was closed without being resolved. That distinction requires scoring on output quality, not just on whether the agent completed its steps. The underlying measurement approach is detailed in what to measure and how to prove it in production agent evaluation.
Prefactor scores spans for quality and risk on both dimensions: structural conformance to the activity schema, and a quality rating on the output itself. Scores are stored alongside the trace, which gives you an audit trail that answers "what did the agent do and how well did it do it" rather than just "did it finish."
Connecting the layers
The three layers are not independent. Schema validation requires span data. Incident scoring requires schema validation results. Teams that try to implement scoring before instrumentation find themselves scoring final outputs only, which catches failures late and misses the step-level information needed to understand why something went wrong.
The step-level accuracy and cascading failures analysis covers what happens when evaluation stops at the output level. The short answer is that cascading failures inside a trace look like a single output failure, which makes remediation harder and regression testing nearly impossible.
For teams weighing where to start, the agent risk audit tool gives you a structured way to assess which gaps in your current instrumentation carry the most exposure. Teams running customer-support agents or internal-ops agents in particular tend to have output-level evaluation in place but no step-level schema validation, which leaves the middle of the pipeline unobserved.
Where to start
Pick one agent in production, instrument its spans, write a schema for its expected behavior, and run a week of traces against it. The violations you find will tell you more about your production readiness than any pre-deployment test suite. You can add scoring after you have a baseline.
Start evaluating your agents or read through the docs to see how span instrumentation and schema validation are configured.