What you get from span-level validation before deployment
Testing an agent's outputs tells you whether the final answer looked right. It does not tell you whether the agent took the right steps to get there, called the right tools in the right order, or stayed within the scope you defined. Span-level validation closes that gap by checking each recorded step in an agent's execution trace against a schema that describes what the agent is supposed to do, in what order, with what constraints.
This article explains how to instrument agent behavior at the span level, what activity schemas need to contain to be useful, and how scoring across spans gives you a signal that pass/fail output testing cannot.
Why output testing is not enough
Two incidents from the first week of August 2026 illustrate the problem clearly.
New Orleans went live in July 2026 with Carbyne's AI triage system for 911 dispatch. The system handles 15 to 18 of every 20 calls about a single car crash, filtering duplicates and prioritizing incidents before a human dispatcher picks up. The system was trained on three months of local recordings to reduce accent misrecognition. That is a narrow, well-defined task, and its output, a triage classification, looks simple. But the steps that produce that classification involve audio processing, intent inference, deduplication against live call queues, and a priority assignment. Each of those steps can produce a plausible-looking output while getting the reasoning wrong. A dispatch system that correctly labels a call as "duplicate" for the wrong reasons is a liability, not an asset.
The same week, reports emerged that Google Gemini was leaking private Google Docs details in agent outputs. The agent's final answers looked coherent. The problem was upstream: the agent was retrieving and surfacing document content it should not have accessed. Output evaluation would score those answers as high quality. Span-level review of the retrieval steps would have flagged the unauthorized access immediately.
These two cases point at the same structural failure. When you only measure what the agent says, you miss what it did.
For more on the gap between what agents appear to do and what they actually do, see measuring what agents actually do versus what you think they do.
What spans record and why that matters
A span is a discrete unit of work in an agent's execution trace: a tool call, a model inference, a retrieval, a state update. OpenTelemetry-compatible instrumentation captures these as structured records with a start time, end time, inputs, outputs, and metadata. The OpenTelemetry standard gives you a common schema across frameworks and model providers.
The span record for a retrieval step in a document agent should include which source was queried, what query was issued, what results came back, and whether any access policy was checked. The span record for a tool call in a dispatch agent should include which tool was invoked, with what arguments, and what the tool returned. Neither of those things is visible in the final output.
When you instrument at the span level, you can answer questions that output evaluation cannot: Did the agent access a resource it was not authorized to use? Did it call a tool out of sequence? Did it skip a required verification step? Did it retry a failing tool call more times than the schema allows?
RAG retrieval agents are particularly exposed here because their span-level behavior, specifically what they retrieve and from where, is entirely invisible in a clean final answer.
Activity schemas: what they need to contain
An activity schema is a specification of what a given agent is supposed to do. It describes the expected sequence of span types, the permitted tools and data sources, the required checks, and the conditions under which the agent should escalate or stop.
A useful schema for a triage agent in a 911 context would specify: the agent must perform deduplication before priority assignment; the agent must not modify any call queue record without a human confirmation span in the trace; the agent must complete classification within a defined latency window. A schema for a document summarization agent would specify: retrieval spans must reference only sources in a permitted list; the agent must not include source identifiers from outside that list in the output; tool calls to external APIs are not permitted.
Schemas written at this level let you validate execution traces mechanically, span by span, before you put the agent in front of production traffic. The validation is not about whether the answer was good. It is about whether the agent's behavior was consistent with what you authorized it to do.
This is the foundation of validating agent behavior against expected outcomes, which becomes more important, not less, as agents acquire more tools and longer execution chains.
Scoring quality across spans, not just at the endpoint
Pass/fail schema validation catches hard violations: a tool that was never supposed to be called, a retrieval from a forbidden source. But many real failures are softer. The agent called the right tools in the right order, but the retrieval quality was poor, the model's reasoning span contained a factual inconsistency, or the agent used 14 tool calls to do what the schema expected in four.
Scoring across spans addresses this. You assign a quality score to each span type based on what you expect from it: retrieval precision, reasoning coherence, tool argument correctness, latency relative to the defined window. You aggregate those scores across the trace to get a run-level quality signal. A run that passes schema validation but scores low on retrieval quality is worth examining before it goes to production.
Step-level accuracy and trajectory evaluation for production agents covers the mechanics of this in detail. The key point here is that span-level scoring gives you a dimensional view of agent quality that a single output score cannot provide.
Prefactor records spans via SDK instrumentation, scores each span against configurable rubrics, and validates traces against activity schemas. The audit trail it keeps is per-run and per-span, which means you can see exactly where in a trace a quality problem originated, not just that the final output was wrong.
The deployment pressure problem
81% of respondents in a June 2026 survey reported feeling pressure to deploy AI agents quickly, even when security or governance is not fully in place. That pressure is real, but the consequences of skipping pre-deployment validation are also documented. Only 14.4% of agents that went live in 2026 received full security and IT approval during deployment.
The Step Finance incident in January 2026 shows what that gap costs. AI trading agents on the Solana DeFi platform executed 261,854 SOL in transfers, between $27 and $40 million, after an executive device was compromised. Only $4.7 million was recovered. The agents did exactly what they were designed to do. The problem was that no schema existed to constrain the conditions under which they could do it, and no span-level validation was in place to catch that the authorization trace was abnormal.
Span validation would not have prevented the device compromise. But an activity schema that required a confirmed human authorization span before any transfer above a defined threshold would have stopped the execution chain at the right point.
For teams managing multiple agents across different risk levels, the agent risk audit is a practical place to understand where your current exposure sits.
Pre-deployment validation is also not a one-time gate. Detecting agent quality decay and production drift describes how to keep schema validation running continuously once an agent is live, so that model updates or data drift do not silently degrade behavior that passed pre-deployment checks.
Ghost actions, cases where an agent does something nobody requested, are another category of span-level failure that output scoring will not catch.
The silent failure gap in agent quality evaluation before production covers the broader pattern of what gets missed when teams stop at output-level testing.
Where to start
Define one activity schema for your highest-risk agent, write down the span sequence it should follow, the tools it is permitted to use, and the checks it must include. Instrument that agent with span-level tracing and validate one week of traces against the schema before expanding coverage.
Start evaluating your agents and read the docs to see how Prefactor records spans, applies schema validation, and scores run quality across traces.