When failure looks like success
A completed span with an empty response body is not a success. It is a failure that your monitoring stack may be treating as one.
On August 22, 2025, researchers at Zenodo documented a regression in Claude Opus 4.6 where the model returned zero-byte responses in 900 of 900 test cases. Not errors. Not refusals. Not safety blocks. Empty strings, indistinguishable from a valid empty response unless you were explicitly checking for the difference. The span closed, the HTTP status was 200, and downstream systems that depended on that output received nothing.
The same week, a separate incident surfaced where an AI agent recommended a malware package during a coding task. That failure was caught by human code review policy, not by any agent-level self-correction or automated output check. The agent did not flag what it had done. It completed normally.
These two incidents represent the same underlying problem from different angles. One agent produced nothing when it should have produced content. Another produced harmful content when it should have produced nothing useful. In both cases, the instrumentation layer saw a completed call and moved on.
This article covers how to close that gap: what to instrument, what to validate, and how to distinguish between a legitimate empty response and an actual regression.
Why empty responses are easy to miss
Most agent telemetry is built to catch crashes. A tool call that throws an exception, a model call that times out, a step that returns a 4xx or 5xx status. These failures are visible because they break the control flow.
Silent failures do not break the control flow. Tool calling fails silently between 3% and 15% of the time in production agents, and those failures are not dramatic crashes. The call returns, the span closes, and the orchestrator moves to the next step. If the output of that step feeds into a downstream action, the downstream action runs on bad input.
The Zenodo void response case is a good illustration of why model-level regressions are particularly hard to catch. The model version did not change mid-deployment. The infrastructure did not change. The only thing that changed was the model's behavior on a specific class of prompt, and the only way to know that behavior had changed was to validate output content, not just output status.
For teams deploying agents across customer service, internal operations, or coding workflows, this matters because production reliability rarely matches benchmark performance. A model that performs well on your evaluation set can silently degrade on the distribution of prompts your users actually send.
What span-level output validation means in practice
Span-level output validation means checking the content of every output at the point it is produced, not just checking whether the call succeeded. There are three things worth validating at each span.
Presence. Did the model return anything? A zero-byte response is not ambiguous. An empty string where your schema requires a populated field is not ambiguous either. This check costs almost nothing to add and would have caught the Zenodo regression immediately.
Schema conformance. If your agent is expected to return a JSON object with specific fields, validate that the fields exist and have the right types before the next step runs. A response that passes presence but fails schema is still a failure, and it will silently corrupt any downstream step that tries to read those fields.
Semantic plausibility. This is harder, but worth doing for high-stakes outputs. A coding agent that returns syntactically valid code is not necessarily returning correct code. A research agent that returns a populated JSON object is not necessarily returning accurate citations. Semantic validation requires a scoring layer, either a secondary model call or a rule-based check against known constraints for the task.
The malware package recommendation case is an example of where presence and schema checks would have passed. The output was present. It was well-formed. It was plausible-sounding. Only a human reviewing the specific package name caught the failure. That is a harder problem to automate, but it is not an impossible one: validating agent behavior against expected outcomes using task-specific rubrics can catch a class of plausible-but-wrong outputs that structural checks miss.
Distinguishing regression from intent
Not every empty response is a failure. A research agent asked whether a document contains a specific clause should be allowed to return an empty result if no clause is found. A summarization agent asked to compress a five-word input may return very little. The instrumentation layer needs to know the difference.
The way to encode that difference is through activity schemas: per-task definitions of what a valid output looks like, what an empty output means in context, and what constitutes an anomaly. An activity schema for a document extraction task might specify that an empty response is valid only when the extraction target field was explicitly absent from the source document, and that any empty response without that annotation should be flagged for review.
This is also how you catch regressions over time rather than just in individual spans. If a model previously returned populated outputs for 98% of a given prompt class and now returns populated outputs for 60%, that shift is a regression even if each individual empty response could be explained away. Detecting quality decay in production deployments requires aggregating span-level results and comparing them against a baseline, not just evaluating each call in isolation.
At Prefactor, this is what the validation layer does: it checks each span's output against the activity schema for that task, scores the result, and writes the outcome to the audit trail. When the distribution of scores shifts, that shift is visible as a trend, not just as individual anomalies.
The cost of not instrumenting
73% of enterprise AI agent deployments experience reliability failures within their first year of production. That figure covers many failure modes, but silent failures are disproportionately costly because they accumulate before anyone notices them.
Consider what happened with the Zenodo regression at scale. If that behavior had appeared in a production deployment rather than a research test, every agent run during the regression window would have produced empty outputs. Downstream systems would have received empty inputs. Depending on how those systems handled missing data, they might have failed silently too, or they might have surfaced errors that looked unrelated to the model behavior that caused them.
The City of Kyle, Texas deployed Agentforce for 311 citizen requests and handled more than 12,000 resident requests since March 2025 with nearly 90% first-call resolution. At that volume, a silent failure rate of even 3% means hundreds of requests handled incorrectly before anyone notices a pattern. The failure does not announce itself. Someone has to be looking for it.
This is why span-level instrumentation matters for production agents, not just for development evaluation. The gap between what your agent does and what you think it does is widest when outputs look normal but content is wrong.
What to add to your instrumentation today
Three additions close most of the gap for most agent deployments.
First, add a presence check to every span that produces output. Log the byte length of the response body. Alert on zero-byte or below-threshold responses for tasks where empty is not a valid answer.
Second, define output schemas for every task your agent performs and validate against them at runtime. If the agent returns a response that does not match the schema, treat that as a failed span regardless of HTTP status.
Third, track output score distributions over time rather than just individual scores. A single low-scoring span may be noise. A shift in the score distribution for a given prompt class over 24 hours is a signal worth investigating. Tools for production drift monitoring exist specifically to surface these shifts before they compound.
For teams running autonomous background agents or multi-agent orchestrators, where failures in one span propagate into downstream spans, adding these checks at each step boundary is the only reliable way to isolate where a failure originated. Tracing and debugging agent failures after the fact is much harder when the only record is a set of successful-looking spans.
Where to start
Pick one agent task your team considers business-critical and add presence and schema checks to its output spans this week. Review the score distribution for that task over the following 48 hours against your existing baseline. If you do not have a baseline yet, the next 48 hours become one.
Start evaluating your agents and read the docs for setup guidance on span instrumentation and activity schemas.