Multi-agent system observability requires instrumentation at every hop, not just at the final output. Propagating trace context across agent boundaries, scoring each agent independently, and validating inter-agent payloads against declared schemas gives you enough signal to isolate which agent caused a failure before it compounds across the chain.
Recording spans across agent boundaries
A span is a timed, attributed record of one unit of work. In a single-agent trace, you get a linear sequence of spans: input received, tool called, output produced. In a multi-agent system, spans branch. Agent A receives a task, routes a sub-task to Agent B, waits for a result, and then continues. If your instrumentation treats each agent as an isolated process, you lose the parent-child relationship between those spans, and you cannot reconstruct what actually happened when something goes wrong.
The fix is to propagate a shared trace context across agent boundaries. When Agent A calls Agent B, it passes a trace ID and a parent span ID. Agent B opens a child span under that parent. Every tool call, retrieval, and model invocation that B makes is nested inside that child span. When B returns, the full subtree is visible in the trace, attributed to B, but linked to A's originating request.
This matters more than it sounds. The insurance underwriting multi-agent system cited in research, which reduced average decision time from three to five days to 12.4 minutes, runs document parsing, risk assessment, and policy decision agents in sequence. A latency spike in document parsing shows up as a delay in the final decision. Without cross-boundary spans, you see the delay but not the cause. With them, the slow span is immediately attributable to the correct agent.
Prefactor instruments this via its SDK, recording each span with the agent identity, the model version, and the tool calls made, so the trace is queryable by agent when you need to isolate a failure.
Distributed tracing for agents follows the same conventions as OpenTelemetry for services, which means existing trace backends can ingest the data without custom adapters, provided the instrumentation is consistent at every hop.
Scoring quality per agent, not just per task
Task-level quality scores tell you whether the final output was acceptable. They do not tell you which agent caused a degradation. If your task score drops from 0.87 to 0.71 across a week of production traffic, you need per-agent scores to find out whether the retrieval agent started returning lower-relevance context, the reasoning agent started hallucinating steps, or the formatter agent introduced structural errors.
Per-agent scoring requires defining what "good" means for each agent independently. A retrieval agent is scored on recall and precision against a reference corpus. A reasoning agent is scored on step validity and conclusion accuracy. A routing agent is scored on whether it dispatched to the correct downstream agent for a given input type.
Williams-Sonoma's deployment of Agentforce agents across its brand portfolio illustrates why this decomposition matters: Otto resolves over 70% of chat inquiries autonomously, while Olive drives a 3x conversion lift for customers who engage with it. Those two numbers are measured separately because the agents do different jobs. A single task-level score would collapse them into a meaningless average.
Evaluating agents in production requires that each agent's scoring criteria be defined before deployment, not derived from failures after the fact. Prefactor records a quality score per span, which means per agent per invocation, and surfaces drift when an agent's score distribution shifts outside the baseline range established in earlier traffic.
Step-level accuracy and compounding failures become especially relevant in long chains: a retrieval agent operating at 90% accuracy, feeding a reasoning agent also at 90%, produces a chain accuracy of roughly 81% by the second step, and lower still as steps accumulate.
Validating inter-agent communication against declared schemas
When Agent A hands a payload to Agent B, both agents need to agree on the structure of that payload. In practice, this agreement is often implicit. The payload is defined in code, the schema is not written down anywhere, and when Agent A's output format drifts, Agent B fails in ways that look like its own bug.
Charter, released by Boundflow on September 12, 2026, addresses this with YAML-based policy definitions for production-safe agent orchestration. Each agent declares its expected input and output schemas, and Charter validates payloads at runtime before they cross an agent boundary. A payload that violates the schema is rejected at the boundary, not silently corrupted three steps later.
This is the same principle that makes API contracts useful in microservice architectures: you catch the mismatch at the interface, not in the downstream consumer. The difference with agents is that the payloads often contain natural language fields alongside structured fields, so schema validation needs to cover both dimensions. A field declared as a `risk_summary` string might technically validate as non-empty while containing a hallucinated value that a downstream underwriting agent will treat as authoritative.
Detecting agent drift through activity schema validation and validating agent behavior against intent before deployment together cover the two failure modes: structural drift, where the payload shape changes, and semantic drift, where the payload shape is valid but the content no longer means what the schema implies.
The September 2026 launch of eight routing and orchestration tools for Claude Code and Codex agent sessions shows that the infrastructure layer is moving toward formalized routing policies, session management, and boundary enforcement. The observability layer needs to keep pace: a routing decision that sends a task to the wrong specialist agent is only detectable if you record which agent was selected and validate that the selection matches the declared routing policy.
Multi-agent orchestration introduces failure modes that single-agent systems do not have, and the tooling is catching up to that fact. The McKinsey State of AI 2025 survey, covering 1,993 participants across 105 countries, found that 64% of organizations cite evaluation and observability as the largest blocker to moving agents from pilot to production. That number does not fall by deploying more agents; it falls by instrumenting the ones you have.
Ghost actions and unintended agent behavior are a direct consequence of missing schema validation: an agent acts on a field it was not intended to receive because no boundary enforcement stopped the payload from arriving.
Where to start
Audit the agent boundaries in your current system and list every handoff that has no schema validation and no cross-boundary span. Start instrumenting there. If you are using Charter or a similar orchestration layer, add schema declarations for the two or three most critical handoffs before expanding coverage.
Start evaluating your agents and read the docs to see how Prefactor records spans, scores quality per agent, and validates behavior against activity schemas in production.