Multi-agent systems fail differently than single agents
When you move from one agent to several, the failure surface changes in ways that your existing evaluation tooling almost certainly does not cover. A single agent fails visibly: it returns a wrong answer, calls the wrong tool, or times out. A multi-agent system can fail while every individual agent reports success. The orchestration layer between them degrades silently, and by the time a user workflow breaks, the trace is cold and the cause is distributed across five spans from three agents.
According to the MAST Failure Taxonomy, which analyzed more than 1,600 execution traces across seven frameworks and was presented at NeurIPS 2025, 79% of observed multi-agent failures originate from specification and coordination issues rather than base-model limitations. The models are doing what they were told. The problem is what they were told, and how the system coordinates the telling.
This matters because most teams evaluate their agents the way they evaluated their first agent: prompt quality, task completion rate, latency. Those metrics answer the wrong questions once coordination is involved.
The three failure categories and what makes them hard to see
The MAST taxonomy organizes multi-agent failures into three root categories. Understanding each one changes what you instrument.
Specification ambiguity is the most common and the most misdiagnosed. It happens when two agents hold inconsistent interpretations of a shared task or shared context. Neither agent is wrong given its own prompt; the gap lives in the handoff. A research agent that considers a source "verified" at 80% confidence hands off to a report-writing agent that assumes "verified" means human-reviewed. The report is confidently wrong. No individual span shows an error.
Coordination breakdowns surface when agents assume state that another agent has not yet written, or when a message broker drops a task during high load and no agent retries it. These failures are timing-dependent, which means they may not appear in your pre-production test suite at all. They emerge at production traffic volumes, or when a dependency is slow on a Tuesday afternoon.
Verification gaps are what happen when no agent in the pipeline is responsible for checking the output of the one before it. In a single-agent system, you can add an evaluator. In a multi-agent pipeline, it is common for no agent to have been explicitly assigned the cross-agent verification role, so it belongs to everyone and is done by no one.
The 40% failure rate within six months of production deployment that the MAST analysis surfaces is not primarily a model quality problem. It is a coordination pattern problem. And coordination patterns require different instrumentation than individual agent quality.
Why standard observability misses this
Distributed tracing and logging tell you what happened inside each agent. They do not tell you whether the handoff between agents preserved intent. A span that records "agent B received input from agent A" does not record whether agent B's interpretation of that input matched what agent A intended to communicate. That gap is semantic, not structural, and it will not appear in your trace unless you instrument for it explicitly.
Evaluating agent behavior requires measuring what agents actually do versus what you intended, and in multi-agent systems that comparison has to happen at the boundary between agents, not just at the terminal output. Most observability stacks are built for the latter.
Teams that evaluate thoroughly before launch but stop monitoring after it consistently experience quality degradation within 30 to 60 days, based on field observations across 2025 and 2026 deployments. The gap between pre-production evaluation and production behavior is a well-documented problem for single agents, and it is wider for multi-agent systems because the number of interacting surfaces grows with each agent added.
What teams that get this right are measuring
The teams with durable multi-agent deployments are not just measuring better. They are measuring different things.
Salesforce's deployment of Agentforce agents in Slack for HR, IT, and knowledge management workflows reached 86% employee adoption after six months of iteration, with a projected 500,000 hours per year in savings. That six-month refinement window is the part most teams skip. Adoption at that scale requires catching coordination failures early, not after they have soured a user's experience.
Notion's internal deployment of more than 2,800 agents, more than their employee headcount, was used explicitly to test evaluation and monitoring under load. Their ticket triage agents coordinating with approval agents reached 95% accuracy and resolved more than 25% of tickets autonomously. That accuracy number reflects ongoing measurement of handoff fidelity, not just individual agent output.
ServiceTitan's Max agentic OS, which routes call-handling agents alongside job-dispatch agents, showed 50% increases in average ticket size and EBITDA margin growth from 18% to 30% for early adopters. The gains depend directly on handoff accuracy between voice agents and dispatch agents. A missed handoff means a missed booking; there is no fallback.
What these deployments share is ongoing measurement at the coordination layer. Continuous evaluation reduces production incidents by 67% compared to periodic evaluation in enterprise AI programs, according to Deloitte's AI Ops Maturity research. The operative word is continuous: not a pre-launch test suite, not a monthly review.
A concrete evaluation framework for multi-agent systems
The framework has four layers, each addressing one of the failure surfaces above.
Boundary validation. At every agent-to-agent handoff, validate that the output of the upstream agent satisfies the input schema of the downstream agent. This is not just type checking. It includes semantic validation: does the downstream agent's interpretation of the payload match the upstream agent's intent? Activity schema validation catches this class of failure before it propagates.
State consistency checks. After each coordination step, verify that shared state is consistent across agents that hold a view of it. A mismatch in state is a precursor to specification ambiguity failures.
Step-level accuracy tracking. Do not aggregate to task-level pass/fail. Step-level accuracy and cascading failures in multi-agent systems mean that an 85% step accuracy rate across a six-step pipeline yields less than 38% end-to-end task accuracy. Each step compounds the risk.
Drift detection over time. Detecting quality decay after production deployment requires a baseline and a signal. Record coordination metrics at launch, set thresholds, and alert when any boundary or state check degrades by more than a defined margin. This is where Prefactor's activity schema validation is useful: it records spans at each coordination boundary, scores them against the expected schema, and surfaces drift without requiring manual log review.
Multi-agent evaluation is also distinct from single-agent evaluation in the risk profile it introduces. A failure in one agent that propagates through five others before surfacing creates an audit trail problem as much as a quality problem. Every step needs to be recoverable.
Where to start
Pick one multi-agent pipeline you have in production and instrument the handoff boundaries first. You do not need to evaluate everything at once; you need to know which coordination point is degrading and by how much. Once you have that baseline, the drift detection layer follows naturally.
Start evaluating your agents and review the docs to see how activity schema validation and span scoring work across multi-agent pipelines.