How large does the golden dataset need to be before canary replays are statistically meaningful?

Thirty to one hundred examples per request category is enough to detect regressions in the five to ten percent range with reasonable confidence. Below thirty examples per category, a two to three percent drift signal is too noisy to act on reliably; above one hundred, marginal coverage gains are usually smaller than the cost of maintaining the additional labeled examples.

Can I use the same LLM that powers the agent as the judge model for canary scoring?

You can, but the judge should be a separate call with a different system prompt that explicitly frames its role as evaluation, not task completion. Using the same model version as both agent and judge introduces correlated failure: if the model version regresses, the judge may not catch it because it is subject to the same regression. Using a different model family for judging removes that correlation.

How do I set thresholds when I have no historical baseline yet?

Run your golden dataset against the agent for five to seven days before setting thresholds, recording the score distribution each day. Use the mean and standard deviation of that initial window to set your level one threshold at roughly 1.5 standard deviations below the mean. Adjust after thirty days once you have enough signal to distinguish genuine drift from normal daily variance.

What is the difference between canary evaluation and standard A/B testing of agent versions?

A/B testing compares two agent versions against each other to choose between them. Canary evaluation compares one agent version against its own historical baseline to detect degradation over time. You need both: A/B testing for deployment decisions and canary evaluation for ongoing production monitoring. Running only A/B tests gives you no signal if both the current and candidate versions have drifted from the original quality level.

Canary evaluation for production AI agents: a practical guide

Canary evaluation is the method most teams are missing when production AI agents start returning wrong answers while every infrastructure dashboard stays green. This article covers the three failure signatures that latency and error-rate monitoring cannot see, the threshold math for tiered alerts, and the cost case for continuous scoring.

Why infrastructure metrics miss quality failures

HTTP 200 means the agent responded. It says nothing about whether the response was correct. A silent agent failure looks identical to a successful request at the infrastructure layer: latency is normal, no exception is thrown, token budget is consumed at the expected rate.

A documented production case from May 2026 makes this concrete. An agent handling support tickets returned wrong answers on roughly one in fourteen requests for nine days before anyone noticed. Infrastructure dashboards stayed green throughout. The cost was approximately $97 per day in token burn, plus the downstream cost of incorrect actions taken on bad answers. The root cause was straightforward: the team had health checks, not behavioral drift detectors.

According to a 2025 Deloitte analysis cited by thinking.inc, evaluation infrastructure costs 10 to 25 percent of agent operating costs, but teams that skip it spend three to five times more on incident response. The math is not close.

The three failure signatures you need to detect

Context drift

Context drift happens when the information flowing into your agent changes without the agent's prompt or model changing. A RAG pipeline that quietly shifts its retrieval results, a vector database reindex that alters document ranking, or a schema change in an upstream API can all make previously correct behavior incorrect. The agent has no way to signal this; it simply starts working from different premises. Read more on how vector database configuration affects retrieval quality.

A reconstructed post-mortem from a May 2026 accounts receivable deployment identified exactly this pattern: an upstream vector database configuration change caused an escalation rate to drift silently from a 12 percent baseline to operationally unacceptable levels. The monitoring gap was not a missing alert; it was a missing detection dimension.

Off-policy renegotiation

An agent that makes sequential decisions can, over time, use its own prior outputs as implicit training signal. In agentic settings where past context is fed back into future prompts, the agent gradually renegotiates its own decision rules. Commonwealth Bank of Australia deployed a continuous evaluation pipeline on their fraud-detection agent for this reason. Their scoring pipeline revealed behavioral drift two full weeks before accuracy dropped below operational thresholds, giving the team time to intervene before any measurable fraud loss increase. Their H1 2026 result was a 20 percent reduction in fraud losses, and the detection lead time was the mechanism that made remediation possible.

Retrieval poisoning and document churn

Retrieval poisoning does not require an adversary. Document churn, where a knowledge base is updated with content that contradicts prior best answers, produces the same symptom: high-confidence wrong answers. RAG-based agents are particularly exposed here because the model does not know what it retrieved last week.

The canary evaluation pattern

Canary evaluation is a hybrid offline-online scoring pattern. You maintain a golden dataset of representative requests with known-good answers. On a scheduled basis, typically daily, you replay those requests against the live agent configuration and score the outputs. Separately, you sample a fraction of live traffic and score it using the same rubric.

The pattern has two loops:

Offline canary replay: Run your golden dataset daily against production model IDs. Getia Consulting's May 2026 field practice report found that daily canary replays catch one to two silent regressions per quarter on average, regressions that infrastructure metrics would never surface. JPMorgan Chase deployed exactly this pattern on their AI-powered legal document review system and caught regressions and deployed fixes in hours rather than the weeks it would have taken via user escalation.
Online sampling with LLM-as-judge: Pull two to five percent of live traffic, score each trace against your rubric using a judge model, and compare against your baseline distribution. This catches distribution shift that the golden set cannot anticipate.

The key design choice is cost tiering. You cannot run an expensive LLM-as-judge call on every production trace at scale. Noveum.ai's internal case study from June 2026 documents a tiered approach that resolved this: cheap groundedness checks ran on all traffic, and expensive judge calls ran only on flagged traces. Their customer support agent's hallucination score moved from 2.8 to 9.5 out of 10; overall success rate rose from 84 to 95 percent. The full loop from failing trace to verified fix ran in minutes.

Threshold design and tiered alerting

Lloyds Banking Group's approach, documented across their 2025 to 2026 deployment, gives a practical reference for threshold calibration: a two percent drift from baseline triggers investigation, and a five percent drift triggers a priority alert. Those numbers came from working with business teams to understand what score degradation actually meant at the output level, not from fitting a curve to historical anomalies.

Your thresholds should be set per request category, not globally. A customer-facing response agent and an internal document summarizer have different acceptable error rates. Set them separately.

Tiered alerting means:

Level one (investigation): Score drops two to three percent below rolling seven-day baseline. Engineer reviews the flagged traces. No pager, no escalation.
Level two (priority alert): Score drops five percent or more, or any single canary request category fails completely. On-call is notified.
Level three (circuit breaker): Score drops ten percent or more across two consecutive canary runs. Automated rollback or human override gate is triggered.

The circuit breaker level is where behavioral validation against activity schemas becomes load-bearing. If your agent is expected to follow a defined workflow, a ten percent quality drop paired with schema violations is stronger evidence for rollback than either signal alone.

Prefactor records span-level scores and validates behavior against activity schemas on each run, which means the level two and level three signals can be generated without building a separate alerting pipeline from scratch.

When continuous evaluation pays for itself

The cost argument is straightforward once you run the numbers for your own agent. Take your daily transaction volume, estimate the fraction that would be affected by a ten percent quality regression, and multiply by the cost of a wrong outcome in your domain. Then compare against the cost of daily canary replay, which for most teams is two to four percent of daily inference cost.

PwC's 2026 AI agent survey found that 79 percent of organizations cannot trace a failure through a multi-step workflow or measure output quality systematically. That is a measurement gap, not a model capability gap. The teams that close it are the ones that treat agent evaluation as an operational discipline, not a pre-launch checklist.

Teams that evaluate thoroughly before launch but stop monitoring post-launch consistently experience quality degradation within 30 to 60 days, according to enterprise deployment practice data from 2026. The degradation window is short enough that a monthly review cadence misses it entirely.

For teams running multi-agent orchestration or autonomous background agents, the window is shorter still, because off-policy renegotiation compounds across agent boundaries.

Where to start

Build your golden dataset first: thirty to one hundred representative requests per request category, with scored reference answers agreed on by the team responsible for the outcome. Run a single offline canary replay against your current production configuration to establish your baseline. Then instrument live traffic sampling at two to five percent and set your level one threshold.

Start evaluating your agents and review the docs for how Prefactor instruments spans and records scores against your activity schemas.