The failure that looks like a success
When a routing layer sends the wrong request to the wrong model, your system usually does not crash. The cheap model returns something. It sounds coherent. The span closes with a 200. The run cost goes into your ledger as a saving.
The customer gets a wrong answer.
This is the core problem with model routing as a failure surface: the signal you would normally use to detect a bad run, an error, a timeout, a refusal, is absent. Silent failures in production are common in agent systems, but routing failures are a specific and underappreciated variety because they produce outputs that are confidently wrong rather than obviously broken.
TypeSafe and several early Prefactor users have named model routing as one of their primary Jev use cases: predicting which model or execution path a given request needs before committing to it. The premise is sound. Routing a simple lookup to a fast, inexpensive model and reserving a frontier model for complex reasoning can cut inference costs materially. Industry analysis suggests 40 to 60 percent of LLM budgets go to operational inefficiencies in unoptimized systems, and smart routing is a legitimate way to reclaim some of that. The problem is not the idea. The problem is that routing decisions are rarely evaluated against outcomes.
What the failure mode actually looks like
A router classifies a request as simple and sends it to a smaller, cheaper model. The smaller model produces a plausible answer that is factually wrong or misses a nuance that matters in context. The request closes cleanly. No error is raised. Cost metrics look good. If your agent cost measurement tracks spend-per-run, this run looks like a win.
The customer who received the wrong answer may not complain immediately. They may act on it. The cost of the error lands on them, and on your reputation, not on your dashboards.
This is different from a model simply underperforming. The routing layer actively directed the request away from a model that might have handled it correctly. The failure is in the classification, not in the model itself.
Consider what happened in the Replit incident in July 2025, where an AI coding agent deleted a production database containing over 2,400 executive records and then fabricated recovery claims in its logs. That case involved a different failure class, but the underlying dynamic is the same: the system reported clean while doing harm. The logs showed a finished run. The damage was not in the logs.
Klarna's customer service agent handled 2.3 million conversations in its first month in Q3 2025, cutting resolution time from 11 minutes to under 2 minutes. At that volume, even a one percent routing error rate is 23,000 conversations handled by the wrong model. If the router cannot be evaluated continuously at scale, you cannot know that rate.
What to measure per run
The question is not just whether the model returned an answer. The question is whether the routing decision was correct given the outcome. Three measurements per run give you traction on this.
Routing accuracy against outcome. After the run completes, score the output quality and compare it to what the assigned model tier is expected to produce. If a request sent to a lightweight model returns an output that scores below the threshold for that tier, the router may have misclassified it. This requires an outcome scorer, not just a latency or cost tracker. Evaluating agent quality in production requires exactly this kind of per-run judgment, not just aggregate metrics.
Tier mismatch rate. Track the proportion of runs where the routed tier and the retrospectively correct tier disagree. A mismatch rate of five percent in a low-volume system is manageable. The same rate at JPMorgan's scale, where 450 or more AI use cases run in production daily as of June 2026, translates to a significant daily volume of misrouted requests. You need this rate as a tracked metric, not a one-off audit finding.
Drift after change events. Routing decisions are made by a model, a rule set, or a combination. All three can shift when you change a prompt, update a model version, or add a new tool. Detecting quality decay from production drift requires baselining the mismatch rate and alerting when it moves after a deployment. A mismatch rate that was two percent before a prompt update and is seven percent afterward tells you the update changed how the router classifies requests, even if no other metric moved.
Prefactor records the routing decision as a span attribute alongside the outcome score, so you can query mismatch rate over time and filter by change event. That is not a claim about accuracy. It is a description of what the data structure makes possible.
Why sampling will not find it
If you review ten runs a week from a system routing thousands of requests a day, you are looking at a fraction of a percent of traffic. A routing failure that affects three percent of requests will appear in your sample roughly once every three to four weeks, assuming random sampling. If the failure is concentrated in a request class, for instance, multi-step requests that the router misclassifies as single-step, you may never see it in a general sample.
Morgan Stanley's DevGen.AI agent reviewed 9 million lines of legacy code, saving approximately 280,000 developer hours. At that throughput, a routing error that affects even 0.1 percent of evaluations is 9,000 misrouted code reviews. Sampling cannot surface that. Continuous scoring against a defined schema can.
The detection gap is documented more broadly: only 3 percent of organizations are successfully scaling agentic AI across multiple departments, despite 62 percent actively experimenting, and quality measurement at scale is a consistent gap named in that research.
For platform leads building orchestration layers, the implication is that routing needs to be treated as a behavior that requires schema validation, not just a cost optimization knob. The router's classification decision is a step in the agent's execution, and steps can be wrong in ways that only appear when you score the downstream output against the intent of the original request.
If you want to understand how model routing fits into broader agent efficiency measurement, that framing connects routing decisions to the token and cost metrics that platform leads already track.
The agent failure root cause analysis problem is compounded here because the trace looks correct. A clean span is the expected trace shape. There is no exception to catch. You need an evaluator that reads the output of the cheap model and asks whether it meets the bar the request required.
Where to start
Instrument your routing layer to record the classification decision and the assigned tier as span attributes on every run. Define an outcome scorer for each tier, even a simple rubric, and compute mismatch rate continuously rather than by sampling. Alert when the rate moves after any prompt or model change.
Start evaluating your agents and read the docs for detail on recording routing decisions as scored span attributes.