Model routing only works if you measure what you are routing
Automated model routing, selecting a frontier model for complex tasks and a lightweight model for simple ones, looks straightforward on paper. In practice, routing decisions compound. A coding agent handling ten thousand requests per day makes ten thousand micro-bets about complexity. If the scoring logic is wrong, you either overspend on frontier calls that a cheaper model could have handled, or you degrade quality on tasks that needed the heavier model. You find out which happened by instrumenting the agent and measuring both sides of that trade-off in real time.
This article covers how to set up span instrumentation to score task complexity, what signals to capture at the routing decision point, and how to validate that routing actually lowers cost without moving quality metrics in the wrong direction.
What routing decisions look like at production scale
Cursor released its intelligent model router in July 2026. The router automatically sends coding requests to frontier or lightweight models based on task complexity. In A/B tests across millions of requests, it achieved 60% cost savings while maintaining frontier-quality performance. Early access customers saw 30 to 50% lower costs. Those figures are meaningful because they came from measuring quality outcomes alongside cost, not cost alone.
The Cursor case illustrates the core constraint: routing decisions are only defensible if you can show, after the fact, that quality held. Without per-request quality scores attached to routing decisions, you have a cost number but no signal about what you traded to get it.
Anthropic's Claude Opus 5 release from July 25, 2026, adds a second dimension. Opus 5 achieved comparable performance to prior frontier models with an 80% reduction in system prompt length. Shorter prompts mean fewer input tokens per call. In a routing setup, this changes the cost calculus for tasks that do get routed to a frontier model: the frontier call is cheaper than it used to be, which shifts the breakeven point between routing tiers. If you are not measuring token counts per span alongside quality scores, you will not see that shift and you will not recalibrate routing thresholds to match.
Instrumenting spans to score complexity at the routing decision point
The routing decision happens before the model call. That means your instrumentation needs to capture the inputs that drive routing, not just the outputs that result from it. Concretely, a span at the routing decision point should record:
- The raw task input and its token count
- Which complexity signals the router evaluated (context length, presence of multi-step reasoning requirements, tool call depth expected)
- The routing outcome: which model tier was selected and why
- A pre-call complexity score if your router produces one
After the model responds, a second span records the output, latency, token usage, and any quality scores your eval layer produces. Linking these two spans by trace ID lets you answer the question that actually matters: for a given complexity score and routing decision, what quality did you get?
This is the pattern Prefactor uses when it instruments agent spans. The SDK records both the routing decision and the downstream output as linked spans, then scores quality against the activity schema you define. The audit trail captures which model handled which task, so you can slice quality metrics by routing tier without manual log correlation.
For more on what to capture at the span level, the token efficiency and cost benchmarking guide covers the instrumentation patterns in detail.
Validating that routing lowers cost without degrading quality
The validation problem has two parts. First, you need a quality signal that is sensitive enough to detect degradation. Second, you need enough volume per routing tier to make the comparison statistically meaningful.
On the quality signal: for coding tasks, test pass rates and lint errors are direct. For summarization or question-answering tasks, you typically need an LLM-as-judge scorer or human spot-checks on a sample. The scorer needs to run on both routing tiers under the same rubric; if your eval method differs between tiers, you cannot compare scores across them. The agent evaluation guide covers how to set up consistent scoring across model tiers.
On volume: routing thresholds calibrated on small samples drift when traffic patterns shift. Klarna's AI customer support agent handles 2.3 million monthly conversations across multiple languages, and reduced cost per transaction by 40% over two years while cutting resolution time from 11 minutes to 2 minutes. That scale makes statistical validation tractable. At lower volumes, say five thousand requests per month per tier, you need wider confidence intervals before acting on quality comparisons.
Production drift is a real risk. Detecting agent quality decay in production describes how quality scores can degrade gradually as task distributions shift, which is exactly what happens when you add a new routing tier and the lightweight model starts receiving tasks that were previously handled by the frontier model.
What the cost and quality data should tell you
Once you have linked span data from both tiers, three questions structure the analysis:
Is the lightweight model handling the tasks you expected? Compare the distribution of complexity scores for tasks routed to each tier. If tasks with high complexity scores are landing on the lightweight model, the routing threshold is miscalibrated, not the model.
Is quality holding at the tier boundary? Plot quality scores against complexity scores for lightweight-routed tasks. Degradation typically appears first at the top of the complexity range for that tier, where tasks are borderline.
Are frontier calls getting cheaper over time? Track input token counts per frontier call. If you have adopted models with shorter required system prompts, like the Opus 5 changes above, average frontier call cost should drop. That creates room to route more tasks to the frontier tier without increasing total spend, which can recover quality at the margin.
51% of enterprises have AI agents in production as of 2026, with another 23% actively scaling. At that adoption level, per-request routing decisions are standard infrastructure, not an optimization experiment. The companies getting consistent results are the ones measuring routing outcomes, not just routing.
The LLM gateways and routers reference covers the infrastructure layer, and the evals versus observability explainer is worth reading alongside this article if you are deciding what your observability stack needs to produce beyond logs.
For multi-agent setups where routing decisions happen across several agents in a pipeline, multi-agent orchestrator patterns describes how cost and quality signals propagate differently when routing is distributed.
The hidden token overhead article covers one frequently missed cost source: system prompt tokens that appear on every call regardless of routing tier, and that distort your per-tier cost comparisons if you do not account for them separately.
Where to start
Pick one agent, instrument its routing decision spans and output spans with a shared trace ID, and attach a quality scorer that runs on both tiers under the same rubric. Run it for two weeks to get enough volume per tier to compare. That gives you the data to validate or adjust your routing thresholds before acting on the cost numbers.
Start evaluating your agents or read the docs to see how Prefactor instruments spans and scores routing outcomes against your activity schemas.