Token efficiency is not what your vendor measured
When JetBrains ran its SkillsBench evaluation in July 2026, it tested RTK (Rust Token Killer), a tool whose marketing promised 60 to 90 percent token savings on Claude Code workloads. Across 425 trials at low reasoning effort, RTK produced a 7.6 percent cost increase. A separate test of the Caveman compression skill, advertised at 65 percent savings, showed 8.5 percent output token reduction on real agentic tasks, not 65 percent.
The gap between claimed and measured performance is not unusual. It reflects a structural problem: vendors benchmark their tools under conditions that minimize the factors agents actually encounter in production, including multi-step context accumulation, branching reasoning chains, and tool-call overhead. If you deploy based on benchmark claims, you are optimizing for someone else's workload.
This article covers the metrics that let you measure token spend against task completion in your own environment, and why the ratio between those two numbers matters more than either one in isolation.
Why agents spend tokens the way they do
A single model call is straightforward to price: input tokens plus output tokens, at published rates. An agent is not one call. Each step in an agentic workflow carries the accumulated context of prior steps, tool responses, memory retrievals, and often the original system prompt. Agent-based approaches consume 5 to 10 times more tokens than zero-shot LLM approaches for this reason, and some estimates put full agentic workflows at 19 to 50 times the cost of a single model call depending on scope and context requirements per step.
This matters because the cost curve is not linear. Add one tool call to a workflow and you may add one step's worth of tokens. Add a reasoning loop or a retrieval step that returns a large context window, and subsequent steps carry that weight for the rest of the session. Compression tools reduce some portion of token volume, but if they add latency or degrade task completion rates, the net cost per completed task can rise even as raw token counts fall, which is exactly what the JetBrains benchmark observed.
The metric you actually need is tokens per completed task, not tokens per call.
The Uber case: when adoption outpaces measurement
In April 2026, Uber reported that it had exhausted its entire 2026 annual AI infrastructure budget in four months through Claude Code and Cursor agent deployments. By that point, 95 percent of engineers were using AI tools monthly, and 70 percent of committed code was coming from AI agents.
The Uber outcome is not a failure story. It is a planning story. The usage spread faster than the measurement infrastructure did. When 95 percent of engineers are generating code through agents, and you have no per-task token accounting, your first signal that the budget is wrong is the budget running out.
This is the pattern that agent observability is designed to prevent. You need span-level records of what each agent step consumed, indexed to the task that step belonged to, before spend reaches a scale where the only correction available is a budget freeze.
The three ratios that matter
Raw token counts tell you volume. To make decisions, you need ratios.
Tokens per completed task. Divide total tokens consumed by the number of tasks the agent completed to the quality threshold you defined. If a compression tool cuts tokens 8 percent but completion rate drops 12 percent, cost per completed task has risen. This is the JetBrains result, expressed as a business number rather than a benchmark number.
Cost per outcome unit. For a coding agent, the outcome unit might be a passing test suite or a merged pull request. For a customer support agent, it might be a resolved ticket without escalation. Klarna's AI customer service deployment, for example, reduced resolution time from 11 minutes to under 2 minutes and cut repeat inquiries by 25 percent, which meant the cost per resolved inquiry fell even as token volume grew with scale. The outcome unit is what makes the spend legible.
Step-level cost variance. Some steps in a workflow consistently cost far more than their contribution to task success warrants. Identifying those steps requires tracing at the span level, not just at the session level. An autonomous background agent running overnight can accumulate a large context window across many tool calls; the expensive steps are often not the ones you would guess from reading the prompt.
What instrumentation actually requires
To compute these ratios reliably, you need four things recorded per span: the model called, the token counts for that call (input and output separately), the task identifier the span belongs to, and the outcome label for the task when it completes. Without the task identifier, you can aggregate token spend but you cannot attribute it to a result. Without the outcome label, you cannot distinguish efficient runs from runs that completed a low-quality task cheaply.
This is where agent evaluation differs from standard observability. Observability tells you what happened. Evaluation tells you whether what happened was worth what it cost.
Prefactor records token counts at the span level and scores task outcomes against the activity schemas you define, so the tokens-per-completed-task ratio is computable from the audit trail rather than derived from sampling. That said, any instrumentation layer that captures span-level token data and links spans to outcomes can support this kind of analysis, whether that is Prefactor, a custom OpenTelemetry pipeline, or another evaluation platform.
For teams building the instrumentation layer from scratch, OpenTelemetry for agent spans is a reasonable starting point: it gives you the trace structure to attach task identifiers and outcome labels to existing span data without rewriting the agent itself.
Evaluating optimization claims before you deploy
The JetBrains RTK result points to a methodology problem that applies to any optimization tool, not just token compression. Before deploying an optimization, run it against a representative sample of your own tasks, not the vendor's benchmark tasks. Measure completion rate alongside token count. Compute cost per completed task for both the baseline and the optimized version. If the optimized version reduces tokens but the ratio rises, the optimization is costing you money.
This does not require a large evaluation set to start. A sample of 50 to 100 representative tasks, scored against a consistent quality rubric, is enough to detect a directional shift. JetBrains ran 425 trials; that is the number that gives you statistical confidence at the level of a publishable benchmark. For an internal go/no-go decision, a smaller set suffices as long as the tasks are representative.
The broader point is that the gap between offline evaluation and production performance is real for optimization tools just as it is for agents themselves. A tool that compresses tokens effectively on the synthetic tasks it was designed for may perform differently on the tool calls, retrieval chains, and context patterns your agents actually generate. Measure in your environment.
For multi-agent orchestrators in particular, where a single user-facing task may spawn several sub-agents, the token accounting needs to trace across the full call graph. A sub-agent that looks efficient in isolation may be generating redundant context that the orchestrator passes to every downstream step. Step-level accuracy and cascading failures in agent chains is worth reviewing if you are running workflows of this kind.
Finally, if you are deploying RAG-based retrieval agents, retrieval chunk size and re-ranking behavior are among the largest drivers of per-step token cost, and they are rarely what compression tools target. Optimizing the retrieval layer directly often produces larger gains than post-hoc token compression.
Where to start
Instrument one agent workflow at the span level, record token counts alongside task outcomes, and compute tokens per completed task for a baseline week. That number gives you something to improve against that vendor benchmarks cannot give you.
Start evaluating your agents or read through the docs to see how span-level token recording and outcome scoring fit into your existing setup.