What you actually need before you can claim ROI
Enterprises are spending at scale on AI agents and reporting almost nothing they can act on. Only 6% of organizations achieve significant enterprise-wide AI impact, defined as a 5% or greater EBIT contribution, despite 88% using AI in at least one business function, according to McKinsey's 2026 State of AI survey of 1,719 respondents. A separate PwC survey of 4,450 CEOs found that 56% report neither increased revenue nor decreased costs from AI investments in the prior 12 months. Those are not adoption problems. Adoption is near-universal. They are measurement problems.
The issue is not that agents fail to produce effects. It is that most organizations have no pre-launch reference point to measure effects against. When you cannot say what a process cost, how long it took, or how often it erred before an agent touched it, you cannot say what the agent changed. Anecdote fills that gap, and anecdote does not survive a board review.
This article explains what an evaluation baseline is, why it has to be built before deployment rather than after, and how to instrument your agents so outcome measurement is possible from day one.
Why the measurement gap is structural, not accidental
The sequence matters. Most teams build an agent, test it in staging, ship it, and then look for evidence it helped. The measurement problem is baked into that order. By the time you ask "did this work?", the pre-agent process is gone. There is no control group. There is no pre-agent cost-per-transaction figure. There is nothing to compare the agent's output against except someone's memory of how things used to feel.
A baseline is a recorded snapshot of process performance taken before the agent runs in production. It captures the metrics the agent is supposed to move: resolution time, error rate, cost per unit of work, escalation frequency, throughput. It is not a narrative. It is a set of numbers with timestamps and a defined measurement method, so the same method can be reapplied after the agent is live.
Without this, teams reach for proxies. They count the number of agent invocations, or they measure token throughput, or they survey users for satisfaction scores. None of those are outcome measurements. Measuring what agents actually cost is a separate problem from measuring what they deliver, and conflating the two is where most ROI analyses collapse.
The companies that report real numbers built the baseline first. Klarna launched its customer service agent in February 2024 against a documented support operation and was able to report $60 million in annual cost savings by Q3 2025, with the agent handling work equivalent to 853 full-time employees, because the pre-agent cost structure was known. The number is credible because the denominator existed before the agent ran.
What a baseline actually contains
A baseline document for an agent deployment covers four areas.
Process cost. What does the human-led version of this process cost per unit? Include fully loaded labor cost, tooling, error correction, and any downstream rework. If you do not have this at transaction level, aggregate over a trailing 90 days and divide.
Quality rate. How often does the current process produce the correct output without rework? Define "correct" precisely, because this definition becomes the scoring rubric you apply to the agent later. Agent behavior validation against a defined schema is what makes this step machine-readable rather than subjective.
Cycle time. How long does a unit of work take from initiation to completion? Capture mean and 90th percentile. Agents often compress mean time dramatically while leaving tail latency unchanged, and you need both figures to report accurately.
Escalation and error frequency. How often does the process require human review, correction, or escalation? This is the metric most likely to shift when an agent takes over, and it is the one most often omitted from pre-launch baselines.
These four figures, recorded with a measurement methodology that can be reapplied, are the minimum viable baseline. They are not a guarantee of measurable ROI. They are the precondition for it.
Instrumenting agents for outcome measurement before launch
Baseline data answers "where were we?" Instrumentation answers "what is the agent doing, and is it working?" Both are required. An agent that runs without instrumentation produces effects you cannot attribute to specific behaviors, which means you cannot improve it deliberately and cannot defend its outputs in a governance or compliance review.
Agent evaluation in production requires span-level recording: each step the agent takes, what it sent and received, what decision it made, and what that cost. This is not just logging for debugging. It is the evidentiary layer that connects agent behavior to outcome metrics.
JPMorgan Chase is running more than 450 agentic AI use cases in production and reported a 20% increase in gross sales from banking AI tools in June 2026. A number like that requires a chain of evidence from agent action to business outcome. You need to know which agent actions preceded the sales interactions that converted, not just that agents were running. That chain is built from instrumented spans.
Detecting agent quality decay over time requires the same infrastructure. Agents degrade in production as prompts drift, model versions change, and the distribution of inputs shifts. If you are not recording what the agent does at the step level, you will not detect decay until it has already moved your outcome metrics in the wrong direction.
Prefactor instruments agents via SDK, recording spans and scoring each step against quality and risk criteria. The scores and the audit trail are what let a platform lead answer "what did the agent do on Tuesday and did it meet the quality threshold?" without relying on inference.
Distinguishing subjective improvement from quantified value
The gap between "this feels better" and "this moved a specific metric by a specific amount" is where most enterprise AI programs stall. Less than 20% of IT leaders report AI initiatives have met or exceeded business goals, according to a 2026 CIO.com survey of 662 IT leaders. That figure reflects, in part, the absence of pre-defined success criteria. If you did not specify what "met business goals" would look like before launch, you cannot measure it afterward.
Quantified value requires three things: a baseline figure, a post-deployment figure measured with the same method, and a clear attribution argument. The attribution argument is where most analyses break down. Agents do not operate in isolation. Other process changes, staffing changes, and seasonal variation all move the same metrics. Your instrumentation needs to be granular enough to isolate the agent's contribution.
Salesforce's deployment of Agentforce for contract review produced a claimed $5 million reduction in legal costs and resolved 380,000 support interactions at 84% without human involvement. Those numbers hold up because the scope was narrow enough to isolate, the volume was large enough to be statistically meaningful, and the pre-launch cost structure was documented.
DXC Technology reported 30 to 50% reduction in complex workflow cycle times using enterprise workflow agents. The range, not a single figure, signals that the measurement was done at the process level with enough granularity to show variance across different workflow types. A single summary figure would have been easier to communicate but less credible.
Start with a narrow scope. One process, one team, one agent. Define the four baseline metrics before the agent touches production. Instrument at the span level. Apply the same measurement method at 30, 60, and 90 days. That sequence, repeated, is how a portfolio of measurable agent deployments gets built.
Understanding the gap between agent deployment and production readiness is a related concern: many agents ship before the measurement infrastructure is in place, and the window for establishing a clean baseline closes immediately after launch.
For teams thinking about the broader governance context, questions every head of AI should ask about agent governance covers the organizational side of this problem, and agent cost measurement across agentic workflows addresses how to account for the full cost side of the ROI equation, not just token spend.
Related reading
- Evals vs observability watching your agents is not evaluating them
- Offline to online evaluation gap agent production failures
Where to start
Before the next agent goes to production, record the four baseline metrics for the process it will touch and commit to a measurement method you can reapply. Then instrument the agent at the span level so behavior is tied to outcome, not inferred from it. Those two steps make every subsequent ROI conversation grounded in data rather than assertion.
Start evaluating your agents and review the docs for the instrumentation setup.