What you actually need to measure in coding agent runs
Speed metrics tell you how fast an agent finished. They do not tell you whether the agent did the right thing. The 1Password research published this week makes that distinction concrete: across 6,080 AI-generated patches evaluated against ChatGPT 5.5 and Claude Opus 4.8, only 26% fully remediated the vulnerability without altering behavior. The remaining 74% either failed completely, introduced new vulnerabilities, or changed program behavior in ways that were not requested.
That number is not an argument against coding agents. Stripe completed a 10,000-line Scala-to-Java migration in four days that would have taken roughly ten engineer-weeks without AI assistance. The question is not whether agents produce value. It is whether your evaluation layer catches the fraction of output that is wrong before it reaches a gate, a deployment pipeline, or a production codebase.
If you are running a coding agent on security-sensitive work and measuring only task completion time, you are missing the signal that matters.
Why tool call spans are the right unit
An agent session produces a sequence of decisions. Each decision involves selecting a tool, constructing arguments, calling the tool, and handling the result. These steps are recorded as spans in the agent's execution trace. A span for a file-write call records what file was targeted, what content was written, and what the agent received back. A span for a bash execution records the command and its exit code.
Completion speed aggregates across all of those spans. It tells you the session took 47 seconds. It does not tell you that on step 9 the agent wrote a patch that left an injection point open, or that on step 12 it silently removed a validation check because the test still passed.
The 1Password findings break down exactly here. Their analysis found that 53.9% of patches failed completely or introduced new vulnerabilities, a category that looks like success from a speed perspective because the agent completed, the file changed, and the test suite often still ran. The failure was in what the patch actually did, not in whether the agent finished.
This is consistent with what step-level accuracy research shows about cascading failures: an error at one tool call propagates silently through subsequent steps because each step trusts the output of the one before it.
What to score at the tool call level
For coding agents specifically, three categories of tool call quality matter:
Patch correctness. Does the change address the root cause, or does it address a symptom? You cannot determine this from the diff alone. You need to trace which vulnerability the agent was given, which code location it targeted, and whether the resulting change eliminates the vulnerable path. This requires scoring the relationship between the input context span, the file-read spans, and the write span that followed.
Behavior preservation. Did the patch change anything beyond the targeted vulnerability? A patch that fixes an injection flaw by removing the function entirely scores zero on behavior preservation even if it passes a security scan. Existing evaluation literature calls this semantic equivalence checking, and it is absent from most current agent pipelines.
Tool selection validity. Did the agent use the tools in a sequence consistent with the task? An agent asked to patch a dependency vulnerability that runs `npm install` without first reading `package.json` or checking lockfile state is exhibiting a planning error that the completed-session metric cannot surface.
Veracode data from May 2026 found that 45% of AI-generated code samples failed security tests overall, with Java performing worst at a 72% failure rate. These failures are not random. They cluster around specific tool use patterns, and those patterns are visible in the span trace.
Where session reconstruction fits in
Platforms like Hoplite and Armature now reconstruct agent sessions from span data, giving you a replay of what the agent decided and when. That is a necessary foundation. What it does not yet provide for most teams is a scoring layer that evaluates each span against task-specific correctness criteria.
The gap is important. Observability tells you what happened. Evaluation tells you whether what happened was correct. These are different problems. A session reconstruction that shows you the agent wrote to `auth.py` at step 7 does not, by itself, tell you whether the write was a valid remediation of the CVE the agent was assigned.
For teams building out this layer, the practical approach is to define activity schemas that describe the expected tool call sequence for a given task type, then validate each session against that schema. A coding agent given a SQL injection patch task should follow a pattern: read the affected file, read surrounding context, write a targeted change, verify the change. Deviations from that pattern, not just failures, are the signal you want to flag for human review.
Anthropic's 2026 State of AI Agents Report found that 57% of organizations use agents to handle multi-stage workflows, but only 16% have progressed to cross-functional processes. The gap between those two numbers is partly an adoption lag. It is also partly a trust deficit that comes from not having a way to know, at the span level, which agent decisions require a human check before proceeding.
Routing human review with scoring
The 1Password researchers concluded that AI-generated security patches cannot be deployed without expert human code review. That is accurate for the current generation of models, but "review everything" does not scale when you have agents running across dozens of repositories.
The scoring approach that actually works is triage-based. You score each session against correctness criteria, and you route sessions above a threshold automatically while flagging sessions below it for human review. The threshold is not set once. It drifts as agent behavior drifts in production, which is why detecting quality decay over time is a separate problem from point-in-time evaluation.
Prefactor records tool call spans via its SDK, scores each span against task-specific schemas, and surfaces sessions where tool selection or output quality falls below configured thresholds. The audit trail it keeps means that when a session is flagged, the reviewer sees the full decision sequence, not just the final diff. This is what makes the human review step tractable at volume, rather than a bottleneck.
Gartner projects that over 40% of agentic AI projects will be canceled by end of 2027 due to escalating costs, unclear business value, or inadequate risk controls. Unclear business value often means the team could not demonstrate that the agent output was correct, not that the agent was slow.
For teams already running autonomous background agents or multi-agent orchestrators on code, adding span-level scoring is the concrete step that turns session reconstruction into a gate with teeth.
Where to start
Instrument your coding agent with a span-recording SDK so you have the raw tool call data. Define a schema for at least one task type, a vulnerability patch or a refactor, and validate production sessions against it to find where your agent's actual behavior diverges from the expected sequence. Then set a review threshold and route accordingly.
Start evaluating your agents and read the docs to see how Prefactor's span instrumentation and scoring layer connect to your existing agent setup.