What retroactive session analysis gives you
Coding agents are merging code at a pace that outstrips the review patterns most teams built for human-authored pull requests. Claude Code achieved an 84% pull request merge success rate in an autonomous deployment study, which sounds strong until you consider what sits in the remaining 16%, and what proportion of the merged 84% carries subtle defects that tests did not catch. The answer depends almost entirely on what you recorded during the session and what you did with that recording afterward.
Retroactive session analysis means replaying what an agent did, step by step, after execution completes, to surface decisions that looked locally correct but produced globally poor outcomes. It is distinct from live monitoring: you are not trying to stop a bad action in flight, you are building the evidence base to understand why a defect entered the codebase and how to prevent the next one. Teams that skip this layer catch bugs at merge review if they are lucky, or in production if they are not.
This article covers how to instrument coding agent sessions so that retroactive auditing is possible, what signals indicate quality gaps, and how the pattern is working in practice.
Why the 84% number needs context
An 84% merge rate is meaningfully higher than what many teams see with AI-generated code today. One dataset from January 2026 found that 67.3% of AI-generated pull requests were rejected, versus 15.6% for manually written code. The gap between those two figures reflects not just model capability differences but instrumentation differences: teams achieving higher merge rates have generally invested in evaluation loops that feed quality signals back before the pull request is opened, not after it is rejected.
The practical implication is that merge rate alone is a weak quality metric. A pull request can merge cleanly, pass CI, and still carry a design flaw that only appears under load or when an adjacent service changes. Retroactive analysis of coding agent tool call quality addresses exactly this gap: it examines not just whether the agent produced code that compiled and passed tests, but whether the sequence of decisions that produced it was coherent.
What to instrument
To make retroactive analysis possible, you need a record that goes beyond the final diff. The session record needs to capture at minimum:
- Every tool call the agent made, in order, with inputs and outputs
- The context window state at each decision point, or a compressed representation of it
- Any intermediate artifacts: scratch files, test runs, linter output, error messages the agent received and acted on
- The agent's stated reasoning where it is surfaced, and any self-correction steps
Open-source tooling like Gage takes this approach for Claude Code sessions specifically: it replays the session log and applies a secondary analysis pass to identify bugs and design flaws that the agent did not flag and that human reviewers frequently miss. The key insight behind Gage is that the agent's own session log contains enough signal to perform a meaningful audit, provided you capture it completely and analyze it with a model that has full context of the session rather than just the final diff.
Instrumenting through OpenTelemetry spans is one way to get this data into a durable, queryable form. Each tool call becomes a span, with attributes carrying the inputs and outputs. The trace for a session then gives you a replay-ready record that you can feed into an analysis pipeline.
Prefactor's SDK instruments agent sessions at the span level, recording tool call sequences, scoring them for quality and risk, and validating the trace against an activity schema that defines what the agent was supposed to do. That validation step is what turns a raw log into a structured audit trail: you can ask not just "what did the agent do" but "did the agent stay within the scope of the task."
Signals that indicate quality gaps
Once you have a session record, the analysis looks for several categories of signal.
Tool call coherence. An agent that reads a file, writes a modification, then reads the same file again before making a second modification is behaving coherently. An agent that makes a modification without reading the relevant file first, or that calls a search tool repeatedly with semantically identical queries, is showing signs of poor context utilization. These patterns are invisible in the final diff but visible in the trace. Step-level accuracy and cascading failures are easier to trace back to specific decision points when you have the full tool call record.
Self-correction rate and loops. Agents that correct themselves frequently may be navigating genuinely complex problems, or they may be stuck in a local loop that produces a lot of churn without making progress. A high self-correction rate with decreasing edit distance between successive versions is a warning sign worth flagging for human review.
Coverage gaps relative to scope. If a task required modifying three files and the agent touched two, the analysis should surface the omission. This requires the analysis step to understand the original task scope, which means capturing the initial prompt and any clarifications in the session record.
Design-level issues in the diff. Gage's approach is to apply a secondary model pass over the full session context, not just the diff, asking whether the implementation approach the agent chose is appropriate given the stated constraints. This catches cases where the agent produced correct code for the wrong abstraction level, for example, solving a problem inline that should have been extracted to a shared function.
Agent behavior gaps between what you think an agent does and what it actually does are most visible at this level. The diff looks fine; the decision sequence reveals the problem.
How teams are using this in practice
Stripe's Minions system, which merged more than 1,300 pull requests per week from autonomous agents with no human-authored code as of February 2026, requires exactly this kind of audit infrastructure to function at that volume. You cannot review 1,300 pull requests per week with human attention on every line. The review capacity has to be concentrated where signals indicate it is needed.
Anthropic's internal deployment, where Claude Code authors roughly 80% of production code and engineers are shipping eight times more code per quarter, faces the same constraint. At that ratio, the engineering team's review time is the bottleneck. Retroactive session analysis is what allows that review time to be applied to the sessions most likely to contain issues rather than distributed uniformly across all of them.
The pattern that emerges from both cases is a triage layer: run session analysis immediately after the agent completes, score the session on coherence and coverage, and route sessions above a risk threshold to human review before the pull request is opened. Sessions below the threshold proceed to CI and standard review. This keeps human attention on the cases where it matters without slowing down the sessions that are straightforwardly clean.
Detecting agent quality decay and production drift requires storing session scores over time so you can see when a model update or a prompt change shifts the distribution. A single session score is informative; a trending score is diagnostic.
Building the audit trail
The audit trail serves two purposes. The first is operational: when a defect reaches production, you need to be able to trace it back to the session that introduced it, understand what the agent was doing and why, and identify whether the failure was a tool call error, a context gap, or a scope misunderstanding. Audit trail design for CI/CD pipelines with AI agents covers the structural requirements in detail.
The second purpose is governance. As agent behavior validation against activity schemas becomes part of engineering standards, teams need to show that every agent session was recorded, scored, and reviewed according to a defined process. That demonstration requires an audit trail with enough fidelity to reconstruct the session, not just a summary of the outcome. Auditability and reproducibility in autonomous workflows describes what that standard looks like in practice.
Related reading
- Evals vs observability watching your agents is not evaluating them
- Agent evaluation in production what to measure and how to prove it
- Evaluating coding agents requires measuring tool call quality not just
- Evaluation gap offline vs production agent failures
- Offline to online evaluation gap agent production failures
- Openai hugging face incident real time agent evaluation
- Silent agent failures production evaluation gaps cost
- Silent failure gap agent quality evaluation before production
- Step level accuracy trajectory evaluation production agents
Where to start
Pick one agent workflow that is already running in your environment and add full tool call capture to its sessions this week. Run a secondary analysis pass over the last ten sessions and count how many contain tool call coherence issues or coverage gaps you did not catch at review. That number will tell you whether the investment in a structured analysis pipeline is warranted.
Start evaluating your agents or read the docs to see how Prefactor instruments sessions and routes them for review.