How Do You Evaluate a Coding Agent?
Outcome-based scoring for agents that write code: did the tests pass, how reliably, and at what cost, on benchmarks and on your own repo.
Coding agent evaluation measures whether an AI coding agent produces changes that actually work: the tests pass, the issue is resolved, and the patch holds up, not whether the diff looks plausible. It is the most outcome-based corner of agent evaluation, anchored by benchmarks like SWE-bench and reliability metrics like pass^k, and it still needs evals on your own repository, because a leaderboard score is not evidence about your codebase.
What is coding agent evaluation?
Coding agent evaluation is the practice of measuring whether an AI coding agent does the job it was given: resolves the issue, passes the tests, and leaves the repository better than it found it. It differs from evaluating code completion, which scores a suggestion a human reviews before accepting; a coding agent plans a change across files, edits, runs tests and iterates, so the unit of evaluation is the completed task, not the suggestion.
It is in one way the easiest corner of agent evaluation, because code comes with a built-in oracle: the test suite either passes or it does not. And in another way the sharpest, because a plausible-looking diff that quietly breaks an edge case is exactly the failure a human reviewer is worst at catching.
Outcome-based scoring: did the tests pass?
The scoring rule that matters is outcome-based: apply the agent's patch, run the tests, and grade the result, resolved or not. This is how SWE-bench grades agents on real GitHub issues, and it is the right model for your own evals too: judge what happened, not what was said. A diff review by a judge model can add signal on style and safety, but the anchor verdict is executable.
Outcome scoring needs care to stay honest. The tests have to actually cover the issue, or an agent gets credit for a patch that satisfies a weak suite; and held-out tests the agent never sees prevent it from overfitting to the checker rather than fixing the problem. The oracle is only as good as the tests behind it.
Benchmarks (SWE-bench) vs your own repo evals
SWE-bench and its variants are the shared scoreboard: real issues from real repositories, graded by whether the test suite passes after the patch. Use them the way benchmarks are meant to be used: to shortlist models and agent harnesses before you invest, and to track the field over time.
But a leaderboard score is evidence about the benchmark's repositories, not yours. Your codebase has its own conventions, its own test culture, and its own definition of done. So build a repo eval: a set of real issues from your own history with known-good resolutions, replayed against every model or harness change. Twenty real issues from your tracker beat any public number, because they measure the only thing you actually ship. For how benchmarks and evals divide the work, see AI Agent Benchmarks.
What metrics matter for coding agents?
Resolution rate is the anchor: the share of tasks where the agent's change passes the tests and resolves the issue, end to end. Report it with pass^k, success across k independent attempts, rather than a single lucky run, because an agent you plan to trust unattended has to succeed dependably, not occasionally.
Then the economics: cost and latency per resolved task, not per attempt. An agent that resolves 70% of issues at low cost per resolution can be worth more than one that resolves 80% while burning tokens on long failed runs, and failed runs are pure cost. Tracking cost per resolution, per agent and per version, is what turns 'the coding agent seems good' into a number a team can manage.
How Prefactor evaluates coding agents in production
Prefactor evaluates coding agents as agents, not as text generators: sessions traced across every step, tool call and test run, outcome-based evals scored per task, and resolution rate, reliability, cost and latency tracked per agent and per version. A model upgrade or prompt change that drops resolution or doubles cost shows up as a trend, not a surprise.
Failed sessions become eval cases, so your repo eval grows from the issues your agent actually fumbled. For the eval mechanics underneath, see Agent Evals; for choosing the agent itself, see the best agentic coding tools.
Track resolution rate and cost for every coding agent
Prefactor helps teams observe, evaluate, and improve their AI agents in production — across every framework and provider.
Book a demo →