What Is an Eval Harness?

What is an eval harness?

An eval harness is the runnable scaffolding that executes your evaluations: it loads a versioned test dataset, runs the system under test against each case, applies scorers, aggregates results into metrics, and reports them. Evals are the measurements; the harness is the machinery that runs them reproducibly.

The distinction matters because most teams have evals long before they have a harness. They exist as a notebook someone ran once, a scratch script with a hardcoded model name, and a Slack message containing the results. That's not a harness, and the tell is that nobody can re-run it after the author goes on leave.

Is lm-evaluation-harness the same thing as an eval harness?

No. EleutherAI's lm-evaluation-harness is a specific open-source project for benchmarking language models on standardised academic tasks. A general eval harness is the pattern you build to evaluate your own system on your own tasks. Different jobs: the first compares models, the second tests your agent.

'Eval harness' means two related things, and searchers land here wanting both. The general pattern is any dataset, run, score, report loop you build for your own system, which is what this page covers. The specific project is EleutherAI's lm-evaluation-harness: an open-source framework for few-shot evaluation of language models across 60+ academic tasks (MMLU, HellaSwag, ARC and others). It's the most widely used evaluation tool in the research community and has backed Hugging Face's Open LLM Leaderboard.

The specific project is excellent at what it does: standardised benchmarking of language models on fixed academic tasks. It's a different job from evaluating your agent on your tasks, which is what the rest of this page covers. If you came looking for the project, that's what you want; if you came looking for how to test your own agent, keep reading.

What does an eval harness contain?

Five parts. A harness missing any of them tends to fail in a predictable way.

Dataset loader: pulls the test cases, versioned. Without it, scores move and you can't tell whether the agent changed or the test set did.

Runner: executes the system under test per case, with fixed config. Without it, results aren't reproducible; two people get two numbers.

Scorers: rule-based checks plus model-graded scoring. Without them, you measure only what's trivially checkable.

Aggregator: rolls per-case scores into metrics. Without it, you have 400 numbers and no verdict.

Reporter: emits to CI, a dashboard, or both. Without it, evals run and nobody notices they regressed.

What should an eval harness for an AI agent include?

Beyond the standard five parts, an agent harness needs repeat runs to handle non-determinism, per-case setup and teardown because agents mutate state, full trajectory capture so you can score the path and not just the output, and cost/latency/step-count recorded per case.

Non-determinism. The same input can produce different trajectories. A single run tells you very little, so the harness needs repeat runs per case and a way to report distribution: a pass rate, not a pass.

Statefulness. Agents call tools that mutate things. The harness needs per-case setup and teardown, or case seven fails because case three left a record behind. This is the one that most often turns a working harness into a flaky one.

Trajectory capture. The harness has to record the full sequence of steps, not just the final output, or trajectory-level scoring is impossible after the fact. Capture more than you currently score: retrofitting a scorer onto traces you never stored means re-running everything.

Cost and latency as first-class results. An agent that gets the right answer in twelve tool calls is not equivalent to one that does it in three. Record tokens, wall time and step count per case alongside the score.

How do you build an eval harness for agents?

Build it in the order that works, one part at a time, and make a single case run end to end before adding breadth.

1. Fix a dataset before writing any runner code. Twenty real cases you've actually seen fail beats two hundred synthetic ones. See Golden Datasets for Agents.

2. Make one case run end to end, scored, reported. Resist building all five parts at once.

3. Add rule-based scorers first: exact match, schema validity, tool-call correctness. They're cheap, deterministic, and catch more than people expect.

4. Add model-graded scoring only where rules genuinely can't reach: tone, helpfulness, faithfulness. See LLM-as-a-judge. Every judge you add is a component that itself needs validating against human labels.

5. Wire it into CI so it runs on prompt and model changes, not just code changes. An agent regresses with no code change at all when a provider ships an update.

6. Then point the same harness at production traffic. The harness you built for release gating is most of what you need for continuous evaluation: the dataset becomes a sample of live traffic.

That last step is where the investment pays off, and it's the reason to build the harness properly rather than leaving it as a script.

Should I build an eval harness or use a platform?

Build if you need exact control and can absorb the ongoing maintenance, which is routinely underestimated. Use a platform if you want trajectory capture, scorers and reporting out of the box and a UI where non-engineers can inspect failures. Many teams end up with a library in CI and a platform for production.

Building gives you exact control and costs ongoing maintenance that is always underestimated: the harness becomes a small internal product with its own bugs. A platform gives you the runner, scorers, aggregation and reporting, plus a UI where someone who isn't the author can inspect a failing run.

Prefactor provides the harness so you don't maintain one: trajectory capture by default, rule-based and model-graded scorers, the same evals running as offline release gates and online against production traffic, and failing scores that link back to the step that caused them. If you'd rather build, the structure above is the one to build. For the wider practice, see Agent Evals: A Practical Guide and AI Agent Testing.

What's the difference between an eval and an eval harness?

An eval is a single measurement: a test case plus a scoring rule. An eval harness is the infrastructure that runs many evals consistently: dataset loading, execution, scoring, aggregation and reporting. You can have evals without a harness, but they won't be reproducible.

How many test cases does an eval harness need?

Start with about twenty real cases drawn from failures you've actually seen, rather than a large synthetic set. Grow the set from production failures. A small, real, versioned dataset run consistently beats a large synthetic one run occasionally.