Docsprefactor.aiGet started, free
Cost of evals in production

What do evals cost once the agent is live?

Three numbers about your agent. One answer: what judging every run costs, and what you keep by letting rules go first.

Tokens per run, in plus out. Edit if you know your number.
Rule checks are not counted. They cost nothing.

Why not just judge every run?

  1. Cost scales with every run and every criterion. That is the left-hand number, and it grows with the business.
  2. Judges are not certain. The best judge models agree with people about 80% of the time (Zheng et al., 2023). A rule is right every time on what it covers, and it is auditable.
  3. Rules give categories, judges give scores. A failed rule names what went wrong. Rules first is not cheaper coverage; it is better coverage, with the judge kept for what needs an opinion.

The five kinds of judge are in learning module 3; the layers, the minimum set of heuristics and the hand-off are in module 4.

Use Prefactor to optimise your evaluation strategy.

Rule checks on every run at no model cost. The judge only where a rule cannot decide. Fewer tokens on evaluation, the same verdict on every run, and the budget goes back into building.

Talk to PrefactorSet up your quality framework
How the numbers are worked out
The judge reads the run plus a token rubric and writes a token verdict. Rule checks resolve % of runs without a judge. Targeted judging sends % of runs to the judge. Share of live runs evaluated: 100% (sampling misses the tail, which is where the complaints live).

Prices: Claude Sonnet $3.00 in, $15.00 out; GPT-5.6 Terra $2.00 in, $12.00 out, per million tokens. From models.dev, refreshed nightly in the real build; the prototype uses fixed sample values. List price, before caching or committed spend. Prefactor's own price is not part of this calculation.