Agent Evaluation Template

The dimensions

Score every run on the dimensions that matter for your agent. Drop the ones that don't apply; add domain-specific ones that do.

Define the levels

For each dimension, set the levels so two reviewers grade the same run the same way. Two worked examples:

Build the baseline

A fixed set of inputs with known-good outputs that you re-run on every change. It is what lets you tell an improvement from a regression.

• 15 to 50 representative inputs, from real traffic where possible
• Cover the common path and the edge cases that have burned you before
• Include known failure inputs: adversarial, ambiguous, out of scope
• Each input has a reference answer or an acceptance rubric
• Stored in version control next to the agent
• Refreshed when the product or policy changes

More on golden datasets for agents →

When to re-run

• On every prompt or agent-logic change (block merge on a regression)
• On every model or model-version change, because providers ship silent updates
• On every tool or integration change
• On a schedule (weekly or monthly) to catch drift with no code change
• Alert when any dimension drops more than a set threshold vs baseline

Offline evals are a spot check; production is the real test. Prefactor runs this cadence continuously, scoring live runs and enforcing thresholds. Seeagent evaluation →

Wire it into Prefactor

The template says what to score. Prefactor is where the scores live in production: each dimension becomes a custom span, next to cost and latency on every run.

import { withSpan } from "@prefactor/core";

// Score one run on a template dimension and send it to Prefactor
// as a custom span. Repeat per dimension you want to track.
await withSpan(
  {
    name: "groundedness check",
    spanType: "eval_groundedness",
    inputs: { answer, sources },
  },
  async () => {
    // your rubric or LLM-as-a-judge returns a 0 to 1 score
    const score = await judgeGroundedness(answer, sources);
    return { score, passed: score >= 0.8 };
  },
);

Or hand this setup prompt to your coding agent

Set up Prefactor evaluation for my agent.
1. Install the Prefactor CLI.
   macOS/Linux: curl -fsSL https://raw.githubusercontent.com/prefactordev/typescript-sdk/main/scripts/install.sh | bash
   Windows:     irm https://raw.githubusercontent.com/prefactordev/typescript-sdk/main/scripts/install.ps1 | iex
2. Run "prefactor login" and paste the API token from my Prefactor
   Dev tier account (25,000 free spans a month), then export it as
   PREFACTOR_API_TOKEN so the SDK can read it.
3. Install prefactor-core in my project and initialise the client.
4. For each dimension in the Prefactor agent evaluation template
   (task success, groundedness, format conformance, policy adherence,
   safety), wrap my agent's output in a withSpan call with
   spanType "eval_<dimension>", returning a 0 to 1 score and a
   passed boolean against a threshold I set.
5. Emit these spans on every run so the scores show up in Prefactor's
   Evaluate view alongside cost and latency.

Want it walked through on your stack? Book a demo →

Frequently asked questions