How to build an evaluation framework for your agent

Seven short modules, each teaching one part of an evaluation framework for an AI agent. Take them in order or pick the one you need. About 60 minutes in all, no code. Five of the seven let you try the idea on a real agent inside a Prefactor workspace.

1. DesignHow to define a good run for an agent and turn it into criteria anyone can check. Thinking, not tooling.8 min
2. BuildHow to build a golden set: real cases with the expected answer at every step, that stands in for "it works".8 min
3. Types of testThe three ways to check a run (heuristics, LLM-as-judge in its five kinds, people) and testing before release against testing in production.7 min
4. TestHow the three layer into one framework: heuristics on every run, a hand-off to a calibrated judge, and how to tell whether each heuristic earns its keep.10 min
5. OperateWhy every live run is checked rather than a sample, how to read failures by category and tail, and how live failures feed the golden set.9 min
6. ImproveHow to treat every change as a version, compare versions on live traffic, decide on the tail, and spot drift when nothing shipped.9 min
7. Common mistakesEight mistakes, why teams make them, what each costs, and where the fix is.4 min

The worked agent handles cancellation requests for a subscription product: it reads the email, looks up the account, decides an offer inside the rules, writes the reply and acts. Every screen shows its real runs.