Quality Assessment

How quality gets measured

Evaluating whether an agent's output was actually good requires knowing what the task was — so there's no single generic score. What Prefactor gives you is the continuous record to measure against: every run, with its inputs, outputs, and the spans that made it up, captured as it happens, scored task by task rather than averaged into a fleet-wide number.

Baselines come from an agent's own prior runs, so a change is read against what that agent normally does — a real shift in behaviour shows up as a shift, not as noise averaged away.

Detecting drift via custom spans

This is the big piece: drift isn't inferred from a single score, it's measured from the custom spans captured by live tracing. Because a custom span records the same operation every time an agent performs it, this week's spans for that operation can be compared directly against last week's — or against the same operation running in a different version or environment. A model update, a prompt change, or a data shift shows up as a difference between those spans, often before it would show up anywhere else.

Qualitative feedback, tied to the moment it's about

A thumbs up or down is only useful if it's attached to the exact thing it's judging. Prefactor captures qualitative feedback natively as a custom span, so a rating attaches to the specific point in a run it's about — one response, one decision, one handoff — rather than a single end-of-conversation rating that leaves you guessing which part it was actually about.

Quality as a review signal

Run history and drift feed into Prefactor's risk monitoring alongside the other signals it already tracks. An agent whose behaviour is drifting can be flagged for review and, where a team has runtime policies configured, routed to the same throttle, sandbox, or escalate actions available there — keeping a human in the loop on what "degraded" actually means for a given task.