How Do You Evaluate a Voice Agent?

What is voice agent evaluation?

Voice agent evaluation is the practice of measuring whether a voice AI agent does its job on real calls: understood the caller, responded appropriately, and resolved the request without needing a human. It is agent evaluation with an extra layer, because a voice agent is a text agent wrapped in speech recognition and synthesis, and every layer can fail independently.

The stakes are different too: not a wrong sentence a user can re-read, but a live conversation where a misheard account number or a three-second silence loses the caller. Evaluation has to cover the conversation as an experience, not just the transcript as text.

What's different about evaluating voice agents?

Four things change when the agent talks. First, speech recognition sits in front of everything: if the ASR layer mishears the caller, the language model reasons perfectly about the wrong input, so word error rate becomes a first-class quality metric. Second, latency is felt, not just measured: a response that takes four seconds reads as broken in a way the same delay in a chat window does not, and turn-taking, knowing when the caller has actually finished speaking, is its own failure mode.

Third, interruptions are normal: callers barge in mid-response, and an agent that cannot stop talking and re-plan fails conversations a text agent never faces. Fourth, tone carries meaning: the same words delivered flat or mistimed read as unhelpful, so open-ended quality scoring has to hear the delivery, not just parse the words.

What metrics matter for voice agents?

Start with the standard anchor: task completion rate, did the call end with the caller's request actually resolved. Add containment rate, the share of calls handled without escalating to a human, which is the business metric most voice deployments are judged on, and read it alongside CSAT, because an agent can contain a call by confidently giving the wrong answer.

Then the voice-specific layer: word error rate on the transcription, and latency at p95, not the average, because the slow tail is what callers remember. A useful discipline is to trace every failure to its layer: a wrong answer caused by a mistranscription is an ASR problem, not a prompt problem, and no amount of prompt work will fix it.

How do you evaluate a voice agent?

Offline, build a golden dataset from real call transcripts, the calls that went well and especially the ones that did not, and replay them through the agent on every change, scoring resolution, groundedness and policy adherence the same way every time. Recorded audio from real calls also tests the ASR layer against real accents, noise and phrasing, which synthetic test audio never quite matches.

Online, sample live calls continuously: score transcripts with the same judges you use offline, track word error rate, latency and containment per agent version, and alert when a rolling metric degrades. Voice agents drift for every reason text agents do, plus the audio stack: a provider update to the speech layer can change behaviour with no change in your code.

How do you test a voice agent before it takes live calls?

Voice agent testing is the offline half of voice agent evaluation: replaying real call transcripts and recorded audio through the agent on every change, before it takes live calls. Build a golden dataset from real calls, the ones that went well and especially the ones that did not, and score resolution, groundedness and policy adherence the same way every time. Use recorded audio rather than synthetic speech, so the speech-recognition layer is tested against real accents, background noise and phrasing.

Test each layer separately, because each can fail independently: word error rate on transcription, latency at p95 rather than the average, turn-taking and interruption handling, and task completion in the language layer. A wrong answer caused by a mistranscription is an ASR problem, not a prompt problem, and the test should say which.

Then keep testing after launch by sampling live calls with the same judges, since a provider update to the speech layer can change behaviour with no change in your code. For the general three-layer approach, see AI Agent Testing: A Practical Guide.

How Prefactor evaluates voice agents in production

Prefactor evaluates voice agents the way it evaluates every agent: continuously, in production, per agent and per version. Call sessions are traced end to end, transcripts are scored with judge-based and golden-dataset evals for resolution, groundedness and policy, and containment, latency and cost are tracked as trends, so a regression shows up in a chart before it shows up in your CSAT.

Failed and low-scored calls flow back into the eval dataset, so the suite grows from real conversations rather than imagined ones. For the eval mechanics underneath, see Agent Evals; for a support voice agent running this loop in practice, see the Voice Customer Support case study.