How Do You Evaluate a Voice Agent?
What changes when the agent talks: transcription accuracy, latency, turn-taking and tone, and how to measure them on real calls.
Voice agent evaluation measures whether a voice AI agent actually resolves calls: did it hear the caller correctly, respond fast enough to feel natural, handle interruptions, and complete the task. It layers voice-specific signals, word error rate, latency, turn-taking, tone, on top of the standard agent evals, because a voice agent can fail in the audio layer before the language layer ever gets a chance.
What is voice agent evaluation?
Voice agent evaluation is the practice of measuring whether a voice AI agent does its job on real calls: understood the caller, responded appropriately, and resolved the request without needing a human. It is agent evaluation with an extra layer, because a voice agent is a text agent wrapped in speech recognition and synthesis, and every layer can fail independently.
The stakes are different too: not a wrong sentence a user can re-read, but a live conversation where a misheard account number or a three-second silence loses the caller. Evaluation has to cover the conversation as an experience, not just the transcript as text.
What's different about evaluating voice agents?
Four things change when the agent talks. First, speech recognition sits in front of everything: if the ASR layer mishears the caller, the language model reasons perfectly about the wrong input, so word error rate becomes a first-class quality metric. Second, latency is felt, not just measured: a response that takes four seconds reads as broken in a way the same delay in a chat window does not, and turn-taking, knowing when the caller has actually finished speaking, is its own failure mode.
Third, interruptions are normal: callers barge in mid-response, and an agent that cannot stop talking and re-plan fails conversations a text agent never faces. Fourth, tone carries meaning: the same words delivered flat or mistimed read as unhelpful, so open-ended quality scoring has to hear the delivery, not just parse the words.
What metrics matter for voice agents?
Start with the standard anchor: task completion rate, did the call end with the caller's request actually resolved. Add containment rate, the share of calls handled without escalating to a human, which is the business metric most voice deployments are judged on, and read it alongside CSAT, because an agent can contain a call by confidently giving the wrong answer.
Then the voice-specific layer: word error rate on the transcription, and latency at p95, not the average, because the slow tail is what callers remember. A useful discipline is to trace every failure to its layer: a wrong answer caused by a mistranscription is an ASR problem, not a prompt problem, and no amount of prompt work will fix it.
How do you evaluate a voice agent?
Offline, build a golden dataset from real call transcripts, the calls that went well and especially the ones that did not, and replay them through the agent on every change, scoring resolution, groundedness and policy adherence the same way every time. Recorded audio from real calls also tests the ASR layer against real accents, noise and phrasing, which synthetic test audio never quite matches.
Online, sample live calls continuously: score transcripts with the same judges you use offline, track word error rate, latency and containment per agent version, and alert when a rolling metric degrades. Voice agents drift for every reason text agents do, plus the audio stack: a provider update to the speech layer can change behaviour with no change in your code.
How Prefactor evaluates voice agents in production
Prefactor evaluates voice agents the way it evaluates every agent: continuously, in production, per agent and per version. Call sessions are traced end to end, transcripts are scored with judge-based and golden-dataset evals for resolution, groundedness and policy, and containment, latency and cost are tracked as trends, so a regression shows up in a chart before it shows up in your CSAT.
Failed and low-scored calls flow back into the eval dataset, so the suite grows from real conversations rather than imagined ones. For the eval mechanics underneath, see Agent Evals; for a support voice agent running this loop in practice, see the Voice Customer Support case study.
Evaluate every call your voice agents take
Prefactor helps teams observe, evaluate, and improve their AI agents in production — across every framework and provider.
Book a demo →