A voice agent can pass every text test you write and still fail on the phone. It hears "four one five" as "for one five", starts talking while the caller pauses to find a card number, or says "you're all set" when the booking call to the backend failed.
Voice agent evaluation measures the whole path from the caller's audio to the action in your systems and back to the agent's voice. This guide covers what to measure at each layer, then the data behind the measurements.
Score each layer and the whole call
Whether your agent is a cascade of speech recognition, a language model and text-to-speech, or one speech-to-speech model, measure the layers separately to see where a failure started. Then measure the whole call, because a misheard digit becomes a wrong lookup and then a confident wrong answer.
End to end, judge what the agent did, not what it said. OpenAI's GPT-Live evaluation guide, written for full-duplex voice agents, makes task completion the guiding measure and follows each request through the tools actually executed to the resulting application state. A natural-sounding confirmation, it notes, does not prove the backend action ran.
| Layer | Metric | How to measure | Data you need |
|---|---|---|---|
| Speech recognition | WER on caller turns; entity accuracy | Score caller transcripts against human references; compare spoken IDs and numbers with the values the agent used | Caller audio in your channel, verified transcripts, entity values per call |
| Latency | Time to first audio, p50 and p95 | End of caller speech to start of agent audio, timed on the caller's side | Calls with each side on its own channel |
| Turn-taking | Cut-ins, missed turns, barge-in stop time, false stops | Voice activity detection per channel on turns with pauses and interruptions | Spontaneous speech with natural pauses and backchannels |
| Task success | Completion, containment, correct transfers, pass^k | Backend state against each scenario's expected end state, over repeated runs | Scenarios with goals and expected outcomes |
| LLM layer | Grounded answers | Claims checked against your sources; LLM judge calibrated on human labels | Questions inside and outside your knowledge base |
| TTS | MOS or A/B preference; pronunciation | Native listener ratings; ASR on agent audio against the intended text | A term list; native listeners per language |
| Slices | Every metric above | Results by accent, noise, channel and speaking rate | Speaker and recording metadata |
Speech recognition: your callers, your entities
Measure word error rate on the caller side of realistic calls against human-verified reference transcripts, not on read or broadcast benchmarks. Our guide to building an ASR test set covers why, how many speakers you need and how to split them.
WER alone misses what matters most. A wrong digit in a 12-digit account number barely moves WER but fails the whole task. Score entity accuracy separately for names, dates, amounts, account numbers and addresses. Check it twice: in the transcript, and in the value the agent finally passed to a tool. The second check catches normalization errors, such as "double two" or "oh" for zero.
Count recoveries too: how often the agent reads a critical value back, and how often a wrong value still reaches the backend.
Latency: time to first audio and end-of-turn detection
Define latency as the time from the end of the caller's speech to the first audio the agent plays, as OpenAI's guide does. It counts a spoken preamble as that first audio, so a quick "let me check" can hide a slow lookup. Whichever rule you pick, keep it fixed across versions.
Measure on the caller's side of the real phone or app path, since telephony and network hops add time that server logs miss. Report the median and 95th percentile, with tool-calling turns split out.
Give end-of-turn detection its own number, separate from model, tool and TTS time, because shortening it is a trade-off: wait less and the agent answers sooner but cuts in more often.
For a sense of human timing: in a study of informal conversation in 10 languages, the most common gap between a yes or no question and its answer was 0 to 200 ms in every language.
Turn-taking: endpointing, barge-in and backchannels
Endpointing fails in two directions. Decide too early and the agent cuts in while the caller pauses between groups of digits. Decide too late and the call drags, or the agent never answers.
Full-Duplex-Bench, an open turn-taking benchmark, scores both. Its pause-handling test draws real turns with internal pauses of 0.4 to 1.0 seconds and measures the takeover rate: how often the model takes the turn when it should stay quiet. It also scores turn-end latency, backchannels and interruptions.
Test three behaviors in your own agent:
- Pauses. Play caller turns with natural mid-turn pauses and count cut-ins. Silence alone is a weak end-of-turn signal, as our guide to full-duplex conversational speech data explains.
- Barge-in. Interrupt the agent mid-sentence. Measure how fast it stops and whether its reply addresses what the caller said instead of resuming its script.
- Backchannels and noise. A caller's "mm-hm", a cough or a television should not stop the agent. Count false stops.
Task success and containment
Give every test scenario an expected end state: the appointment booked for the right slot, the payment posted, the ticket created with the right fields. Score the backend, not the agent's last sentence.
Containment, the share of calls resolved without a human, needs a partner metric, because an agent that never transfers scores perfectly. Count correct transfers too: calls where the caller asked for a person or the request was outside policy, and the agent handed over.
Run each scenario several times, because language models vary between runs. τ-bench, a 2024 benchmark of tool-using agents talking with simulated users, introduced pass^k, the chance that all k trials of a task succeed. At publication, even GPT-4o agents succeeded on fewer than half of its tasks, and pass^8 was below 25% in its retail domain.
Voice agents show the same pattern. Across 12 systems in EVA-Bench, ServiceNow's 2026 voice agent benchmark, peak accuracy (pass@k) exceeded reliable accuracy (pass^k) by a median of 0.44.
Grounding in the language model layer
Every fact the agent states, from a fee to an opening time, should trace back to your sources. Include questions your knowledge base cannot answer, where the right move is to say so or transfer.
LLM judges make this scale, but check them first. A 2026 study of LLM judges for voice agents found their reliability depends on the metric and the setup, and recommends keeping human reviewers for judgments that need context.
TTS quality: naturalness and pronunciation
Rate naturalness with a mean opinion score, where listeners score samples from 1 to 5, or with A/B preference tests, which separate close systems more easily. Use native listeners and the agent's real responses, played through the channel callers hear. Our explainer on how speech synthesis is evaluated covers MOS and its pitfalls.
Pronunciation needs its own test. Have the agent say each product name, street name, amount and confirmation code in context, transcribe the audio with ASR, compare it with the intended text, and have native listeners flag errors. ServiceNow's open-source EVA framework, the code behind EVA-Bench, scores the same thing, with attention to entities such as confirmation codes and dollar amounts.
Slice every metric
Report each metric by accent or region, background noise, channel (landline, mobile or VoIP, narrowband or wideband) and speaking rate, not only as one number per release. EVA-Bench ran accent and noise perturbations and found gaps that varied by architecture, system and metric, so one agent's weak slice is not another's.
The data: a frozen set of realistic caller audio
You will change prompts, models and speech providers often, and scores only compare across versions on the same calls. So build a frozen, versioned evaluation set: caller audio for each scenario, human-verified transcripts, the entity values spoken, the expected end state, and speaker and recording metadata.
Tune prompts and thresholds on a separate development set. Keep production failures in a separate regression suite, where they won't skew the held-out averages.
Where test calls come from
Scripted calls by your own team. Fast and good for smoke tests, but your staff know the flows, speak clearly, call from quiet offices and are a handful of voices, so scores come out high.
Replays of real calls. The closest match to production, but they carry callers' personal data, so you need consent for this use and redaction, as our guide to voice data privacy covers. A replay also can't answer the agent's new questions. OpenAI's guide notes that replaying a saved recording can't establish multi-turn clarification, yielding, backchannel handling or recovery.
Role-played calls recorded by native speakers. Speakers improvise inside scenario scripts as caller and agent rather than reading lines, so the audio carries the hesitations, restarts and interruptions of unscripted speech, while the scenarios control which entities and edge cases come up. Names and account details are fictional, so there is no customer data to redact.
SpeechData.ai records role-played call center data to spec: native speakers as agent and customer, industry scenario scripts, one channel per speaker, narrowband telephony or wideband. Held-out evaluation sets for production calls are one of its listed uses. Like replays, these calls can't react to your agent, so use them for recognition, entity and timing tests and as the human reference for your simulator.
Simulated callers. A language model plays a caller with a goal and persona and speaks through TTS, so it reacts to the agent. It is the practical way to rerun multi-turn tests after every change.
EVA's authors list the limits of their own simulator: it may not fully replicate real callers' disfluencies, hesitations or emotions, results may not generalize to other languages or accents, and one provider's voices may favor certain speech recognition systems. Full-Duplex-Bench-v3, built entirely from real human audio annotated for disfluencies, found self-correction handling among the most consistent failures. A simulated caller only corrects itself when its language model writes the correction in, and TTS then speaks it cleanly.
Calibrate the simulator against people. If your ASR scores much lower WER on simulated caller turns than on human ones for the same scenarios, treat the other simulated results as optimistic too.
Benchmarks mix sources the same way. Full-Duplex-Bench takes most of its pause tests and all of its turn-taking tests from Candor, an 850-hour corpus of spontaneous two-channel conversation, and synthesizes speech for interruptions, which public datasets rarely contain.
Cover every language you ship
Build a set for each language and dialect, with native speakers from your callers' regions. Translated English scripts voiced by TTS measure your TTS voice, not your callers.
For a conversational baseline per language, SpeechData.ai's catalog covers 60 languages and dialects of spontaneous two-person conversation with one channel per speaker, time-aligned and human-verified transcripts, and speaker metadata (age, gender, region, recording device). The commercial license covers evaluation as well as training. These are everyday conversations rather than service calls, so they suit recognition and turn-taking baselines, with scenario tests on top.
How often to refresh
- Rerun the full set on every change to a model, prompt, speech provider, endpointing setting or telephony route.
- Pin the simulated caller's model, persona prompts and TTS voices, so a score change comes from the agent and not the caller.
- Compare the set's mix of languages, call types and channels with production every month or quarter.
- Add scenarios and audio when you launch a new language, market, product, call type or channel.
- Replace any test call that ends up in the agent's prompt examples or training data.
Voice agent launch checklist
- Entity accuracy measured on the values passed to tools
- Time to first audio at p50 and p95 from the caller's side
- Cut-ins, barge-in stop time and false stops counted
- Task completion scored against backend state over repeated runs
- Containment reported next to correct transfers
- LLM judges checked against human labels
- Domain terms checked by native listeners
- Results sliced by accent, noise, channel and speaking rate
- Simulator calibrated against human caller audio
- A frozen evaluation set in every launch language, under a license that covers evaluation
