A speech recognition model can post a low single-digit word error rate on a public benchmark and still get more than one word in four wrong on phone calls. Neither number is false. They measure different audio, and only one of them sounds like your users.
An ASR test set is a held-out collection of recordings with reference transcripts that you score every candidate model against. If it is built from the kind of speech your product hears, its WER predicts production accuracy. If it isn't, it predicts accuracy on something else.
This guide covers the data side of ASR evaluation: what goes into the test set, how big it needs to be, how to keep it clean, and when to buy one. For the metric itself, see our word error rate guide, and for the rest of a voice agent, from latency to task success, see voice agent evaluation.
Why benchmark scores overstate speech recognition accuracy
The clearest evidence is in OpenAI's Whisper paper. The authors compared Whisper large-v2 with a wav2vec 2.0 Large model fine-tuned on LibriSpeech. Both scored 2.7% WER on LibriSpeech's clean test set.
On conversational telephone speech they came apart. On Switchboard the fine-tuned model scored 28.3% against Whisper's 13.8%, and on CallHome 34.8% against 17.6%. Across the other test sets in the comparison, Whisper made 55.2% fewer errors on average. The benchmark score was identical and told you nothing about which model to ship.
Three things drive the gap.
Speaking style. LibriSpeech is read speech from LibriVox audiobooks. The Open ASR Leaderboard ranks models by average English WER over eight test sets: two LibriSpeech splits, TED talks, European Parliament speeches, earnings calls, financial meetings, GigaSpeech's mix of audiobooks, podcasts and YouTube, and the AMI meeting corpus. That is a broad mix, but it has no customer service calls or voice agent sessions, and the separate multilingual track covers five European languages.
Who was recorded. LibriSpeech's test-clean split has 40 speakers with about eight minutes each. Its authors ranked every speaker by how well an existing model recognized them, labeled the easier half "clean", and drew the clean test speakers from that half. They describe the clean pool as, on average, better recorded and closer to US English accents. It is a fair test of clean read English, and nothing more.
Contamination. Public test sets are public. LibriSpeech's audio comes from public domain LibriVox recordings and its texts from Project Gutenberg, both freely available online. A model trained on web-scale audio and text may have seen material from a benchmark's test set. The Whisper authors de-duplicated transcripts between their training data and TED-LIUM 3 for exactly this reason. You can't inspect a closed model's training data, so only audio that has never been published rules this out.
What a representative ASR test set contains
Start from your production traffic, not from what is easy to collect. Use your usage statistics, or your launch plan, to write the target mix down as percentages: languages, devices, environments, call types. Then build or buy audio that matches it, and over-sample small groups you need to report on separately.
| Test set dimension | Why it matters | How to cover it |
|---|---|---|
| Languages and dialects | Varieties differ in sounds, words and spelling; one "Arabic" score hides them | A subset per variety you ship, from native speakers of that region |
| Accents | Errors cluster by accent, and an average hides the worst groups | Speakers from every region your users come from, with region recorded per speaker |
| Speaking style | Spontaneous speech has fillers, restarts and overlap that read speech lacks | Conversation for conversational products; read prompts only if users read aloud |
| Devices and channels | Microphone, codec and sample rate change the signal; 8 kHz telephony drops everything above 4 kHz | Match the capture path (phone line, app, headset, far-field) and log the device per file |
| Noise and environment | Background speech, traffic and reverberation raise errors unevenly | Record in the places users talk, and label the condition of each recording |
| Speaker demographics | Accuracy can differ by age, gender and community | Quotas for age band and gender, with metadata for every speaker |
| Domain vocabulary | Names, products, amounts and jargon carry the meaning but barely move WER | Scenarios that make speakers use your terms, plus a term list for scoring term recall separately |
| Code-switching | Mixed-language speech trips single-language models and spelling conventions | Include it at the rate your users produce it, tagged by language in the reference |
How big an ASR test set needs to be
A test set has to tell two models apart, or tell you whether one meets a target. Two counts decide how precisely it can: words in the references and distinct speakers.
Words matter because WER is a share of reference words. Speakers matter more, because errors aren't independent. A speaker with a strong accent, fast speech or a poor microphone produces errors across all of their utterances, so each extra minute from the same person tells you less.
An illustrative example, not a measurement: suppose a model's WER on your traffic is 15%, each speaker contributes 1,000 words, and WER measured per speaker varies with a standard deviation of 6 points. The table shows the approximate 95% confidence interval two ways: treating every word as independent, and accounting for the spread between speakers.
| Speakers × words each | Total words | If every word were independent | Accounting for speaker spread |
|---|---|---|---|
| 10 × 1,000 | 10,000 | ±0.7 points | ±4.3 points |
| 20 × 1,000 | 20,000 | ±0.5 points | ±2.8 points |
| 20 × 5,000 | 100,000 | ±0.2 points | ±2.8 points |
| 40 × 1,000 | 40,000 | ±0.3 points | ±1.9 points |
| 50 × 1,000 | 50,000 | ±0.3 points | ±1.7 points |
| 100 × 1,000 | 100,000 | ±0.2 points | ±1.2 points |
Under these assumptions, ignoring speakers is far too optimistic. With 20 speakers the honest interval is about ±2.8 points, roughly 12% to 18% for a model measured at 15%. Five times the audio from the same 20 speakers leaves it at ±2.8, while doubling to 40 speakers narrows it by about a third. The same 100,000 words give ±2.8 from 20 speakers and ±1.2 from 100. For scale, LibriSpeech test-clean, one of the most quoted test sets in speech recognition, has 40 speakers.
How the columns are computed:
- Independent words: 1.96 × √(0.15 × 0.85 / total words).
- Speaker spread: t × s / √n for n speakers, where s is the standard deviation of per-speaker WER and t is Student's t at 97.5% with n - 1 degrees of freedom. With equal words per speaker, overall WER is the mean of the per-speaker WERs.
- More words per speaker: at 5,000 words each, s falls only from 6 to about 5.9 points, because extra words shrink each speaker's sampling noise (about 1.1 points at 1,000 words), not the differences between speakers.
Your own spread will differ, so measure it on a pilot set before you fix the size.
Three rules follow:
- Count speakers per slice. If you will report WER for each accent or device, each slice needs enough speakers on its own. Under the assumptions above, 50 speakers gives roughly ±1.7 points, a sensible floor for a slice you will make decisions on.
- Resample speakers, not utterances. Liu and Peng point out that when utterances are dependent, for example because they come from the same speaker, a bootstrap over single utterances gives confidence intervals much narrower than they should be. They resample whole blocks, such as all of one speaker's utterances, instead.
- Compare models speaker by speaker. On the same speakers, the difference between two models usually varies much less than either model's WER, so a paired comparison can detect smaller gaps than the table suggests.
Keep test speakers out of training
If a speaker appears in both your training data and your test set, the model has already adapted to their voice, microphone and room, and the test overstates how it will do on new users. Make every split speaker-disjoint. The LibriSpeech authors went as far as excluding multi-reader audiobook chapters so that no speaker could cross from training into test.
In practice:
- Assign speakers and conversations to splits together. Both sides of a two-person call go to the same split, and so does every other call either speaker appears in.
- If the data is scripted or role-played, keep scripts and scenarios disjoint too, or the model will have seen the test vocabulary in the same sentences.
- If test and fine-tuning data come from the same source, hold out whole speakers before any ASR training starts.
All of this depends on reliable speaker IDs, so confirm every file maps to one consistent speaker. Held-out speakers also come out of your fine-tuning budget; how much audio data you need to train ASR covers the training side.
Reference transcripts decide the score
WER is measured against a reference, so mistakes and inconsistencies in the reference become your model's errors.
- Use one written transcription convention, with fillers and word fragments marked so they can be scored as optional. Our comparison of verbatim and clean verbatim transcription shows the same model output scoring 0% or almost 30% depending only on the reference style.
- Have a second transcriber check every test reference, and exclude segments marked unclear rather than scoring against a guess.
- Fix the text normalizer and store its version with the test set, so scores stay comparable over time.
Slice the results instead of averaging them
A single WER hides who the model fails. Koenecke and colleagues tested speech recognition from Amazon, Apple, Google, IBM and Microsoft on interviews with 42 white and 73 Black speakers, and found an average WER of 0.35 for Black speakers against 0.19 for white speakers (PNAS, 2020).
Report WER for every slice your metadata supports: accent or region, device and channel, gender, age band, noise condition and speaking style. Add term recall for your domain vocabulary, because a wrong drug name or account number costs a single word in WER. This is why per-speaker metadata matters as much as the audio: you can only slice by what was recorded.
Keep the set private, frozen and versioned
A test set loses value every time it touches training.
- Never train on it or tune decoding settings, prompts or vocabulary lists against it; tune on a separate development set.
- Restrict access. Store it apart from training data, outside any bucket that training jobs read from.
- Freeze it with a version number, a file list with checksums, the transcription guideline and the normalizer. Compare scores only within one version.
Refresh it when production changes: a new market, a new device, new product vocabulary. Build the next version alongside the old one and score both for a release or two to bridge the numbers before retiring the old one. Retire a set at once if any of it leaks into training data.
Build or buy the evaluation set
Production audio is the most representative source, but users must have consented to this use, the recordings hold personal data, and every file still needs a careful reference transcript. Many teams can use little or none of it.
Bought evaluation data has to meet four conditions:
- A license that explicitly covers evaluation. Research licenses often don't. The Linguistic Data Consortium states that non-member licensees cannot use its data to develop or test products for commercialization.
- Speaker IDs and per-speaker metadata, so you can split by speaker and slice results.
- Audio that matches your traffic in speaking style, channel and language.
- No overlap with training data. Catalog data is licensed to more than one buyer, which can include the providers of models you are comparing. If that matters, ask vendors whether exclusive terms are available, and get them in the contract.
SpeechData.ai licenses spontaneous two-person conversations in 60 languages and dialects, recorded in-country with one channel per speaker. Transcripts are time-aligned and human-verified, every speaker has age, gender, region and recording device metadata, and the commercial license covers evaluating ASR models as well as training them. Free samples from the conversational speech catalog let you check the audio against your target mix before you buy.
For contact-center products, call center speech data is recorded to order: native speakers role-play agent and customer in your industry's scenarios, each on their own channel, as narrowband telephony or wideband audio, with fictional names and account details. It is built for voice agents, contact-center ASR and held-out evaluation sets for production calls. For other domains, devices or rooms, custom speech data collection records to your specification.
ASR test set checklist
- Target mix taken from production data: languages, accents, devices, environments, speaking style
- Enough speakers in every reported slice, checked with a confidence interval
- No speaker or conversation shared with training or tuning data
- References in one written convention, second-checked, unclear segments excluded
- One fixed normalizer, versioned with the set
- Age, gender, region and device metadata for every speaker
- Per-slice results with intervals from speaker or conversation resampling
- Restricted access, never used for training or tuning
- A new version whenever production changes
- A license that explicitly covers evaluation
