ASR training data is recorded speech paired with transcripts of what was said, cut into short segments with timestamps, plus metadata about every speaker and recording, and the rights to train on it. A speech recognition model learns the mapping from sound to text from those pairs, so it also learns everything else in them: the speaking style, the microphones, the voices and the transcribers' habits.
Three things decide whether a dataset will improve your model: how closely its audio matches what the model will hear in production, how accurate and consistent its transcripts are, and how many different speakers it covers. Hours matter, but only after these three. This guide goes through each part of an ASR dataset, shows how a mismatch shows up in word error rate, and ends with a check you can run on a vendor's sample in an afternoon and a spec table for your RFP.
The five parts of an ASR dataset
Every supervised speech recognition dataset, from LibriSpeech to a commercial corpus, comes down to the same parts:
- Audio: the recordings, with a known sample rate, channel layout and format.
- Transcripts: the text of what was said, written to one convention.
- Segments and timestamps: where each utterance starts and ends, so audio and text can be cut into training examples.
- Metadata: a speaker ID on every segment, and facts about each speaker and recording.
- Rights: consent from every speaker and a license that covers training.
Datasets differ in how well each part is done, and more of one part doesn't make up for a weak one.
Audio: it should sound like production
The audio should sound like the traffic your model will transcribe: the same speaking style, the same channel (phone line, app microphone, headset or far-field), and similar background noise. A model learns the acoustics of its training data along with the words.
Specify the native sample rate, the channel layout and a lossless format such as WAV or FLAC, and ask for the recording device of every file. Our guide to sample rate, bit depth and channel layout has the numbers. The short version: record at your production rate or higher, never upsample, and in conversations give each speaker a channel of their own.
Variety in the audio is fine, and useful if production is varied. Variety in the transcripts is not. The authors of OpenAI's Whisper paper make exactly this distinction: diverse audio quality can help a model cope with diverse conditions, but diverse transcript quality is "not similarly beneficial", so they filtered transcripts hard.
Transcripts: one convention, applied the same way everywhere
Decide the convention before anyone transcribes. Verbatim keeps fillers, false starts and repetitions; clean verbatim removes them. Then write down how numbers, abbreviations, casing, punctuation, overlaps, unclear words and non-speech sounds are handled. Our comparison of verbatim and clean verbatim transcription shows what each choice does to training and to your scores.
Consistency matters as much as the choice. If one transcriber writes "twenty five dollars" and another "$25", the model gets two answers for the same sound. Formatting alone moves the numbers a lot: the Whisper team built a text normalizer so that formatting differences would stop counting as errors, and on several datasets it cut measured WER by up to 50 percent, usually because of a quirk such as references that split contractions from words with a space.
Check that people wrote the transcripts. The Whisper team found that many transcripts on the internet are the output of existing ASR systems, and removed them with heuristics to avoid learning what they called "transcript-ese". One of their signals: an all-uppercase or all-lowercase transcript is very unlikely to be human-generated.
Training transcripts and test references can be held to different standards. The Linguistic Data Consortium's overview of its transcription methods puts the cost of its quickest method at about 5 times real time and its most careful at about 50. The quick method was used to create the 2003 Fisher English corpus, and LDC reports that a 2002 pilot with Switchboard calls found quick transcripts good enough for system training. A test set is different, because every error in a reference is scored as a model error.
Segments and timestamps
Models train on short pieces of audio, each paired with the text spoken inside it. Whisper trains on 30-second segments paired with the part of the transcript that falls within each one. So every transcript needs timestamps, at least per utterance, accurate enough to cut the audio without slicing words in half.
Ask what level the timestamps are at (utterance, segment or word), how boundaries were set and what tolerance was allowed. Then ask how the vendor finds segments where text and audio disagree. The LibriSpeech authors ran a second alignment pass just to drop such segments, caused by errors in the book texts, readers adding or skipping words, and disfluencies. Conversational transcripts are written after the fact rather than taken from a script, so check their alignment at least as closely.
Segmentation can also shift scores. LibriSpeech split its training audio at any pause longer than 0.3 seconds but split its test audio only at sentence breaks, which its authors expected to be easier for a language model. If you compare WER across datasets, compare how they were cut.
Speakers, metadata and splits
A model needs many voices, not many hours from a few people. LibriSpeech, 1,000 hours of read audiobooks, capped each speaker at 25 minutes in its two clean training sets "to avoid major imbalances in per-speaker audio duration". Ask for the number of unique speakers, the hours per speaker and a speaker ID on every segment.
Per-speaker metadata (age band, gender, region or dialect, recording device) lets you balance the training mix and later slice your error rates to see whom the model fails. A field that wasn't recorded at collection is hard to add afterwards.
Splits must be speaker-disjoint: no voice in both training and test. IBM researchers pointed out that 36 of the 40 speakers in the Switchboard part of the NIST 2000 test set, a standard conversational benchmark, also appear in the training data, some in as many as eight conversations.
Plan your test set in speakers rather than hours. The datasets in the SpeechData.ai catalog have between 50 and 200 speakers each, so holding out 10 percent of speakers gives a test split of only 5 people in the smallest and 20 in the largest. That is well short of the 50 speakers per reported slice that our guide to ASR test sets suggests as a starting point, and the guide also covers how to split two-person calls.
Consent and commercial rights
Check rights before anything else. Ask for the consent wording speakers signed and a license that names training and evaluating commercial models. A dataset you can't ship a model on is worth nothing to a product team, however good the audio.
Read speech vs conversation: how mismatch shows up
Read speech and conversation are different inputs. In read speech the text exists first. In conversation people hesitate, restart, overlap, speed up and reduce words, and nobody has written down what they will say. Our article on conversational vs scripted speech data covers the differences in detail.
The Whisper paper puts numbers on the gap. It compares a wav2vec 2.0 Large model, pretrained and fine-tuned on 960 hours of LibriSpeech's read audiobooks, with Whisper large-v2, trained on 680,000 hours of audio from the web. Both score 2.7% WER on LibriSpeech's clean test set. On other audio they come apart:
| Test set | What it is | wav2vec 2.0, LibriSpeech only | Whisper large-v2 |
|---|---|---|---|
| LibriSpeech test-clean | Read audiobooks | 2.7% | 2.7% |
| Switchboard | Phone calls between newly acquainted people on an assigned topic | 28.3% | 13.8% |
| CallHome | Phone calls between friends and family | 34.8% | 17.6% |
| AMI IHM | Meetings, close-talking microphone on each speaker | 37.0% | 16.9% |
| AMI SDM1 | The same meetings, from one far-field microphone | 67.6% | 36.4% |
| CHiME-6 | Dinner parties in real homes | 65.8% | 25.5% |
Word error rates from the Whisper paper's Table 2 and Appendix D, after its text normalizer. Descriptions from Microsoft's Switchboard study, the AMI corpus and the CHiME-5 paper.
Three things stand out. The model trained only on read speech made 10 to 25 times as many errors on conversational audio as on audiobooks. Whisper, trained on far more varied audio, made 55.2% fewer errors on average across the paper's comparison, but still lost accuracy on every kind of conversation. And moving from close-talking microphones to one far-field microphone roughly doubled the error rate of both models on the same meetings.
The lesson for a buyer: if your product hears people talking to each other or to a voice agent, train on conversation recorded on the channel you deploy on. Read prompts still have a job, covering names, product terms and digit strings that conversation won't produce reliably.
Check a vendor sample in an afternoon
Our guide to buying speech data includes a 30-minute listening audit. Before you commit real budget, this longer check puts numbers on the same questions. Ask for a sample you choose from the vendor's file list rather than a showcase, and set your pass and fail thresholds before you open it.
- Listen to 20 random files. Pick them with a random number generator, spread across speakers. Note whether the speaking style, channel and noise match the spec sheet, and whether each channel holds one speaker.
- Re-transcribe 10 minutes. Take short stretches from several files and transcribe them yourself, or have a native speaker do it, following the vendor's own guideline. Allow time: the IBM study estimated 12 to 14 times real time for a professional first pass on phone conversations, so 10 minutes of audio is about two hours of work.
- Score the vendor against your version. Normalize both transcripts the same way (casing, punctuation, numbers) and compute the word error rate of the vendor's transcript, with yours as the reference.
- Read every difference. Some disagreement is normal: on conversational telephone speech, two careful transcribers in an LDC study disagreed on 4.1 to 4.5 percent of words, and quick transcripts on 9.6 percent. When LDC sorted the disagreements in its quick English transcripts, about 65 percent were insignificant (capitalization, punctuation, spelling of hesitations), nearly 20 percent were judgment calls, and about 15 percent were transcriber errors. Count only real errors against the vendor, and weigh the ones that change meaning: names, numbers, negations and missing stretches of speech.
- Check speakers against the metadata. Count distinct speaker IDs, compare age, gender and region with the claimed distribution, and listen to two files per ID to confirm it is the same voice throughout.
- Look for duplicates and silence. Hash every audio file to find exact copies, and search the transcripts for long repeated passages. Run a voice activity detector to measure how much of each file is speech, and compare speech-only hours with the hours you would pay for.
- Check the files. Confirm sample rate, channel count and duration against the manifest, and open a few files in a spectrogram: energy that stops sharply at 4 kHz in a 16 kHz file means narrowband audio relabeled. Check that timestamps still line up near the end of a long file.
A difference in the range careful transcribers show, with few real errors, is a good result. A sample that fails on style, channel or speaker IDs can't be rescued by good transcripts.
How much ASR training data you need
It depends on where you start: adapting a strong pretrained model to a new domain takes far less audio than reaching production quality in a language the model handles poorly. Our article on how much audio data you need to train ASR gives the bands and the caveats. Whatever the number, settle audio match, transcripts and speakers first, then buy in tranches and measure on a held-out set before you scale the order.
ASR training data spec checklist
| Spec line | What to ask for | How to check it on the sample |
|---|---|---|
| Speaking style | Spontaneous conversation, read prompts or commands, matching production | Listen to 20 random files |
| Channel and devices | Phone line (narrowband or wideband), app, headset or far-field, logged per file | Compare metadata with what you hear |
| Audio format | Native sample rate, lossless WAV or FLAC, nothing upsampled | File headers and a spectrogram |
| Channel layout | One channel per speaker for conversations | Play each channel alone |
| Transcription convention | Verbatim or clean, with written rules for numbers, casing, fillers, overlap and unclear words | Read the guideline, then check three files against it |
| Transcript accuracy | A WER target against an independent re-transcription, with the method stated | Re-transcribe 10 minutes and score |
| Timestamps | Utterance or word level, with a boundary tolerance | Check the start, middle and end of a long file |
| Speakers | Minimum unique speakers, maximum hours per speaker, quotas for age, gender and region | Count speaker IDs and compare with metadata |
| Metadata | Speaker ID on every segment, plus age band, gender, region and device | Every field filled and plausible |
| Hours | Total hours and speech-only hours, both stated | Voice activity detection on the sample |
| Duplicates | No repeated recordings or transcripts | Hash files, search for repeated passages |
| Rights | Commercial-use consent from every speaker, a license covering training and evaluation | Read the consent wording and the license |
Paste the first two columns into the data specification of your speech data RFP, and use the third as the acceptance test for the sample and for every delivery.
Every dataset in the SpeechData.ai conversational speech catalog includes spontaneous two-person conversations recorded in-country, one channel per speaker, time-aligned human-verified transcripts, speaker metadata (age, gender, region and recording device) and signed commercial-use consent. Request a free sample, with matching transcripts and the metadata schema, and run these checks on it before you buy.
