Voice agents answer phones, qualify leads, book appointments, and handle support. The models behind them have improved quickly. The thing that most often decides whether a voice agent works in production or frustrates every caller is not the model architecture. It is whether the speech data it was trained on looks like the calls it actually receives.
This article covers what training data voice agents and call-center AI need, why the usual mistakes happen, how to assemble a dataset that holds up on real calls, and where to get it. It is written for the teams building these systems and the buyers deciding what data to invest in.
Real Calls Are Not Clean Audio
The defining feature of a voice agent's job is that it operates on live, two-party, spontaneous conversation over a phone line. Every part of that sentence has consequences for the data.
Two-party and overlapping. Real conversations have turn-taking, interruptions, back-channels ("mm-hm", "right"), and stretches where both people talk at once. A model that has only heard one clean speaker at a time struggles the moment a caller talks over the agent.
Spontaneous and disfluent. People planning what to say as they say it produce false starts, self-corrections, filler words, and unusual phrasing. This is the normal texture of speech, and a model needs to have learned from it.
Over a phone channel. Classic telephone audio is narrowband: the G.711 codec samples at 8 kHz, so nothing above 4 kHz reaches the model, and calls also carry compression, line noise, and sometimes echo or dropouts. Wideband codecs such as G.722 sample at 16 kHz and carry audio up to 7 kHz, but a compressed call is still a different signal from a clean microphone recording, and the difference alone can sink an otherwise good model.
The most common and most expensive mistake in this space is training on scripted, single-speaker, studio-clean speech because it is easy to collect, and then being surprised when the agent falls apart on real calls. The audio has to match production, and production is messy. Published numbers show the size of the gap: in OpenAI's Whisper paper, a model trained on LibriSpeech audiobooks and Whisper both scored 2.7% word error rate on LibriSpeech's clean test set of read speech, but on Switchboard telephone conversations the audiobook-trained model scored 28.3% and Whisper 13.8%.
What the Training Set Needs to Contain
A voice agent dataset is not just hours of audio. It is audio plus the structure that makes it learnable. The components that matter:
- Conversational, two-party recordings that reflect the real dynamics of your calls, including overlap and interruption.
- Channel match. Telephone-bandwidth audio for phone agents. If your agent runs over VoIP or a specific codec, represent that.
- Accurate transcripts, verbatim, with the disfluencies preserved rather than cleaned away.
- Speaker separation, ideally one channel per speaker, or diarization labels for mixed audio, so the model can learn who said what and you can separate agent and caller turns.
- Domain vocabulary. Your product names, account terms, place names, and jargon need to appear enough times that the model recognizes them. Out-of-the-box ASR routinely mangles exactly the words your business depends on.
- Accent and demographic coverage that matches your actual caller base, not an idealized one.
- Intent and outcome labels, if you are training the dialogue side, so the system learns to map speech to what the caller wants.
That metadata is not optional polish. For fine-tuning, a well-labeled 100 hours usually does more than an unlabeled 1,000, because the labels are what let you target training, measure accuracy per accent or call type, and find where the model fails.
How Much Data, and Where the Gains Come From
The honest answer depends on whether you are fine-tuning or building from scratch, but the shape of the answer is consistent: coverage beats raw volume.
If you are fine-tuning a strong base speech recognition model, which is what most teams should do, you can see clear improvements from tens to a few hundred hours of well-matched conversational audio in your domain and channel. The gains come disproportionately from data that covers what the base model gets wrong: your vocabulary, your accents, your noise conditions, your call types. We go deeper on sizing in our article on how much audio data you need to train ASR.
The trap is measuring progress in hours. Two hundred hours of the same three accents saying similar things will not fix failures on a fourth accent or an unusual call type. When you plan a dataset, plan the coverage matrix first: which accents, which call scenarios, which noise conditions, which vocabulary, and then collect or license enough of each. That framing catches the gaps that an hours target hides.
The Domain-Vocabulary Problem
One issue is worth calling out on its own because it burns so many voice-agent projects. General speech recognition is now good at everyday language and still often wrong on the specific terms a given business cares about: product SKUs, drug names, local street names, account identifiers spoken as strings of digits and letters. These are exactly the words a call hinges on, and they are underrepresented in generic training data.
The fix is targeted: make sure your training and fine-tuning data contains enough real examples of your critical vocabulary, spoken naturally in context, by a range of voices. This is one of the places where a small amount of the right data, real callers saying your real terms, moves accuracy more than another hundred hours of generic conversation.
Where to Get Voice Agent Training Data
There are a few realistic sources, each covering a different part of the brief, and most mature teams combine them.
Your own call recordings. If you already run a call center, your recordings are the most representative data possible, because they are literally your production distribution. The catch is consent and privacy. Call recordings are full of personal information, and using them to train models requires proper consent, redaction of sensitive data, and a defensible legal basis. This is not a step to skip, and we cover it in our guide on voice data privacy.
Licensed conversational datasets. Off-the-shelf data gives you natural two-party conversation and broad language and accent coverage right away, already transcribed and consented. SpeechData.ai's catalog holds 60 datasets, one per language or dialect, totaling 73,000 hours from 7,250 native speakers. Each is spontaneous two-person conversation recorded in-country, with one channel per speaker, time-aligned and human-verified transcripts, and age, gender, region, and recording device metadata for every speaker. The commercial license covers training and evaluation, and prices, from $60 to $95 per hour of audio, are published on each dataset page. These are everyday conversations rather than customer service calls: they build conversational recognition and turn-taking, and your call types and vocabulary go on top.
Role-played calls recorded to your scenarios. Role-played customer service calls give you call dynamics without real customer data, and control over coverage that a call archive can't give. Native speakers play agent and customer and improvise inside scenario scripts for your industry, such as banking, insurance, telecom, or healthcare scheduling. Each side is recorded on its own channel, as narrowband telephony or wideband audio to match your calls, and names and account details are fictional, so there is nothing to redact. Intent, call outcome, sentiment, and PII labels can be added. The calls are recorded per project rather than sold from stock, so plan for the time a collection takes. Other languages, domains, and recording setups can be collected to your spec the same way.
Two-channel conversation for full-duplex agents. An agent that listens while it talks learns turn-taking, backchannels, and barge-in from conversations with each speaker on a separate channel and transcripts aligned to each channel. Published full-duplex models rely on exactly this: Kyutai's Moshi learned real two-speaker interaction in a fine-tuning stage on Fisher, 2,000 hours of telephone conversations with each side recorded on its own channel. Our guide to full-duplex conversational speech data covers what to check before you buy.
A held-out test set. Whatever you train on, keep a set of calls the model never sees in training: recorded in production conditions, with speakers who don't appear in the training data, and sliced by accent, call type, and channel. Score every model and every candidate dataset against it, and check that the data license covers evaluation as well as training. Our guide to building an ASR test set covers size, splits, and slicing, and the one on voice agent evaluation covers latency, turn-taking, and task success.
| Source | When it fits | What to ask |
|---|---|---|
| Your own call recordings | You run a contact center and have a legal basis to train on calls | What callers consented to, how PII is redacted, which call types are covered |
| Licensed conversational datasets | You need a broad conversational base now, in one language or many | Channel layout, native sample rate, transcript conventions, consent wording, a free sample |
| Role-played calls | You need call dynamics and domain terms without real customer data | Scenario scripts, telephony or wideband capture, speaker quotas, labels |
| Two-channel conversation | Your agent is full-duplex or speech-to-speech | A separate channel per speaker, per-channel timestamps, how overlaps are transcribed |
| Held-out test set | Every project | Speakers kept out of training, production audio conditions, a license that covers evaluation |
The Bottom Line
A voice agent is only as good as its ear, and its ear is trained by data. The data that works is real conversational audio, in your channel, covering your accents and your vocabulary, transcribed and separated by speaker so it is actually learnable, and collected with consent you can stand behind. Scripted, clean, single-speaker audio is comfortable to collect and will not get you there.
Start from real, matched conversational data, whether that is your own consented call recordings, licensed conversational datasets, or role-played calls recorded to your scenarios. Keep a test set that never touches training, and then invest specifically in the domain vocabulary and call types your agent lives or dies on.
