An audio deepfake detector needs two classes of training data: bona fide speech from real people, and spoofed speech made with text-to-speech, voice conversion or replayed recordings. It learns whatever separates those two classes, whether or not the difference has anything to do with synthesis. When the bona fide side doesn't match production audio in language, accent, phone channel, codec, speaking style and range of speakers, the detector learns shortcuts and fails in the field.
This article is for teams that build or fine-tune a detector: what public deepfake detection datasets contain, why detectors trained on them break, how to specify the real speech yours learns from, and what to ask a vendor.
What an audio deepfake detector learns from its data
In Speech is Silver, Silence is Golden (ASVspoof 2021 Workshop), Müller and colleagues showed that bona fide clips in the ASVspoof 2019 logical access data tend to have longer leading and trailing silences than spoofed clips. A small network given only the length of the leading silence reached 15.1% equal error rate (EER) on the evaluation set, where random guessing scores 50%. When they trimmed silence before training RawNet2, an official ASVspoof 2021 baseline, its EER rose from 3.6% to 15.5%.
The same weakness showed up in the ASVspoof 2021 deepfake (DF) task. Participants had to train on ASVspoof 2019 data, whose bona fide speech all comes from the VCTK corpus, but the DF evaluation set added real speech from the 2018 and 2020 Voice Conversion Challenge (VCC) corpora. In the ASVspoof 2021 summary paper, the five top submissions and the best baseline scored EERs of roughly 15% to 34% whenever VCC bona fide speech was among the trials. Without it, their EERs were far lower, under 1% for one system. The organizers suggest the systems learned real speech at least partly from the silence pattern of their only bona fide source.
The ASVspoof 5 organizers searched their own data for such cues and, as their database paper describes, found five potential shortcuts: peak amplitude, leading non-speech length, trailing non-speech length, total duration and energy. They reduced these differences in the development and evaluation sets and deliberately left the training set as it was.
The spoofed class gets most of the attention, but the bona fide class defines what "real" means to the model. If that class is narrow, real speech outside it starts to look fake.
Public deepfake detection datasets and their real speech
| Dataset | Bona fide speech | Spoofed speech | Size | License as published |
|---|---|---|---|---|
| ASVspoof 2019 LA and PA | VCTK: 107 English speakers reading newspaper sentences, recorded in a hemi-anechoic chamber | LA: 19 TTS and voice conversion attacks; PA: simulated replay | LA: 121,461 utterances | ODC-By 1.0 |
| ASVspoof 2021 LA and DF | LA: 2019 VCTK speech sent over real VoIP and phone networks; DF: VCTK plus VCC 2018 and 2020 speech from 26 more speakers, coded with mp3, m4a and ogg | LA: 13 attacks; DF: more than 100 attack algorithms | Evaluation sets only; training uses ASVspoof 2019 LA | ODC-By (LA), ODbL (DF) |
| ASVspoof 5 (2024) | Multilingual LibriSpeech English: LibriVox audiobooks read by volunteers on their own devices, about 2,000 speakers | 32 algorithms, including adversarial attacks; evaluation set coded under 11 codec conditions | Training: 400 speakers, 19k bona fide and 164k spoofed utterances | ODC-By; bona fide audio CC BY 4.0 |
| In-the-Wild (2022) | Speeches, podcasts and other public clips of 58 English-speaking celebrities and politicians | Deepfakes of the same people, collected online | 37.9 hours, 17.2 of them fake | Apache 2.0, stated by the authors for clips collected from the web |
| WaveFake (2021) | Not redistributed: LJSpeech (one female English reader) and JSUT (one female Japanese speaker) | Vocoders such as MelGAN, HiFi-GAN and WaveGlow, plus one full TTS pipeline | 104,885 clips, about 175 hours | CC BY-SA 4.0 |
| FoR (2019) | Arctic, LJSpeech, VoxForge and the lab's own recordings | TTS systems such as Deep Voice 3 and Google WaveNet | More than 195,000 utterances; one version re-recorded to simulate a voice channel | None stated on the dataset page |
| ADD 2023 Track 1.2 | Mandarin: AISHELL-3, a 218-speaker TTS corpus, for training and development; AISHELL-1, THCHS-30 and others in test | TTS and voice conversion | Training: 3,012 real and 24,072 fake utterances | CC BY-NC-ND 4.0 |
| MLAAD v11 | Not included: paired with M-AILABS audiobooks and public speeches in 8 languages | 205 TTS models in 54 languages | 1,153.6 hours of synthetic speech | CC BY-NC 4.0 |
Most of the real speech in the table is read aloud: newspaper sentences in a studio chamber, audiobooks and TTS recording prompts. In-the-Wild comes closest to production audio, and its authors intend it as evaluation data. None of these bona fide sets is spontaneous conversation recorded over phone channels in many languages.
The real side of the training sets is also small. ASVspoof 2019 LA trains on 2,580 bona fide utterances from 20 speakers against 22,800 spoofed ones.
Licenses need a second read. ODC-By and ODbL explicitly include commercial use, but both preambles say they govern the database, not its individual contents, and that privacy or data protection rights may need separate clearance. The real recordings come from other corpora with their own terms, such as VCTK, the VCC data and Multilingual LibriSpeech. WaveFake's Japanese clips are generated from JSUT, whose audio terms cover non-commercial research. MLAAD and ADD 2023 exclude commercial use, and the FoR page states no license. In-the-Wild's clips of public figures come from social networks and video platforms, so its Apache 2.0 license comes from the dataset's authors, not from the people speaking or the original publishers.
None of these licenses settles speaker consent or publicity rights on its own. Have counsel check each dataset's terms, its source corpora and those rights before any commercial use.
Why detectors trained on ASVspoof fail in the wild
In Does Audio Deepfake Detection Generalize? (Interspeech 2022), Müller and colleagues trained twelve architectures on ASVspoof 2019 LA, then tested them on ASVspoof's own evaluation split and on In-the-Wild.
| Model, trained on ASVspoof 2019 LA | EER on ASVspoof 2019 eval | EER on In-the-Wild |
|---|---|---|
| RawGAT-ST, full-length input | 1.2% | 37.2% |
| RawNet2, 4-second input | 4.4% | 33.9% |
| LCNN-LSTM, CQT spectrogram, full-length input | 6.2% | 61.5% |
EERs rose by about 200% to 1,000% across models, often to no better than random guessing. Retraining RawNet2 on all three ASVspoof 2019 splits, evaluation data included, gave 33.1% EER on In-the-Wild, no better than before. The authors point out that every ASVspoof split is built from the same corpus, VCTK. More data from the same source doesn't fix a source mismatch; different real speech does.
A spec for the bona fide set
Write the bona fide spec from production audio, not from what is easy to download: list what the detector will hear, then check that the training set covers it.
| Dimension | What to match | Why it matters |
|---|---|---|
| Languages and accents | The language mix of your traffic, with regional accents in proportion | Unseen real speech broke the ASVspoof 2021 DF systems |
| Channel and codec | Narrowband telephony for phone calls, wideband for apps and media, and the codecs you actually see, such as G.711, GSM, AMR, Opus or mp3 | ASVspoof 5 tests 11 codec conditions, four of them at 8 kHz |
| Speaking style | Spontaneous conversation if production is conversation: fillers, hesitations, overlap and turn-taking | Public bona fide sets are mostly read, so a model can learn fluent reading as real and a hesitant caller as unusual |
| Devices and rooms | Phones, headsets, laptops, cars and noisy rooms, in the proportions you expect | Recording conditions can become the cue instead of synthesis artifacts |
| Speakers | Hundreds of speakers rather than tens, balanced by gender and age | ASVspoof 2019 LA trains on 20 speakers' bona fide speech |
| Metadata | Age, gender, region, device and channel for every file | Needed to balance training and to break EER down by group |
| Processing | The same resampling, loudness normalization, silence handling and codec chain for both classes | Silence length, peak amplitude, duration and energy are documented shortcut cues |
Where you can, record the channel instead of simulating it, as ASVspoof 2021 LA did by sending both classes through real VoIP and phone networks. Codec augmentation is the fallback, and ASVspoof 5 leaves it to the user by coding only its evaluation set. Our guide to speech data sample rates covers narrowband and wideband capture.
Splits and per-condition evaluation
A test score only means something if the test set holds what the training set did not.
- Split by speaker. ASVspoof 2019 and ASVspoof 5 both use speaker-disjoint partitions, so the test measures voices the model has never heard.
- Split by generation system. ASVspoof 2019 LA trains on 6 attack systems and evaluates on 13, of which 11 are unseen, and ASVspoof 5 uses different algorithms in each partition.
- Hold out a source. Keep one test set from a separate collection, so you see the kind of shift that hit ASVspoof 2021 DF before customers do.
- Report EER per condition. EER is the error rate at the threshold where the share of spoofs accepted as real equals the share of real speech rejected as fake. Break it down by codec, channel, language, speaking style, gender, age band and attack family; a pooled number can hide a condition where the detector fails.
- Audit for shortcuts. Train a one-feature classifier on leading silence, duration, peak amplitude or energy. If it scores far below 50% EER, fix the data before you trust the model.
Our guide to held-out ASR test sets applies the same rules to speech recognition.
The spoof side: current voice spoofing attacks, refreshed
Generate the spoofed class with the text-to-speech and voice conversion systems an attacker would use, including zero-shot voice cloning, and pass it through the same channels as the bona fide class. Public sets age fast: WaveFake was published in 2021, and even MLAAD's 205 TTS models are a snapshot. Regenerate spoofs as new systems appear, and keep generation systems disjoint across splits. Here synthetic speech is the label, not a stand-in for real data; our comparison of synthetic and real speech data covers the other uses.
Cloning the voices in your bona fide set gives matched real and fake pairs, and it carries the most obligations. Consent to be recorded for one purpose does not automatically cover cloning, as our guide to voice cloning training data explains, and our overview of voice data privacy covers the wider rules.
Before you generate synthetic speech from licensed recordings, ask the vendor, including us, to confirm in writing whether the license and the speakers' consent permit it. If they don't, use those recordings only as the bona fide class and generate spoofs from voices you have the rights to synthesize.
What to ask a speech data vendor
Before you license bona fide speech for a detector, get written answers to five questions:
- Style and speakers. Is the speech spontaneous or read, how many speakers are there per language, and how are gender and age balanced?
- Capture. Which devices, channels and codecs were used, and is each speaker on a separate channel?
- Metadata. Does every file carry age, gender, region and device, so you can balance training and break EER down by group?
- Consent and license. Is there consent documentation for every speaker, and does the license cover training and evaluating a commercial detection model?
- Synthesis. Do the license and the speakers' consent permit generating synthetic speech from the recordings?
Then ask for a sample and run the shortcut audit above on it before you buy.
Where SpeechData.ai fits
SpeechData.ai sells real speech data, so weigh this section with that bias in mind: we supply bona fide speech, not fakes.
Our catalog has 60 off-the-shelf datasets of spontaneous two-person conversations in 60 languages and dialects, recorded in-country. Each recording is dual-channel with one channel per speaker and comes with time-aligned, human-verified transcripts, speaker metadata (age, gender, region and recording device) and consent documentation for every speaker, under a commercial license for training and evaluating ASR, TTS and other voice AI models. Each dataset holds 500 to 2,000 hours from 50 to 200 speakers, at published prices of $60 to $95 per hour of audio. US English, for example, is 1,500 hours from 150 speakers at $90 per hour, and you can request a free sample from every dataset page.
For a contact-center detector, we also record to spec through custom speech data collection, including role-played customer service calls with agent and customer on separate channels, captured as narrowband telephony or wideband audio as the brief specifies. The calls are role-played, not real customer recordings; our page on call center speech data sets out the scope.
Whoever supplies your real speech, write the bona fide spec from production audio first. A detector only knows what real speech sounds like from the examples you give it.
