A full-duplex voice model listens while it talks. It can say "mm-hm" while you explain something, stop when you cut in, and answer a fraction of a second after you finish instead of waiting for a long silence. Kyutai reports a theoretical latency of 160 ms for its Moshi model, and 200 ms in practice.
None of that timing comes from text. The model learns it from recordings of people talking to each other, and those recordings have to keep each person's voice on its own track.
The data these models need is spontaneous two-person conversation recorded with one channel per speaker, with transcripts aligned in time to each channel. In Moshi and SyncLLM it was less than 1% of the training audio, but it was the stage that taught the model to take turns. dGSLM trained on nothing else.
What full-duplex models learn from conversation
A cascaded voice agent runs speech recognition, a language model and text-to-speech in sequence, and treats enough silence as the end of the user's turn. The dGSLM authors liken this setup to a walkie-talkie. Full-duplex and speech-to-speech dialogue models instead process both sides of the conversation at once. Moshi, for example, models its own speech and the user's speech as parallel streams.
To sound natural, such a model has to learn the timing of real conversation:
- Turn-taking. The gap between one person stopping and the other starting. The dGSLM authors point out that pauses inside a turn tend to be longer on average than gaps between turns, so silence alone is a weak signal that someone has finished.
- Overlap. Stretches where both people speak at once, at turn boundaries and in the middle of turns.
- Backchannels. Short listener responses such as "mm-hm", "yeah" or "right", often spoken over the other person without taking the turn.
- Interruptions and barge-in. One speaker starts while the other is mid-turn, and the first one yields or keeps going. For a voice agent, this is the caller cutting in while the agent talks.
- Pauses and hesitations. Silent and filled pauses inside a turn that do not invite a reply.
- Response latency. How quickly a reply starts, which varies with the kind of answer.
Full-Duplex-Bench, a 2025 benchmark for these systems, scores close to this list: pause handling, backchanneling, turn-taking and interruption management. A model can only learn these behaviors if the training data contains them and keeps them measurable.
Why one channel per speaker matters
A mono recording of two people mixes them into one signal. Wherever they overlap, the voices are summed, and nothing in the file says which sound belongs to whom. Dual-channel recording keeps each person on a separate track, which gives you three things.
A clean target for each speaker. Moshi and dGSLM both represent a conversation as two streams, one per speaker. dGSLM uses a dual-tower transformer with one tower per channel. Training them needs each stream as its own signal, not an estimate pulled out of a mix.
Overlap that survives. Starting from mono audio means rebuilding the two streams with diarization and source separation, and both fail most often on overlapped speech and short backchannels, as our guide to speaker diarization explains. Moshi's authors used this route for one training stage. They diarized single-stream audio into two streams and note that the result "contains no overlap, and the stream of an inactive speaker is perfectly silent". They then fine-tuned on real two-channel telephone conversations to teach the model real interaction.
Timing labels without annotation. With separate channels, voice activity detection on each channel gives you every turn, gap, pause and overlap. The dGSLM paper measures turn-taking this way, defining a unit of speech as a stretch on one channel bounded by more than 200 ms of silence. No annotator has to judge where an overlap begins.
Synthetic dialogue does not close the gap. SyncLLM, from Meta AI and the University of Washington, trains in three stages. In the second, synthetic dialogue is laid out as two streams with the listener completely silent during the other speaker's turn, and the authors note that at this point the model "cannot learn turn-taking cues such as back-channeling or overlaps between two speakers yet". The third stage, on real two-channel conversation, is where it learns the distribution of pauses between turns and backchanneling.
How much data published systems used
These three papers describe their training data in enough detail to compare. All three used the same real conversational corpus: Fisher, about 2,000 hours of English telephone calls with each side recorded on its own channel.
| System | Real two-channel conversation | Other training data |
|---|---|---|
| dGSLM (Meta AI, TACL 2023) | Fisher, 2,000 hours | None; the dialogue model trained on raw audio with no text or labels |
| Moshi (Kyutai, 2024) | Fisher, 2,000 hours, in a fine-tuning stage | 7 million hours of mostly English audio for pre-training; more than 20,000 hours of synthetic dialogue for instruction tuning, voiced by a TTS model fine-tuned on 170 hours of separate-channel conversations |
| SyncLLM (Meta AI and University of Washington, EMNLP 2024) | Fisher, 2,000 hours | 212,000 hours of synthetic spoken dialogue generated from text dialogue |
Three things stand out. In Moshi and SyncLLM, real two-channel conversation is less than 1% of the training audio, yet it is the stage each paper relies on for natural turn-taking. All three systems used the same English corpus, so there is little published evidence for other languages. And Fisher is 8 kHz telephone audio, while Moshi's audio codec works at 24 kHz. Its authors upsampled Fisher with AudioSR, a super-resolution model, which estimates the frequencies the phone line removed rather than recovering them.
About 2,000 hours of real two-channel conversation is a useful reference point when you size a purchase for one language. Outside English there is little published work to lean on, so plan on two-channel data in each language you ship.
Open sources and their limits
Fisher and Switchboard. The Linguistic Data Consortium (LDC) distributes both as two-channel 8 kHz telephone audio. Fisher English Parts 1 and 2 hold 11,699 conversations of up to 10 minutes, about 1,960 hours, recorded in 2003 between participants who usually did not know each other and were given a topic. Switchboard-1 Release 2 holds about 2,400 conversations, roughly 260 hours from 543 speakers, collected in 1990 and 1991.
Licensing is the main constraint. LDC states that non-members "cannot use LDC data in any commercial product or for any commercial purpose". Commercial technology development is allowed only with data an organization received while it was an LDC for-profit member, and some corpora carry their own agreements on top. Our post on Switchboard and Fisher alternatives covers membership costs and the open corpora, license by license.
Transcripts are the other limit. The dGSLM authors note that Fisher's quick transcripts contain some inaccuracies and untranscribed portions, and Moshi's authors transcribed Fisher again with a timestamped Whisper pipeline to get reliable word timings.
Newer open full-duplex sets. These are small or scripted so far. A 2025 paper released two dual-track datasets of spontaneous conversation recorded in isolated rooms: about 10 hours in Chinese and 5 in English, from 14 speakers. The ICASSP 2026 HumDial challenge released more than 100 hours of dual-channel Chinese and English dialogue, written by an LLM to include interruptions and asides and then performed by professional actors. Both are useful for benchmarking and research. Check each license before training a commercial model on them.
What to check before you buy
Spec sheets blur most of these requirements, so put every vendor through the same questions and check the answers in a sample you choose yourself.
| Requirement | Why it matters | What to ask the vendor |
|---|---|---|
| One channel per speaker | Clean per-speaker targets; overlap preserved | Was each speaker on a separate microphone? How much bleed is there? |
| Whole conversations | Turn-taking plays out across many turns, not single utterances | Do you deliver full recordings or cut segments? How long is a typical conversation? |
| Native sample rate | Generation needs wideband audio; phone agents need telephony audio | What rate was it recorded at? Was anything upsampled? |
| Spontaneity | Timing, backchannels and interruptions come from unscripted talk | Were speakers given topics, scenarios or scripts? |
| Overlap in transcripts | Overlapped words and backchannels are often dropped or merged | How are overlaps and backchannels transcribed? Can I see an example? |
| Timestamp granularity | Speech-text models such as Moshi align each word to the audio | Are timestamps per turn, segment or word, and per channel? |
| Speaker metadata | Balance voices and build fair test splits | Which fields come with every speaker? |
| Consent and license | A generation model learns to produce voices from the data | Can I read the consent wording? Does the license cover speech generation? |
| Language and accent | The published systems above trained on English | Was it recorded in-country by native speakers? Which regions? |
| Hours and speakers | Many voices help generalization; more hours per voice help generation | How many hours and unique speakers, and how is talk time spread between them? |
Two rows deserve a closer look. "Two channels" can mean two microphones or one recording split by software afterwards, so ask how each channel was captured and listen for bleed, the other speaker faintly audible on a channel. And because a speech-to-speech model produces voices, check that the consent wording and the license cover training models that generate speech, not only recognition.
Spontaneous and role-played data do different jobs. Free conversation gives the most natural timing. Role-play, where speakers act out a scenario, gives you the agent and customer dynamics a voice agent will face. Read speech substitutes for neither. Our article on speech data for voice agents covers the rest of that brief, from telephone audio to domain vocabulary.
Where SpeechData.ai fits
SpeechData.ai records the format that taught these models to take turns. Every dataset in the catalog is spontaneous two-person conversation recorded in-country on dual-channel audio, one channel per speaker. There are 60 datasets, one per language or dialect, totaling 73,000 hours from 7,250 native speakers.
Transcripts are time-aligned and human-verified, and every speaker comes with age, gender, region and recording device metadata. Every speaker signed explicit commercial-use consent, the full consent chain is provided for legal review, and the license covers training and evaluating ASR, TTS and other voice AI models.
Prices are published on each dataset page, from $60 to $95 per hour of audio. The US English conversations, for example, are 1,500 hours from 150 speakers at $90 per hour. Before buying for a full-duplex model, ask for a free sample and test it against the checklist above, in particular the sample rate, timestamp granularity and how overlaps are transcribed.
Natural conversation teaches general turn-taking, but not what a caller does to an agent. For that, SpeechData.ai records role-played customer service calls to spec: native speakers play agent and customer through industry scenarios, each on their own channel, in narrowband telephony or wideband, with fictional names and account details so no real customer data is involved.
Languages, dialects, domains or recording conditions outside the catalog can be collected to your spec, with feasibility and a quote within one business day.
