Speaker diarization is the task of taking an audio recording and answering "who spoke when": partitioning the timeline into segments and assigning each segment a speaker label. The labels are anonymous (Speaker A, Speaker B) because diarization does not need to know who the speakers actually are, only that they are different people.
That distinction separates diarization from speaker identification (matching a voice against known, enrolled speakers) and speaker verification (confirming a claimed identity). Diarization is unsupervised at inference time: the system usually doesn't even know how many speakers are present.
Diarization is what turns a raw transcript into a usable record of a conversation. Meeting notes, call-center analytics, medical scribing, and conversational training-data preparation all depend on it. It also remains meaningfully harder than transcription itself, because natural conversation is full of overlaps, interruptions, and half-second back-channels.
What is speaker diarization? Definition and output
Given an audio file, a diarization system outputs a list of (start time, end time, speaker label) tuples, e.g.:
0.00-4.20 SPEAKER_00
4.05-9.80 SPEAKER_01
9.80-10.30 SPEAKER_00
Note the overlap between 4.05 and 4.20: both speakers active at once. Whether a system can even represent overlap is one of the main dividing lines between architectures. Combined with an ASR transcript, this yields speaker-attributed transcription: who said what, when.
The classic pipeline: VAD → embeddings → clustering
Most production diarization systems are modular pipelines with four stages:
- Voice activity detection. Find speech regions and discard silence and noise (see our VAD guide).
- Segmentation. Cut speech into short chunks, ideally at speaker-change points; modern systems (pyannote) use a neural segmentation model that also detects overlapping speech.
- Speaker embeddings. Map each chunk to a fixed-length vector that encodes voice identity. The historical progression: i-vectors → x-vectors (TDNN-based, 2018) → ECAPA-TDNN, the current workhorse, trained on large speaker-verification corpora like VoxCeleb so that same-speaker chunks land close together in embedding space.
- Clustering. Group embeddings by speaker, typically with agglomerative hierarchical clustering or spectral clustering, estimating the number of clusters from the data unless the caller supplies it.
This pipeline is robust and interpretable, but it has a structural weakness: steps 3-4 assume each chunk contains exactly one speaker. Overlapping speech violates that assumption, which is why classic pipelines historically just lost the overlapped speaker.
End-to-end neural diarization (EEND)
End-to-end neural diarization replaces the pipeline with a single network that ingests audio and directly outputs, for each frame, the activity of each speaker. Overlap becomes a first-class citizen rather than an afterthought. Training uses permutation-invariant loss, since speaker label order is arbitrary.
In practice the field has converged on hybrids: a local EEND-style model handles segmentation and overlap within short windows, and clustering stitches speaker identities across the full recording. pyannote 3.x works this way (its "powerset" segmentation model classifies each frame into speaker combinations, including overlap sets), and NVIDIA's Sortformer (in NeMo) is a prominent end-to-end model that resolves the permutation problem by sorting speakers by arrival time. Pure EEND still struggles with long recordings and unbounded speaker counts, which keeps hybrid designs dominant.
Speaker diarization tools in 2026
| Tool | Approach | Strengths | Notes |
|---|---|---|---|
| pyannote.audio | Neural segmentation, speaker embeddings and clustering | Open pipeline with published benchmarks | Model access, hardware needs and scoring results depend on the release |
| NVIDIA NeMo | Clustering pipelines, MSDD, Sortformer (end-to-end) | Strong overlap handling, GPU-optimized, streaming variants | Heavier engineering lift |
| WhisperX | Whisper ASR + Silero VAD + pyannote diarization | One-command speaker-attributed transcripts; fast batched inference | Diarization quality is pyannote's; alignment adds word-level timestamps |
| Commercial APIs (AssemblyAI, Deepgram, Speechmatics, Google, AWS) | Proprietary | No infrastructure; diarization bundled with ASR | Speaker-count limits and overlap behavior vary |
Does Whisper support speaker diarization?
No. Whisper does not diarize. This is one of the most common misconceptions about it. Whisper is a sequence-to-sequence transcription model: audio in, text (with timestamps) out. It has no speaker representation anywhere in its architecture and will happily merge five speakers into one undifferentiated transcript.
The standard workaround is to run diarization alongside Whisper and merge the outputs: run pyannote to get speaker-labeled time segments, run Whisper to get word-level timestamps (Whisper's native segment timestamps are too coarse, so tools force-align them), then assign each word to the speaker segment it falls inside. WhisperX packages exactly this (batched Whisper inference, forced alignment for accurate word timing, and pyannote diarization) and is the most popular way to get "Whisper with speakers."
Measuring accuracy: Diarization Error Rate (DER)
DER measures missed speech, false alarms and speaker confusion against a reference annotation, using durations rather than word counts:
DER = (missed speech + false alarm speech + speaker confusion) / total reference speaker time
- Missed speech: reference has speech, system says nothing (overlaps are the usual culprit).
- False alarm: system claims speech where there is none.
- Speaker confusion: speech detected, but assigned to the wrong speaker.
The pyannote.metrics definition sums reference speech duration over speakers, so two people speaking at once contribute two speaker durations. DER is not a word attribution error rate, and missed speech or false alarms can raise it even when speaker labels are correct.
State the test set, boundary collar and overlap policy alongside each result. Excluding overlap or ignoring a tolerance window around boundaries changes what is scored. There is also JER (Jaccard Error Rate), which weights speakers equally rather than by talk time.
Why natural conversation is the hard case
Diarization demos look great on podcasts: two speakers, good microphones, polite turn-taking. Real conversation breaks every one of those assumptions:
- Overlap is constant. In spontaneous dialogue, speakers overlap during turn transitions and back-channels; depending on the corpus, overlapped speech is commonly on the order of 5-20% of speech time. Single-channel systems must detect and separate it; many simply miss it.
- Back-channels are short. "mm-hmm", "yeah", "right" last 200-500 ms, often shorter than the analysis window used for embeddings, so they get absorbed into the other speaker's turn.
- Similar voices confuse clustering. Two same-gender speakers with similar vocal characteristics on a narrowband phone line push embedding clusters together.
- Channel effects. Telephony codecs strip the spectral detail that embeddings rely on; far-field reverb smears speaker cues.
This is why evaluation on scripted or read speech tells you nearly nothing about production performance, a theme we cover in depth in conversational vs. scripted speech data.
Practical tips for better diarization results
A few adjustments consistently move DER more than swapping models:
- Pass the speaker count if you know it. Estimating the number of speakers is one of the largest error sources in clustering; pyannote and most APIs accept
num_speakers(or min/max bounds) and improve markedly when given them. - Check how the pipeline handles channels. If dual-channel audio truly isolates one person per channel, channel identity can supply speaker attribution. Speech boundaries still need detection or annotation. The pyannote Community-1 model card says it averages multichannel input to mono, so keep the original channels separately if you need them to build a reference.
- Mind the audio front-end. Aggressive noise suppression and AGC can erase the speaker cues embeddings depend on; diarize the least-processed audio available.
- Evaluate on your own audio. Published DERs come from specific benchmarks with specific scoring rules. Twenty minutes of hand-labeled production audio, scored with
pyannote.metricsordscore, tells you more than any leaderboard. - Reconcile word boundaries carefully when merging with ASR. Most "diarization errors" users see in speaker-attributed transcripts are actually word-to-segment assignment errors at turn boundaries, exactly where timestamps from both systems are least reliable.
The training-data angle: building references from dual-channel audio
Building or fine-tuning diarization systems needs reference annotations of per-speaker activity, including overlaps. Those boundaries need an agreed annotation policy and review, especially for quiet speech, breaths and short backchannels.
Dual-channel recording helps separate speaker identity from speech timing. When each channel contains one speaker, its channel mapping supplies identity. A voice activity detector or annotator still has to locate speech on each channel; crosstalk, echo and background noise can affect that decision. Review the activity labels before using them as a benchmark reference. You can then mix the channels into mono input while retaining the checked per-speaker annotations. Two-channel conversation also supports full-duplex voice models that learn turn-taking from separate streams.
The SpeechData.ai datasets are recorded this way: spontaneous two-party conversations, one channel per speaker, with time-aligned transcripts and speaker metadata, across 60 languages and dialects, from US English to Hindi and Arabic (MSA). If you're deciding how much of it you need, our guide on how much audio data you need is a good starting point.
Need training data?
SpeechData.ai licenses dual-channel conversational audio with time-aligned, human-verified transcripts and speaker metadata. Check a sample against your speech-boundary and overlap policy before using it for diarization evaluation. Explore the dataset catalog or get in touch to talk through your setup.
