Forced alignment takes a recording and a transcript of it, and works out when each word, and often each phone, was spoken. The output is the same text with a start and end time on every unit.
It is how speech corpora get word timestamps, how many TTS pipelines get phone durations, and how a finished script becomes timed captions. Getting boundaries you can trust on conversational speech takes care, and most of that care goes into the transcript, not the tool.
Forced alignment vs. ASR timestamps
Speech recognition decides what was said and when. Forced alignment only decides when. The transcript is treated as correct, or in the words of NVIDIA's explainer on CTC forced alignment, "we provide the reference text ourselves."
Use it when you have human-verified transcripts and need timing. ASR timestamps sit on the recognizer's output, so a misheard word still gets a time, and the times can be coarse. The WhisperX README says Whisper's own timestamps are "at the utterance-level, not per word, and can be inaccurate by several seconds," which is why WhisperX adds a separate alignment step.
How a forced aligner works
An aligner scores how well each short frame of audio matches each sound unit, turns the transcript into a sequence of those units, and searches for the most probable path through the frames. Two families dominate.
GMM-HMM aligners with a pronunciation dictionary. The Montreal Forced Aligner (MFA), built on the Kaldi toolkit, looks up each word's phones in a dictionary, aligns them with hidden Markov models, then adapts to each speaker and aligns again. It works on 10 ms frames, so every boundary falls on a 10 ms grid, and current versions let a phone be as short as one frame. Words missing from the dictionary need a grapheme-to-phoneme (G2P) model, or MFA writes them out as <unk>.
A first English run takes three commands, with each WAV file next to a .lab or .TextGrid transcript of the same name:
mfa model download acoustic english_us_arpa
mfa model download dictionary english_us_arpa
mfa align ~/my_corpus english_us_arpa english_us_arpa ~/my_corpus_aligned
The output is a TextGrid per file, with word and phone tiers for each speaker.
CTC aligners. Neural models trained with connectionist temporal classification (CTC) output, for every frame, a probability for each character or token, plus a "blank". The aligner lays the transcript's tokens against the frames and finds the best path with the Viterbi algorithm, allowing blanks between tokens. No pronunciation dictionary is needed, which is why these aligners stretch to many languages, but the text has to be written in the model's alphabet. torchaudio's multilingual tutorial, for example, romanizes text with uroman before aligning it with Meta's MMS model.
Forced alignment tools compared
| Tool | Approach | Languages | License | Best for |
|---|---|---|---|---|
| Montreal Forced Aligner | Kaldi GMM-HMM, pronunciation dictionary, G2P for unknown words | Pretrained models for 41 languages; train or adapt your own | Code MIT; MFA models CC BY 4.0, Common Voice-based models CC0 | Word and phone boundaries, TTS durations, phonetics |
| torchaudio forced_align with MMS_FA | CTC Viterbi alignment on a wav2vec 2.0 model | MMS_FA trained on 31,000 hours in 1,130 languages | Code BSD 2-Clause; MMS_FA weights CC BY-NC 4.0 | Many languages, for research or non-commercial use |
| WhisperX | Whisper transcribes, then a wav2vec 2.0 CTC model aligns | Default models for 41 languages (5 via torchaudio, 36 from Hugging Face) | Code BSD 2-Clause; each alignment model has its own license | Word timestamps when there is no transcript |
| NeMo Forced Aligner | Viterbi on NeMo CTC models, or hybrid models in CTC mode | Depends on the model; the docs say only English has been tested | Code Apache 2.0; each model has its own license | Token, word and segment timings as CTM; long files |
| aeneas | Synthesizes the text with TTS, then aligns MFCCs with dynamic time warping | Confirmed working on 38 languages | AGPL v3 | Sentence or paragraph sync for audiobooks and captions |
| ctc-segmentation | Finds utterance boundaries in long audio using a CTC model | Whatever the CTC model covers | Apache 2.0 | Splitting long recordings before word alignment |
Check the model license, not only the code license. Meta's MMS release puts its code and weights under CC BY-NC 4.0, a non-commercial license, and each community model WhisperX downloads has its own terms. If your timestamps feed a commercial model, clear the alignment model first.
The torchaudio 2.8 docs mark forced_align as deprecated, part of the library's move into a maintenance phase, but the project's tracking issue later confirmed the function was kept. Gentle, an older Kaldi-based aligner under the MIT license, tolerates loose transcripts by marking words it can't find as not-found-in-audio. It sees few updates, the last in February 2025.
How accurate is forced alignment?
The clearest recent comparison is Rousso et al. (2024). They scored MFA, WhisperX and MMS word boundaries against manual alignments in TIMIT, a read speech corpus, and Buckeye, a corpus of spontaneous conversational English. WhisperX and MMS were scored only on words they had recognized correctly, so transcription errors don't explain the gaps.
On TIMIT, MFA placed 72.8% of word boundaries within 25 ms of the manual ones, against 52.7% for WhisperX and 43.5% for MMS. Within 50 ms, the figures were 89.4%, 82.4% and 75.7%. MFA led at every tolerance on both corpora. The authors suspect the end-to-end models' architecture and training: CTC, for one, learns without any pre-aligned data.
Input length matters too. Buckeye's utterances averaged 531 seconds, against 3.1 seconds in TIMIT. On Buckeye, MFA's median word boundary error was 13.6 ms but its mean was 976.5 ms: most words landed close, and a few far-off ones pulled the average to almost a second. Leaving out words more than 500 ms off brought the mean down to 27.8 ms. Report the tail, not only the average, and cut long recordings into shorter segments before aligning them.
Why forced alignment fails
- The transcript doesn't match the audio. The aligner has to place every sound somewhere. An untranscribed "um" or a skipped repetition gets absorbed into a neighboring word or pause and drags its boundaries.
- Word fragments and unknown words. Fragments such as "Tue-", names, product terms and digits are missing from most dictionaries. MFA writes unknown words out as
<unk>, lists them inoovs_found.txt, and can give cut-off words partial pronunciations with--use_cutoff_model. WhisperX can't time words with characters that aren't in its alignment model's dictionary. - Overlapping speech. On a mono mix, two voices share the same frames, but the aligner follows one sequence of words. The WhisperX README says overlap "is not handled particularly well".
- Code-switching. A dictionary or G2P model usually covers one language, so switched words become unknown or get the wrong phones.
- Noise and non-speech events. Laughter, coughs and background talk need a place in the model. MFA's dictionary can map a tag such as
{LG}tospn, the phone it uses for unknown words and sounds like laughter or coughing. - Segment length. Errors build up over long stretches, and the NeMo docs tie the longest file you can align to your machine's memory. Segments cut too tight fail the other way: CTC needs at least one frame per token, plus one for each doubled token such as the "ll" in "all", so a short clip with too much text can't be aligned at all.
Verbatim transcripts align better
A clean transcript drops fillers, false starts and repetitions that are still in the audio, and the aligner stretches other words over them. That is one more argument for tagged true verbatim, the style our comparison of verbatim and clean verbatim recommends for speech data.
A tagged verbatim transcript can be prepared for alignment by rule. Fillers become dictionary words, event tags such as [laugh] map to a noise phone, fragments go to a cutoff model, and segments marked as unclear are left out.
One channel per speaker makes alignment easier
Overlap is the hardest case on a mono recording and far easier on dual-channel audio. With each speaker on their own channel, you align each transcript against its own track, so a backchannel spoken over the other person gets its own timing, apart from any crosstalk between microphones.
A channel per speaker also gives MFA a clean speaker grouping for its adaptation pass. Our guide to full-duplex conversational speech data explains why dialogue models need the same separation.
Phone-level alignment for TTS
TTS models with an explicit duration predictor need to know how many frames each phone lasts, and forced alignment is a common source. FastSpeech 2 replaced the durations its predecessor took from a teacher model with durations from MFA. On 50 hand-aligned clips, the average phone boundary error fell from 19.68 ms to 12.47 ms, and the voice trained on MFA durations scored higher in a listening test.
Some models skip the external aligner. VITS learns its alignment during training with monotonic alignment search. Either way, the transcript has to match the audio exactly, a requirement our guide to TTS datasets treats as essential.
How to check alignment quality
- Hand-align a reference sample. Take a few minutes per language and recording condition, including fast speech, overlap and noise, and correct the boundaries by hand in a tool such as Praat.
- Measure the share within tolerance. Report the percentage of boundaries within your tolerance, such as 25 or 50 ms for words, plus the median and mean error. Since version 3.4,
mfa compare_alignmentscompares two sets of alignments for the same files. Against hand-corrected files the result measures accuracy; between two automatic runs it measures agreement. - Listen to the risky words. Play words at segment edges, very short and very long words, and anything that came back as
<unk>or unaligned. - Automate the rest. Flag words with zero or implausible durations, overlapping timestamps on one channel and timestamps past the end of the file.
What to specify when buying or outsourcing alignment
"Time-aligned" can mean one timestamp per utterance or one per phone. Write these down in the scope:
- Granularity: segment, word or phone, and whether every word needs both a start and an end.
- Tolerance and measure: for example, 90% of word boundaries within 50 ms of a hand-aligned reference, checked on a random sample you choose. Uncorrected MFA reached 84.9% on Buckeye's conversations, so a target like this may need human correction.
- Transcript convention: verbatim or clean, and how fillers, fragments and events are tagged.
- Format: Praat TextGrid, JSON or CTM for training data, or SRT and VTT for captions. A NIST CTM line gives each word a channel, a start time and a duration, not an end time.
- Overlap and failures: how overlapping speech, unclear stretches and words that couldn't be aligned are marked, instead of quietly given a guessed time.
- Channels and phone set: which channel each speaker is on and, for phone alignment, which phone set.
Paste these lines into the timestamp and transcription fields of our speech data RFP template. If you already have recordings, SpeechData.ai's audio annotation services deliver word- or segment-level timestamps with boundary rules and tolerances agreed in writing, as JSON, SRT or VTT.
Every dataset in our catalog of 60 languages and dialects pairs dual-channel audio with time-aligned, human-verified transcripts. Ask for a free sample and check the alignment granularity against your spec before you build word- or phone-level work on top of it.
