Inter-Annotator Agreement in Speech Data: Kappa, Alpha and WER

Inter-annotator agreement (IAA), also called inter-rater reliability, measures how often two or more annotators give the same label to the same item when they work independently. In speech data, the item might be an utterance tagged with an intent, a clip tagged for laughter, a recording split into speaker turns, or a whole transcript.

It matters because labels are both what a model learns from and what it is scored against. If two careful annotators disagree on one utterance in five, about one label in five in a single-annotated test set could reasonably have gone the other way. Once a model's score nears that level of human agreement, further gains are hard to read. Agreement works as a practical ceiling on the accuracy a test set can show, though not as a hard limit on what a model can learn.

What agreement measures, and what it doesn't

Raw percent agreement is the share of items where annotators chose the same label. It flatters, because some matches happen by chance. If 90% of clips contain no laughter, two annotators who mostly answer "no" will match often without listening closely. Chance-corrected measures such as kappa and alpha discount the agreement you would expect by chance, given how often each label is used.

Agreement also shows consistency, not correctness. High agreement is necessary for trustworthy labels but doesn't prove them right: two annotators who share a misreading of the guideline agree perfectly and are both wrong. That is why agreement checks run alongside gold sets: items with expert-verified labels, scored per annotator.

Which measure fits which label type

Categorical labels: intent, sentiment, emotion, event tags

Cohen's kappa compares two annotators on nominal labels (Cohen, 1960). It is κ = (p_o - p_e) / (1 - p_e), where p_o is the observed agreement and p_e the agreement expected by chance from each annotator's label distribution.

Fleiss' kappa extends this to any fixed number of ratings per item, and the raters can differ from item to item (Fleiss, 1971). That fits a pool where each clip goes to three of its annotators, as in the IEMOCAP emotion corpus: six evaluators, three per turn.

Krippendorff's alpha works with any number of annotators, missing labels, and nominal, ordinal, interval or ratio scales (Krippendorff, 2011). Use it when items get different numbers of labels, or when labels are ordered, such as a 1 to 5 arousal rating, where a 4 against a 5 is a smaller miss than a 1 against a 5.

Transcripts: WER between independent transcripts

For transcription, two annotators transcribe the same audio independently and you score one against the other with word error rate. The Linguistic Data Consortium (LDC) calls the result a word disagreement rate, since not every difference is a mistake. Three details decide whether the number means anything:

• Normalize both sides the same way. In its 2010 consistency study, LDC ignored differences in capitalization and punctuation.
• Fix the direction. WER divides by the reference's word count, so it isn't symmetric. Name one transcript the reference, or score both against an adjudicated version.
• Pick the unit. For languages written without spaces between words, such as Mandarin or Japanese, use character error rate.

LDC's 2010 paper on transcription consistency shows what to expect (Glenn et al., 2010). Citing an earlier LDC study, it reports careful transcription at 1.3% disagreement between two transcribers on English broadcast news and 4.1% to 4.5% on conversational telephone speech, where 95% of the differences were judgment calls about contractions, rapid or difficult speech, or disfluencies. In the paper's own comparison, quick transcription of English telephone speech reached 9.63%.

Segments, speaker turns and event boundaries

Two annotators rarely place a boundary on the same millisecond, so time-based labels need a tolerance written into the guideline. For speaker turns, treat one annotation as the reference and compute diarization error rate (DER) for the other: missed speech, false alarm and speaker confusion as a share of reference speech time. Our speaker diarization guide explains the metric.

Scoring tools let you set a collar, a no-score zone around each reference boundary. The DIHARD scoring tool, dscore, defaults to no collar and scores overlapping speech, so report your settings with the number.

For sound events such as laughter or music, one common approach is event-based F1 with a time tolerance. The sed_eval toolkit, for example, defaults to a 200 ms collar on onsets, and for offsets accepts the larger of 200 ms and half the reference event's length. Report boundary agreement and label agreement separately, so "same event, different edges" doesn't look like "different event".

A worked Cohen's kappa example

Two annotators label the sentiment of the same 100 utterances from customer calls. Rows are annotator A, columns annotator B:

              B: pos  B: neu  B: neg   Total
A: positive     24       4       2       30
A: neutral       5      38       7       50
A: negative      1       3      16       20
Total           30      45      25      100

1. Observed agreement is the diagonal: p_o = (24 + 38 + 16) / 100 = 0.78.
2. Chance agreement multiplies the two annotators' totals for each class: p_e = (30 × 30 + 50 × 45 + 20 × 25) / 100² = (900 + 2,250 + 500) / 10,000 = 0.365.
3. Kappa: κ = (0.78 - 0.365) / (1 - 0.365) = 0.415 / 0.635 = 0.65.

The annotators match on 78% of utterances, and kappa says they closed 65% of the gap between chance agreement and perfect agreement. The matrix also shows where to fix the guideline: the largest off-diagonal cell is the 7 utterances A called neutral and B called negative, so the line between a mild complaint and a neutral statement needs a rule and examples.

Now a rare tag. The same two annotators mark 100 clips for laughter. Both say yes on 2, both say no on 90, and they split on the other 8, four each way.

1. Observed agreement: p_o = (2 + 90) / 100 = 0.92.
2. Chance agreement: each annotator said yes 6 times and no 94 times, so p_e = (6 × 6 + 94 × 94) / 100² = (36 + 8,836) / 10,000 = 0.8872.
3. Kappa: κ = (0.92 - 0.8872) / (1 - 0.8872) = 0.0328 / 0.1128 = 0.29.

Of the 10 clips where either annotator heard laughter, they agreed on 2. Kappa caught what 92% hid.

How to read kappa values

The most quoted scale comes from Landis and Koch (1977): below 0.00 is poor agreement, 0.00 to 0.20 slight, 0.21 to 0.40 fair, 0.41 to 0.60 moderate, 0.61 to 0.80 substantial and 0.81 to 1.00 almost perfect. A value below zero means agreement worse than chance. On that scale, the sentiment example is substantial and the laughter example fair.

Landis and Koch called these divisions "clearly arbitrary" and offered them as benchmarks for discussion, not as a test. Use the scale as vocabulary, not as a pass mark:

• The bands aren't tied to model quality. They name ranges. Nothing in them says what a 0.65 does to your classifier.
• Kappa depends on the label distribution. As the laughter example shows, a skewed tag can score low at high raw agreement.
• One number hides where the trouble is. Report agreement per class and read the confusion matrix.

Set a threshold for each label layer from a pilot, write it into the scope, and compare only like with like: the same measure, label set and kind of audio.

Why emotion labels agree less than transcripts

A transcript has an answer in the audio: the speaker said the word or didn't. An emotion label records a listener's perception, and listeners differ. Published emotion corpora show how much.

IEMOCAP holds about 12 hours of scripted and improvised two-person sessions recorded with ten actors. Three evaluators labeled each turn using nine emotion categories plus "other", and could pick more than one. Fleiss' kappa across the database was 0.27. After merging happiness with excitement and moving fear, disgust and surprise into "other", it rose to 0.35. A single most-voted label emerged on 74.6% of turns.

The final 2.0 release of the MSP-Podcast corpus has 409 hours of naturalistic podcast speech, with at least five annotations per speaking turn. It reports a kappa of 0.411 for eight primary emotion classes, which the authors call high given how natural the recordings are. Krippendorff's alpha for the attribute ratings is 0.508 for valence, 0.441 for arousal and 0.386 for dominance.

So don't hold emotion or sentiment labels to a transcript's standard. Plan more annotators per item and keep the full vote distribution, not only the majority label, so a model can learn that some turns are ambiguous.

How to run an inter-annotator agreement study

1. Pilot first. Double-annotate a pilot batch, compute agreement per label layer and read every disagreement. Most trace back to a rule the guideline doesn't state. For transcripts, that usually means fillers, fragments, numbers and unclear speech, the conventions covered in our guide to verbatim vs. clean verbatim transcription.
2. Size the sample by class, not by hours. Kappa for a class seen only a handful of times swings with every item. Make sure each label you care about appears dozens of times in the double-annotated set. If you oversample rare classes to get there, say so with the result, since kappa shifts with class balance.
3. Keep a double-annotated share in production. A fixed share of every batch, such as 5% to 10%, shows drift while it can still be fixed.
4. Keep the passes blind. Annotators shouldn't see each other's labels. If both correct the same ASR draft, agreement partly measures the draft.
5. Adjudicate. A senior annotator resolves each disagreement, and the adjudicated label becomes the reference. Tag every difference as an error, a judgment call or an insignificant difference, as LDC's adjudicators did, so you know whether to retrain people or rewrite rules.
6. Score against gold. Seed expert-labeled items into every queue to catch shared mistakes.
7. Version the guideline. Recompute agreement after each revision and record the guideline version with every number.

What to ask an annotation vendor about agreement

A bare "95% accuracy" tells you little. Ask:

• Which measure do you use for each label layer, and why that one?
• How many items were double-annotated, how were they chosen, and were the passes blind?
• Can we see per-class and per-annotator figures, not only an average?
• What normalization, collar or tolerance sits behind each number?
• What threshold triggers rework or a guideline change, and is it in the contract?
• How do you score annotators against gold items?
• Will you share the raw double-annotated labels so we can recompute the figures?

For pricing, pilots and contract terms, see our guide to choosing an audio annotation vendor.

If you have recordings that need labels, SpeechData.ai's audio annotation services transcribe and label your own audio, from verbatim transcripts, timestamps and speaker turns to non-speech events, intent, sentiment and emotion. Native-speaker annotators work under a written guideline, we measure agreement on duplicated items, and the written scope states the sample, the threshold and what triggers rework. Pricing follows a review of a sample of your audio.

Whoever does the work, agree the measure and the threshold for each label layer before the first batch.