Voice command data is recordings of many different people saying what your product must understand: every intent in many phrasings, with the slot values filled in (names, numbers, dates, places, product names), plus requests the product should decline. Each recording is labeled with its transcript, its intent, its slots and the words that carry them, and the speaker, device and environment it came from.
The decisions to make before anyone records are the intents, the phrasings per intent, the slot values, the speakers and their quotas, the devices and environments, and the labels. This guide works through each one, using what public spoken language understanding (SLU) corpora did and what later research found missing, and ends with a spec table you can paste into a brief. It covers the commands that follow the wake word; for the wake phrase itself, see our guide to wake word training data.
What a voice command dataset contains
Take one smart home request. The Fluent Speech Commands paper labels "turn on the lights" with three slots: action "activate", object "lights", location "none". A complete voice command dataset holds four kinds of recordings.
- In-scope commands. Every intent, said in many phrasings, with slot values varied across speakers.
- Near-miss commands. Requests one word away from a supported one that the product should not act on, such as a room it doesn't know or a setting outside the allowed range.
- Out-of-scope requests. Things users will ask that the product does not do, so it learns to decline instead of guessing.
- A held-out test set. New speakers and new phrasings, recorded in the conditions of use.
What public voice command datasets show
Four public corpora are the usual reference points. None holds your intents, but their design choices show what an utterance collection needs.
| Corpus | What it holds | How the wording was written | How it was recorded |
|---|---|---|---|
| Fluent Speech Commands (2019) | 30,043 English utterances, 19 hours, 97 speakers, 31 intents | 248 phrasings decided in advance by asking Fluent.ai employees, native and non-native English speakers, how they would say each intent | Crowdsourced; each speaker said every phrasing twice, in random order |
| Snips (2018) | A text benchmark of over 16,000 crowdsourced queries for 7 intents; spoken test sets for a smart lights and a weather assistant | Crowd workers wrote a query for a given intent and fixed slot values; at least two of three other workers had to confirm it | The test sets were crowdsourced as spoken versions of the assistants' queries |
| SLURP (2020) | 72,277 English recordings, 58 hours, 177 speakers, 18 scenarios, 46 actions | Mechanical Turk workers answered 200 prompts such as "How would you ask for the time?" | Participants read the text aloud from a tablet, captured by a microphone array at a distance and, for some, a headset |
| Speech-MASSIVE (2024) | 12 languages, with 2,033 dev and 2,974 test utterances per language, plus training sets for French and German | Professional translators localized SLURP's text into other languages for MASSIVE | Native speakers recruited on Prolific read the sentences; a second group validated each recording |
Three lessons follow from them.
The written phrasings set the ceiling on variety. By SLURP's count, the 248 Fluent Speech Commands sentences hold 96 distinct tokens, and SLURP's 17,181 sentences hold 6,467. SLURP's authors "carefully designed the prompts to avoid lexical priming": the prompt named a goal and left the words to the worker.
Every speaker read a script. In all four corpora, the text was written first, by employees, crowd workers or translators, and speakers read it aloud. The recordings added voices, microphones and rooms, not new ways of asking.
Check the license. SLURP's audio is released under CC BY-NC 4.0, though its repository says a less strict license can be requested, and Speech-MASSIVE under CC BY-NC-SA 4.0. As released, both are non-commercial, so they are best used to compare model architectures.
How to write the utterance list
Whether you call it an utterance list, a prompt list or a set of intent utterances for a voice assistant, it is the core of the spec. Build it from the intent schema outward.
Start with intents and slots, not sentences. Write down every intent, the slots it takes and the values each slot can hold. In Fluent Speech Commands an intent is a combination of three slot values (action, object and location), 31 in all. The Snips smart lights assistant had six intents, a room slot with 34 values, a color slot with 4, and brightness as a built-in number.
Write phrasings, then get other people to write more. The Fluent Speech Commands authors note that "it is generally impractical to try to imagine every possible wording for a particular intent". The Snips paper says writing up to 50 queries by hand is easy, but coming up with "hundreds or thousands of diverse formulations of the same intent is nearly impossible", and that template grammars produce queries limited by "the imagination of the person having created it". Snips widened its pool by sending each writing task to as many contributors as possible and limiting how many tasks one person could take.
Mix prompt types. In a Clinc study of crowd-written text for 47 personal finance intents, some prompts described a generic situation ("You want to know about your account balance") and others asked for a detail ("Please specify the account you want to inquire about in your responses"). Mixing both raised SVM accuracy to about 75%, from 66 to 69% with one type alone, with every training set the same size (about 4,700 examples). Seeding paraphrase tasks with real user queries instead of engineer-written examples gave about 8 points more. These were text classifiers, but the same advice applies to any text you plan to have read aloud.
Rotate the words around each slot value. A carrier phrase is the frame a value sits in, such as "set a timer for ___". If every date in the list follows "remind me on", the data never shows the model a date anywhere else. Rotate frames and values independently, so each value appears in several phrasings and each phrasing with many values.
Choose slot values like test cases. Names, numbers, dates, places and product names are where commands fail. In the Snips weather assistant, the date and time slot was the weakest, with an F1 of 0.78 to 0.80 from speech against 0.93 to 0.96 when the language understanding model read the true text. Values the model never heard are risky too: the Fluent Speech Commands authors report that a model trained only on "bedroom" and "washroom" guesses "bedroom" when it hears "bathroom", because the two sound alike, although "washroom" is the right meaning. List the synonyms and spoken forms users will say for each value, and record them.
Localize the values for each language. For MASSIVE, translators decided per slot whether to translate the value ("tomorrow"), localize it (a film by its local title, a song by one known in the region) or keep it unchanged (names such as "Taylor Swift"). Do the same in every language you collect: slot values should be what users in that market say.
Write out-of-scope and near-miss requests on purpose. The CLINC150 dataset pairs 22,500 in-scope queries over 150 intents with 1,200 out-of-scope ones. On this text benchmark, BERT scored 96% or more in scope, but the best system reached only 66% recall on out-of-scope queries, even with extra out-of-scope training data. They are hard to write as well: only about 69% of queries collected with prompts aimed at out-of-scope topics turned out to be out of scope, so each one needs checking against the intent list. A useful set covers:
- Requests from neighboring domains the product does not handle.
- Supported actions on unsupported things, and unsupported actions on supported things.
- Near-misses: a room or device that does not exist, or a value outside the allowed range.
Scripted, paraphrased or goal prompts
How the prompt is shown decides whose words end up in the audio.
| Prompt type | What the speaker sees | What you get | What to watch |
|---|---|---|---|
| Scripted | "Set the living room to 21 degrees." | A known transcript and even coverage of every intent and value | Every speaker says the same words in the same order |
| Paraphrase | The same sentence, with "say this your own way" | More phrasings per session | An example sentence can prime its own words, which SLURP's prompts were designed to avoid |
| Goal | "The living room feels cold. Ask for 21 degrees." | The words speakers choose for themselves | Some will say a different intent; transcribe and label after recording |
Goal prompts move the variety into the recording session, along with the errors. Snips found that crowd workers writing to an intent sometimes wrote the wrong one, such as "Switch off the lights" when the task asked for switching them on, or added a slot value nobody asked for. Expect the same from speakers, so goal-prompted audio needs a transcript and an intent check before it counts.
A practical mix is a scripted core that covers every intent and slot value evenly, plus goal prompts for phrasing variety. Give each speaker a different random subset and present it in random order, as Fluent Speech Commands did with its phrases, so no two speakers read the same list in the same sequence.
What labels to ask for
Ask for these on every file, and agree the conventions before recording starts.
- Transcript. Set the convention for numbers, dates and times. Speech-MASSIVE reports discrepancies between Whisper's output and the MASSIVE references, such as numbers that the references write out in letters.
- Intent, for what was said. Label what the speaker said, not what the prompt asked for. Fluent Speech Commands removed recordings that a separate group of crowd workers judged unintelligible or found to contain the wrong phrase, and SLURP checked automatically whether each participant said the right, complete query.
- Slots with spans. Each slot's type, the exact words that express it and, for dates, times and numbers, a normalized value. SLURP labels each slot as a tag with its words, such as [date: Saturday] and [timeofday: morning].
- Speaker metadata. Speaker ID, age band, gender, region, accent and native language. Fluent Speech Commands ships age range, gender and speaking ability for its anonymized speakers.
- Device and environment. Device, microphone distance, room, speaker position and noise. SLURP encouraged participants to sit, stand or walk and sometimes not to speak directly to the microphone array, but "these parameters are not logged with the dataset", so results can't be broken down by them.
When SpeechData.ai records to spec, transcripts are time-aligned and human-verified in the buyer's convention, so the rule for numbers and dates is yours to set.
Speakers, accents and devices
A command model has to work for every voice the product is sold to, and public corpora show how easily a test misses that. A Carnegie Mellon study noted that all of Fluent Speech Commands' held-out test speakers are native English speakers, while accented speakers appear only in training. Running Google's speech recognizer over the dataset, the authors measured a word error rate of 10.8% for speakers whose first language is English and 27.8% for the others.
Set quotas in the brief by language, region, accent, age band and gender, with a minimum number of speakers per language. For comparison, Speech-MASSIVE's test sets range from 30 speakers in Vietnamese to 151 in Polish. If your users span several regional accents, name them in the quotas; our page on recruiting speakers by accent shows how that is checked before anyone records.
Devices matter as much. The Snips authors write that most available training data "is clear close-field speech", while assistants are used in noise, "from a distance of several meters in far-field conditions, and in rooms or cars with reverberation", which they had to simulate. SLURP recorded both, 34,603 close-range and 37,674 far-range files, and moved the microphone array to a new position for each of the four parts of a session.
SpeechData.ai's wake word and voice command collection records on phone, headset, far-field or in-car audio, as the brief specifies, with native speakers recruited to quotas for age, gender, region and accent, each passing an ID check and a microphone test.
Build the test set before you train
- Hold out speakers. Fluent Speech Commands splits its data so that "no speaker appears in more than one split", and CLINC150 puts every in-scope query from a given crowd worker in a single split. Do the same by speaker ID.
- Hold out phrasings too. In Fluent Speech Commands, every test transcript also appears in training, so the test measures new voices only. On the Carnegie Mellon team's new splits, the same model scored 99.1% on the original test set, 94.2% on unseen speakers matched to the training demographics, 88.2% on unseen phrasings and 78.3% on the phrasings that shared the fewest words with training. Their splits opened gaps of up to 10% between systems that were within 1% of each other on the original test.
- Score per intent and per slot. Report exact match (all slots right, the accuracy Fluent Speech Commands uses), then each intent and each slot, as the Snips paper does. One average hides the date slot that fails.
- Score out-of-scope recall separately. CLINC150's results show a model can be accurate in scope and still act on most requests it should decline.
- Slice by speaker group and device. The rules in our guide to ASR test sets apply here: count speakers per slice, not just utterances, and keep the set frozen and versioned.
A worked example
The numbers here are an illustration, not a recommendation. Say the product has 25 intents and you write 12 phrasings for each: 300 prompt templates. Hold out 2 phrasings per intent (50 templates) and 100 of 300 speakers for the test set.
- The 200 training speakers each record 120 different prompts drawn at random from the 250 training templates, with fresh slot values each time. That makes 24,000 utterances, about 96 per template, each from a different speaker.
- The 100 test speakers each record 60 prompts drawn from all 300 templates. That makes 6,000 utterances, of which about 1,000 use phrasings the model never trained on.
- 100 test speakers allow two slices of 50, for example two accent groups or two device classes. More slices need more test speakers.
Because templates and speakers are held out separately, the test can tell a new voice from a new phrasing.
A voice command data spec to paste into a brief
Fill in each row for your product. The reference points come from the corpora above and are planning references, not rules.
| Item | What to specify | Reference point |
|---|---|---|
| Intents and slots | Every intent, its slots and the allowed values for each slot | Fluent Speech Commands: 31 intents from three slots. SLURP: 18 scenarios and 46 actions |
| Phrasings | Phrasings per intent, who writes them, which are held out for testing | Fluent Speech Commands: 248 phrasings, eight per intent on average |
| Slot values | Values per slot, synonyms, spoken forms of numbers and dates, local values per language; each value in several carrier phrases | Snips smart lights: 34 room values |
| Out-of-scope and near-miss | Share of prompts, categories, who checks them against the intent list | CLINC150: 1,200 out-of-scope queries beside 22,500 in scope |
| Prompt format | Scripted, paraphrase or goal prompts and the mix; prompts per speaker; random order | Fluent Speech Commands presented phrases in random order |
| Languages, speakers and quotas | Languages and dialects; speakers per language; quotas by age band, gender, region and accent | Speech-MASSIVE test sets: 30 to 151 speakers per language |
| Repetitions | Takes per prompt per speaker | Fluent Speech Commands: each phrasing twice |
| Devices and environments | Phone, headset, far-field or in-car; distance; rooms; noise | SLURP: a headset and a far-field microphone array |
| Transcripts | Convention for numbers, dates, times, hesitations and false starts | Speech-MASSIVE: numbers written out in letters were one source of mismatches |
| Labels | Intent; slot type, span and normalized value; speaker ID; device and environment per file | SLURP did not log speaker position, so it can't be analyzed |
| QA | Each recording says the prompt (scripted) or a valid intent (goal prompts) | Fluent Speech Commands removed unintelligible and wrong-phrase audio |
| Consent | Written consent from every speaker covering commercial model training | Ask to see the consent wording before recording starts |
| Test set | Held-out speakers and phrasings; per-intent, per-slot and out-of-scope scores | Fluent Speech Commands: one model scored 99.1% on seen phrasings, 88.2% on held-out ones |
Planning a voice command collection
SpeechData.ai records the commands, slot values and out-of-scope requests in your prompt list with native speakers recruited to your accent and demographic quotas, on phone, headset, far-field or in-car audio, and you can send your brief through our wake word and voice command collection page for a feasibility assessment and a quote within one business day.
