A multilingual ASR model is one speech recognition model that transcribes several languages. For most products that need a handful of languages, the practical choice is one multilingual model, fine-tuned on data from every language it must serve, with a separate test set and score for each language. Sharing a model lets languages borrow from each other. It doesn't remove the need for data in each one: accuracy in a language still follows how much audio of that language the model learned from.
So multilingual speech data is planned language by language. Each language needs its own hours, speakers and varieties, every language follows one transcription and metadata convention so the sets combine, and each language is tested on its own. This guide sets out the evidence, the decisions, a worked budget from our catalog and a checklist you can reuse.
One multilingual model or one model per language
The research gives a consistent picture: sharing parameters across languages costs small models accuracy and helps large ones. In the Whisper paper, OpenAI found that small models trained with moderate compute did worse on English when trained jointly on many languages and tasks than English-only models given the same compute. The largest joint models outperformed their English-only counterparts.
The authors of XLS-R saw the same pattern. Their 300-million-parameter multilingual model trailed an English-only model of the same size on LibriSpeech, which they attribute to capacity dilution and interference, and the 1-billion-parameter version largely caught up.
Meta's Massively Multilingual Speech (MMS) paper shows a middle path. When a model with fully shared parameters grew from 61 to 1,107 languages, its average character error rate (CER) on 61 FLEURS languages rose by 5.1 points, mainly because it confused languages with each other. Small language-specific adapters, about 2% extra parameters per language, plus a per-language output layer cut that rise to 0.4 points.
For a buyer, the architecture changes the training recipe, not the shopping list:
- One shared model suits a product that serves all its languages from one stack, has some languages with little data, or hears users switch languages mid-conversation.
- Per-language models, fine-tunes or adapters suit a language that carries most of the traffic, tight size or latency limits, or languages served by separate deployments.
- Either way, the data plan is per language. Each language still needs matched audio, its own test set and its own score.
Data per language decides accuracy per language
The Whisper paper measured this directly. Of Whisper's 680,000 hours of training audio, 117,000 hours covered 96 languages other than English, spread very unevenly. Across languages, the authors found a squared correlation of 0.83 between the log of the word error rate (WER) on the FLEURS benchmark and the log of training hours in that language. Their fit implies that WER halves for every 16-fold increase in training data.
Many of the languages that did worse than the trend predicts have their own scripts and are more distantly related to the Indo-European languages that make up most of the data, such as Hebrew, Telugu, Chinese and Korean.
Here is the trend for seven languages, all of them in our catalog. Hours come from the paper's training data statistics, and error rates from its FLEURS results for Whisper large-v2.
| Language | Whisper training hours | Whisper large-v2 WER on FLEURS |
|---|---|---|
| German | 13,344 | 4.5% |
| Indonesian | 1,014 | 7.1% |
| Vietnamese | 691 | 10.3% |
| Malay | 382 | 8.7% |
| Tagalog | 75 | 13.8% |
| Hindi | 12 | 21.5% |
| Swahili | 5.4 | 39.3% |
The trend is not exact: Malay scores better than Vietnamese on about half the hours. And FLEURS is read speech. Its paper describes native speakers reading the same English Wikipedia sentences, translated, in 102 languages, about 12 hours per language. Treat these numbers as a measure of how much each language starts with, not as the accuracy your users will get in conversation.
Meta's Omnilingual ASR paper, which covers more than 1,600 languages, shows the same effect at the low end. More than 90% of languages with 10 or more hours of labeled training data reached a CER below 10. Among languages with fewer than 10 hours, only 34 to 36% did, depending on the model.
Open data shows why a language count is not a data plan. Common Voice, where contributors read pre-written sentences aloud, lists 295 languages and 29,295 validated hours in Scripted Speech release 27.0, with data up to September 2026. In that release's statistics, 108 of the languages have fewer than 10 validated hours and 39 have 100 or more. Catalan has 3,392 validated hours, Vietnamese 7.8 and Malay 0.2.
Hours aren't the whole story either. The labeled data MMS built is readings of the New Testament in 1,107 languages, with an average of 32 training hours per language, and its authors name the downside: one narrow domain, with most recordings from a single speaker. When a vendor or corpus says it covers many languages, ask for hours, distinct speakers and speaking style for each language you need.
Balancing languages in training
Your data won't be balanced, and neither is production traffic. Multilingual training handles this with a sampling rule. XLS-R samples each language with a probability proportional to its share of the data raised to a power, which the paper calls an upsampling factor that controls the trade-off between high- and low-resource languages. It used 0.5, both across languages and across the corpora they came from.
The Omnilingual ASR team swept the same settings, where an exponent of 1 keeps the natural mix and 0 samples every language equally. With no upsampling, low-resource languages did by far the worst. Uniform sampling scored best on the language-based evaluations, but it over-weighted the narrow audio domain of the MMS corpus at the expense of noisier, more varied corpora such as Babel and Common Voice. The final models used 0.5 across corpora and 0.25 across languages.
You will also see the term temperature-based sampling: mSLAM, which pre-trains on speech and text, uses it to upsample lower-resource languages in its text data.
What this means for buying: upsampling repeats the hours a language already has. A language with 20 hours from 30 speakers, sampled up, still teaches the model 30 voices. Sampling can rebalance training, but it can't add speakers, accents or recording conditions the data doesn't contain. Record the exponent with every training run, and buy hours where per-language tests show gaps.
Make the languages combinable
Data from several languages combines cleanly only if every language follows the same rules. Write these once, for all languages, before anything is recorded or bought:
- Transcription convention. Verbatim or clean, and one tag set for disfluencies, noise, overlap, unintelligible speech and foreign words, used in every language.
- Normalization. How each language writes numbers, casing, punctuation and abbreviations, applied the same way to training transcripts and test references. The Whisper authors note that non-English text needs language-specific normalizers, which they lacked the linguistic knowledge to build, so they fell back to a basic standardization.
- Language tags. A BCP 47 tag on every file and every code-switched span, with a script subtag where a language has two scripts, such as sr-Latn and sr-Cyrl for Serbian. The MMS team treated Serbian recordings in Latin and Cyrillic script as separate languages for ASR.
- Metadata. The same fields in every language: speaker ID, age, gender, region, recording device and channel.
- Audio. One sample rate, file format and channel layout across languages, with one channel per speaker for conversations.
- Delivery. One folder layout and manifest format, so a new language drops into the pipeline without new code.
Then add a short appendix per language for what the shared rules can't settle.
Scripts and word boundaries. Japanese, Thai, Chinese, Lao and Burmese don't put spaces between words, so a word error rate first needs a segmentation rule. Whisper's evaluation puts a space between every character in those languages, which in effect measures character error rate, and MMS adds Khmer to that list. Japanese transcripts also need rules for when to write kanji and when kana.
Tones and diacritics. Vietnamese writes most of its tones as marks on vowels, and careful Yoruba writing marks tones with accents that most text online leaves out. Decide whether transcripts carry the marks, and check that your scoring normalizer keeps them, or tone errors vanish from the score.
Code-switching and varieties within a language
In many markets, people switch language inside a sentence. Everyday Tagalog in Manila is usually Taglish, Malaysians mix Malay with English, and Singapore English borrows words from Hokkien, Malay and Tamil. Decide whether switched speech stays in each language's training set, how switched words are written, and how the spans are tagged. Our guide to code-switching in speech recognition explains why it breaks systems built for one language at a time.
A language name is not a target either. Spanish in Spain and in Mexico differs in pronunciation and vocabulary, and Portuguese differs between Portugal, Brazil and Angola. That is why our catalog's 60 datasets cover 49 languages: English, Spanish, French, Portuguese, Dutch and Arabic each come in more than one variety, each recorded in-country. List the varieties your users speak and set hours for each. Our article on accent and dialect coverage explains how to find the gaps.
Evaluate each language separately
A multilingual model needs a test set per language, drawn from the people and conditions that language will serve, with test speakers kept out of training. Our guide to building an ASR test set covers size, speakers and splits. For a multilingual model, repeat it for every language, and for every variety where varieties differ.
Report the score for each language: WER, or CER where words aren't separated by spaces. Don't lead with one average. The unweighted average of the seven Whisper scores in the table above is 15.0%, a number that describes neither German at 4.5% nor Swahili at 39.3%. Set an acceptance threshold per language, and report the worst language next to any average.
Public benchmarks still help. FLEURS has the same translated sentences in 102 languages, so content stays constant across languages, but it is read speech from people who are not your users. Use it beside your own test sets, not instead of them.
A worked budget for five languages
Say a voice product launches in five Southeast Asian markets, and you want 200 hours of conversational speech in each language for fine-tuning. Here is the cost from our datasets for Southeast Asian languages, at list price.
| Language | Hours in the dataset | Price per hour | 200 hours |
|---|---|---|---|
| Indonesian | 2,000 | $65 | $13,000 |
| Malay | 2,000 | $65 | $13,000 |
| Thai | 2,000 | $75 | $15,000 |
| Vietnamese | 2,000 | $65 | $13,000 |
| Tagalog | 2,000 | $65 | $13,000 |
| Total, 1,000 hours | $67,000 |
The same 1,000 hours for a European product, in German, French, Spanish, Italian and Dutch from the datasets recorded in Germany, France, Spain, Italy and the Netherlands, comes to $89,000: 200 hours each at $95, $95, $85, $75 and $95. These are list prices from our pricing page, and volume and multi-language discounts are available on request.
200 hours per language is an illustration, not a recommendation. The Whisper figures suggest where to shift budget: if you fine-tune Whisper, Tagalog started with 75 training hours against 1,014 for Indonesian, so Tagalog is the likelier place for extra hours to pay off. Run your base model on a test set for each language first, then split the budget by the gaps you measure.
Multilingual ASR data planning checklist
| Decision | What to specify |
|---|---|
| Languages and varieties | Each language and variety by country, with a BCP 47 tag, for example es-ES and es-MX |
| Hours per language | Set per language from base-model gaps and test results, not one number for all |
| Speakers per language | Distinct speakers per language, and which speakers are held out for testing |
| Speaking style | Conversational or read, per language, matched to how the product is used |
| Code-switching | Kept or removed, how switched words are written, how spans are tagged |
| Transcription convention | One guideline for all languages, plus a per-language appendix for script, word boundaries, numbers and tone marks |
| Metadata | The same fields in every language: speaker ID, age, gender, region, device, channel |
| Audio | The same sample rate, format and channel layout in every language |
| Balancing | The sampling exponent across languages and corpora, recorded with each training run |
| Evaluation | A test set per language from target users, WER or CER per language, worst language reported |
| License and consent | The same license terms and consent documentation for every language |
Where to get the data
SpeechData.ai's catalog has spontaneous two-person conversations in 60 languages and dialects, with the same components in every language: dual-channel audio, time-aligned and human-verified transcripts, and the same speaker metadata fields (age, gender, region and recording device). Ask for a sample in each language you need and check its transcripts against your shared convention before you combine them. For a language or variety outside the catalog, we collect speech data to spec with native speakers, scripted or spontaneous.
