Medical speech recognition, also called healthcare speech recognition, is speech-to-text built for clinical speech: a physician dictating a note, a doctor and patient talking during a visit, or a patient calling to book an appointment. Under the hood it is the same kind of ASR model used everywhere else. What makes it medical is the vocabulary, the stakes, and the data it was trained and tested on.
That last part is where many teams get stuck. General models stumble on drug names and doses, and the recordings that would fix this, real patient encounters, are some of the hardest audio to collect and license. This guide covers what each use case needs from its data, the realistic ways to get it, and how to test the result.
What medical speech recognition covers
Speech recognition in healthcare covers three different products, and their audio differs a lot. Decide which one you are building before you collect anything, because data that suits one rarely suits another.
| Use case | Who speaks | Typical audio | What the data must contain |
|---|---|---|---|
| Dictation | One clinician, speaking to the system | Close microphone or phone; fast and dense with terms | Specialty vocabulary, doses, abbreviations, spoken formatting commands, your clinicians' accents |
| Ambient scribe | Clinician and patient, sometimes family or an interpreter | One device in the room, at a distance from both speakers, with overlap | Natural multi-party conversation, lay and clinical wording, marked overlap, speaker turns |
| Calls and voice agents | Patient and a scheduler, nurse line or bot | Phone line, often 8 kHz narrowband | Names, dates of birth, times, phone numbers, booking and refill intents, telephony audio |
Dictation is the oldest and most forgiving case: one speaker who knows they are talking to software. Ambient documentation is the hardest, because the model hears an unscripted conversation from across the room and has to catch every term that ends up in the note. Healthcare call centers sit in between. The speech is closer to customer service than to clinical talk, but callers routinely give names, dates of birth and insurance details.
Why general speech recognition fails on clinical speech
A general model can post a respectable word error rate on clinical audio and still be unusable, because the words it gets wrong are the ones that matter. The usual failure points:
- Rare terms. Drug names, procedures and anatomy are rare in general training data, so the model substitutes a common word that sounds close.
- Confusable drug names. The FDA recommends tall man lettering for look-alike pairs such as hydrALAZINE and hydrOXYzine, or predniSONE and prednisoLONE, so people can tell them apart in print. A recognizer gets no capital letters, only a syllable or two of difference in fast speech.
- Numbers and doses. "Fifteen" and "fifty", "point five" and "five", milligrams and micrograms. One wrong digit can become a medication error, and doses have to be written consistently for anything downstream to parse them.
- Abbreviations and shorthand. Clinicians say "b.i.d.", "PRN", "A-fib" and specialty shorthand that general text normalizers mangle.
- Accents and pace. Clinical staff come from many language backgrounds, and dictation is fast and clipped. A model tuned on broadcast-style speech handles neither well.
- Conversation. In an exam room people interrupt, talk over each other, trail off and use lay terms like "water pill" or "my sugar". Equipment, doors and a distant microphone add noise.
- Small words with big meaning. "No chest pain" and "chest pain" differ by one word. So do the left and right knee.
What training data each use case needs
Dictation. Record the clinicians you will serve, or speakers who match them in accent and specialty. Cover your specialty's term list on purpose rather than hoping it turns up, including doses in the forms people actually say them. Decide how spoken commands such as "new paragraph" or "period" are transcribed, because the model learns whatever convention the transcripts use.
Ambient documentation. Capture each speaker on a separate channel so the reference transcripts are clean, and also capture the single room microphone your product will hear, so you can train and test on the mixed signal. Mark overlap and speaker turns. Vary visit types, lengths and the amount of small talk, and include patients who describe symptoms in everyday words.
Healthcare calls and voice agents. Use telephony audio at the sample rate your system runs on, with agent and caller on separate channels. Scripts should cover booking, rescheduling, cancellations, refill requests and insurance questions, with names, dates of birth and phone numbers spoken the way callers really say them. Label intents and personal details, so the same data can also train a redaction step.
Why real patient recordings are hard to use
The most realistic data is recordings of real care, which is also the most regulated. What follows is a short, general summary, not legal advice. Check with your counsel before you rely on any of it.
In the US (HIPAA). HIPAA defines health information as information "whether oral or recorded in any form or medium," so an identifiable recording of a patient visit can be protected health information. A vendor that creates, receives, maintains or transmits that information on a provider's behalf is generally a business associate, and the provider has to get satisfactory assurances from it in writing, usually a business associate agreement.
De-identification under HIPAA. The Privacy Rule does not apply to information de-identified by one of two methods: an expert determination, or the Safe Harbor method, which, among other conditions, requires removing 18 types of identifiers of the patient and of their relatives, employers and household members.
The list includes names, all elements of dates except the year, phone numbers, medical record numbers and "biometric identifiers, including finger and voice prints." Many of these are spoken aloud during a visit, and the voice itself is on the list, so scrubbing the transcript does not settle what to do with the audio.
In the EU (GDPR). Data concerning health is a special category of personal data, and so is biometric data processed to uniquely identify a person. Explicit consent is one of several conditions that can allow processing, and EU or national law may rule it out in some cases. Pseudonymised data that can still be linked back to a person remains personal data.
Consent in practice. A patient who agreed to be recorded for their own care record has not necessarily agreed to a third party training a commercial model on their voice. Our primer on voice data privacy covers consent and redaction in more depth.
None of this rules real data out. It usually means real data arrives slowly, through a health-system partner, with agreements in place and a plan for de-identifying the audio as well as the transcript.
Where to get medical speech data
Real data, de-identified, through a health-system partner. This is the closest match to production and the best final test set if you can get it. Expect long legal timelines and limited volume.
Open datasets. A few public sets exist. Check each license on the original source, not on a mirror.
- PriMock57. 57 mock primary care consultations, 8 hours 38 minutes in total, between 7 clinicians and 57 company employees playing patients from case cards. It includes manual utterance-level transcripts and the clinicians' notes, under CC BY 4.0. Clinicians used laptops and patients used mobile phones in an office with background noise, and the telemedicine software exported each side on its own channel as compressed audio.
- Fareez et al. (2022). 272 simulated OSCE-style interviews, 3,309 minutes (about 55 hours), mostly respiratory cases. Resident doctors and senior medical students played both doctor and patient, so most patient voices belong to people in their twenties. Sessions were recorded over Microsoft Teams, transcribed automatically, corrected by hand and reviewed again by a team of two. The figshare record lists a CC0 license.
- MultiMed. About 150 hours in five languages, taken from medical channels on YouTube: lectures, interviews, podcasts, news and similar formats. The Hugging Face page carries an MIT tag, but the authors describe collecting and using the data for non-commercial research. A license tag on a hub page does not settle the rights to the underlying audio.
All three help with benchmarking. None was recorded on a single device in an exam room, which is what most ambient scribes hear.
Simulated clinical conversations recorded to your script. Write case scripts that cover your specialties, medication list and dose patterns, then have consenting speakers perform them with fictional patient details, on the devices and in the rooms you choose. Decide who plays each role: people with clinical training say terms the way clinicians do, while lay speakers sound more like real patients. Simulated visits are tidier than real ones, so keep a realistic held-out test set to check against.
General conversational speech. Spontaneous conversation across many accents teaches a model how people actually talk, which matters for the patient side of an ambient visit. It will not teach drug names.
Annotation of audio you already have. If you hold recordings with the right permissions, transcribing them to a strict guideline is often the fastest route to a usable training and test set.
Where SpeechData.ai fits
We do not sell patient recordings, and we do not have a ready-made medical dataset. What we offer comes from services we already run.
- Simulated consultations and dictation. Our custom collection service records them to a script or scenario you supply, with fictional patient details written into the script. Consenting speakers are recruited to your age, gender, region and accent quotas, speech can be scripted or spontaneous, and devices include phones, headsets and far-field setups. Transcripts follow your conventions, and consent documentation ships with the data.
- Healthcare scheduling calls. Healthcare scheduling is one of the example scenarios for our call center data: role-played calls by native speakers, one channel per speaker, narrowband or wideband, with fictional names and details and optional intent and PII labels.
- Accents and natural conversation. Our catalog holds spontaneous two-person conversations in 60 languages, one channel per speaker, with time-aligned, human-verified transcripts, speaker metadata and a commercial license. It helps a model cope with accents and natural conversation, not with medical vocabulary.
- Audio you already have. We annotate it under a written guideline that covers numbers and domain terms, and the contract sets who can access the audio, where it is stored and how long it is kept. If your audio contains patient information, have your counsel decide what agreement is required before anything is shared.
How to evaluate medical speech recognition
Keep WER as the overall measure and add metrics that track what clinicians care about. The PriMock57 authors did this in 2022: alongside WER, they extracted clinical concepts linked to SNOMED CT from the reference and machine transcripts and compared the two. Their baseline scores are dated now, but the method holds up.
- Medical term recall. Build a list of the terms that matter (drugs, doses, conditions, procedures, anatomy) and measure what share of their occurrences in the reference come out correctly. Also count terms that appear in the output but were never said, since an invented drug name can do as much harm as a missed one.
- Dose accuracy. Score the value and the unit together, after a normalizer that writes "five milligrams" and "5 mg" the same way.
- Negation and laterality. Check "no", "denies", "left" and "right" in context.
- Slices. Report results by speaker role, accent, microphone and specialty. One blended number hides the regressions that matter.
- A matched, held-out test set. Same devices, same rooms, same patient population, and never seen in training.
Buyer checklist
| Question | Why it matters | A good answer |
|---|---|---|
| Which use case is the data for? | Dictation, ambient and call audio differ | One named use case per dataset |
| Who are the speakers? | Accent and role drive errors | Speaker metadata that matches your users |
| How was it recorded? | Mic distance and channels shape what the model learns | Your devices, rooms and sample rate; one channel per speaker |
| Does it cover your terms? | Rare terms will not appear by chance | Scripts built from your term and dose list |
| Are patient details real or fictional? | Real patient details can bring HIPAA or GDPR obligations | Fictional, or real under agreements your counsel approved |
| What did speakers consent to? | Consent to care is not consent to training | Documented consent to AI training and commercial use |
| What are the transcript conventions? | Doses and abbreviations must be consistent | A written guideline you approved |
| What does the license allow? | Open datasets vary one by one | Commercial training, confirmed at the source |
| How will you test it? | WER alone hides term errors | A matched held-out set with term recall and dose accuracy |
