Benchmarking speech-to-text robustness in noisy emergency medical dialogues: an evaluation of models under realistic acoustic conditions.
JAMIA Open · 2025-11-19 · Method & validation
Abstract
OBJECTIVES: To evaluate the transcription accuracy of 6 German-capable speech-to-text (STT) systems in simulated emergency medical services (EMS) environments, focusing on clinically relevant performance under noisy and multilingual field conditions. MATERIALS AND METHODS: We generated a corpus of 99 synthetic emergency dialogues and overlaid them with ecologically valid noise types-crowd chatter, traffic, public spaces, and ambulance interiors-at 5 signal-to-noise ratios (SNRs), producing 1980 noisy audio samples. Each was transcribed by 6 STT systems (recapp, Vosk, Whisper v3 variants, and RescueSpeech). We assessed performance using 5 metrics: Word Error Rate (WER), Medical Word Error Rate (mWER), TF-IDF Cosine Similarity, BLEU, and semantic embedding similarity. Statistical models quantified the effects of system, noise, and SNR on transcription fidelity. RESULTS: recapp consistently outperformed all other systems across metrics. Among open-source models, Whisper v3 Turbo achieved the lowest mWER and strongest phrase-level accuracy (BLEU), while Whisper v3 Large preserved semantic content best. RescueSpeech and Vosk underperformed. "Inside crowded" noise had the most degrading impact on performance, while "talking" noise had minimal effect. Performance degradation was most pronounced at the lowest SNR (-2 dB). DISCUSSION: STT model accuracy is highly sensitive to acoustic conditions. Clinically salient transcription errors (mWER) were most frequent under dense environmental noise. Whisper v3 Turbo balances accuracy and efficiency, suggesting strong potential for EMS applications. CONCLUSION: This study introduces a clinically grounded, noise-robust benchmark for STT evaluation in EMS settings. It highlights the importance of domain-specific metrics and acoustic realism for deploying STT systems where transcription errors carry safety-critical consequences.
Abstract © the authors or publisher, via PubMed. Reuse and removal
Tags
Communication & handover · Rural & remote · Telemedicine
Related articles
- Telemedicine and decision support in emergency ambulances in Uppsala. (J Telemed Telecare, 2000)
- Pre-hospital technology research: reflecting on a collaborative project between ambulance service and academia (Journal of Paramedic Practice, 2015)
- A Pathophysiological Model-Driven Communication for Dynamic Distributed Medical Best Practice Guidance Systems. (J Med Syst, 2016)
- A Smart Glass Telemedicine Application for Prehospital Communication: User-Centered Design Study. (J Med Internet Res, 2024)
- Technical and organisational feasibility of a multifunctional telemedicine system in an emergency medical service - an observational study. (J Telemed Telecare, 2011)
- Experiences with the quality of telemedical care in an offshore setting - a qualitative study. (BMC Health Serv Res, 2023)