When Audio-Language Models Fail to Leverage Multimodal Context for Dysarthric Speech Recognition
Pehuén Moure, Niclas Pokel, Bilal Bounajma, Yingqiang Gao, Roman Boehringer, Longbiao Cheng, Shih-Chii Liu
cs.AI, cs.CL, eess.AS
Submitted: 2026-08-18
Updated: 2026-08-20
Code: https://github.com/fixie-ai/ultravox
License: http://creativecommons.org/licenses/by/4.0/
The gist: Automatic speech recognition (ASR) systems remain brittle on dysarthric and other atypical speech.
Terminology
Abstract
Automatic speech recognition (ASR) systems remain brittle on dysarthric and other atypical speech. Recent audio-language models raise the possibility of improving performance by conditioning on additional clinical context at inference time, but it is unclear whether these models can make use of such information. We introduce a benchmark built on the Speech Accessibility Project (SAP) dataset that tests whether diagnosis labels, clinician-derived speech ratings, and progressively richer clinical descriptions improve transcription accuracy for dysarthric speech. Across matched comparisons on nine models, we find that current models do not meaningfully use this context: diagnosis-informed and clinically detailed prompts yield negligible improvements and often degrade word error rate. We complement the prompting analysis with context-dependent fine-tuning, showing that LoRA adaptation with a mixture of clinical prompt formats achieves a WER of 0.066, a 52% relative reduction over the frozen baseline, while preserving performance when context is unavailable. Subgroup analyses reveal significant gains for Down syndrome and mild-severity speakers. These results clarify where current models fall short and provide a testbed for measuring progress toward more inclusive ASR.
Sources
- Zero-Shot Recognition of Dysarthric Speech Using Commercial Automatic Speech Recognition and Multimodal Large Language Models
- Qwen2-Audio Technical Report
- A Survey on In-context Learning
- Gemini: A Family of Highly Capable Multimodal Models
- Audio Flamingo 3: Advancing Audio Intelligence with Fully Open Large Audio Language Models
- Robust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised Pre-Training
- LoRA: Low-Rank Adaptation of Large Language Models
- Voxtral
- Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs
- Adversarial NLI: A New Benchmark for Natural Language Understanding
- GPT-4o System Card
- Aligning ASR Evaluation with Human and LLM Judgments: Intelligibility Metrics Using Phonetic, Semantic, and NLI Approaches
- Adapting Foundation Speech Recognition Models to Impaired Speech: A Semantic Re-chaining Approach for Personalization of German Speech
- Robust Speech Recognition via Large-Scale Weak Supervision
- Fast Conformer with Linearly Scalable Attention for Efficient Speech Recognition
- Qwen3-ASR Technical Report
- Personalized Fine-Tuning with Controllable Synthetic Speech from LLM-Generated Transcripts for Dysarthric Speech Recognition
- Qwen3-Omni Technical Report
- MiniCPM-V 4.5: Cooking Efficient MLLMs via Architecture, Data, and Training Recipe
- BERTScore: Evaluating Text Generation with BERT
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems