Encoder Awakening via Adapters: Effective Domain-Adaptive Fine-tuning of Speech-LLMs

arXiv:2609.17981 · eess.AS, cs.CL · Submitted 2026-09-16 · Read on arXiv

eess.AS, cs.CL

Submitted: 2026-09-16

Updated: 2026-10-08

Comments: Accepted to IEEE SLT 2026

Code: https://github.com/morganshi/EAVA

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Terminology

• Fine-Tuning — 47× in this paper · explained in 5 episodes
• Adaptation — 24× in this paper · explained in 1 episode
• Alignment — 12× in this paper · explained in 1 episode
• Automatic Speech Recognition — 4× in this paper
• Domain Adaptation — 4× in this paper · explained in 5 episodes
• Domain Shift — 4× in this paper · explained in 4 episodes
• Automatic Speech Recognition (ASR) — 3× in this paper · explained in 2 episodes
• Foundation Model — 3× in this paper · explained in 3 episodes
• Foundation Models — 3× in this paper · explained in 7 episodes
• Generalizability — 3× in this paper · explained in 6 episodes
• Loss Function — 3× in this paper · explained in 2 episodes
• Ablation — 2× in this paper · explained in 1 episode
• Ablation study — 2× in this paper · explained in 1 episode
• Benchmarking — 2× in this paper · explained in 3 episodes
• Connectionist Temporal Classification — 2× in this paper
• Generative AI — 2× in this paper · explained in 2 episodes
• Activation Function — 1× in this paper · explained in 1 episode
• Benchmark — 1× in this paper · explained in 2 episodes
• Centered Kernel Alignment — 1× in this paper
• Department of Education — 1× in this paper
• Fidelity — 1× in this paper · explained in 2 episodes
• Generalization — 1× in this paper · explained in 13 episodes
• Institute of Education Sciences — 1× in this paper
• International Conference on Machine Learning — 1× in this paper
• LLM Cross-Entropy — 1× in this paper
• Large Language Models (LLMs) — 1× in this paper · explained in 6 episodes
• Low-Rank Adaptation (LoRA) — 1× in this paper · explained in 9 episodes
• Reinforcement Learning — 1× in this paper · explained in 4 episodes
• Representation Learning — 1× in this paper · explained in 2 episodes
• Residual Adapters — 1× in this paper
• Self-Supervised Learning — 1× in this paper · explained in 8 episodes
• Transformer — 1× in this paper · explained in 2 episodes
• Zero-Shot Generalization — 1× in this paper · explained in 5 episodes
• alpha — 1× in this paper · explained in 1 episode

Sources

• The Llama 3 Herd of Models
• Phi-4 Technical Report
• Qwen3 Technical Report
• An Embarrassingly Simple Approach for LLM with Strong ASR Capacity
• Gumbel-BEARD: Automatic Layer Selection for Self-Supervised Adaptation of Whisper in Low-Resource Domains
• Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs
• Searching for Activation Functions
• GC-LoRA: Gated Convolutional LoRA for Parameter-Efficient Acoustic Adaptation

Related papers

• Rethinking Entropy Allocation in LLM-based ASR: Understanding the Dynamics between Speech Encoders and LLMs
• X-VC: Zero-shot Streaming Voice Conversion in Codec Space
• Autoregressive Guidance of Deep Spatially Selective Filters using Bayesian Tracking for Efficient Extraction of Moving Speakers
• VM-ARRAYDPS: Virtual Microphone Augmented Diffusion Posterior Sampling for Unsupervised Blind Speech Separation
• Conditional Flow Matching for Visually-Guided Acoustic Highlighting
• Anonymization, Not Elimination: Utility-Preserved Speech Anonymization