Rethinking Entropy Allocation in LLM-based ASR: Understanding the Dynamics between Speech Encoders and LLMs
The study analyzes how training allocates entropy reduction between speech encoders and LLMs in ASR. It introduces three metrics to diagnose suboptimal training regimes characterized by representation drift or offloading uncertainty. The proposed solution is a capability-boundary-aware multi-stage training strategy involving phoneme pretraining and an iterative alignment stage, which improves parameter efficiency and reduces hallucinations.