Rethinking Entropy Allocation in LLM-based ASR: Understanding the Dynamics between Speech Encoders and LLMs

arXiv:2604.08003 · eess.AS, cs.CL, cs.SD · Submitted 2026-04-09 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Rethinking Entropy Allocation in LLM-based ASR".

Jane: The gist The study reexamines LLM-based ASR from an entropy allocation perspective and proposes a capability-boundary-aware multi-stage training strategy to optimize parameter efficiency and hallucination robustness.

Tom: First, who's behind it and why it matters.

Paper summary: Jane: So, to wrap up, the authors of "Rethinking Entropy Allocation in LLM-based ASR: Understanding the Dynamics between Speech Encoders and LLMs" are proposing a framework that explicitly decouples the speech encoder and the LLM.

Tom: They use a capability-boundary-aware approach. It involves a phoneme-prioritized pretraining and an IA-SFT stage to manage those entropy dynamics across modules better than previous methods.

Lu: The implication is that you don't just throw a giant language model onto an existing speech encoder and hope for the best performance, because you run into problems with representation drift and uncertainty.

Meng: It means for someone looking at deployment, this suggests that parameter efficiency isn't just about making the model smaller; it’s about designing the training process so the parts don't interfere with each other negatively.

Jane: And they found that by focusing on keeping the encoder acoustically grounded, you can get strong results with a smaller model size. They show competitive performance using only two point three billion parameters while significantly lowering hallucination rates across Mandarin and English benchmarks.

Tom: It’s about moving away from just scaling up the LLM and instead focusing on how the information flows between the audio understanding part and the language generation part of an ASR system.

Lu: This work sets a new direction for how we think about integrating these models, suggesting that principled training strategies tied to those entropy metrics are necessary to unlock their full potential reliably.

Conclusion: Tom: So, we've seen how this paper looks at how information gets split between the speech encoder and the LLM in AI speech recognition models.

Jane: Right, it’s about those entropy allocations—how much "understanding" is put into the audio part versus the language part.

Tom: The authors are calling it rethinking that allocation using three specific metrics to see where things go wrong in current systems.

Lu: They introduce Normalized spectral entropy, PAI for phonetic information, and CSAI for conditional semantic information to measure how much uncertainty stays in each module.

Meng: It looks like they found these two main trouble spots: one where the encoder drifts because it leans too much on the LLM’s guesses, and another where the system just offloads all its confusion onto the language model itself.

Jane: And their whole proposal is a multi-stage training strategy that tries to fix that by setting clear boundaries between those two parts.

Tom: They are focusing on capability-boundary awareness, meaning they want the encoder to get good, acoustic understanding first before the LLM starts taking over everything.

Lu: The method uses a phoneme-level CTC pretraining and then adds this iterative stage that runs while other things are happening, which is called IA-SFT.

Jane: That iterative part specifically helps keep the encoder from drifting by forcing it to see different types of audio representations in a controlled way, without letting it lose its acoustic grounding.

Tom: The results show that this decoupling works really well for parameter efficiency and hallucination control on both Mandarin and English benchmarks.

Meng: It’s interesting because they manage to get competitive performance using only two point three billion parameters, which is pretty small for this kind of system.

Lu: But the real win, according to their analysis, is how much lower their hallucination rates are compared to other models because the design fundamentally stops the encoder from getting messy.

Jane: So they’ve shown that by carefully managing these entropy flows and keeping those boundaries sharp, we can get better accuracy without just throwing more parameters at it.

Tom: It really makes you wonder what this means for building real-world ASR systems that actually work reliably in tricky situations.

Yuan Xie, Jiaqi Song, Guang Qiu, Xianliang Wang, Ming Lei, Jie Gao, Jie Wu

Advanced Intelligent Systems Group, NIO

eess.AS, cs.CL, cs.SD

Submitted: 2026-04-09

Updated: 2026-10-09

Comments: Accepted at NeurIPS 2026 Main Track

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 92/100

The gist: The gist The study reexamines LLM-based ASR from an entropy allocation perspective and proposes a capability-boundary-aware multi-stage training strategy to optimize parameter efficiency and

Key concepts

Entropy Allocation Metrics
These are three specific metrics—Normalized spectral entropy (NSE), Phonetic accessible information (PAI), and Conditional semantic accessible information (CSAI)—used to measure how much uncertainty is reduced in the speech encoder versus the LLM during training. They reveal if the system is drifting or inefficiently using parameters.
Capability-Boundary-Aware Design
This design principle dictates that the speech encoder must first learn acoustically grounded, low-entropy representations before interacting with the LLM. During joint training, a functional boundary must be maintained so that encoder compression remains focused on acoustic quality rather than sacrificing representation fidelity.
IA-SFT Stage
This is an iterative asynchronous stage placed between alignment and joint SFT. It freezes the encoder while training the adaptor and LLM. This stage uses CKA-driven hot-swapping to expose the LLM to diverse encoder representations, helping constrain representation drift by forcing a curriculum-like exposure.
Phoneme-level CTC Pretraining
This pretraining phase uses a Conformer encoder with a linear CTC head, trained on phoneme units instead of text. This approach is motivated by the idea that phoneme representations offer a more universal and acoustically grounded interface for ASR than using grapheme or subword units.

Terminology

Summary

The gist The study reexamines LLM-based ASR from an entropy allocation perspective and proposes a capability-boundary-aware multi-stage training strategy to optimize parameter efficiency and hallucination robustness.

Entropy Allocation Metrics

The researchers introduce three metrics to characterize how training paradigms allocate entropy reduction between the speech encoder and the LLM <ref:2604.080035, Normalized spectral entropy (NSE) quantifies the spectral entropy of encoder representations, while phonetic accessible information (PAI) and conditional semantic accessible information (CSAI) serve as probe-inspired proxies for linearly accessible phoneme17, NSE (%) 86.8 85.1 NSE FireRed Encoder (Before e2e SFT) <ref:2604.080037, PAI (bits) 23.50 PAI FireRed Encoder (Before e2e SFT) <ref:2604.080037, CSAI (bits) 11.37 CSAI FireRed Encoder (Before e2e SFT) <ref:2604.08003#pg8>. These metrics reveal two characteristic suboptimal regimes in prevailing LLM-ASR systems: one suffers representation drift and over-reliance on semantic priors, the other offloads residual uncertainty onto the LLM at the cost of parameter efficiency.

Design Principle

Motivated by this diagnosis, they propose a capability-boundary-aware design principle with two requirements: The encoder should be guided toward low-entropy, acoustically grounded representations before being exposed to LLM-dominated joint optimization. During joint optimization, the functional boundary between modules should be explicitly maintained, ensuring that further encoder compression proceeds along an acoustic-grounded direction rather than at the expense of acoustic representation quality.

Multi-Stage Training Paradigm

The proposed training pipeline features two core components: Phoneme-level CTC pretraining and an additional IA-SFT stage that runs as asynchronously in parallel with the pretraining phase. The phoneme-level CTC Pretraining starts from a Conformer encoder initialized with FireRedASR-AED weights, replacing the autoregressive decoder with a lightweight linear CTC head trained under the CR-CTC (Yao et al., 2024) objective. This pretraining is motivated by prior findings that phoneme-based representations can offer a more universal and acoustically grounded interface than grapheme or subword units.

Alignment and IA-SFT

To explicitly constrain representation drift, they introduce an additional IA-SFT stage between alignment and joint SFT. This stage runs as asynchronously in parallel with pretraining. During IA-SFT, the encoder is frozen while the adaptor and LLM are trained. This stage serves an alignment purpose by performing CKA-driven iterative encoder hot-swapping to expose the LLM to diverse encoder representations in a curriculum-like fashion.

Empirical Analysis of Metric Dynamics

During phoneme-level CTC pretraining, NSE drops noticeably, as the CTC bottleneck compels the encoder to form compact, low-entropy representations. Meanwhile, PAI decreases while CSAI rises steadily. During joint SFT, NSE continues to decrease along their training trajectory. Furthermore, a key pattern emerges in their trajectory: PAI recovers notably while CSAI remains flat or slightly declines — consistent with their design expectations.

Hallucination Mitigation

A core contribution of their method is its principled approach to hallucination mitigation. They report that their model achieves substantially lower hallucination rates than all baselines. This suggests that their design mitigates hallucination at a fundamental level, owing to the explicit constraint during training that enforces the encoder to remain acoustically grounded, thereby suppressing representation drift. They further ablate the IASFT module and identify it as a major contributor to their low hallucination rate.

Layer-wise Alignment

Figure 4 visualizes the progression of semantic alignment by computing linear CKA scores between ground-truth text embeddings and representations at each encoder layer as well as the post-adaptor embedding. For FireRedASR-LLM, the CKA score decreases after adaptor projection, indicating that high-entropy encoder representations are difficult to map cleanly into the text embedding space. In contrast, their model consistently shows a pronounced CKA increase at this interface.

Conclusion

In this work, they revisit LLM-based ASR from an entropy allocation perspective and propose a capability-boundary-aware framework that explicitly decouples the encoder and LLM to resolve acoustic uncertainty and semantic ambiguity. By implementing a phoneme-prioritized encoder pretraining and an IA-SFT paradigm, they improve the entropy reduction dynamics across modules. Experiments on Mandarin and English benchmarks show that their approach can achieve competitive performance with lower hallucination rates using only 2.3B parameters. Future work will extend this analysis to large-scale LALMs and investigate how reinforcement learning further reshapes entropy allocation.

--- Page 1 ---

Rethinking Entropy Allocation in LLM-based ASR: Understanding the Dynamics between Speech Encoders and Large Language Models

Multi-Stage Training Paradigm

The proposed training pipeline features two core components: Phoneme-level CTC pretraining and an additional IA-SFT stage that runs as asynchronously in parallel with the pretraining phase.

Improvements for AI systems

  1. Adapt pretraining to utilize phoneme-level CTC rather than character-level CTC to mitigate premature semantic coupling, resulting in more language-agnostic and acoustically grounded representations.

  2. Implement an iterative asynchronous SFT (IA-SFT) stage between alignment and joint SFT, which explicitly narrows the speech-text modality gap and acts as a representational buffer to prevent representation drift during joint optimization.

  3. Employ CKA-driven iterative encoder hot-swapping during IA-SFT, where the system monitors CKA scores between encoders (0.50 M-1.01 M) and reference checkpoint (0.50 M) to trigger updates when the score falls below a predefined threshold, ensuring robust adaptation across evolving encoder states.

  4. Optimize for a capability-boundary-aware design principle where the encoder is guided toward low-entropy, acoustically grounded representations before being exposed to LLM-dominated joint optimization, thereby preserving acoustic grounding and suppressing premature semantic anchoring.

  5. Achieve competitive performance with significantly lower hallucination rates by enforcing a division of labor where the encoder primarily reduces acoustic uncertainty while the LLM handles higher-level semantic disambiguation, leading to a system that is efficient and robust for real-world deployment.

Sources

Related papers