LLM-HB: Language-Aware LLM-Guided Hotword Biasing for Code-Switching ASR

The gist The framework incorporates a mixture-of-experts (MoE) adaptor and an auxiliary language prediction head to better capture and differentiate multilingual representations, while hotwords are injected through LLM-based prompting.

Introduction

Code-switching (CS) speech, characterized by language alternation within a single utterance, is prevalent in multilingual communities and poses challenges to automatic speech recognition (ASR) due to rapid language transitions Code-switching (CS) refers to the alternation between multiple languages within a single utterance or discourse, Compared with monolingual settings, automatic speech recognition (ASR) faces additional challenges in CS scenarios due to frequent inter- and intra-sentence language transitions Although conventional ASR systems can be applied to codeswitching speech similarly to monolingual data, early studies typically perform language identification prior to recognition or jointly model language identification and transcription More recent CS-ASR approaches mitigate cross-language confusion by incorporating language-aware components into the ASR architecture Prior work has explored various strategies to address crosslanguage interference in CS-ASR Beyond explicit structural separation, language-aware frameworks dynamically specialize representations for different languages Mixture-of-Experts (MoE) adaptors enable dynamic expert specialization to enhance language-dependent representation learning and distinguish language boundaries Other approaches incorporate frame- or token-level language posteriors into acoustic representations prior to decoding

Methodology

The LLM-HB framework integrates several components for CS-ASR Given an input speech spectrogram X ∈ R M×T, where M denotes the number of Mel-frequency bins and T the number of time frames, Acoustic features S are extracted using a pre-trained Whisper encoder The encoded features are passed through an MoE adaptor to produce language-specialized speech embeddings ES ∈ R T ×D, where D is aligned with the LLM representation dimension The hotword list H = h0, h1,..., hN, containing N hotwords and a special token, is tokenized using a frozen LLM tokenizer The tokenized sequence is encoded by the LLM text encoder to obtain hotword embeddings EH ∈ R N′×D The speech embeddings ES and hotword embeddings EH are fed into a LoRA-adapted [27] LLM Special marker tokens ⃝H, ⃝T, and ⃝E denote the beginning of hotword embeddings, transcription embeddings, and the end-of-sequence token, respectively During training, the LLM input sequence is constructed as [ES, ⃝H, EH, ⃝T, ET, ⃝E] The LLM outputs hidden states et, which are projected to the vocabulary space for token prediction An auxiliary language prediction head is attached to et to provide explicit language supervision

Language-Specialized MoE Adaptation and Auxiliary Supervision

The MoE adaptor performs dynamic expert selection and aggregation to produce languagespecialized speech embeddings Given acoustic features S, a router computes top-K gating weights, and the selected experts process the input independently Their weighted outputs are aggregated into an intermediate representation hmoe, defined as gt = Softmax(Top-K(Router(S), k)) (1) hmoe = X i∈K gt,i FFNi(S) (2) The aggregated representation hmoe is further transformed by a feed-forward layer followed by layer normalization (the Aggregation block in Figure 1(b)) to obtain the final speech embeddings ES: ES = LayerNorm(FFN(hmoe)) (3) To provide explicit language supervision, an auxiliary language prediction head is attached to the LLM hidden states et The language posterior is computed as Plt = Softmax(LayerNorm(FFN(et))) (4) where Plt denotes the predicted language distribution for the t-th token

LLM-Guided Hotword Biasing and Loss Functions

To encourage hotword recognition, an LLMguided biasing objective is introduced The biasing loss is defined as LBias = − X t∈H log p(ytX, H) (5) where yt denotes the ground-truth token at position t, and H represents the set of token positions corresponding to hotwords in the target sequence The overall training objective jointly optimizes the ASR loss, the hotword biasing loss, and the auxiliary language supervision loss within a unified end-to-end framework: L = LASR + λ1LBias + λ2LLan (6) The ASR loss LASR is defined as the standard crossentropy (CE) loss over the target transcription tokens The auxiliary language loss LLan is also formulated as a CE loss over the predicted language distribution at each output token

Experiment and Results

Experiments were conducted on three benchmark datasets: the 1000-hour Mandarin corpus AISHELL-2 [28], the 960-hour English corpus LibriSpeech [29], and the 200-hour codeswitching dataset ASRU2019-CS For the proposed LLM-HB framework, hyperparameters were set as follows We employed Whisper-medium3 as the speech encoder for its strong multilingual capability in codeswitching scenarios As the LLM backbone, we adopted Qwen3-4B4, which provides a favorable trade-off between efficiency and representational capacity The Whisper encoder produces 1024-dimensional acoustic embeddings, which are projected to 2560 dimensions via a dense MoE adaptor with two experts and top-2 gating to match the LLM input space During training, we optimized the model using AdamW with a triangular learning rate schedule The loss weights in Equation 6 were set to λ1 = 0.6 and λ2 = 0.3

Main Results of LLM-HB Method

When trained only on the 200-hour code-switching dataset, the complete LLM-HB further improves MER to 7.36%, demonstrating that the proposed framework remains effective even under limited training data Notably, the hotword prompt delivers the most significant individual gain, reducing MER to 7.48% and substantially lowering B-MER from 26.91% to 16.00%, confirming its strong biasing capability in low-resource conditions Under larger-scale multi-dataset training (2,200 hours), the complete LLMHB model, integrating all three components, achieves the best MER of 5.85%, corresponding to a 20.30% relative reduction over the baseline (7.34%) The improvements are particularly pronounced on hotwordrelated metrics, with B-MER decreasing from 22.11% to 8.99%, B-CER from 19.26% to 3.01%, and B-WER from 22.90% to 10.93

Ablation Studies and Analysis

The results indicate that the MoE adaptor alone leads to a slight performance degradation (MER increases from 9.59% to 9.65%), suggesting that the additional parameters introduced by the adaptor may require larger-scale data for effective optimization The single-expert configuration (1 choose 1) achieves a MER of 7.23%, which is inferior to the proposed multi-expert setting The complete LLM-HB model, integrating all three components, achieves substantial improvements over the backbone, reducing MER from 7.34% to 5.85% Ablation studies further show that dense top-2 expert activation is more effective than sparse alternatives in bilingual settings

Conclusion

In this work, we presented LLM-HB, a framework designed to enhance code-switching ASR through LLM-guided hotword biasing To the best of our knowledge, this is the first attempt to incorporate LLM-guided biasing into code-switching ASR The core components include a MoE adaptor that dynamically integrates speech and text representations, an auxiliary language head that provides language identity supervision, and a hotword prompt that introduces targeted biasing signals Experimental results on ASRU2019-CS demonstrate that the complete model achieves substantial improvements over the backbone, reducing MER from 7.34% to 5.85% Ablation studies further show that dense top-2 expert activation is more effective than sparse alternatives in bilingual settings, and that the bias loss weight regulates the trade-off between character-level and word-level accuracy These findings confirm the effectiveness of LLM-guided biasing for code-switching ASR and provide insights for future multilingual speech research. The paper concludes by stating that in future work, they plan to extend the approach to address near-homophones and to scale to larger hotword vocabularies.

Acknowledgments

This work was supported in part by China NSFC project under Grants No.

1. Bold Header: Language-Specialized Acoustic Representation Learning

The MoE adaptor is used to produce language-specialized speech embeddings ES, which are then fed into a LoRA-adapted LLM, enabling dynamic expert selection and aggregation to produce languagespecialized speech embeddings.

2. Bold Header: Explicit Language Supervision

An auxiliary language prediction head is attached to the LLM hidden states, and the language posterior is computed as Plt = Softmax(LayerNorm(FFN(et))) (4), which serves to provide explicit language supervision and stabilize expert specialization.

3. Bold Header: LLM-Guided Hotword Biasing

The model is optimized using the biasing loss defined as LBias = − X t∈H log p(ytX, H) (5), which encourages the model to increase the posterior probability of hotword tokens during decoding.