Learning to Hear Hesitation: Continual Learning for Disfluency-Aware ASR

arXiv:2606.14391 · cs.CL, cs.AI, cs.SD · Submitted 2026-06-12 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Learning to Hear Hesitation".

Jane: The gist This work addresses disfluent speech recognition by leveraging continual learning (CL) with explicit disfluency tokens to mitigate catastrophic forgetting and improve performance across varying disfluency distributions.

Tom: First, who's behind it and why it matters.

Title and authors: Jane: We've talked about how this paper, "Learning to Hear Hesitation: Continual Learning for Disfluency-Aware ASR," addresses the issue of information loss when ASR models are optimized to omit disfluencies by introducing explicit tokens and using continual learning to manage forgetting.

Tom: Right, and we saw that this is done across three distinct speech datasets—SME, Pitt, and Delaware—and they tested four different continual learning strategies like EWC, ER, A-GEM, and WA against them.

Lu: It’s interesting that they are using the SME Corpus from Malaysian university students to test how these models handle a specific type of language variation and disfluency distribution. That’s a very specific starting point for their continual learning setup.

Meng: So, the initial question we need to ask is what exactly is it like when a model tries to learn both general speech recognition and these new disfluency tokens simultaneously?

Jane: It's like training a student who has to learn standard English grammar while also trying to memorize every single way that someone might pause or repeat themselves. The paper seems interested in the mechanics of how that happens internally.

Tom: They looked into internal specialization using head-masking, and they found that when markers are successfully trained, a few specialized decoder cross-attention heads consistently handle the marker emission across all methods.

Lu: That suggests it’s not just one method being inherently superior; there's a shared cross-attention mechanism in the architecture that emerges once those specialized circuits are established.

Jane: So, if we look at this from a user's perspective, what does knowing there’s a shared attention pattern mean for the system I use every day?

Tom: It means that whatever learning strategy you pick will eventually lead to a similar internal structure for handling those tokens, which is good because it makes the resulting model more predictable.

Meng: But from my side, if we are deploying this at scale, we need to know if that shared mechanism is computationally efficient or if it introduces bottlenecks when processing long audio streams.

Jane: That’s a fair concern. We need to ensure that finding these specialized circuits doesn't just add complexity in terms of real-time processing costs for our production models.

Lu: The paper does explore the internal model dynamics during training, which gives us a good picture of how the model evolves from a general ASR system into one that can handle those specific speech events.

Tom: It’s like watching the model learn to recognize patterns in an audio stream; it starts by hearing words, then gradually starts paying attention to the extra stuff like fillers and disruptions.

The paper's summary: Jane: So, summarizing what we just covered, this paper is about using continual learning with explicit tokens to improve ASR for disfluent speech by showing that models can adapt across different datasets while keeping their general performance high.

Tom: It’s not just about the training; it's about the results across those three datasets—SME, Pitt, and Delaware—and how different CL methods handle these varying distributions of disfluency.

Lu: The key summary is that they are trying to bridge the gap between verbatim transcription and general-domain adaptation by using these tokens as a consistent signal during training.

Meng: So, what's the big picture takeaway for us in the AI industry? Are we just adding more labels, or is this fundamentally changing how ASR models are trained?

Jane: It suggests it’s more than just adding labels; it’s about creating a structured way to teach the model to value those extra pieces of information consistently across different speech contexts.

Tom: The paper shows that the introduction of these tokens into a pretrained backbone model creates stable token mechanisms, which is a necessary first step before any complex continual learning can happen.

Lu: And then they test how methods like A-GEM, ER, and WA perform on adapting to those different data distributions while preserving the knowledge from the original training.

Jane: It’s about showing that you can create a system that learns incrementally without forgetting its old skills as it learns new ones.

Tom: The paper demonstrates this capability by using these tokens across all three datasets, and it shows how different learning methods manage the distribution of those tokens differently during adaptation.

The paper's improvements: Jane: Now let’s talk about what the authors suggest is better than what they found, moving beyond just their initial findings in "Learning to Hear Hesitation: Continual Learning for Disfluency-Aware ASR."

Tom: They highlight that Weight Averaging achieved the strongest pWER reduction while also showing the most stable pWER performance, which they showed with their Average Incremental Word Error Rate.

Lu: That’s a strong finding because it means WA is really excellent at preserving general knowledge while adapting to new data, which is what we want for robust models.

Meng: For deployment, that stability is crucial; I’d argue that if a model keeps its base accuracy steady across different user populations, that’s more valuable than a massive jump in accuracy on one specific test set.

Jane: So it gives us a clear choice: prioritize the stability offered by WA for general robustness or look at other methods for when you need to push performance on a specific task.

Tom: On the other hand, they also found that Experience Replay showed the best marker performance with a macro-F1 score of zero point four nine during sequential adaptation, even though its pWER wasn't the absolute lowest overall <ref:2606.14391#pg1>.

Jane: So it’s not a clear winner; it really depends on whether you care more about getting high accuracy or getting high marker prediction scores.

Lu: And they also showed that ER achieved the best performance for disfluency-marker F1, showing strong generalization for markers with nearly no forgetting, as indicated by an FM score of zero point zero two during those sequential adaptation steps <ref:2606.14391#pg1>.

Meng: That suggests that if our primary goal is accurate tagging of speech events, we might want to lean towards ER because it seems to generalize the marker learning well without catastrophic forgetting on the general ASR part.

Tom: So, in short, WA is for stable baseline performance across all domains, and ER seems better when you need strong marker generalization across those challenging distributions.

Jane: The paper also points out that they found a consistent trade-off where methods that maintained the strongest pWER performance tended to under-introduce markers while those that reliably emitted markers did so at a measurable ASR cost.

Conclusion: Tom: So, wrapping up, the main thing is that CL improved sequential training for both ASR and disfluency-marker retention using explicit tokens in the paper "Learning to Hear Hesitation: Continual Learning for Disfluency-Aware ASR."

Jane: The best method really depended on your primary objective—WA showed the strongest pWER reduction while ER showed the best marker generalization and retention.

Lu: This work offers one of the first studies of continual learning specifically for modeling disfluency, giving us practical guidance for verbatim ASR adaptation and mechanistic insights into how these token behaviors are acquired in pretrained models.

Meng: For me, it’s about understanding the performance trade-offs so we can deploy models that match our specific needs to avoid wasting resources on a general solution that doesn't fit our specific requirement.

Tom: I think the consistent cross-attention pattern is a really important mechanistic finding; it explains the difference between methods like ER and WA in terms of how they handle parameter updates during adaptation.

Jane: So, we’ve seen how to use explicit tokens to make ASR models more robust against forgetting while still capturing valuable speech information.

Lu: This paper provides a solid foundation for moving forward in this area, especially regarding the structural understanding of these learned circuits.

Meng: We need to keep pushing on figuring out how to make that marker learning more efficient so it doesn't become a bottleneck at scale.

University of Hamburg

cs.CL, cs.AI, cs.SD

Submitted: 2026-06-12

Updated: 2026-06-12

Comments: Accepted at Interspeech 2026

DOI: 10.21437/Interspeech.2026-2080

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 75/100

The gist: The gist This work addresses disfluent speech recognition by leveraging continual learning (CL) with explicit disfluency tokens to mitigate catastrophic forgetting and improve performance across

Key concepts

Continual Learning (CL): CL is a technique where a model learns new tasks sequentially without forgetting what it learned from previous tasks. In this paper, it was used to adapt an ASR model to recognize different types of speech disfluencies across various datasets while preserving its core language understanding.
Disfluency Tokens: These are specific markers (like 'um', 'uh', or coughs) explicitly added as tokens into the ASR model's input. This allows the system to learn how to recognize and handle these non-speech elements, which is crucial for verbatim transcription and understanding spoken language nuances.
Weight Averaging (WA): WA is a continual learning method that keeps old model weights in memory while training a new version on new data. After training, it updates the weights by averaging the old and new versions. The study found WA was effective at reducing overall speech error rates (pWER) while maintaining stability.
Cross-Attention Heads: These are specific components within the ASR model that focus on different parts of the input data during processing. The research discovered that when markers are successfully learned, a small, consistent set of these heads specializes in emitting disfluency tokens, suggesting a shared mechanism for handling markers.

Terminology

Continual Learning — 2× in this paper · explained in 3 episodes
Automatic Speech Recognition — 1× in this paper
Automatic Speech Recognition (ASR) — 1× in this paper · explained in 2 episodes
Catastrophic Forgetting — 1× in this paper · explained in 6 episodes
Hallucination — 1× in this paper · explained in 4 episodes

Summary

The gist

This work addresses disfluent speech recognition by leveraging continual learning (CL) with explicit disfluency tokens to mitigate catastrophic forgetting and improve performance across varying disfluency distributions.

Datasets and Setup

The study utilizes three datasets from the TalkBank repository, including the Standard Malaysian English (SME) Corpus, the Pitt Corpus, and the Delaware Corpus, which were selected based on data availability, number of samples, disfluency distribution, and domain <ref:2606.14391#pg3>. All three use the unified CHAT Transcription Format <ref:2606.14391#pg3>. The SME Corpus contains 11.79 h of English speech from Malaysian university students who are second-language learners, representing a common subgroup of speakers who are generally healthy but may exhibit disfluencies in their speech production due to the language gap <ref:2606.14391#pg3>. The Pitt Corpus contains 21.30 h of speech from elderly speakers with dementia or Alzheimer’s disease, while the Delaware Corpus contains 9.72 h of speech from people with mild cognitive impairment <ref:2606.14391#pg3>. In the CL setup, a small subset of LibriSpeech data is included for the rehearsal buffer in replay-based methods and as a representation of unperturbed speech <ref:2606.14391#pg3>.

Continual Learning Methods

Four continual learning methods are employed to preserve the pretrained performance of whisper-small.en1 and handle different disfluency distributions: Elastic Weight Consolidation (EWC) <ref:2606.14391#pg3>, Experience Replay (ER) <ref:2606.14391#pg3>, A-GEM <ref:2606.14391#pg3>, and Weight Averaging (WA) <ref:2606.14391#pg3>. EWC is a regularization-based method that identifies model parameters important to previously learned tasks and penalizes updates to those parameters using the diagonal Fisher information matrix <ref:2606.14391#pg3>. Experience Replay (ER) keeps a small percentage of old data in memory (rehearsal buffer) and uses it to supplement the training data for the new task, preventing drastic gradient updates <ref:2606.14391#pg3>. A-GEM is a gradient-based method that calculates the dot product between the gradients of the rehearsal buffer and the training data of the current dataset to penalize gradients in opposing directions <ref:2606.14391#pg3>. Weight Averaging (WA) keeps weights of old models in memory while another version is trained on the new task, and once training is complete, it updates with an average of the old and the new weights <ref:2606.14391#pg3>.

Evaluation Metrics

General ASR performance is evaluated using preprocessed WER (pWER), where punctuation, special characters, and disfluency markers are removed from transcripts before the metric is computed <ref:2606.14391#pg3>. To assess memory stability and learning plasticity during CL, metrics defined by Wang et al. are used with minor modifications to accommodate pWER and marker F1 in place of accuracy <ref:2606.14391#pg3>. Average Word Error Rate (A-WER) and Average Incremental WER (AI-WER) measure overall pWER performance at the current step and across the full CL trajectory, respectively <ref:2606.14391#pg3>. For marker prediction, Average F1 (A-F1) and Average Incremental F1 (AI-F1) replace per-task accuracy with per-task marker F1, calculated using a bag-of-words approach <ref:2606.14391#pg3>. Backward Transfer (BWT) and Forgetting Measure (FM) assess memory stability, whereas Forward Transfer (FWT) and Intransigence Measure (IM) assess the model’s learning plasticity <ref:2606.14391#pg3>.

Disfluency Token Introduction

The first experiment involves introducing four distinct token types into a pretrained model: filler words (FILLER; e.g., “um”, “uh”), repetitions/revisions (REP; word- and phoneme-level), disruptions (DISRUPT; e.g., coughs, laughs), and pauses (PAUSE) <ref:2606.14391#pg3>. In the SME dataset with disfluency tokens, Weight Averaging (WA) outperformed the other methods by retaining and even slightly improving the ASR model performance on LS while achieving the lowest pWER on SME at 9.64% <ref:2606.14391#pg4>. However, both methods failed to integrate markers reliably, as ER and A-GEM consistently produced marker micro-F1 scores between 0.73 and 0.75 <ref:2606.14391#pg4>. This suggests a trade-off between pWER and marker performance, indicating that marker training may require parameter updates that conflict with low-cost general ASR domain adaptation <ref:2606.14391#pg4>.

Sequential Continual Adaptation

The sequential continual adaptation experiment involves fine-tuning the best model from the token introduction stage on the Pitt and Delaware corpora, which vary in domain and disfluency distribution <ref:2606.14391#pg3>. The results show that all CL methods reduce pWER relative to the FT baseline, as reflected in lower A-WERs across all methods <ref:2606.14391#pg5>. Weight Averaging (WA) achieves an A-WER of 18.90%, while ER shows the best marker performance with a macro-F1 score of 0.49 <ref:2606.14391#pg5>. WA shows the strongest pWER reduction while exhibiting the most stable pWER performance, indicated by AI-WER <ref:2606.14391#pg5>. ER achieves the best performance for disfluency-marker F1, showing strong generalization for markers with nearly no forgetting as indicated by an FM score of 0.02 <ref:2606.14391#pg5>.

Mechanistic Insights

The investigation into internal specialization mechanisms using head-masking shows that when markers are successfully trained, a few specialized decoder cross-attention heads are consistently assigned to marker emission, suggesting a shared cross-attention mechanism rather than method-specific strategies <ref:2606.14391#pg5>. Targeted ablations confirmed this by showing that removing around 50% of the emitted FILLER markers could be done while retaining similar pWER values <ref:2606.14391#pg5>. The paper concludes that successful marker learning is linked to a consistent cross-attention pattern, providing a mechanistic explanation for method differences in marker introduction <ref:2606.14391#pg5>.

Conclusion

Overall, CL improved sequential training performance for both ASR and disfluency-marker retention <ref:2606.14391#pg5>. The best CL method depended on the primary objective, as WA showed the strongest pWER reduction while ER showed the best marker generalization and retention <ref:2606.14391#pg5>. This work presents one of the first studies of CL for disfluency modeling, offering practical guidance for verbatim ASR adaptation and mechanistic insights into how disfluency token behaviors are acquired in pretrained models <ref:2606.14391#pg5>. The findings indicate a consistent tradeoff between pWER and marker performance across CL settings <ref:2606.14391#pg5>.

Acknowledgements

The authors gratefully acknowledge funding from Horizon Europe under the MSCA grant agreements No 101072488 (TRAIL), No 101226624 (GREET), and No 101168792 (SWEET), as well as from the German Research Foundation (DFG), project number 551629603. Data collection for the Delaware Corpus was supported by the National Institute of Aging of the National Institutes of Health under award number RF1AG083823 (PIs: MacWhinney and Lanzi). The Pitt Corpus is part of DementiaBank, which is funded by NIH grants NIA AG03705 and AG05133. The paper also includes a section on Generative AI Use Disclosure.

References

The references listed in the paper are [1] through [27] <ref:2606.

Improvements for AI systems

The improved system can perform verbatim transcription of disfluent speech by integrating explicit disfluency tokens into a pretrained ASR model, as described by We first introduce these tokens into a pretrained ASR model to establish stable token mechanisms. This allows the model to capture clinically relevant speech information that is often lost when state-of-the-art systems are optimized to omit disfluencies, leading to information loss and hallucinations.
The system can maintain robust performance across evolving domains and speaker variations through continual learning (CL) strategies like Weight Averaging (WA). This is evidenced by WA exhibiting the strongest pWER reduction while also exhibiting the most stable pWER performance, as indicated by AI-WER, suggesting superior memory stability for preserving general ASR knowledge.
The system can identify and isolate specialized circuits within the model that handle disfluency events through attention head analysis. By using the head-masking approach of Michel et al., the system can determine whether disfluency handling concentrates in a small set of heads (specialized circuitry) and verify this mechanism by showing that we can consistently remove around 50% of the emitted FILLER markers while retaining similar pWER values.
The system can dynamically adapt to new, unseen disfluency distributions by employing sequential continual adaptation across heterogeneous datasets. This capability allows the model to approximate lifelong model adaptation to changing domains and speaker variations through sequential training on more difficult datasets, specifically demonstrating that ER [Experience Replay] achieves the best marker generalization and retention.
The system can provide a mechanistic explanation for performance trade-offs between ASR quality and marker learning. The results show a consistent tradeoff where methods that maintained the strongest pWER performance tended to under-introduce markers, while those that reliably emitted markers did so at a measurable ASR cost, linking successful marker learning to a consistent cross-attention pattern.

Abstract

Despite advances in large-scale Automatic Speech Recognition (ASR), disfluent speech remains challenging, as state-of-the-art systems are often optimized to omit disfluencies, leading to information loss and hallucinations. Prior work has focused on verbatim transcription and the integration of disfluency markers, but adapting models on limited datasets can lead to catastrophic forgetting of general-domain knowledge. We address this gap by leveraging continual learning (CL) with explicit disfluency tokens. We first introduce these tokens into a pretrained ASR model to establish stable token mechanisms, and then continue training on additional datasets with varying disfluency distributions. Through a detailed analysis of model dynamics during training, we identify a trade-off between marker learning and ASR performance, and a consistent cross-attention head mechanism shared across CL methods.