MoDiCoL: A Modular Diagnostic Continual Learning Dataset for Robust Speech Recognition

arXiv:2606.14459 · cs.CL, cs.AI, cs.SD · Submitted 2026-06-12 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "MoDiCoL: A Modular Diagnostic Continual Learning Dataset for Robust Speech Recognition".

Jane: The gist The MoDiCoL dataset introduces a Modular Diagnostic Continual Learning dataset designed for controlled analysis of linguistic content, speaker characteristics,

Tom: First, who's behind it and why it matters.

Title and authors: Jane: So what is this paper about, really? It introduces the MoDiCoL dataset for studying how models handle drift when they learn new things sequentially.

Tom: Yeah, it’s called "MoDiCoL: A Modular Diagnostic Continual Learning Dataset for Robust Speech Recognition," and the main idea is that standard benchmarks ignore how recording conditions, accents, and noise all happen at the same time in the real world.

Lu: They argue that model robustness isn't static; it should be treated as something that develops continually, and this dataset lets them study linguistic content, speaker characteristics, and acoustic environments in a controlled way.

Meng: So instead of testing one thing at a time, they are setting up a system to test how these different factors interact when the model has to learn new tasks in order.

Tom: Exactly; they build this dataset using an L27 orthogonal array following the Taguchi design method, which means they have one hundred eight distinct run configurations that cover all those combinations of factors.

Jane: That's a lot of variation—medical vocabulary mixed with elderly speakers in a noisy fan—and it gives researchers a very specific way to isolate exactly what causes performance drops.

Lu: They are using these runs to create this modular diagnostic set, so they can pinpoint whether the failure is coming from the linguistic side, the speaker side, or the acoustic environment.

The paper's summary: Jane: So what does this dataset actually contain and how do they use it in their experiments? It seems they are setting up a controlled sequence of learning tasks.

Tom: They define continual learning as a set of tasks, T, where the model learns them one after another, and D is the stream of data for each task with its own unknown distribution.

Lu: They start by defining a control setting called t0, which is adult speech, healthy speakers with no pauses or disfluencies in clean recording conditions.

Jane: Then they define a curriculum that introduces four types of drift: acoustic environment drift where you add babble and fan noises and varied signal-to-noise ratios.

Meng: The next one is speaker characteristics drift, where you introduce new groups like children or impaired speakers with different accents while keeping the acoustic conditions moderate.

Tom: Then there is linguistic content drift, which exposes the model to new domains like medical speech or different speaking styles such as spontaneous and conversational speech.

Lu: And they have compound drift, which is when you combine multiple types of drift together, like an elderly speaker far away in a noisy environment with specialized vocabulary.

Jane: These four drifts are the core components of their continual learning curriculum, and they allow them to test how the model handles these changes sequentially.

The paper's improvements: Tom: So what are the actual methods they propose for dealing with these sequential drifts? They look at three different continual learning strategies.

Lu: They evaluate experience replay, which maintains a memory buffer with a fraction of data from previous tasks, and representation-level regularization, which constrains the encoder to preserve old representations by minimizing cosine similarity loss.

Meng: Then there’s orthogonal gradient descent, which tries to stop forgetting by making sure the parameter updates go in directions that don't interfere with what the model already learned.

Jane: They measure how well these methods work using A-WER and AIWER for performance, plus memory stability metrics like Forgetting Measure and Backward Transfer, and learning plasticity things like Forward Transfer.

Tom: The key finding here is that the unadapted model does best on the control setting t0, but performance drops significantly when you introduce acoustic drift t1 or speaker characteristics drift t2.

Lu: But they also found that linguistic content drift t3 causes a dramatic increase in word error rate, showing it’s a really sensitive area for performance.

Meng: Interestingly, compound drift t4 didn't produce the worst performance result, which suggests that these different types of drift don't just pile up in difficulty when they are applied together.

Conclusion: Jane: So to wrap this up, what is the main point about using MoDiCoL for testing speech recognition systems?

Tom: The main point is that MoDiCoL acts as a diagnostic tool; it lets you control every variable—linguistic content, speaker characteristics, and acoustics—to see exactly which specific drift type is causing performance issues.

Lu: And when we look at the continual learning strategies they tested, experience replay with a replay buffer size of ten percent was the most balanced approach for preserving robustness across all those different drifts.

Meng: That finding is important because it suggests that keeping a moderate amount of memory doesn't hurt performance on new tasks; it actually helps stabilize training, especially when you get to compound drift.

Jane: And the orthogonal gradient descent method showed that constraining the parameter updates to directions that don't interfere with old tasks really helped mitigate forgetting in a different way.

Tom: So, this work shows continual learning isn't just about making a model robust; it’s actually becoming a diagnostic tool for finding out exactly how the model is forgetting things.

Lu: Because they measured metrics like forgetting measure FM and backward transfer BWT, which tells you what kind of representational drift you’re dealing with.

Jane: So, by using this MoDiCoL dataset and these continual learning strategies, we can move past just seeing that a model performs well on its original training data.

Tom: We can actively probe the mechanisms of forgetting in pretrained speech models under realistic distribution shifts.

Meng: It’s a good way to know what breaks at scale, because you’re testing it against things like babble and fan noise, not just clean audio recordings.

Lu: The paper makes the MoDiCoL dataset and the continual learning curriculum available on Hugging Face for other researchers to use.

Jane: That accessibility is key; it lets us test these ideas without having to build all that complex data pipeline from scratch.

Tom: So, the full picture here is that MoDiCoL gives us a rigorous way to study robustness in Automatic Speech Recognition systems by letting us control every variable.

Meng: It’s a solid setup for figuring out how we can deploy models that handle real-world audio variation without them forgetting what they learned.

Lu: Moving forward, this framework opens up the door to studying how different drift types interact when training models sequentially.

Jane: That interaction is going to be a big area of research, so keep an eye on how these continual learning strategies evolve as people try to solve those complex drift scenarios.

University of Hamburg

cs.CL, cs.AI, cs.SD

Submitted: 2026-06-12

Updated: 2026-06-12

Comments: Accepted at Interspeech 2026

DOI: 10.21437/Interspeech.2026-2111

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 76/100

The gist: The gist The MoDiCoL dataset introduces a Modular Diagnostic Continual Learning dataset designed for controlled analysis of linguistic content, speaker characteristics, and acoustic environments to

Key concepts

MoDiCoL Dataset: A speech dataset built using a factorial design to rigorously control linguistic content (vocabulary/style), speaker traits (age, accent, impairment), and acoustic conditions (noise type, SNR). It consists of 8100 samples designed for controlled analysis of how different real-world factors affect ASR performance.
Continual Learning Curriculum: A structured training sequence that incrementally introduces four types of drift: acoustic noise changes, new speaker groups, linguistic styles, and compound drifts. This simulates the continuous evolution of a model's operating conditions in a real-world scenario.
Experience Replay (ER): A continual learning strategy that maintains a memory buffer of past data samples. During training on new data, it interleaves these old samples with the current batch, helping the model retain knowledge from previous tasks and stabilize performance under drift.

Terminology

Continual Learning — 4× in this paper · explained in 2 episodes
Robustness — 3× in this paper · explained in 5 episodes
Benchmark — 2× in this paper · explained in 2 episodes
Automatic Speech Recognition (ASR) — 1× in this paper · explained in 2 episodes
Distribution Shift — 1× in this paper · explained in 9 episodes
Distribution Shifts — 1× in this paper · explained in 2 episodes

Summary

The gist The MoDiCoL dataset introduces a Modular Diagnostic Continual Learning dataset designed for controlled analysis of linguistic content, speaker characteristics, and acoustic environments to study model robustness in Automatic Speech Recognition systems.

MoDiCoL Dataset Construction

MoDiCoL is a speech dataset constructed using a systematic factorial design that enables rigorous evaluation of linguistic, speaker, and acoustic variation with clearly defined experimental runs <ref:2606.14459#pg3>. The dataset covers three primary factor categories: Linguistic Content, Speaker Characteristics, and Acoustic Environment <ref:2606.14459#pg4>. Linguistic content includes vocabulary domain (Medical, Air Traffic Control, Other) and speech style (Read, Spontaneous, Conversational) <ref:2606.14459#pg4>. Speaker characteristics describe demographic and speech-production factors such as age group (Child, Adult, Elderly), accent (English, South-East Asian, Other), health (Healthy or Impaired), pauses (Yes/No), and disfluencies (Yes/No) <ref:2606.14459#pg4>. Acoustic environment encompasses noise type (Clean, Babble, Fan) and signal-to-noise ratio (SNR) level (Clean, 10dB, or 20dB), as well as the distance between the speaker and the microphone (Close/Far) <ref:2606.14459#pg4>. The structure utilizes an L27 orthogonal array following the Taguchi design method [17] and two foldover dimensions F0 and F1 to accommodate six three-dimensional factors and four two-dimensional factors, resulting in 108 distinct run configurations <ref:2606.14459#pg3>. The total dataset comprises 8100 samples, with each run filled by 75 samples, a standard heuristic for mixed-level fractional factorial designs <ref:2606.14459#pg3>.

Data Compilation and Augmentation

The dataset compilation involves populating runs with open-source speech datasets and generating synthetic speech using text datasets for missing combinations <ref:2606.14459#pg4>. Healthy children’s speech is sourced from Non-native Children English Speech (NNCES) [19], Children Speech Recordings [20], and Ultrax Typically Developing Children (UXTD) [21, 22] <ref:2606.14459#pg4>. Impaired children’s speech is sourced from the Cleft dataset [10] <ref:2606.14459#pg4>. Healthy adult and elderly speech is taken from Svarah [8], Trustworthy Intent in Speech (TIS) [23], TEDLIUMv2 [24], and Common Voice [25] <ref:2606.14459#pg4>. ATC data is obtained from the ATCO-2 1-hour test split [27] and the UWB-ATCC dataset [28] <ref:2606.14459#pg4>. Medical speech is collected from United-Syn-Med [29] and Eka Medical ASR Evaluation Dataset [30], supplemented with texts from MedRAG, MedDialog, NoteChat, and simulated medical exams <ref:2606.14459#pg4>. Synthetic speech is generated using XTTS-v2 [36], a multilingual zero-shot text-to-speech model that uses reference audio to match speaker characteristics <ref:2606.14459#pg4>. The augmentation pipeline involves several steps, including Denoising using a DNN-HMM hybrid system [39], Disfluency Insertion by synthesizing filler words with XTTSv2, Impairment Simulation via prosodic and spectral modifications like jitter and shimmer, Pause Insertion/Removal by detecting silent segments or randomly selecting insertion points, Distance Simulation through adding reverberation to model a larger room, and Noise Injection based on the predefined SNR level <ref:2606.14459#pg4>.

Continual Learning Curriculum

The continual learning curriculum is defined by sampling from LibriSpeech [41] to establish a control setting t0 against which drift types are compared <ref:2606.14459#pg4>. The curriculum introduces four distinct drift types: t1 Acoustic Environment Drift, involving babble and fan noises, varied SNR levels, and microphone distances <ref:2606.14459#pg4>. t2 Speaker Characteristics Drift, which includes new speaker groups like children, the elderly, and impaired speakers with different accents while keeping acoustic conditions moderate <ref:2606.14459#pg4>. t3 Linguistic Content Drift, exposing the model to new domains and speaking styles such as spontaneous and conversational speech, and specialized vocabulary <ref:2606.14459#pg4>. t4 Compound Drift, which is a combination of multiple drift types, such as elderly or impaired speakers at a far distance in noisy environments with conversational or domain-specific speech <ref:2606.14459#pg4>. Two additional permutations of the curriculum are generated for evaluation while keeping task t4 fixed <ref:2606.14459#pg4>.

Continual Learning Strategies and Evaluation

Three continual learning (CL) methods are employed to analyze sequential drift: Experience Replay (ER) [42], Representation-level Regularization (RLR) [43], and Orthogonal Gradient Descent (OGD) [44] <ref:2606.14459#pg4>. Experience Replay maintains a memory buffer containing a fraction of data from previous tasks, interleaving buffered samples with current training data <ref:2606.14459#pg4>. Representation-level Regularization introduces a regularization term that constrains the encoder to preserve previously learned representations by minimizing the cosine similarity loss between mean-pooled encoder outputs <ref:2606.14459#pg4>. Orthogonal Gradient Descent mitigates forgetting by constraining parameter updates to directions that do not interfere with prior tasks, projecting gradients onto orthogonal subspaces <ref:2606.14459#pg4>. Performance is measured using Average Word Error Rate (A-WER) and Average Incremental WER (AIWER), memory stability via Forgetting Measure (FM) and Backward Transfer (BWT), and learning plasticity using Forward Transfer (FWT) and Intransigence Measure (IM) <ref:2606.14459#pg4>. Evaluations are conducted in an online and streaming CL setting where the batch size is limited to one, with training split into train (70%), validation (20%), and test (10%) sets at a learning rate of 1e − 5 <ref:2606.14459#pg4>.

Key Findings on Robustness

The unadapted model performs best on the control setting t0, but performance deteriorates substantially under distribution shifts, with acoustic drift (t1) degrading moderately and speaker (t2) and linguistic drift (t3) causing dramatic increases in WER <ref:2606.14459#pg5>. Compound drift (t4), despite combining all previous factors, does not produce the worst performance, suggesting that drift effects do not accumulate in difficulty when applied together <ref:2606.14459#pg4>. The results show that ER with a replay buffer size of 10% is the most balanced approach, outperforming other CL methods and even the joint training baseline <ref:2606.14459#pg5>. A-WER for ER-10% is 17.31±0.48, indicating strong retention of robustness across drift types <ref:2606.14459#pg4>. OGD outperformed RLR across most metrics and achieved the best AI-WER score (21.19 ± 1.18) overall, suggesting that interference within gradient subspaces is a factor contributing to performance loss <ref:2606.14459#pg5>. The sequential introduction of drifts improved the model’s learning plasticity, and robustness across drift types benefits most from moderately sized memory and replay mechanisms <ref:2606.14459#pg4>.

Conclusion

Continual learning can serve as a diagnostic tool for investigating forgetting mechanisms in pretrained models by preserving model robustness to distributional drift <ref:2606.14459#pg4>. The study demonstrates that ER with a buffer of 10% stabilizes training and outperforms the joint training baseline, while OGD suggests that interference within gradient subspaces is a factor contributing to performance loss <ref:2606.14459#pg4>. Overall, the findings show that CL can not only preserve model robustness to distributional drift but also serve as a diagnostic tool for investigating forgetting mechanisms in pretrained models <ref:2606.14459#pg4>. The authors make MoDiCoL, its augmentation pipeline, and the CL curriculum available on Hugging Face <ref:2606.14459#pg4>.

Improvements for AI systems

Bold header: Robustness Diagnostics via MoDiCoL Dataset

The MoDiCoL dataset allows for a controlled evaluation of different factors that influence ASR performance: linguistic content, speaker characteristics, and acoustic environment. This enables the development of diagnostic tools that can pinpoint exactly which drift factor—linguistic, speaker-related, or acoustic—is causing a specific degradation in model performance.

Bold header: Real-World Inspired Continual Learning Curriculum

The proposed curriculum allows for simulating incremental updates and study how robustness is acquired, transferred, and forgotten across sequential distribution shifts like acoustic drift (t1), speaker characteristics drift (t2), or linguistic content drift (t3). This facilitates the creation of models that are not only robust but also demonstrate learning plasticity by showing how exposure to previous drifts generalizes to others.

Bold header: Strategy-Specific Robustness Acquisition Analysis

By applying three distinct CL methods—Experience Replay (ER), Representation-level Regularization (RLR), and Orthogonal Gradient Descent (OGD)—researchers can compare how different mechanisms impact forgetting. For instance, OGD is shown to mitigate forgetting by constraining parameter updates to directions that do not interfere with prior tasks, suggesting a method for building models where knowledge from previous conditions is preserved through constrained gradient subspaces.

Bold header: Targeted Performance Stabilization via Replay Buffer

The finding that ER with a buffer size of 10% of previous task data stabilized training on new tasks and even outperformed the joint training baseline suggests that maintaining a moderate memory buffer is crucial for preserving robustness across drifts, leading to A-WER improvements in complex scenarios like compound drift (t4).

Bold header: Diagnostic Tool for Model Forgetting

The framework can serve as a diagnostic tool for investigating forgetting mechanisms in pretrained models by analyzing metrics like Forgetting Measure (FM) and Backward Transfer (BWT). This allows researchers to determine whether performance loss is due to representational drift (as suggested by RLR's high forgetting scores) or interference within gradient subspaces (as suggested by OGD's success).

Abstract

Modern Automatic Speech Recognition (ASR) systems have made remarkable progress on standard benchmarks, yet performance gaps have emerged under real-world distribution shifts, caused by recording conditions, accents, speech impairments, and noise. Existing datasets and benchmarks typically isolate these factors, which overlooks their co-occurrence in real-world applications. In this paper, we argue that model robustness can be treated as a dynamic capability that continually develops, and we introduce MoDiCoL, a Modular Diagnostic Continual Learning dataset designed for controlled analysis of linguistic content, speaker characteristics, and acoustic environments. Furthermore, we propose a real-world-inspired continual learning curriculum to simulate incremental updates and study how robustness is acquired, transferred, and forgotten. We evaluate three continual learning strategies and provide detailed insights into robustness under evolving conditions.