Exploring the potential and limitations of Model Merging for Multi-Domain Adaptation in ASR

summary

The gist

Model merging is presented as a scalable alternative to multi-task training that combines specialized models into a single model…

In short

The study explored model merging as a scalable alternative to multi-task training for large speech models adapted to different domains. Eleven merging algorithms were tested across ten European Portuguese domains, showing that BoostedTSV-M achieved the best accuracy while maintaining good generalization across unseen languages and improving out-of-distribution performance compared to full fine-tuning.

Key concepts

Model Merging
Combining specialized models into a single model instead of training separate models for each task. This is proposed as an efficient way to adapt large speech foundation models to many different domains, offering scalability over traditional multi-task training methods.
Parameter-Space Methods (PS)
Merging techniques that operate directly on the full set of model parameters as a single vector. Examples include Model Soups and Karcher mean, which adjust the entire model structure based on pre-trained and fine-tuned versions.
BoostedTSV-M
A new merging algorithm that improves upon TSV-M by using singular-value boosting. This technique prevents 'rank collapse' by clamping small singular values to a lower bound, ensuring task-specific information is not suppressed during the merging process.

Terminology used across episodes

This episode discusses

The paper

Exploring the potential and limitations of Model Merging for Multi-Domain Adaptation in ASR · Read on arXiv

Carlos Carvalho, Francisco Teixeira, Thomas Rolland, Alberto Abad

Instituto Superior Tecnico, Universidade de Lisboa, Portugal

Model merging is a scalable alternative to multi-task training that combines the capabilities of multiple specialised models into a single model. This is particularly attractive for large speech foundation models, which are typically adapted through domain-specific fine-tuning, resulting in multiple customised checkpoints, for which repeating full fine-tuning when new data becomes available is computationally prohibitive. In this work, we study model merging for multi-domain ASR and benchmark 11 merging algorithms for 10 European Portuguese domains, evaluating in-domain accuracy, robustness under distribution shift, as well as English and multilingual performance. We further propose BoostedTSV-M, a new merging algorithm based on TSV-M that mitigates rank collapse via singular-value boosting and improves numerical stability. Overall, our approach outperforms full fine-tuning on European Portuguese while preserving out-of-distribution generalisation in a single model.

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Exploring the potential and limitations of Model Merging for Multi-Domain Adaptation in ASR".

Jane: Here's the gist of the paper. Model merging is presented as a scalable alternative to multi-task training that combines specialized models into a single model, which is particularly attractive for large speech foundation models adapted through domain-specific fine-tuning This work studies model merging for multi-domain ASR and benchmarks eleven merging algorithms across ten European Portuguese domains, evaluating in-domain accuracy, robustness under distribution shift, as well as English and multilingual performance The authors propose BoostedTSV-M, a new merging algorithm that mitigates rank collapse via singular-value boosting and improves numerical stability Overall, the approach outperforms full fine-tuning on European Portuguese while preserving out-of-distribution generalisation in a single model.

Tom: Model Merging Paradigms Model merging is categorized into three broad parameter space paradigms: parameter-space (PS), τ-space merging (τSpa), and τ-subspace merging (τSub) Parameter-space methods operate directly on the full model parameters θt as a flattened vector, including approaches such as Model Soups (SOUP)

29: , Model Stock

30: , Karcher mean

31: , and Multi-Spherical Linear Interpolation (SLERP)

24: τ-space merging methods operate over the taskvector, defined as the difference between fine-tuned and pre-trained parameters, including Task Arithmetic (TA)

32: , TrIm, Elect, and Sign (TIES)

33: , Parameter Competition Balancing (PCB)

34: , and Select, Calculate and Erase (SCE)

35: Finally, τ-subspace methods operate over τ matrices in their original form to perform merging in low-rank subspaces, including Task Singular Vectors Merging (TSV-M)

27: , Isotropic Merging in Common Subspace (ISO-C), and Isotropic Merging in Common and Task-Specific Subspaces (ISO-CTS)

36: .

Jane: BoostedTSV-M Mechanism The BoostedTSV-M algorithm enhances the TSV-M method by incorporating a singular-value boosting scheme inspired by prior work in CV

28: This enhancement addresses rank collapse, which occurs when singular values decay sharply after truncation and concatenation, leading to the suppression of task-specific signals The scheme defines a function ci,t(s) as the fraction of total energy explained by the first s singular values for each task vector matrix Wi It then finds the smallest index s∗ such that the cumulative energy ci,t(s) exceeds a threshold β ∈

0, 1: The singular values below σ(s∗)i,t are then boosted by clamping them to that value, effectively setting a lower bound on the singular values and preventing the suppression of task-specific information carried by smaller singular values.

Tom: Performance Trade-offs The experimental results reveal a clear trade-off between target-domain specialisation and cross-lingual robustness when comparing merging to full fine-tuning Full fine-tuning substantially improves EP performance over the zero-shot model, reducing WER from 15.62% to 8.54% on the ID test sets and from 25.21% to 17.65% on EP OOD However, Full-FT worsens performance on all non-EP OOD data except AAP, indicating that specialising the model to EP can harm multilingual robustness and potentially cause catastrophic forgetting Merging approaches generally preserve – and in several cases improve – OOD performance, while narrowing the EP performance gap relative to joint fine-tuning Superiority of BoostedTSV-M The proposed BoostedTSV-M achieves the best EP Full Avg.

Paper summary: Tom: So we're looking at this paper now called Exploring the potential and limitations of Model Merging for Multi-Domain Adaptation in ASR. Basically, they’re talking about merging specialized models into one single model instead of training many separate ones for every domain.

Jane: That sounds like a way to save a ton of computational effort when you have a bunch of different speech domains you want to support, right? What's the main idea here, though?

Tom: The thesis is that model merging is this scalable alternative to multi-task training where you combine the capabilities of multiple specialized models into one single model.

Lu: It’s particularly attractive for large speech foundation models because these big models usually get adapted through domain-specific fine-tuning, which results in multiple customized checkpoints, and repeating full fine-tuning when new data comes available is just computationally prohibitive <ref:2603.05354#pg1>.

Meng: So it’s about avoiding that massive retraining cost for every single new domain adaptation. But what are they actually testing?

Tom: They study model merging for multi-domain ASR and benchmark eleven different merging algorithms across ten European Portuguese domains.

Jane: And they aren't just looking at accuracy on those domains, either; they are evaluating robustness under distribution shift, and English and multilingual performance too.

Lalam: It sounds like they are trying to see if you can get one model that works well everywhere without needing all those separate checkpoints <ref:2603.05354#pg1>.

Tom: They propose a new algorithm called BoostedTSV-M, which builds on the TSV-M method but adds this singular-value boosting scheme to mitigate rank collapse and improve numerical stability.

Lu: That singular value boosting scheme is key because it addresses rank collapse, which happens when the singular values decay sharply after you truncate and concatenate things together, suppressing those task specific signals <ref:2603.05354#pg1>.

Jane: So they’re trying to make sure that when you merge things, you don't accidentally throw away the important information specific to one of those domains.

Meng: If it works well, that means we might be able to deploy one single model instead of managing ten separate ones for European Portuguese domains <ref:2603.05354#pg3>. But what about performance trade-offs?

Tom: The results show a clear trade-off between target domain specialisation and cross-lingual robustness when comparing merging to full fine-tuning.

Jane: Full fine-tuning substantially improves European Portuguese performance over the zero shot model, reducing word error rate from fifteen point six two percent to eight point five four percent on the in domain test sets, and from twenty-five point two one percent to seventeen point six five percent on European Portuguese out of distribution data <ref:2603.05354#pg2>.

Lu: But they also found that full fine-tuning actually worsens performance on all non european portuguese out of distribution data except for AAP, which indicates that specializing the model to European Portuguese can harm multilingual robustness and potentially cause catastrophic forgetting <ref:2603.05354#pg1>.

Paper summary: Meng: That’s a real concern for production. If you specialize it too much, you risk losing its general ability to handle other languages or unseen data entirely.

Tom: Merging approaches generally preserve and in several cases improve out of distribution performance while narrowing the European Portuguese performance gap relative to joint fine-tuning.

Jane: That’s interesting because it suggests that merging isn't just about getting good at one thing; it’s about maintaining a better overall baseline for unseen things.

Lalam: The paper also looks at cross-lingual generalization using the FLEURS dataset, where a large number of merged models actually outperform the WhisperLv3-X baseline.

Tom: That suggests that through model merging, it is possible to obtain gains even for languages that are not related to the fine tuning data.

Lu: The authors hypothesize that these improvements stem from shared acoustic characteristics captured by the diverse adaptation domains, which can improve robustness and transfer knowledge to unrelated languages <ref:2603.05354#pg1>.

Jane: So what does this mean for someone just listening to the show? It means we might get a single AI model that’s strong in European Portuguese but also surprisingly good at other things, without needing separate models for every language.

Tom: And they put BoostedTSV-M up against full fine-tuning and found it achieves the best European Portuguese full average word error rate of fifteen point five five percent, slightly improving over the eighteen point five eight percent word error rate of full fine tuning with statistical significance <ref:2603.05354#pg2>.

Meng: That’s a strong number for the target domain, Tom. But Lu mentioned that TSV-M without NS actually outperformed BoostedTSV-M on all European Portuguese out of distribution corpora <ref:2603.05354#pg2>.

Lu: Exactly, the ablation study shows that in domain performance improves as beta decreases, with diminishing returns beyond beta approximately zero point six, while European Portuguese out of distribution performance generally degrades as beta decreases <ref:2603.05354#pg1>.

Jane: So the authors are showing that there’s a sweet spot they have to find for that boosting parameter to get the best balance between the two things we talked about earlier.

Tom: Overall, these findings show that while joint fine-tuning gives you the best European Portuguese accuracy, it can substantially degrade performance on other Portuguese varieties and on English multilingual benchmarks <ref:2603.05354#pg1>.

Jane: In contrast, most merging approaches preserve and in several cases improve out of distribution performance while narrowing the European Portuguese performance gap relative to joint fine-tuning.

Lalam: The paper proposes BoostedTSV-M as a practical way to produce a single deployable model that achieves strong European Portuguese accuracy and improved generalization across unseen domains and languages.

Tom: Yeah, so the main thing here is that model merging provides a practical way to get one deployable model that balances being good in its target domain with having better general performance everywhere else.

Conclusion: Tom: So, we’re wrapping up this session by talking about the paper “Exploring the potential and limitations of Model Merging for Multi-Domain Adaptation in ASR” and what it actually means for us.

Jane: It sounds like this study looked at taking specialized speech recognition models and combining them into one single model instead of training many separate ones for every new domain.

Lu: Exactly, it’s about moving away from that multi-task training idea where you have a pile of models, and figuring out if merging them is a good way to handle different language or domain tasks.

Meng: I mean, the paper proposes this BoostedTSV-M method as a specific way to do the merging part, trying to fix some issues like rank collapse when you combine those task vectors.

Tom: It seems they’re finding that while full fine-tuning is great for European Portuguese accuracy on its own, it can hurt how well the model handles other languages or unseen data.

Jane: So the big implication here is that merging isn't just about getting one thing perfect; it’s about finding a way to keep good performance across different kinds of data without losing general ability.

Lu: They suggest that most merging methods actually keep, and in some cases even improve, performance when you test on data you haven't seen before or in other languages.

Meng: That’s interesting because I worry about deployment; if we merge too aggressively, we might accidentally break the model’s ability to handle a new acoustic environment entirely.

Tom: So the authors are saying that this merging technique gives us a practical way to build one single deployable model that's strong in its target area and also better at generalizing everywhere else.

Jane: It really shows that we don't always have to pick one approach over another; sometimes combining techniques can give you a better overall system.

Lu: And they’re pointing toward this BoostedTSV-M method as the strongest merging technique they tested for European Portuguese accuracy, which is a good finding for those specific domain needs.

Meng: If that boosts the target domain score without tanking everything else, then it could actually be something we consider moving towards in production pipelines.

More episodes

← Home