Exploring the potential and limitations of Model Merging for Multi-Domain Adaptation in ASR
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "Exploring the potential and limitations of Model Merging for Multi-Domain Adaptation in ASR".
Jane: Here's the gist of the paper. Model merging is presented as a scalable alternative to multi-task training that combines specialized models into a single model, which is particularly attractive for large speech foundation models adapted through domain-specific fine-tuning This work studies model merging for multi-domain ASR and benchmarks eleven merging algorithms across ten European Portuguese domains, evaluating in-domain accuracy, robustness under distribution shift, as well as English and multilingual performance The authors propose BoostedTSV-M, a new merging algorithm that mitigates rank collapse via singular-value boosting and improves numerical stability Overall, the approach outperforms full fine-tuning on European Portuguese while preserving out-of-distribution generalisation in a single model.
Tom: Model Merging Paradigms Model merging is categorized into three broad parameter space paradigms: parameter-space (PS), τ-space merging (τSpa), and τ-subspace merging (τSub) Parameter-space methods operate directly on the full model parameters θt as a flattened vector, including approaches such as Model Soups (SOUP)
29: , Model Stock
30: , Karcher mean
31: , and Multi-Spherical Linear Interpolation (SLERP)
24: τ-space merging methods operate over the taskvector, defined as the difference between fine-tuned and pre-trained parameters, including Task Arithmetic (TA)
32: , TrIm, Elect, and Sign (TIES)
33: , Parameter Competition Balancing (PCB)
34: , and Select, Calculate and Erase (SCE)
35: Finally, τ-subspace methods operate over τ matrices in their original form to perform merging in low-rank subspaces, including Task Singular Vectors Merging (TSV-M)
27: , Isotropic Merging in Common Subspace (ISO-C), and Isotropic Merging in Common and Task-Specific Subspaces (ISO-CTS)
36: .
Jane: BoostedTSV-M Mechanism The BoostedTSV-M algorithm enhances the TSV-M method by incorporating a singular-value boosting scheme inspired by prior work in CV
28: This enhancement addresses rank collapse, which occurs when singular values decay sharply after truncation and concatenation, leading to the suppression of task-specific signals The scheme defines a function ci,t(s) as the fraction of total energy explained by the first s singular values for each task vector matrix Wi It then finds the smallest index s∗ such that the cumulative energy ci,t(s) exceeds a threshold β ∈
0, 1: The singular values below σ(s∗)i,t are then boosted by clamping them to that value, effectively setting a lower bound on the singular values and preventing the suppression of task-specific information carried by smaller singular values.
Tom: Performance Trade-offs The experimental results reveal a clear trade-off between target-domain specialisation and cross-lingual robustness when comparing merging to full fine-tuning Full fine-tuning substantially improves EP performance over the zero-shot model, reducing WER from 15.62% to 8.54% on the ID test sets and from 25.21% to 17.65% on EP OOD However, Full-FT worsens performance on all non-EP OOD data except AAP, indicating that specialising the model to EP can harm multilingual robustness and potentially cause catastrophic forgetting Merging approaches generally preserve – and in several cases improve – OOD performance, while narrowing the EP performance gap relative to joint fine-tuning Superiority of BoostedTSV-M The proposed BoostedTSV-M achieves the best EP Full Avg.
Paper summary: Tom: So we're looking at this paper now called Exploring the potential and limitations of Model Merging for Multi-Domain Adaptation in ASR. Basically, they’re talking about merging specialized models into one single model instead of training many separate ones for every domain.
Jane: That sounds like a way to save a ton of computational effort when you have a bunch of different speech domains you want to support, right? What's the main idea here, though?
Tom: The thesis is that model merging is this scalable alternative to multi-task training where you combine the capabilities of multiple specialized models into one single model.
Lu: It’s particularly attractive for large speech foundation models because these big models usually get adapted through domain-specific fine-tuning, which results in multiple customized checkpoints, and repeating full fine-tuning when new data comes available is just computationally prohibitive <ref:2603.05354#pg1>.
Meng: So it’s about avoiding that massive retraining cost for every single new domain adaptation. But what are they actually testing?
Tom: They study model merging for multi-domain ASR and benchmark eleven different merging algorithms across ten European Portuguese domains.
Jane: And they aren't just looking at accuracy on those domains, either; they are evaluating robustness under distribution shift, and English and multilingual performance too.
Lalam: It sounds like they are trying to see if you can get one model that works well everywhere without needing all those separate checkpoints <ref:2603.05354#pg1>.
Tom: They propose a new algorithm called BoostedTSV-M, which builds on the TSV-M method but adds this singular-value boosting scheme to mitigate rank collapse and improve numerical stability.
Lu: That singular value boosting scheme is key because it addresses rank collapse, which happens when the singular values decay sharply after you truncate and concatenate things together, suppressing those task specific signals <ref:2603.05354#pg1>.
Jane: So they’re trying to make sure that when you merge things, you don't accidentally throw away the important information specific to one of those domains.
Meng: If it works well, that means we might be able to deploy one single model instead of managing ten separate ones for European Portuguese domains <ref:2603.05354#pg3>. But what about performance trade-offs?
Tom: The results show a clear trade-off between target domain specialisation and cross-lingual robustness when comparing merging to full fine-tuning.
Jane: Full fine-tuning substantially improves European Portuguese performance over the zero shot model, reducing word error rate from fifteen point six two percent to eight point five four percent on the in domain test sets, and from twenty-five point two one percent to seventeen point six five percent on European Portuguese out of distribution data <ref:2603.05354#pg2>.
Lu: But they also found that full fine-tuning actually worsens performance on all non european portuguese out of distribution data except for AAP, which indicates that specializing the model to European Portuguese can harm multilingual robustness and potentially cause catastrophic forgetting <ref:2603.05354#pg1>.
Paper summary: Meng: That’s a real concern for production. If you specialize it too much, you risk losing its general ability to handle other languages or unseen data entirely.
Tom: Merging approaches generally preserve and in several cases improve out of distribution performance while narrowing the European Portuguese performance gap relative to joint fine-tuning.
Jane: That’s interesting because it suggests that merging isn't just about getting good at one thing; it’s about maintaining a better overall baseline for unseen things.
Lalam: The paper also looks at cross-lingual generalization using the FLEURS dataset, where a large number of merged models actually outperform the WhisperLv3-X baseline.
Tom: That suggests that through model merging, it is possible to obtain gains even for languages that are not related to the fine tuning data.
Lu: The authors hypothesize that these improvements stem from shared acoustic characteristics captured by the diverse adaptation domains, which can improve robustness and transfer knowledge to unrelated languages <ref:2603.05354#pg1>.
Jane: So what does this mean for someone just listening to the show? It means we might get a single AI model that’s strong in European Portuguese but also surprisingly good at other things, without needing separate models for every language.
Tom: And they put BoostedTSV-M up against full fine-tuning and found it achieves the best European Portuguese full average word error rate of fifteen point five five percent, slightly improving over the eighteen point five eight percent word error rate of full fine tuning with statistical significance <ref:2603.05354#pg2>.
Meng: That’s a strong number for the target domain, Tom. But Lu mentioned that TSV-M without NS actually outperformed BoostedTSV-M on all European Portuguese out of distribution corpora <ref:2603.05354#pg2>.
Lu: Exactly, the ablation study shows that in domain performance improves as beta decreases, with diminishing returns beyond beta approximately zero point six, while European Portuguese out of distribution performance generally degrades as beta decreases <ref:2603.05354#pg1>.
Jane: So the authors are showing that there’s a sweet spot they have to find for that boosting parameter to get the best balance between the two things we talked about earlier.
Tom: Overall, these findings show that while joint fine-tuning gives you the best European Portuguese accuracy, it can substantially degrade performance on other Portuguese varieties and on English multilingual benchmarks <ref:2603.05354#pg1>.
Jane: In contrast, most merging approaches preserve and in several cases improve out of distribution performance while narrowing the European Portuguese performance gap relative to joint fine-tuning.
Lalam: The paper proposes BoostedTSV-M as a practical way to produce a single deployable model that achieves strong European Portuguese accuracy and improved generalization across unseen domains and languages.
Tom: Yeah, so the main thing here is that model merging provides a practical way to get one deployable model that balances being good in its target domain with having better general performance everywhere else.
Conclusion: Tom: So, we’re wrapping up this session by talking about the paper “Exploring the potential and limitations of Model Merging for Multi-Domain Adaptation in ASR” and what it actually means for us.
Jane: It sounds like this study looked at taking specialized speech recognition models and combining them into one single model instead of training many separate ones for every new domain.
Lu: Exactly, it’s about moving away from that multi-task training idea where you have a pile of models, and figuring out if merging them is a good way to handle different language or domain tasks.
Meng: I mean, the paper proposes this BoostedTSV-M method as a specific way to do the merging part, trying to fix some issues like rank collapse when you combine those task vectors.
Tom: It seems they’re finding that while full fine-tuning is great for European Portuguese accuracy on its own, it can hurt how well the model handles other languages or unseen data.
Jane: So the big implication here is that merging isn't just about getting one thing perfect; it’s about finding a way to keep good performance across different kinds of data without losing general ability.
Lu: They suggest that most merging methods actually keep, and in some cases even improve, performance when you test on data you haven't seen before or in other languages.
Meng: That’s interesting because I worry about deployment; if we merge too aggressively, we might accidentally break the model’s ability to handle a new acoustic environment entirely.
Tom: So the authors are saying that this merging technique gives us a practical way to build one single deployable model that's strong in its target area and also better at generalizing everywhere else.
Jane: It really shows that we don't always have to pick one approach over another; sometimes combining techniques can give you a better overall system.
Lu: And they’re pointing toward this BoostedTSV-M method as the strongest merging technique they tested for European Portuguese accuracy, which is a good finding for those specific domain needs.
Meng: If that boosts the target domain score without tanking everything else, then it could actually be something we consider moving towards in production pipelines.
Carlos Carvalho, Francisco Teixeira, Thomas Rolland, Alberto Abad
Instituto Superior Tecnico, Universidade de Lisboa, Portugal
cs.CL, eess.AS
Submitted: 2026-03-05
Updated: 2026-10-30
Comments: Accepted at INTERSPEECH 2026
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 81/100
The gist: Model merging is presented as a scalable alternative to multi-task training that combines specialized models into a single model…
Key concepts
- Model Merging
- Combining specialized models into a single model instead of training separate models for each task. This is proposed as an efficient way to adapt large speech foundation models to many different domains, offering scalability over traditional multi-task training methods.
- Parameter-Space Methods (PS)
- Merging techniques that operate directly on the full set of model parameters as a single vector. Examples include Model Soups and Karcher mean, which adjust the entire model structure based on pre-trained and fine-tuned versions.
- BoostedTSV-M
- A new merging algorithm that improves upon TSV-M by using singular-value boosting. This technique prevents 'rank collapse' by clamping small singular values to a lower bound, ensuring task-specific information is not suppressed during the merging process.
Terminology
Summary
Model merging is presented as a scalable alternative to multi-task training that combines specialized models into a single model, which is particularly attractive for large speech foundation models adapted through domain-specific fine-tuning This work studies model merging for multi-domain ASR and benchmarks eleven merging algorithms across ten European Portuguese domains, evaluating in-domain accuracy, robustness under distribution shift, as well as English and multilingual performance The authors propose BoostedTSV-M, a new merging algorithm that mitigates rank collapse via singular-value boosting and improves numerical stability Overall, the approach outperforms full fine-tuning on European Portuguese while preserving out-of-distribution generalisation in a single model. p. 0 In this work,
we study model merging for multi-domain ASR and benchmark
11 merging algorithms for 10 European Portuguese domains,
evaluating in-domain accuracy, robustness under distribution
shift, as well as English and multilingual performance. We further propose BoostedTSV-M, a new merging algorithm based
on TSV-M that mitigates rank collapse via singular-value boosting and improves numerical stability.Where the paper says it
Model Merging Paradigms
Model merging is categorized into three broad parameter space paradigms: parameter-space (PS), τ-space merging (τSpa), and τ-subspace merging (τSub) Parameter-space methods operate directly on the full model parameters θt as a flattened vector, including approaches such as Model Soups (SOUP) [29], Model Stock [30], Karcher mean [31], and Multi-Spherical Linear Interpolation (SLERP) [24] τ-space merging methods operate over the taskvector, defined as the difference between fine-tuned and pre-trained parameters, including Task Arithmetic (TA) [32], TrIm, Elect, and Sign (TIES) [33], Parameter Competition Balancing (PCB) [34], and Select, Calculate and Erase (SCE) [35] Finally, τ-subspace methods operate over τ matrices in their original form to perform merging in low-rank subspaces, including Task Singular Vectors Merging (TSV-M) [27], Isotropic Merging in Common Subspace (ISO-C), and Isotropic Merging in Common and Task-Specific Subspaces (ISO-CTS) [36].
BoostedTSV-M Mechanism
The BoostedTSV-M algorithm enhances the TSV-M method by incorporating a singular-value boosting scheme inspired by prior work in CV [28] This enhancement addresses rank collapse, which occurs when singular values decay sharply after truncation and concatenation, leading to the suppression of task-specific signals The scheme defines a function ci,t(s) as the fraction of total energy explained by the first s singular values for each task vector matrix Wi It then finds the smallest index s∗ such that the cumulative energy ci,t(s) exceeds a threshold β ∈ [0, 1] The singular values below σ(s∗)i,t are then boosted by clamping them to that value, effectively setting a lower bound on the singular values and preventing the suppression of task-specific information carried by smaller singular values.
Performance Trade-offs
The experimental results reveal a clear trade-off between target-domain specialisation and cross-lingual robustness when comparing merging to full fine-tuning Full fine-tuning substantially improves EP performance over the zero-shot model, reducing WER from 15.62% to 8.54% on the ID test sets and from 25. p. 1 For AAP, we use the AAP subset of Portugues Falado [37] ˆ
(∼3. For BP, we evaluate on the BP subset of Portugues Fal- ˆ
ado as well as the CORAA [38] and MuPe [39] test sets (total
∼46h).Where the paper says it
Superiority of BoostedTSV-M
The proposed BoostedTSV-M achieves the best EP Full Avg. (11.55% WER), slightly improving over Full-FT (11. p. 2 Our proposed modification of TSV-M, BoostedTSV-M
achieves the best EP Full Avg. 55% WER), slightly improving over Full-FT (11. The fact that TSV-M w/ NS outperforms
BoostedTSV-M on all EP OOD corpora, and the opposite happening for most ID corpora, strengthens our claim that boosting trades-off target-domain performance for EP OOD performance. Moreover, while neither method is able to achieve the
performance of Full-FT or ID-FT on ID data, we argue that the
method’s ability to preserve the capabilities of the base foundation model compensates the ID performance gap. ID performance improves as β decreases, with
diminishing returns beyond β ≈ 0. 6, while EP OOD performance generally degrades as β decreases.Where the paper says it
Where the paper says it
Where the paper says it
Cross-Lingual Generalization
The last column of Table 2 reports the average results obtained for 21 languages of the FLEURS dataset, where a large number of merged models actually outperform the WhisperLv3-X baseline This suggests that through model merging it is possible to obtain gains even for languages that are not related to the fine-tuning data The authors hypothesize that these improvements stem from shared acoustic characteristics captured by the diverse adaptation domains, which can improve robustness and transfer knowledge to unrelated languages.
Conclusion
The findings show that while joint fine-tuning yields the best EP accuracy, it can substantially degrade performance on other Portuguese varieties and on English/multilingual benchmarks In contrast, most merging approaches preserve – and in several cases improve – OOD performance, while narrowing the EP performance gap relative to joint fine-tuning Building on TSV-M, this work further proposed BoostedTSV-M, which mitigates rank collapse via singularvalue boosting, allowing it to achieve the strongest EP performance of all model merging methods under consideration Overall, the findings show that model merging provides a practical way to produce a single deployable model that achieves strong EP accuracy and improved generalization across unseen domains and languages.
Acknowledgements
Work supported by Portuguese national funds through Fundac¸ao para a Ci ˜ encia e a Tecnologia (FCT), with references UIDB/50021/2020 and 2022/12328/BD, as well as by the Portuguese Recovery and Resilience Plan (RRP) through project C644865762-00000008 (Accelerat.AI)
References
[1] A. Radford, J. p. 4 References
[1] A. Radford, J. Radford, J. Brockman, C. Brockman, C. McLeavey, and
I.Where the paper says it
Where the paper says it
Where the paper says it
[6] C. Carvalho, F. p. 0 Exploring the potential and limitations of Model Merging for Multi-Domain
Adaptation in ASR
Carlos Carvalho ID 1,2
, Francisco Teixeira ID 1
, Thomas Rolland ID 1
, Alberto Abad ID 1,2
1
INESC-ID & 2
Instituto Superior Tecnico, Universidade de Lisboa, Portugal ´
carlos. carvalho@inesc-id. Exploring the potential and limitations of Model Merging for Multi-Domain
Adaptation in ASR
Carlos Carvalho ID 1,2
, Francisco Teixeira ID 1
, Thomas Rolland ID 1
, Alberto Abad ID 1,2
1
INESC-ID & 2
Instituto Superior Tecnico, Universidade de Lisboa, Portugal ´
carlos. carvalho@inesc-id. Teixeira, C. Botelho, A. Botelho, A. Pompili, R. Pompili, R. SoleraUrena, S.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
[9] D Lopez-Paz and M Ranzato Gradient episodic memory for continual learning in Proc NeurIPS 2017 pp 6467–6476
[13] A Tang, E Yang, L Shen, Y Luo, H Hu, B Du and D Tao Merging models on the fly without retraining: A sequential approach to scalable continual model merging in Proc NeurIPS 2025
[14] W Ruan T Yang Y Zhou T Liu and J Lu From taskspecific models to unified systems: A review of model merging approaches arXiv 2503.08998 2025
[27] A A Gargiulo D Crisostomi M S Bucarelli S Scardapane F Silvestri and E Rodola Task singular vectors: Reducing task interference in model merging in Proc CVPR 2025 pp 18 695–18 705
[28] R Skorobogat K Roth and M-I Georgescu Subspace-boosted model merging arXiv 2506.
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
BoostedTSV-M can achieve an EP Full Avg WER of 11.55%, slightly improving over Full-FT (11.58% WER) with statistical significance (p < 0.001). This allows the system to produce a single deployable model that achieves strong EP accuracy while narrowing the EP performance gap relative to joint fine-tuning.
-
The improved model can preserve and in several cases improve OOD performance, as seen by TSV-M w/ NS reducing the average EP OOD WER by 1.58 points compared to Full-FT. This allows the system to maintain robustness under domain and language shift across unseen data like English and other Portuguese varieties.
-
By using BoostedTSV-M, the system can achieve the strongest EP performance of all model merging methods under consideration, specifically reaching 9.27% WER on in-domain EP data while maintaining improved generalization capabilities compared to simpler merging techniques.
Abstract
Model merging is a scalable alternative to multi-task training that combines the capabilities of multiple specialised models into a single model. This is particularly attractive for large speech foundation models, which are typically adapted through domain-specific fine-tuning, resulting in multiple customised checkpoints, for which repeating full fine-tuning when new data becomes available is computationally prohibitive. In this work, we study model merging for multi-domain ASR and benchmark 11 merging algorithms for 10 European Portuguese domains, evaluating in-domain accuracy, robustness under distribution shift, as well as English and multilingual performance. We further propose BoostedTSV-M, a new merging algorithm based on TSV-M that mitigates rank collapse via singular-value boosting and improves numerical stability. Overall, our approach outperforms full fine-tuning on European Portuguese while preserving out-of-distribution generalisation in a single model.
Sources
- Granite-speech: open-source speech-aware LLMs with strong English ASR capabilities
- Qwen3-ASR Technical Report
- From Task-Specific Models to Unified Systems: A Review of Model Merging Approaches
- Low-Rank and Sparse Model Merging for Multi-Lingual Speech Recognition and Translation
- Subspace-Boosted Model Merging
Related papers
- ALARM: Audio-Language Alignment for Reasoning Models
- ThinkSum: Probabilistic reasoning over sets using large language models
- Coherence boosting: When your pretrained language model is not paying enough attention
- From Facts to Insights: A Persona-Driven Dual Memory Framework and Dataset for Role-Playing Agents
- Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization
- The Art of Asking: Multilingual Prompt Optimization for Synthetic Data