Dynamic Model Merging Made Slim
summary
The gist
The gist The DiDi-Merging framework is a slim dynamic merging framework that leverages differentiable rank allocation to balance shared and expert parameters, achieving near-original accuracy with
In short
DiDi-Merging is a slim dynamic model merging framework that balances shared and task-specific parameters using differentiable rank allocation. It optimizes parameter budgets through rank selection and refines performance with a data-free step, achieving near-original accuracy at 1.24x the storage of one backbone across vision, language, and multimodal tasks.
Key concepts
- Differentiable Rank Allocation
- This technique treats parameter budgeting as an optimization problem where the optimal size (rank) for shared and expert modules is determined continuously. Instead of guessing ranks, the model learns them end-to-end by minimizing a loss function that balances reconstruction quality against a compression penalty.
- Heavy Shared Knowledge
- This approach involves merging task vectors into a single shared knowledge vector. The goal is to keep the shared component small while allocating more capacity to task-specific expert components, which are defined as the residuals between the full model and this shared representation.
- Data-Free Parameter-Space Refinement
- After initial low-rank truncation, a data-free refinement step is used to recover task fidelity. This process uses a lightweight optimization (like Adam) to update only the low-rank factors, allowing the system to regain performance lost during aggressive compression without needing access to the original training data.
Terminology used across episodes
This episode discusses
- Dynamic Model Merging Made Slim · Paper Radio
- Program Synthesis with Large Language Models
- Qwen Technical Report
- Evaluating Large Language Models Trained on Code
- Vocabulary Hijacking in LVLMs: Unveiling Critical Attention Heads by Excluding Inert Tokens to Mitigate Hallucination
- Training Verifiers to Solve Math Word Problems
- End-to-end Rain Streak Removal with RAW Images
- The Llama 3 Herd of Models · Paper Radio
- Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators
- To See a World in a Spark of Neuron: Disentangling Multi-task Interference for Training-free Model Merging
- GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
- Are We Done with MMLU?
- Delta Decompression for MoE-based LLMs Compression
- EMR-Merging: Tuning-Free High-Performance Model Merging
- Editing Models with Task Arithmetic
- Dataless Knowledge Fusion by Merging Weights of Language Models
- D-QRELO: Training- and Data-Free Delta Compression for Large Language Models via Quantization and Residual Low-Rank Approximation
- From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline
- Multi-objective Large Language Model Alignment with Hierarchical Experts
- Video-LLaVA: Learning United Visual Representation by Alignment Before Projection
- When Models Learn to Ask Why: Adaptive Causal Reasoning for Trustworthy Medical Vision-Language Models
The paper
Dynamic Model Merging Made Slim · Read on arXiv
Guodong Du Wanyu Lin
The Hong Kong Polytechnic University
Model merging consolidates independently fine-tuned checkpoints without joint training or access to task data. Dynamic methods retain task-specific capacity, but must divide a limited storage budget between shared and expert parameters. We propose DiDi-Merging, which jointly allocates low-rank capacity to these components through differentiable rank selection, followed by data-free parameter-space refinement. On the eight-task ViT-B/32 benchmark, the small configuration reaches 89.3% average accuracy (98.7% normalized retention) at 1.24x the storage of one backbone, including that backbone; the medium configuration reaches 89.8% at 1.4x. Removing refinement reduces the medium configuration by 4.3 percentage points, while static-merge plus uniform-recovery controls are 4.7-8.9 points below the small configuration. Additional evaluations cover language and multimodal models at their separately reported budgets. The main results assume known task identity; practical routing and accuracy under larger task counts at fixed capacity remain distinct evaluation questions. Code is available at https://github.com/WanyuGroup/NeurIPS2026-DiDi-Merging.
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: Hi, I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: We're looking at the paper: "Dynamic Model Merging Made Slim".
Jane: Here's the gist of the paper. The gist The DiDi-Merging framework is a slim dynamic merging framework that leverages differentiable rank allocation to balance shared and expert parameters, achieving near-original accuracy with minimal overhead and outperforming prior dynamic baselines across vision, language, and multimodal tasks<ref:2605.18904#pg2>.
Tom: How it works DiDi-Merging addresses the two failure modes in existing dynamic merging methods by balancing shared and task-specific components through differentiable rank allocation<ref:2605.18904#pg3>.
Jane: It formulates parameter budgeting as differentiable rank optimization in low-rank modules and introduces a data-free refinement step to recover task fidelity<ref:2605.18904#pg5>.
Tom: The process involves three key steps: (1) Task vectors are statically merged into a shared knowledge vector, with per-task residuals as expert vectors (exact but parameter-heavy)<ref:2605.18904#pg6>, (2) Low-rank decomposition recasts parameter budgeting as a differentiable rank allocation problem, learned end-to-end against a task-vector reconstruction objective<ref:2605.18904#pg7>, and (3) The truncated low-rank parameters are refined in a data-free pass to recover task-specific performance<ref:2605.18904#pg8>.
Paper summary: Tom: So we’re moving on to summarizing this paper's core argument. The authors of "Dynamic Model Merging Made Slim" are proposing a slim dynamic merging framework that reuses fine-tuned models without joint training or access to original data, which is the whole thesis.
Jane: They claim they improved flexibility by selectively activating task-relevant parameters and efficiently composing experts across multiple tasks, but they point out that existing dynamic methods have problems because they either keep a full shared model with tiny experts or they allocate too much capacity to those experts.
Lu: DiDi-Merging specifically targets the balanced shared–expert router regime, where both the shared component and the per-task experts are kept small, which is achieved through differentiable rank optimization.
Meng: It seems like they’re trying to solve a specific trade-off problem that most dynamic merging papers struggle with when trying to balance parameter efficiency and accuracy.
Jane: They claim that by formulating parameter budgeting as differentiable rank optimization in low-rank modules and adding a data-free refinement step, DiDi-Merging matches prior dynamic baselines at only one point two four times the parameters of a single fine-tuned model and surpasses them at one point four times <ref:2605.18904#pg1,the parameters of a single fine-tuned model and surpasses them at>.
Tom: That's the big number here—matching or beating previous methods at just one point two four times the parameter count, which is substantially more compact than methods that require two times or more storage <ref:2605.18904#pg1,substantially more compact than methods>.
Jane: And they show this applies across vision, language, and multimodal tasks, which makes it a pretty comprehensive result for merging techniques.
Lu: The idea is that by decomposing the knowledge into low-rank modules and using learnable ranks to control how much information is kept in each module, you can achieve better parameter efficiency than fixed allocation methods.
Meng: So, the key mechanism here isn't just merging; it’s this differentiable rank optimization that lets the system decide dynamically how to allocate its budget between the shared knowledge and the task-specific parts.
Jane: It really is about making sure you don't waste capacity on a component when it’s not needed for a specific task, which is exactly what they are trying to achieve here.
Tom: So, the paper lays out this idea of balancing those two knowledge components using rank optimization as the central mechanism before showing the actual performance gains.
Conclusion: Tom: Wrapping up this discussion on "Dynamic Model Merging Made Slim," the authors are Guodong Du Wanyu Lin, and this framework is about using differentiable rank allocation to find the right balance between shared and expert parameters so you get near-original accuracy with minimal overhead.
Jane: The implication is that for deploying specialized AI models, this technique could drastically reduce both the storage costs and the serving costs associated with running collections of these models.
Lu: It means that we can deploy expert ensembles without having to retrain or exposing patient data, which is particularly promising in high-stakes multimodal scenarios like medical diagnosis.
Meng: It’s about lowering the barrier for deploying specialized models because you don't have to retain all that raw data just to use the merged version.
Jane: So, in simple terms, this paper shows a way to make dynamic model merging much more parameter efficient by using smarter ways to budget those parameters through differentiable rank optimization.
More episodes
- 2604.04385-How Alignment Routes: Localizing, Scaling, and Controlling Policy Circuits in Language Models
- 2604.07667-From Debate to Decision: Conformal Social Choice for Safe Multi-Agent Deliberation
- 2605.08827-Mental Health AI Safety Claims Must Preserve Temporal Evidence
- 2605.10641-LLaVA-CKD: Bottom-Up Cascaded Knowledge Distillation for Vision-Language Models
- 2605.10765-Dynamic Cross-Modal Prompt Generation for Multimodal Continual Instruction Tuning
- 2605.12682-Learning Transferable Latent User Preferences for Human-Aligned Decision Making
- 2605.17826-CounterCount: Diagnosing Knowledge Bias in How Vision-Language Models Count Attributes
- 2605.21187-NVIDIA Spectrum-X: Hardware-Accelerated Multiplane Networking for Large-Scale AI Training
- 2605.29458-What a One-Hour Interview Does Not Tell an LLM About You: Interviews Elicit Traits, Not Decision Policies
- 2605.30251-Same Evidence, Different Answers: Canonical-Context On-Policy Distillation for Multi-Turn Language Models