Model Merging Does Not Require Uniform LoRA Ranks
cs.LG, cs.AI
Submitted: 2026-11-04
Updated: 2026-11-11
Terminology
Sources
- Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch
- Model merging with SVD to tie the Knots
- Arcee's MergeKit: A Toolkit for Merging Large Language Models
- Saliency-Aware Model Merging
- ImPart: Importance-Aware Delta-Sparsification for Improved Model Compression and Merging in LLMs
- PreLort: Prefix-Nested LoRA for Federated Fine-Tuning under Rank Heterogeneity
- Compress then Merge: From Multiple LoRAs into One Low-Rank Adapter
- MergeBench: A Benchmark for Merging Domain-Specialized LLMs
- Are we Merging the Right Models? Impact of Expert Training Duration on Model Merging for LLMs
- Qwen3 Technical Report
- DART-Math: Difficulty-Aware Rejection Tuning for Mathematical Problem-Solving
- Training Verifiers to Solve Math Word Problems
- Tulu 3: Pushing Frontiers in Open Language Model Post-Training
- Instruction-Following Evaluation for Large Language Models
- Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis