A Margin Perspective on LoRA: Robustness to Catastrophic Forgetting and Adapter Merging (MaLoRA)
cs.LG
Submitted: 2026-11-10
Updated: 2026-11-10
Code: https://github.com/simonxu97/MaLoRA
Terminology
Sources
- LoRA Learns Less and Forgets Less
- Nuanced Metrics for Measuring Unintended Bias with Real Data for Text Classification
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Implicit Bias in Leaky ReLU Networks Trained on High-Dimensional Data
- Neural Collapse for Unconstrained Feature Model under Cross-entropy Loss with Imbalanced Data
- LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Composition
- Editing Models with Task Arithmetic
- LoRA Training in the NTK Regime has No Spurious Local Minima
- Deeper, Broader and Artier Domain Generalization
- Decoupled Weight Decay Regularization
- Gradient Descent Maximizes the Margin of Homogeneous Neural Networks
- LoRA-Flow: Dynamic LoRA Fusion for Large Language Models in Generative Tasks
- Continual Learning with Low Rank Adaptation
- Understanding the Learning Dynamics of LoRA: A Gradient Flow Perspective on Low-Rank Adaptation in Matrix Factorization
- TIES-Merging: Resolving Interference When Merging Models
- The Expressive Power of Low-Rank Adaptation
- Few-Shot Adaptation of Pre-Trained Networks for Domain Shift
- LoraRetriever: Input-Aware LoRA Retrieval and Composition for Mixed Tasks in the Wild
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis