Maximizing Capacity Utilization of LoRA Adapters for Mitigating the Learning-Forgetting Trade-off
cs.LG, cs.CL
Submitted: 2026-11-12
Updated: 2026-11-12
Code: https://github.com/tomohiro-sawada/lora-capacity-utilization
Terminology
Sources
- Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning
- LoRA-XS: Low-Rank Adaptation with Extremely Small Number of Parameters
- DeLoRA: Decoupling Angles and Strength in Low-rank Adaptation
- Training Verifiers to Solve Math Word Problems
- Linear Mode Connectivity and the Lottery Ticket Hypothesis
- A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA
- VeRA: Vector-based Random Matrix Adaptation
- ReLoRA: High-Rank Training Through Low-Rank Updates
- PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models
- Learning to Reason in 13 Parameters
- 2 OLMo 2 Furious
- LoRA vs Full Fine-tuning: An Illusion of Equivalence
- Singular Value Fine-tuning: Few-shot Segmentation requires Few-parameters Fine-tuning
- Orthogonalizing Convolutional Layers with the Cayley Transform
- MiLoRA: Harnessing Minor Singular Components for Parameter-Efficient LLM Finetuning
- Norm-Bounded Low-Rank Adaptation
- Qwen2.5 Technical Report
- AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis