Beyond LoRA vs. Full Fine-Tuning: Gradient-Guided Optimizer Routing for LLM Adaptation
The Mixture of LoRA and Full (MoLF) framework solves limitations of using only LoRA or Full Fine-Tuning by dynamically routing updates between both methods at the optimizer level. It uses a momentum-based scoring function to decide which expert receives gradient updates, ensuring every part of the model benefits from training. This unified approach consistently achieves performance matching or exceeding static methods.