Beyond Low-Rank Parameterization: Narrowing the Gap Between LoRA and Full Fine-Tuning via Gradient Decomposition
cs.AI
Submitted: 2026-09-29
Updated: 2026-09-29
Terminology
Sources
- Fira: Can We Achieve Full-rank Training of LLMs Under Low-rank Constraint?
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Parameter-Efficient Fine-Tuning with Discrete Fourier Transform
- LoRA+: Efficient Low Rank Adaptation of Large Models
- LoRA: Low-Rank Adaptation of Large Language Models
- LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models
- A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA
- Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning
- DoRA: Weight-Decomposed Low-Rank Adaptation
- RoBERTa: A Robustly Optimized BERT Pretraining Approach
- Fine-Grained Visual Classification of Aircraft
- Gemma: Open Models Based on Gemini Research and Technology
- AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning
- LoRA-One: One-Step Full Gradient Could Suffice for Fine-Tuning Large Language Models, Provably and Efficiently
- GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems