LoRA's Second Descent Extends Beyond Parameter Parity
cs.LG
Submitted: 2026-10-03
Updated: 2026-10-03
Code: https://github.com/huggingface/pytorch-image-models
Sources
- Frozen Overparameterization: A Double Descent Perspective on Transfer Learning of Deep Neural Networks
- Unifying Grokking and Double Descent
- Procedural Knowledge Is Not Low-Rank: Why LoRA Fails to Internalize Multi-Step Procedures
- PoLoRA: A Preconditioned Orthogonalized LoRA Optimizer
- A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA
- SDS-LoRA: Overcoming Anisotropic Gradient Scaling in Low-Rank Adaptation
- Rethinking the Rank Threshold for LoRA Fine-Tuning
- Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets
- Qwen2.5 Technical Report
- Rank-Accuracy Trade-off for LoRA: A Gradient-Flow Analysis
- Double Descent Demystified: Identifying, Interpreting & Ablating the Sources of a Deep Learning Puzzle
- Why LoRA Resists Label Noise: A Theoretical Framework for Noise-Robust Parameter-Efficient Fine-Tuning
- LoRA-FA: Efficient and Effective Low Rank Representation Fine-tuning
- LoRA vs. Full Fine-Tuning: A Theoretical Perspective
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks