Frozen Factor or Spectral Band? Disentangling Two Choices in Low-Rank LoRA
cs.LG, cs.AI, cs.CL
Submitted: 2026-10-05
Updated: 2026-10-05
Code: https://github.com/hjunseoh/PiCa
Sources
- LoRA-XS: Low-Rank Adaptation with Extremely Small Number of Parameters
- Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
- The Llama 3 Herd of Models
- Flora: Low-Rank Adapters Are Secretly Gradient Compressors
- LoRA+: Efficient Low Rank Adaptation of Large Models
- The Impact of Initialization on LoRA Finetuning Dynamics
- Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
- LoRA: Low-Rank Adaptation of Large Language Models
- PiCa: Parameter-Efficient Fine-Tuning with Column Space Projection
- Mistral 7B
- VeRA: Vector-based Random Matrix Adaptation
- Sharp Generalization Bounds for Foundation Models with Asymmetric Randomized Low-Rank Adapters
- Beware of the Batch Size: Hyperparameter Bias in Evaluating LoRA
- SVFT: Parameter-Efficient Fine-Tuning with Singular Vectors
- Decoupled Weight Decay Regularization
- PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models
- Parameter Efficient Fine-tuning via Explained Variance Adaptation
- Initialization using Update Approximation is a Silver Bullet for Extremely Efficient Low-Rank Fine-Tuning
- Least but not Last: Fine-tuning Intermediate Principal Components for Better Performance-Forgetting Trade-Offs
- Qwen2.5 Technical Report
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks