Static Gradient Attribution Underperforms a Density-Matched Random Mask on Loss-Based Forgetting Within LoRA B-Matrix
cs.LG
Submitted: 2026-10-08
Updated: 2026-10-08
Sources
- The Multiple Ticket Hypothesis: Random Sparse Subnetworks Suffice for RLVR
- Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning
- QLoRA: Efficient Finetuning of Quantized LLMs
- When One Adapter Speaks for Many: Discovering Low-Rank Redundancy in Continual Fine-Tuning
- Orthogonal Gradient Descent for Continual Learning
- The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
- LoRA+: Efficient Low Rank Adaptation of Large Models
- LoRA: Low-Rank Adaptation of Large Language Models
- SNIP: Single-shot Network Pruning based on Connection Sensitivity
- InfLoRA: Interference-Free Low-Rank Adaptation for Continual Learning
- Attribution-Guided Continual Learning for Large Language Models
- PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models
- Gradient Projection Memory for Continual Learning
- Qwen2.5 Technical Report
- Picking Winning Tickets Before Training by Preserving Gradient Flow
- Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch
- LoRA-FA: Efficient and Effective Low Rank Representation Fine-tuning
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks