Static Gradient Attribution Underperforms a Density-Matched Random Mask on Loss-Based Forgetting Within LoRA B-Matrix
The study compared gradient attribution against a density-matched random mask for parameter updates in LoRA, focusing on minimizing catastrophic forgetting using loss-based measures. The findings show that the uniform random mask consistently resulted in lower loss-based forgetting than gradient attribution across multiple seeds and architectures. This suggests that for standard zero-initialized LoRA, a simple random selection rule is superior to gradient attribution when restricting comparisons to the B-Matrix parameters.