OPLoRA: Orthogonal Projection LoRA Prevents Catastrophic Forgetting during Parameter-Efficient Fine-Tuning
Orthogonal Projection LoRA (OPLoRA) prevents catastrophic forgetting during fine-tuning by constraining low-rank adaptation updates. It uses Singular Value Decomposition to find dominant weight directions and then projects the LoRA updates onto the orthogonal complement of these dominant subspaces. This mathematically guarantees that essential pre-trained knowledge is preserved while maintaining high performance on tasks like reasoning and code generation.