SOS-LoRA: Static Orthogonal-Subspace Low-Rank Adaptation with Fixed Multi-Scale Scaling
SOS-LoRA addresses how standard LoRA couples different adaptation behaviors under a fixed rank budget by decomposing it into K static experts. It introduces fixed multi-scale scaling and orthogonal initialization to ensure these experts have diverse input directions, leading to consistent performance gains on reasoning and NLU benchmarks while maintaining LoRA's mergeability.