PGSB: Pretrained-Guided Shared Basis for LoRA Model Merging
PGSB is a two-stage framework for merging LoRA model updates by first finding shared directions across task-specific updates and then refining these directions using the pretrained weights' geometry. This process creates a compatible basis that allows for better merging, leading to state-of-the-art performance on vision and language tasks.