Are Parameter-Efficient Fine-tuning Methods Really Different?
The study compared six parameter-efficient fine-tuning methods across language and diffusion models to see how their mathematical parameterizations affect task performance, forgetting, and changes in weight geometry. It found that explicit geometric preservation is not the main differentiator; instead, the operating regimes and functional consequences of learned weight geometry are what truly distinguish these methods.