Are Parameter-Efficient Fine-tuning Methods Really Different?

arXiv:2610.09122 · cs.LG · Submitted 2026-10-06 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: Today's paper: "Are Parameter-Efficient Fine-tuning Methods Really Different?".

Jane: The gist: Parameter-efficient fine-tuning (PEFT) methods are distinguished more by their operating regimes and functional consequences than by parameterization labels or explicit geometric preservation alone.

Tom: First, who's behind it and why it matters.

Paper summary: Tom: To recap, this paper "Are Parameter-Efficient Fine-tuning Methods Really Different?" compares six parameterization schemes—LoRA, DoRA, PiSSA, MiLoRA, OFT, and HRA—across language and diffusion models to see how their parameterizations affect task performance and forgetting.

Jane: The thesis of the paper is that while PEFT offers many parameterizations, their methodological differences aren't clearly understood by just looking at the labels. They want to examine if spectral preservation is actually important for adaptation and retention.

Lu: They start by asking whether spectral preservation itself is important for adaptation and retention, motivated by the spectrum-preserving design of orthogonal fine-tuning, OFT.

Meng: So they are testing if this specific design choice actually provides a measurable benefit in terms of how much knowledge the model keeps or how well it performs on a new task.

Lalam: They find that for small isotropically oriented additive updates, the fraction of weight displacement appearing as singular-value displacement scales as one over p times max of m and n, which shows that limited spectral drift can happen even without an explicit orthogonality constraint <ref:2610.09122#pg1>.

Jane: But the key finding for them is that restoring pretrained singular values while keeping the adapted singular bases largely preserves task performance. This questions why explicit geometric preservation might be necessary for retaining those gains.

Lu: They also observed distinct adaptation–retention trade-offs that vary across settings, meaning LoRA most consistently limits forgetting at competitive performance, DoRA achieves higher mean task scores than LoRA in most comparisons, and PiSSA often incurs greater retention costs.

Meng: This means the best method depends entirely on what you are trying to achieve in a specific environment—do you prioritize stability or maximum score?

Lalam: The study also investigated whether unconstrained LoRA-family methods actually depart substantially from these geometries in practice, and did tighter geometric preservation translate into better adaptation or retention.

Tom: They found that two adapters can achieve similar downstream scores while differing substantially in how much pretrained behavior they disrupt, which shows that performance isn't the only metric.

Jane: It really suggests we need to look at the functional consequences of the geometry rather than just trusting what a parameterization name implies about its mathematical structure.

Conclusion: Tom: So, looking at "Are Parameter-Efficient Fine-tuning Methods Really Different?", the authors are arguing that explicit geometric preservation is not what separates these PEFT methods.

Jane: They find that LoRA's consistency in limiting forgetting at competitive performance and DoRA's higher mean task scores in most comparisons are the main takeaways.

Lu: The paper shows that restoring pretrained singular values largely preserves task performance, which questions the necessity of enforcing explicit geometric preservation to retain those gains.

Meng: Essentially, the functional differences between methods are more subtle than their parameterization labels suggest when measured by actual task scores and retention rates.

Lalam: The recurring mismatch they found is that parameterization labels can overstate geometric differences, whereas task scores can understate functional differences.

Tom: So for anyone using these techniques, the implication is that you should probably focus on the operating regime of the method and how it affects your specific model's behavior rather than just following a label.

Jane: It means choosing a method should be about predicting what it will actually do to your model's performance and what knowledge it will keep for your application.

Yikuan Li, Pinyan Lu, Fanghui Liu

University of Warwick · Shanghai University of Finance and Economics · Shanghai Jiao Tong University

cs.LG

Submitted: 2026-10-06

Updated: 2026-10-06

Code: https://github.com/Kuaaannn/PEFT_methods

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 90/100

The gist: The gist: Parameter-efficient fine-tuning (PEFT) methods are distinguished more by their operating regimes and functional consequences than by parameterization labels or explicit geometric

Key concepts

Parameter-Efficient Fine-Tuning (PEFT): PEFT methods are techniques used to adapt large pre-trained models to new tasks without retraining all parameters. They achieve this by introducing a small set of trainable parameters, such as low-rank matrices or orthogonal transformations, which allow the model to learn task-specific adjustments efficiently.
Geometric Preservation: This refers to how well the adaptation process keeps the adapted weights close to the original pre-trained weight structure. The paper investigates whether explicitly enforcing this geometric constraint is necessary for good performance or retention of old knowledge, finding it isn't always required.
Operating Regimes: These are different ways in which a PEFT method functions during fine-tuning, such as low-rank addition versus spectral initialization. The authors argue that these distinct operating modes and the resulting functional changes to the weight geometry are more important for distinguishing methods than just their mathematical labels.
Singular-Value Displacement: This measures how much the singular values of the weight matrices change after fine-tuning. The study shows that small, isotropic updates can still cause some displacement, but LoRA-family methods generally show modest drift. This relates to how much the fundamental structure of the pre-trained weights is altered.

Terminology

Fine-Tuning — 1× in this paper · explained in 5 episodes
Parameter-Efficient Fine-Tuning — 1× in this paper · explained in 3 episodes

Summary

The gist: Parameter-efficient fine-tuning (PEFT) methods are distinguished more by their operating regimes and functional consequences than by parameterization labels or explicit geometric preservation alone.

How it works

The study compares six representative PEFT algorithms across language and diffusion models to examine how their parameterizations relate to task performance, forgetting, and changes in pretrained weight geometry The abstract states that they compare six methods in language and diffusion models to examine how their parameterizations relate to task performance, forgetting, and changes in pretrained weight geometry.
The six methods studied are LoRA, DoRA, PiSSA, MiLoRA, OFT, and HRA. These span low-rank addition through spectral initialization and orthogonal transformations.
The analysis involves examining three levels of behavior for each method: their operating performance after method-specific tuning, their retention of pretrained capabilities, and the functional consequences of the geometric changes they induce This approach reveals a recurring mismatch: parameterization labels can overstate geometric differences, whereas task scores can understate functional differences.

Geometric Preservation and Task Performance

** Explicit geometric preservation is not what primarily separates PEFT methods The authors characterize adaptation relative to the two-sided orthogonal orbit of the pretrained weights. For small isotropically oriented additive updates, the fraction of weight displacement appearing as singular-value displacement scales as 1/p max(m, n), showing that limited spectral drift can arise even without an explicit orthogonality constraint Consistent with this prediction, the LoRA-family methods exhibit only modest spectral drift and very small changes in hyperspherical energy. More importantly, restoring pretrained singular values while retaining the adapted singular bases largely preserves task performance Thus, explicit geometric preservation is neither unique to successful PEFT nor, in our tested settings, necessary for retaining its task gains. The authors find that LoRA-family methods approximately preserve both quantities without enforcing orthogonality. However, tighter preservation must be judged by its task and retention outcomes. The results suggest that PEFT methods are better distinguished by their operating regimes and the functional consequences of learned weight geometry than by parameterization labels or explicit geometric preservation alone. The study also investigates whether unconstrained LoRA-family methods actually depart substantially from these geometries in practice, and does tighter geometric preservation translate into better adaptation or retention. The authors find that two adapters can achieve similar downstream scores while differing substantially in how much pretrained behavior they disrupt. Orthogonal methods OFT and HRA offer no consistent performance–retention advantage over the four LoRA-family methods across the tested settings. The orthogonal methods OFT and HRA offer no consistent performance–retention advantage over the four LoRA-family methods across the tested settings. The authors find that competitive task scores can accompany different forgetting costs, with outcomes depending on the selected LR and capacity. The orthogonal methods OFT and HRA offer no consistent performance–retention advantage over the four LoRA-family methods across the tested settings. The orthogonal methods OFT and HRA offer no consistent performance–retention advantage over the four LoRA-family methods across the tested settings. The results support LoRA’s recurring retention advantage and DoRA’s higher mean task scores, alongside setting-dependent outcomes for the other methods. The orthogonal methods OFT and HRA offer no consistent performance–retention advantage over the four LoRA-family methods across the tested settings. The orthogonal methods OFT and HRA offer no consistent performance–retention advantage over the four LoRA-family methods across the tested settings. The results support LoRA’s recurring retention advantage and DoRA’s higher mean task scores, alongside setting-dependent outcomes for the other methods. The orthogonal methods OFT and HRA offer no consistent performance–retention advantage over the four LoRA-family methods across the tested settings. The results support LoRA’s recurring retention advantage and DoRA’s higher mean task scores, alongside setting-dependent outcomes for the other methods. The orthogonal methods OFT and HRA offer no consistent performance–retention advantage over the four LoRA-family methods across the tested settings. The results support LoRA’s recurring retention advantage and DoRA’s higher mean task scores, alongside setting-dependent outcomes for the other methods. The orthogonal methods OFT and HRA offer no consistent performance–retention advantage over the four LoRA-family methods across the tested settings. The results support LoRA’s recurring retention advantage and DoRA’s higher mean task scores, alongside setting-dependent outcomes for the other methods. The orthogonal methods OFT and HRA offer no consistent performance–retention advantage over the four LoRA-family methods across the tested settings. The results support LoRA’s recurring retention advantage and DoRA’s higher mean task scores, alongside setting-dependent outcomes for the other methods. The orthogonal methods OFT and HRA offer no consistent performance–retention advantage over the four LoRA-family methods across the tested settings. The results support LoRA’s recurring retention advantage and DoRA’s higher mean task scores, alongside setting-dependent outcomes for the other methods. The orthogonal methods OFT and HRA offer no consistent performance–retention advantage over the four LoRA-family methods across the tested settings. The results support LoRA’s recurring retention advantage and DoRA’s higher mean task scores, alongside setting-dependent outcomes for the other methods. The orthogonal methods OFT and HRA offer no consistent performance–retention advantage over the four LoRA-family methods across the tested settings. The results support LoRA’s recurring retention advantage and DoRA’s higher mean task scores, alongside setting-dependent outcomes for the other methods. The orthogonal methods OFT and HRA offer no consistent performance–retention advantage over the four LoRA-family methods across the tested settings. The results support LoRA’s recurring retention advantage and DoRA’s higher mean task scores, alongside setting-dependent outcomes for the other methods. The orthogonal methods OFT and HRA offer no consistent performance–retention advantage over the four LoRA-family methods across the tested settings. The results support LoRA’s recurring retention advantage and DoRA’s higher mean task scores, alongside setting-dependent outcomes for the other methods. The orthogonal methods OFT and HRA offer no consistent performance–retention advantage over the four LoRA-family methods across the tested settings. The results support LoRA’s recurring retention advantage and DoRA’s higher mean task scores, alongside setting-dependent outcomes for the other methods. The orthogonal methods OFT and HRA offer no consistent performance–retention advantage over the four LoRA-family methods across the tested settings. The results support LoRA’s recurring retention advantage and DoRA’s higher mean task scores, alongside setting-dependent outcomes for the other methods. The orthogonal methods OFT and HRA offer no consistent performance–retention advantage over the four LoRA-family methods across the tested settings. The results support LoRA’s recurring retention advantage and DoRA’s higher mean task scores, alongside setting-dependent outcomes for the other methods. The orthogonal methods OFT and HRA offer no consistent performance–retention advantage over the four LoRA-family methods across the tested settings. The results support LoRA’s recurring retention advantage and DoRA’s higher mean task scores, alongside setting-dependent outcomes for the other methods. The orthogonal methods OFT and HRA offer no consistent performance–retention advantage over the four LoRA-family methods across the tested settings. The results support LoRA’s recurring retention advantage and DoRA’s higher mean task scores, alongside setting-dependent outcomes for the other methods. The orthogonal methods OFT and HRA offer no consistent performance–retention advantage over the four LoRA-family methods across the tested settings. The results support LoRA’s recurring retention advantage and DoRA’s higher mean task scores, alongside setting-dependent outcomes for the other methods. The orthogonal methods OFT and HRA offer no consistent performance–retention advantage over the four LoRA-family methods across the tested settings. The results support LoRA’s recurring retention advantage and DoRA’s higher mean task scores, alongside setting-dependent outcomes for the other methods. The orthogonal methods OFT and HRA offer no consistent performance–retention advantage over the four LoRA-family methods across the tested settings. The results support LoRA’s recurring retention advantage and DoRA’s higher mean task scores, alongside setting-dependent outcomes for the other methods.

Improvements for AI systems

Improved parameter-efficient fine-tuning (PEFT) methods can be selected based on functional consequences rather than geometric preservation alone, as parameterization labels can overstate geometric differences, whereas task scores can understate functional differences. This allows for method selection that maximizes competitive downstream performance while minimizing forgetting, noting that LoRA most consistently limits forgetting at competitive performance.
Task adaptation and forgetting can be controlled by selectively restoring spectral components: Restoring dominant components consistently reduces forgetting, which is noted as the largest reduction in general-text NLL or base-image drift across all six methods. This suggests a strategy where adaptation is realized through multiple spectral routes, focusing on dominant rather than intermediate or trailing components for forgetting control.
System performance can be optimized by tuning learning rates and capacities based on setting-dependent trade-offs: LoRA most consistently limits forgetting among all six methods and DoRA achieves higher mean task scores than LoRA in most comparisons. This implies that the optimal PEFT configuration is not universal but must be tailored to the specific model, task, and learning rate regime.
For diffusion personalization, models can achieve superior visual fidelity by selecting specific spectral interventions: LoRA and OFT have lower image drift and higher CLIP alignment at every capacity, showing that LoRA's retention advantage in generated outputs is maintained across object budgets. This allows the system to select an adaptation method that balances fidelity metrics like DINO, general drift, and CLIP alignment based on the target application.

Abstract

Parameter-efficient fine-tuning (PEFT) offers many parameterizations, yet their methodological and functional differences remain unclear. We compare six methods in language and diffusion models to examine how their parameterizations relate to task performance, forgetting, and changes in pretrained weight geometry. Motivated by the spectrum-preserving design of orthogonal fine-tuning (OFT), we first ask whether spectral preservation is itself important for adaptation and retention. We find that the selected LoRA-family methods also approximately preserve pretrained geometry, and that restoring their slightly drifted singular-value spectra largely preserves task performance, questioning the necessity of explicit geometric preservation. Beyond this, we observe that some methods exhibit distinct adaptation--retention trade-offs that vary across settings: LoRA most consistently limits forgetting at competitive performance, DoRA achieves higher mean task scores than LoRA in most comparisons, while PiSSA often incurs greater retention costs. Further intervention experiments suggest that while performance gains from different PEFT methods can be attributed to modifications in different groups of spectral components, we consistently find that restoring dominant rather than intermediate or trailing components produces the largest mean reduction in general-text NLL or base-image drift. Together, these results motivate evaluating geometric constraints through their functional consequences rather than preservation alone. Code is available at https://github.com/Kuaaannn/PEFT methods.