C-LoRA: Continual Low-Rank Adaptation for Pre-trained Visual Models

The gist The Continual Low-Rank Adaptation (C-LoRA) method enables a single, shared LoRA adapter to handle sequential tasks without catastrophic forgetting by using a learnable routing matrix R that explicitly controls how each rank-one subspace contributes to the weight update

How it works

The core innovation of C-LoRA is a learnable routing matrix R that decomposes into a stability component (Rbase) and a plasticity component (Rδ), which drives adaptation to the current task without disrupting consolidated knowledge The update is formulated as ∆W = ARB, where R controls how each rank-one subspace contributes to the weight update This matrix R is decomposed into a stability component (Rbase), which accumulates knowledge from prior tasks and is excluded from gradient flow during new-task training The plasticity component (Rδ) drives adaptation to the current task without disrupting consolidated knowledge

Gradient Flow Control and Stability

The standard LoRA update implicitly assigns a uniform weight to every rank-one subspace, causing all subspaces to be equally exposed to gradient updates from new tasks Introducing the routing matrix R addresses this by allowing the model to learn differential subspace weights Theorem 1 shows that the gradients flowing into the shared bases A and B scale with∥R∥2 As knowledge accumulates,∥R∥2 grows, and the subspaces most critical for stability are paradoxically the most vulnerable to disruption The decomposition strategy breaks this conflict by stopping gradient flow through Rbase during new-task training, modulating updates only by Rδ Theorem 3 demonstrates that if σ2base > σ2δ, then the total variance of the gradient updates modulated by Rδ is strictly smaller than if they were modulated by Rbase

Importance-Aware Gating

To further mitigate gradient magnitude, C-LoRA introduces an importance-aware gate Γ ∈ [0, 1]r×r that suppresses gradient flow into Rδ at dimensions important to prior tasks The importance of each subspace is estimated by the magnitude of the corresponding element, Iij = Rbase,ij This importance score is then Z-score normalized to compute Γ using an element-wise sigmoid function The effective routing component for the current task, R′δ, is calculated as R′δ = Γ ⊙ Rδ + (1 − Γ) ⊙ ϕ(Rδ), where ϕ(·) denotes the stop-gradient operator

Empirical Performance and Efficiency

C-LoRA demonstrates competitive performance across multiple benchmarks, achieving high Last-Acc values on CIFAR-100, ImageNet-A, CUB-200, and CAR196 datasets The fixed parameter budget of C-LoRA is a key advantage over expansion methods like EASE or SD-LoRA, which incur linear parameter growth with the number of tasks For instance, C-LoRA maintains a constant overhead parameter regardless of sequence length Furthermore, the analysis shows that routing through Rδ produces updates that are both smaller in magnitude and lower in variance compared to routing through Rbase The method's ability to reduce forgetting is quantified by the Backward Transfer (BWT) metric, where C-LoRA achieves a BWT of-3.78 on ImageNet-A

Conclusion

C-LoRA successfully mitigates the stability-plasticity conflict within a single, fixed-size adapter by decoupling R into Rbase and Rδ By employing a learnable routing matrix R, C-LoRA significantly enhances knowledge retention during sequential task adaptation The work concludes that the design of the routing matrix is crucial for resolving gradient amplification and achieving robust continual learning performance

--- Page 1 ---

C-LoRA: Continual Low-Rank Adaptation for Pre-trained Visual Models Xin Zhang Institute of Intelligent Information Processing Shanxi University Taiyuan 030006, China zhangxin8572@sxu.edu.cn Liang Bai∗ Institute of Intelligent Information Processing Shanxi University Taiyuan 030006, China bailiang@sxu.edu.cn Xian Yang Alliance Manchester Business School The University of Manchester Manchester M13 9PL, United Kingdom xian.yang@manchester.ac.uk Abstract Pre-trained visual models have become fundamental in computer vision, but they face challenges in continual learning scenarios where data and tasks evolve over time Low-Rank Adaptation (LoRA) offers efficient fine-tuning capabilities but remains limited for such dynamic environments Standard LoRA cannot distinguish important subspaces, causing critical knowledge to be overwritten in sequential training We propose Continual Low-Rank Adaptation (C-LoRA), a method that enables a single, shared LoRA adapter to handle sequential tasks without catastrophic forgetting—without requiring any module selection or fusion at inference The core of C-LoRA is a learnable routing matrix R that explicitly controls how each rank-one subspace contributes to the weight update This matrix is decomposed into a stability component (Rbase), which preserves knowledge from prior tasks, and a plasticity component (Rδ), which drives adaptation to the current task—providing direct control over the stability-plasticity trade-off We analyze how R governs gradient flow during sequential training, and demonstrate competitive performance across multiple benchmarks The code is available at https://github.com/lambor9973/C-LoRA 1 Introduction Pre-trained visual models Dosovitskiy [2020], Wang et al. [2022a,b] have revolutionized computer vision applications, but they face fundamental limitations when deployed in dynamic environments where tasks and data distributions continuously evolve Parisi et al. [2019] This challenge, known as continual learning Wang et al. [2024], requires models to progressively accumulate knowledge while preserving performance on previously learned tasks Low-Rank Adaptation (LoRA) Hu et al. [2021] is a parameter-efficient fine-tuning method that adapts large-scale pre-trained models to downstream tasks by introducing low-rank matrices to update a small subset of parameters while keeping the core pre-trained weights frozen This approach preserves the general knowledge captured during pre-training, reduces computational and storage costs, and mitigates the risk of overfitting by limiting updates to task-specific components Ayoub Kamal et al. [2502.17920] LoRA has proven highly effective in fields such as computer vision, where pre-trained models are commonly deployed for diverse and evolving applications Yu et al. [2502.17920], Zhou et al. [2502.17920], Aljundi et al. [2502.17920], Hu et al. [2502.

1. Continuous knowledge preservation in sequential tasks: The C-LoRA framework enables a single, shared LoRA adapter to handle sequential tasks without catastrophic forgetting, specifically by decomposing the routing matrix into a stability component (Rbase) and a plasticity component (Rδ), allowing Rbase to accumulate knowledge from prior tasks and is excluded from gradient flow during new-task training.

2. Gradient flow control for stability: The system mitigates catastrophic forgetting through decoupled routing, where updates are modulated by Rδ alone, which ensures that the resulting parameter shifts are significantly smaller compared to joint strategies, as shown by Theorem 2: If Rδ is initialized such that∥Rδ∥2 ≪∥Rbase∥2, the decomposition strategy achieves a significantly tighter upper bound on the stability loss.

3. Importance-aware subspace gating: The C-LoRA system protects critical knowledge by introducing an importance-aware gate Γ, which suppresses gradient flow into Rδ at dimensions important to prior tasks, allowing for fine-grained control as defined by R′δ = Γ ⊙ Rδ + (1 − Γ) ⊙ ϕ(Rδ).

4. Fixed parameter budget efficiency: Unlike existing approaches that incur linear parameter growth, C-LoRA maintains a constant overhead parameter regardless of sequence length, which is demonstrated by the finding that C-LoRA (Ours) maintains a fixed parameter budget while others grow linearly with the number of tasks.

5. Enhanced performance across task sequences: The method achieves superior continual learning performance, as evidenced by its results on CUB-200 where C-LoRA’s Last-Acc slightly increases with the number of sessions, and it shows competitive results against state-of-the-art methods like Adam+adapter.

6. Reduced gradient variance: The decomposition strategy leads to smaller in magnitude and lower in variance updates, as formalized by Theorem 3: "If σ2 base > σ2 δ, then the total variance of the gradient updates modulated by Rδ is strictly smaller than if they were modulated by Rbase."