Task-Driven Subspace Decomposition for Knowledge Sharing and Isolation in LoRA-based Continual Learning

The gist The proposed Low-rank Decomposition and Adaptation (LoDA) framework performs a task-driven decomposition to build general and truly task-specific LoRA subspaces by solving two energy-based objectives, decoupling directions for knowledge sharing and isolation.

How it works

The LoDA framework addresses the limitations of existing methods that either overlook task-shared directions or fail to capture truly effective task-specific directions by performing a task-driven decomposition to build general and truly isolated LoRA subspaces. This is achieved by revealing that the impact of a LoRA update on the loss is governed by the projection energy of the task features onto the down-projection’s row subspace, which motivates a data-driven design to freeze down-projections as a gate that selects learnable new feature components while controlling interference with past tasks.

The decomposition involves two subspaces:

1. A general subspace UG that yields high projection energy across both old and new tasks, capturing directions salient across all tasks and enabling knowledge transfer. This is derived by maximizing the overall projection energy (Eold + Enew) of both old and new tasks onto UG=span(UG).

2. An isolated subspace UI that exhibits the largest new-to-old relative energy (UI = arg max U(Enew/Eold)), identifying updates effective for the new task with little impact on past tasks. This is derived by maximizing the ratio between (i) the projection energy Enew of task t and (ii) the accumulated energy Eold of all previous tasks 1: t − 1: UI = arg max U∈RD×r XtU2F / Xt−1i=1 XiU2F.

LoRA Optimization and Integration

Based on this decomposition, LoDA builds a dual-branch LoRA module where each branch’s down-projection is fixed on decomposed subspace bases and the up-projection is learnable. To steer the learning of up-projections toward robust and cross-class conflict-free directions, Gradient Aligned Optimization (GAO) is designed to encourage its gradient consistency across label-disjoint subsets.

The training pipeline involves several steps:

1. Before training, the frozen backbone f(Wt−1) on Dt is run to cache per-layer inputs and compute statistics S t.

2. The general and isolated subspace bases UG and UI are derived via optimization objectives in Eq. 4 and Eq. 6 using S 1:t−1 and S t.

3. LoRAG is initialized by freezing the down-projection on UG, while LoRAI is initialized by fixing the down-projection on UI.

4. During training, trainable parameters BG and BI are optimized on Dt with GAO.

5. After training, LoDA integrates the decoupled updates into the backbone: the general branch is recalibrated by a closed-form rescaling matrix ΛG to approximate a feature-level joint optimum, while the isolated branch is directly merged.

Key Theoretical Insights

The analysis reveals several key theoretical findings regarding the LoRA mechanism:

**-Theorem 3.1 reveals that Updating B changes the output Y exactly along the steepest descent direction −∂L/∂Y; L(Y′) − L(Y) = -η AX⊤22 ∂L/∂Y + o(∆Y). (B.1.7). This shows that the magnitude of change is modulated by the projection energy E = AX⊤22, which quantifies how much the input feature X lies in the subspace selected by A. If rows of A are orthonormal, AX⊤22 is exactly the squared Euclidean norm of the projection of X onto the row space of A. This motivates controlling plasticity on new data and interference with past tasks by designing the down-projection A in a task-driven manner. (B.2.5) The optimal solution for the general subspace UG is achieved when U′ = Ir0 and UG = US[:,1:r], i.e., UG consists of the top-r singular vectors of matrix S 1:t−1 + S t. (B.2.6) The optimal merged update for each rank-1 unit is a re-scaling of the current unit, i.e., B(j)⋆G = γ(j)⋆B(j)G. This coefficient γ(j)⋆ is determined by the energy of the current-task covariance S t and the accumulated past covariance S 1:t−1 along the same input-side direction A(j)G. (B.3.6) The GAO algorithm implements a stochastic first-order optimization of gradient alignment, which directly promotes larger cross-group inner products g⊤i gj. This encourages shared directions across groups effectively enlarges the safe set of update directions in which the current task is less likely to be harmed, improving robustness to future optimization steps. (B.4.9) The closed-form solution for Eq. 12 reveals that the optimal merged update for each rank-1 unit is a re-scaling of the current unit, i.e., B(j)⋆G = γ(j)⋆B(j)G. (B.3.6) The coefficient γ(j)⋆ is determined by the energy of the current-task covariance S t and the accumulated past covariance S 1:t−1 along the same input-side direction A(j)G. (B.3.6) The visualization of loss landscapes demonstrates that the isolated direction lies in a region where the loss on D1 varies minimally while the loss on D2 changes obviously, indicating that it indeed captures truly task-specific knowledge that effectively adapts to the new task with limited interference to previous tasks. (Fig. 12) The general direction induces large loss variations on both tasks, reflecting its role as a shared subspace that captures general knowledge. (Fig. 12) The isolated direction lies in a region where the loss on D1 varies minimally while the loss on D2 changes obviously, indicating that it indeed captures truly task-specific knowledge that effectively adapts to the new task with limited interference to previous tasks. (Fig. 12) The visualization of the loss landscapes demonstrates that LoDA’s effectiveness on finding a joint optimum during continual adaptation. (Fig. 6) The visualization of the loss landscapes on the first task D1, the second task D2, and two tasks on 10S-ImageNetA. (Fig. 12) The general direction induces large loss variations on both tasks, reflecting its role as a shared subspace that captures general knowledge. (Fig. 6) The isolated direction lies in a region where the loss on D1 varies minimally while the loss on D2 changes obviously, indicating that it indeed captures truly task-specific knowledge that effectively adapts to the new task with limited interference to previous tasks. (Fig. 6) The visualization of the loss landscapes along the general directions (LoRAG) and the isolated directions (LoRAI) on 10S-ImageNetA and 10SImageNetR datasets. (Fig. 12) The isolated direction lies in a region where the loss on D1 varies minimally while the loss on D2 changes obviously, indicating that it indeed captures truly task-specific knowledge that effectively adapts to the new task with limited interference to previous tasks. (Fig.

1. Better stability-plasticity trade-off management: The LoDA framework decomposes the update space into general and truly isolated subspaces by solving two energy-based objectives, which allows for a principled control over knowledge sharing and isolation, as visualized in Figure 12 where the isolated direction lies in a region where the loss on D1 varies minimally while the loss on D2 changes obviously.

2. Robust task adaptation via Gradient-Aligned Optimization (GAO): The GAO approach encourages gradient consistency across label-disjoint subsets, which is shown to improve robustness by ensuring that any future update whose gradient direction admits the cone decomposition (B.4.7) tends to induce a smaller increase or even a decrease on the current-task loss.

3. Joint optimum approximation for general updates: The method derives a closed-form rescaling matrix to recalibrate it by minimizing feature optimization errors on all tasks to approximate a joint optimum, which addresses the issue that the general update inevitably induces feature drift of old tasks when learning the new task.

4. Efficient and parameter-free inference: LoDA achieves an efficient inference stage because, after training, we derive a closed-form diagonal rescaling matrix ΛG to rescale each rank-1 unit in LoRAG, allowing the system to operate with no extra parameters and overhead for inference (0.00M, 17.60 GFLOPs), while achieving the best performances.

5. Scalability across model sizes: The framework is model size-agnostic CL framework that brings consistent performance improvements, demonstrating that LoDA delivers consistent and clear improvements over Seq-LoRA across 4 model sizes on both ALast and AAvg, with gains ranging from +7.21% to +28.52% on ALast.

Continual Learning (CL) requires models to sequentially adapt to new tasks without forgetting old knowledge. Recently, Low-Rank Adaptation (LoRA), a representative Parameter-Efficient Fine-Tuning (PEFT) method, has gained increasing attention in CL. Several LoRA-based CL methods reduce interference across tasks by separating their update spaces, typically building the new space from the estimated null space of past tasks. However, they (i) overlook task-shared directions, which suppresses knowledge transfer, and (ii) fail to capture truly effective task-specific directions since these ``null bases" of old tasks can remain nearly inactive for new task under correlated tasks. To address this, we study LoRA learning capability from a projection energy perspective, and propose Low-rank Decomposition and Adaptation (LoDA). It performs a task-driven decomposition to build general and truly task-specific LoRA subspaces by solving two energy-based objectives, decoupling directions for knowledge sharing and isolation. LoDA fixes LoRA down-projections on two subspaces and learns robust up-projections via a Gradient-Aligned Optimization (GAO) approach. After each task, before integrating the LoRA updates into the backbone, LoDA derives a closed-form recalibration for the general update, approximating a feature-level joint optimum along this task-shared direction. Experiments indicate that LoDA outperforms existing CL methods. Our code is available at https://github.com/HHHLF/LoDA ICML2026.