BISECT: Bilevel Spectral Decomposition for Low-Rank Continual Learning
cs.LG
Submitted: 2026-11-02
Updated: 2026-11-02
License: http://creativecommons.org/licenses/by-sa/4.0/
The gist: Exemplar-free continual learning on pre-trained Vision Transformers increasingly relies on Low-Rank Adaptation (LoRA)…
Terminology
Abstract
Exemplar-free continual learning on pre-trained Vision Transformers increasingly relies on Low-Rank Adaptation (LoRA), yet prevailing methods govern the low-rank update space through strict subspace isolation. By projecting each incoming task into the approximate null space of historical token activations, isolation-based learners equate any cross-task feature overlap with catastrophic interference. We show that this orthogonality premise induces a structural bottleneck we term spectral starvation: when down-projection bases are locked before task optimization, the gradient norm reaching the trainable up-projection is upper-bounded by the energy of current-task features captured by the chosen basis. Because natural visual tasks share low- and mid-level covariance structure, historical null tails contain barely 5.5% of incoming task variance, starving new-class plasticity while blocking positive consolidation along shared directions. We propose BISECT (Bilevel Spectral Decomposition), a continual LoRA framework that partitions a fixed rank- r budget into two orthogonal functional blocks: an invariant carrier spanned by the dominant modes of the joint historical-and-current Gram matrix, and a novelty complement extracted via generalized Rayleigh whitening followed by orthogonal complement projection. After single-task training, BISECT folds both blocks back into the frozen backbone using a closed-form precision gate that scales each rank-1 invariant atom by its current-to-joint Rayleigh ratio while preserving novelty atoms at unit gain. Across eight incremental settings on ImageNet-R, ImageNet-A, CIFAR-100, CUB-200, and DomainNet, BISECT outperforms prompt, adapter, and LoRA baselines with zero extra inference parameters or FLOPs, and remains complementary to sparse analytic expansion.
Sources
- Continual Learning with Pre-Trained Models: A Survey
- Structural features of the fly olfactory circuit mitigate the stability-plasticity dilemma in continual learning
- Dynamic Cross-Modal Prompt Generation for Multimodal Continual Instruction Tuning
- Miles: Metric Learning with Expandable Subspace for Pre-Trained Model-Based Class-Incremental Learning
- CKAA: Cross-subspace Knowledge Alignment and Aggregation for Robust Continual Learning
- One Adapter, Many Tasks: Task-Conditioned Feature Transformations for Continual Learning
- SDS-LoRA: Overcoming Anisotropic Gradient Scaling in Low-Rank Adaptation
- SD-LoRA: Scalable Decoupled Low-Rank Adaptation for Class Incremental Learning
- Continuous Subspace Optimization for Continual Learning
- Train with Perturbation, Infer after Merging: A Two-Stage Framework for Continual Learning
- SplitLoRA: Balancing Stability and Plasticity in Continual Learning Through Gradient Space Splitting
- Task-Driven Subspace Decomposition for Knowledge Sharing and Isolation in LoRA-based Continual Learning
- Janus-LoRA: A Balanced Low-Rank Adaptation for Continual Learning
- KeepLoRA++: Continual Learning with Layer-Scaled Residual Gradient Adaptation
- LoRA in LoRA: Towards Parameter-Efficient Architecture Expansion for Continual Visual Instruction Tuning
- Harmonious Parameter Adaptation in Continual Visual Instruction Tuning for Safety-Aligned MLLMs
- ACE-LoRA: Adaptive Orthogonal Decoupling for Continual Image Editing
- Spectral-Aware Analytic Class-Incremental Learning for Long-Tailed Distributions
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis