Task-Oriented Rank Adaptation for Continual Learning in Text Classification
cs.CL
Submitted: 2026-10-01
Updated: 2026-10-01
Comments: Preprint submitted to CIARP2026
License: http://creativecommons.org/licenses/by/4.0/
The gist: Continual learning (CL) in text classification faces two critical challenges: catastrophic forgetting and negative transfer across sequential tasks.
Terminology
Abstract
Continual learning (CL) in text classification faces two critical challenges: catastrophic forgetting and negative transfer across sequential tasks. Parameter-Efficient Fine-Tuning (PEFT) methods such as LoRA enable efficient adaptation by learning low-rank updates of the model parameters. However, these compact representations are normally trained in isolation, limiting their reuse across related tasks. We introduce Task-Oriented Rank Adaptation (TORA), a geometric routing framework that leverages the low-rank structure of LoRA adapters to decide whether to transfer knowledge from the most compatible expert (Boosting) or isolate the new task (Shielding) based on structural similarity. Evaluated across 15 diverse text classification benchmarks, TORA consistently avoids harmful routing decisions: compatible tasks exceed their isolated performance while reducing training time, and structurally distant tasks are protected from interference with no loss in accuracy. With a single geometric threshold and no reliance on task identities or predefined sequences, TORA provides a simple and effective approach for dynamic adapter routing in sequential text classification systems.
Sources
- On Tiny Episodic Memories in Continual Learning
- Gradient Descent Happens in a Tiny Subspace
- Scaling Down to Scale Up: A Guide to Parameter-Efficient Fine-Tuning
- An overview of gradient descent optimization algorithms
- DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering