CoDe-LoRA: Mitigating the Orthogonality Dilemma in Continual Learning of LLMs via Knowledge Consolidation and Decoupling

arXiv:2610.08312 · cs.AI, cs.CL, cs.LG · Submitted 2026-10-06 · Read on arXiv

cs.AI, cs.CL, cs.LG

Submitted: 2026-10-06

Updated: 2026-10-06

Code: https://github.com/Estrellajer/CoDe-LoRA

Terminology

• Continual Learning — 1× in this paper · explained in 2 episodes
• Decoupling — 1× in this paper · explained in 2 episodes

Sources

• LoRA: Low-Rank Adaptation of Large Language Models
• LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Composition
• The Power of Scale for Parameter-Efficient Prompt Tuning
• Prefix-Tuning: Optimizing Continuous Prompts for Generation
• Decoupled Weight Decay Regularization
• LFPT5: A Unified Framework for Lifelong Few-shot Language Learning Based on Prompt Tuning of T5
• Progressive Prompts: Continual Learning for Language Models
• Gradient Projection Memory for Continual Learning
• Llama 2: Open Foundation and Fine-Tuned Chat Models
• Orthogonal Subspace Learning for Language Model Continual Learning
• TRACE: A Comprehensive Benchmark for Continual Learning in Large Language Models
• Finetuned Language Models Are Zero-Shot Learners
• Qwen3 Technical Report

Related papers

• MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
• Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
• Agents' Last Exam
• The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
• MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
• Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems