A Deafening Silence: Catastrophic Forgetting Lives in the Output Embeddings of Tokens the Data Never Speaks
cs.CL, cs.AI
Submitted: 2026-10-07
Updated: 2026-10-07
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Terminology
Sources
- Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning
- LoRA Learns Less and Forgets Less
- MoFO: Momentum-Filtered Optimizer for Mitigating Forgetting in LLM Fine-Tuning
- On Empirical Comparisons of Optimizers for Deep Learning
- Investigating Catastrophic Forgetting During Continual Training for Neural Machine Translation
- Scalable Strategies for Continual Learning with Replay
- Catastrophic Forgetting is Low-Rank: A Function-Space Theory for Continual Adaptation
- LoRA: Low-Rank Adaptation of Large Language Models
- Editing Models with Task Arithmetic
- Overcoming catastrophic forgetting in neural networks
- Continual Learning Optimizations for Auto-regressive Decoder of Multilingual ASR systems
- Mechanistic Analysis of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning
- Class-incremental learning: survey and performance evaluation on image classification
- Gradient Projection Memory for Continual Learning
- Orthogonal Subspace Learning for Language Model Continual Learning
- Robust fine-tuning of zero-shot models
- Large Scale Incremental Learning
- Tensor Programs IVb: Adaptive Optimization in the Infinite-Width Limit
- Spurious Forgetting in Continual Learning of Language Models
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering