Hippocampal consolidation without replay in fully fine-tuned networks
cs.AI
Submitted: 2026-11-01
Updated: 2026-11-01
Comments: 37 pages, 5 figures, 1 table; Extended Data and Supplementary Information included. Code: https://github.com/kkt94/deltahippo
Code: https://github.com/kkt94/deltahippo
License: http://creativecommons.org/licenses/by/4.0/
The gist: Networks that keep learning overwrite what they know.
Terminology
Abstract
Networks that keep learning overwrite what they know. Prevailing remedies replay stored data, distil from old models or freeze the backbone. The brain suggests a different division of labour: the hippocampus indexes where experience lies in cortex while the cortex keeps learning. We show that this division lets a fully trainable network keep learning without cross-task replay, a teacher or inference-time memory. After each task, a training-only hippocampal memory indexes the input geometry of every group of synapses sharing an input, residual energy included, and limits each synapse's movement per direction before and after the optimizer. This class-free geometry carries most of the protection; with conflict detection on top, the method cuts absorption of established classes by near-duplicates several-fold against every compared method. To our knowledge, this is the first demonstration of one principle keeping whole networks learning across four architecture families and models up to 8 billion parameters. Methods that freeze the backbone hold up only as far as its representation already separates the classes; here the whole network keeps learning to tell near-duplicates apart, leading the frozen-backbone baseline by 17.5 points on TACRED. Without the memory, much old knowledge stays decodable but beyond the network's reach.
Sources
- Sculpting Subspaces: Constrained Full Fine-Tuning in LLMs for Continual Learning
- When Forgetting is not Catastrophic: On the Mechanics of Spurious Forgetting
- Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer
- RoBERTa: A Robustly Optimized BERT Pretraining Approach
- A Tropical Geometry View of Forgetting: A Per-Unit Projector for Knowledge-Preserving Fine-Tuning
- PLATE: Plasticity-Tunable Efficient Adapters for Geometry-Aware Continual Learning
- Representation Finetuning for Continual Learning
- Geometry Conflict: Explaining and Controlling Forgetting in LLM Continual Post-Training
- Three scenarios for continual learning
Related papers
- Multi-Objective Structured Pruning of LLMs for Latency and Model Size Optimization
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Embeddings for Preferences, Not Semantics
- Can We Trust Item Response Theory for AI Evaluation?
- Diagnosing Task Insensitivity in Language Agents