A Step Towards Forgetting: Optimiser History and the Loss of Answer Mass
cs.CL, cs.AI
Submitted: 2026-10-02
Updated: 2026-10-02
Sources
- SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Model
- Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling
- MoFO: Momentum-Filtered Optimizer for Mitigating Forgetting in LLM Fine-Tuning
- Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability
- How Weight Resampling and Optimizers Shape the Dynamics of Continual Learning and Forgetting in Neural Networks
- OLMo: Accelerating the Science of Language Models
- Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution
- Revisiting Catastrophic Forgetting in Large Language Model Tuning
- Learning without Forgetting
- Cautious Optimizers: Improving Training with One Line of Code
- Gradient Episodic Memory for Continual Learning
- The AdEMAMix Optimizer: Better, Faster, Older
- Qwen2.5 Technical Report
- Qwen3 Technical Report
- Large Scale Incremental Learning
- Gradient Surgery for Multi-Task Learning
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering