Weight-norm Criticality: A Mechanism for Loss Spikes Induced by the Normalization and Weight Decay
cs.LG, cs.NE
Submitted: 2026-07-23
Updated: 2026-11-11
Terminology
Sources
- Layer Normalization
- Adaptive Preconditioners Trigger Loss Spikes in Adam
- A very short proof of Cauchy's interlace theorem for eigenvalues of Hermitian matrices
- The large learning rate phase of deep learning: the catapult mechanism
- An Exponential Learning Rate Schedule for Deep Learning
- Decoupled Weight Decay Regularization
- A Theory on Adam Instability in Large-Scale Machine Learning
- Llama 2: Open Foundation and Fine-Tuned Chat Models
- L2 Regularization versus Batch and Weight Normalization
- A Walk with SGD
- Anchor function: a type of benchmark functions for studying language models
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis