Mind the Drift: Diagonal Linear Networks Under Large Learning Rates
cs.LG
Submitted: 2026-10-05
Updated: 2026-10-05
Code: https://github.com/nelaturuharsha/TurboPrune
Sources
- SGD with Large Step Sizes Learns Sparse Features
- Understanding Gradient Descent on Edge of Stability in Deep Learning
- Stable Signal Recovery from Incomplete and Inaccurate Measurements
- More is Less: Inducing Sparsity via Overparameterization
- Robust Implicit Regularization via Weight Normalization
- Understanding Optimization in Deep Learning with Central Flows
- Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability
- Adaptive Gradient Methods at the Edge of Stability
- Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stability
- Conflicting Biases at the Edge of Stability: Norm versus Sharpness Regularization
- Flatland: The Adventures of Gradient Descent with Large Step Sizes
- Large Spikes in Stochastic Gradient Descent: A Large-Deviations View
- Non-Euclidean Gradient Descent Operates at the Edge of Stability
- Gradient Descent Monotonically Decreases the Sharpness of Gradient Flow Solutions in Scalar Networks and Beyond
- Gradient Descent with Large Step Sizes: Chaos and Fractal Convergence Region
- Implicit Bias of SGD in Multivariate ReLU Networks: Effective Width Collapse
- Generalization Below the Edge of Stability: The Role of Data Geometry
- Rod Flow: A Continuous-Time Model for Gradient Descent at the Edge of Stability
- Singular-limit analysis of gradient descent with noise injection
- Generalization at the Edge of Stability
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks