Watch Your Step: Adam's Margin Maximization Bias Depends on the Learning Rate Schedule
cs.LG, stat.ML
Submitted: 2026-11-01
Updated: 2026-11-13
Code: https://github.com/EitanGronich/adam-implicit-bias-lr-schedule
Terminology
Sources
- Towards Understanding Adam Convergence on Highly Degenerate Polynomials
- The Geometry of Sign Gradient Descent
- Old Optimizer, New Norm: An Anthology
- Implicit Bias of Gradient Descent for Non-Homogeneous Deep Networks
- Implicit Bias of Spectral Descent and Muon on Multiclass Separable Data
- ImpMIA: Leveraging Implicit Bias for Membership Inference Attack
- The Implicit Bias of Adam and Muon on Smooth Homogeneous Neural Networks
- Which Optimizer Leaks Most? Optimizer Implicit Bias as a Training-Data Privacy Controller
- Gradient descent aligns the layers of deep linear networks
- The Asymmetric Maximum Margin Bias of Quasi-Homogeneous Neural Networks
- Scaling Laws for Gradient Descent and Sign Descent for Linear Bigram Models under Zipf's Law
- Gradient Descent Maximizes the Margin of Homogeneous Neural Networks
- Flavors of Margin: Implicit Bias of Steepest Descent in Homogeneous Neural Networks
- Kernel and Rich Regimes in Overparametrized Models
- A Unifying View on Implicit Bias in Training Linear Neural Networks
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis