High dimensional theory of two-phase optimizers
cs.LG, math.ST, stat.TH
Submitted: 2026-03-27
Updated: 2026-10-05
Terminology
Sources
- Understanding Outer Optimizers in Local SGD: Learning Rates, Momentum, and Acceleration
- DiLoCo: Distributed Low-Communication Training of Language Models
- Communication-Efficient Language Model Training Scales Reliably and Robustly: Scaling Laws for DiLoCo
- SNOO: Step-K Nesterov Outer Optimizer - The Surprising Effectiveness of Nesterov Momentum Applied to Pseudo-Gradients
- High dimensional analysis reveals conservative sharpening and a stochastic edge of stability
- Smoothing DiLoCo with Primal Averaging for Faster Training of LLMs
- Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability
- Adaptive Gradient Methods at the Edge of Stability
- Dimension-adapted Momentum Outscales SGD
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks