Factored One-sided Adam-Moment for Practical and Scalable SOAP
cs.LG
Submitted: 2026-11-12
Updated: 2026-11-12
Code: https://github.com/naver-ai/foam
Terminology
Sources
- On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes
- The Polar Express: Optimal Matrix Sign Methods and Their Application to the Muon Algorithm
- Scalable Second Order Optimization for Deep Learning
- Benchmarking Neural Network Training Algorithms
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Kronecker-Factored Approximate Curvature for Modern Neural Network Architectures
- Purifying Shampoo: Investigating Shampoo's Heuristics by Decomposing its Preconditioner
- What Really Matters in Matrix-Whitening Optimizers?
- A Stable Whitening Optimizer for Efficient Neural Network Training
- Fast Approximate Natural Gradient Descent in a Kronecker-factored Eigenbasis
- Towards Efficient Optimizer Design for LLM via Structured Fisher Approximation with a Low-Rank Extension
- Shampoo: Preconditioned Stochastic Tensor Optimization
- Gaussian Error Linear Units (GELUs)
- Measuring Mathematical Problem Solving With the MATH Dataset
- Training Compute-Optimal Large Language Models
- Accelerating Neural Network Training: An Analysis of the AlgoPerf Competition
- SOAP, Muon, and Beyond: Pushing LLM Pretraining Scales
- Adam: A Method for Stochastic Optimization
- Limitations of the Empirical Fisher Approximation for Natural Gradient Descent
- Heavy-Tailed Class Imbalance and Why Adam Outperforms Gradient Descent on Language Models
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis