Optimizer Geometry Sets the Pace: Spectral Learning Dynamics in Matrix Factorization
cs.LG
Submitted: 2026-10-03
Updated: 2026-10-03
Sources
- On the Implicit Bias of Adam
- Muon Optimizes Under Spectral Norm Constraints
- To Use or not to Use Muon: How Simplicity Bias in Optimizers Matters
- Clarifying Shampoo: Adapting Spectral Descent to Stochasticity and the Parameter Trajectory
- Continuous-Time Analysis of Adaptive Optimization and Normalization
- The Implicit Bias of Adam and Muon on Smooth Homogeneous Neural Networks
- The Quantization Benefits of Residual-Free Transformers
- Uniform Spectral Growth and Convergence of Muon in LoRA-Style Matrix Factorization
- Depth Without the Magic: Inductive Bias of Natural Gradient Descent
- SOAP, Muon, and Beyond: Pushing LLM Pretraining Scales
- Adam: A Method for Stochastic Optimization
- Muon is Scalable for LLM Training
- Reassessing Muon for Matrix Factorization
- Muon learns balanced solutions in matrix factorization without slow saddle-to-saddle dynamics
- Analysis of Convolutions, Non-linearity and Depth in Graph Neural Networks using Neural Tangent Kernel
- Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
- Kimi K2: Open Agentic Intelligence
- Implicit Bias of AdamW: $\ell_\infty$ Norm Constrained Optimization
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks