MALT: Lightweight Curvature-Aware Muon via Diagonal Preconditioning
Tongle Wu, Huanyu Dong, Ying Sun, Ziye Ma
cs.LG
Submitted: 2026-08-05
Code: https://github.com/karpathy/nanoGPT
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Terminology
Sources
- Disentangling Adaptive Gradient Methods from Learning Rates
- The Polar Express: Optimal Matrix Sign Methods and Their Application to the Muon Algorithm
- Scalable Second Order Optimization for Deep Learning
- On the Opportunities and Risks of Foundation Models
- MuonEq: Balancing Before Orthogonalization with Lightweight Equilibration
- TrasMuon: Trust-Region Adaptive Scaling for Orthogonalized Momentum Optimizers
- DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models
- RMNP: Row-Momentum Normalized Preconditioning for Scalable Matrix-Based Optimization
- Towards Quantifying the Hessian Structure of Neural Networks
- The Newton-Muon Optimizer
- What Really Matters in Matrix-Whitening Optimizers?
- Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities
- Accelerating Newton-Schulz Iteration for Orthogonalization via Chebyshev-type Polynomials
- Low-rank Orthogonalization for Large-scale Matrix Optimization with Applications to Foundation Model Training
- Spectra: Rethinking Optimizers for LLMs Under Spectral Anisotropy
- Scaling Laws for Neural Language Models
- PolarGrad: A Class of Matrix-Gradient Optimizers from a Unifying Preconditioning Perspective
- Variance-Adaptive Muon: Pre-Orthogonalization Variance Modulation for Efficient Language Model Pretraining
- NorMuon: Making Muon more efficient and scalable
- Muown: Row-Norm Control for Muon Optimization
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks