Iterative Orthogonalization Scaling Laws
summary
The gist
The paper investigates the scaling behavior of hyper-parameters in the muon optimizer, specifically concerning weight decay and learning rate…
In short
The paper investigates how hyperparameters scale in Muon's optimizer, focusing on weight decay and learning rate. It analyzes Muon's iterative orthogonalization method, which uses a polynomial to manipulate matrix singular values. The study finds that this method struggles at larger model sizes because the scaling of singular values complicates coefficient selection, suggesting the need to adjust the polynomial for better performance.
Key concepts
- Iterative Orthogonalization
- Muon uses a clever polynomial involving odd-degree matrix exponents to iteratively orthogonalize momentum. This process transforms a matrix operation into an expression like U(a+bS2+cS4)SV^T, aiming to improve the optimization path before applying it to the model.
- Singular Value Scaling Law
- This law mathematically describes how singular values behave in large random matrices. The paper proves that for a gradient normalized by its Frobenius norm, the singular values scale proportionally to 1/√d, where d is the matrix dimension. This scaling is crucial for understanding the behavior at different scales.
- Marchenko-Pastur Theorem
- This theorem provides a mathematical framework for understanding the distribution of eigenvalues (related to singular values) in matrices with i.i.d entries and a fixed aspect ratio. It allows researchers to calculate the exact scaling behavior of these values, confirming that singular values scale as 1/√in.
- Unlikely Directions
- These refer to specific directions within the parameter space that are less probable or less likely to be captured by the current polynomial approximation in Muon's preconditioner. The paper suggests that at larger scales, capturing these directions might be necessary for optimal performance.
Terminology used across episodes
This episode discusses
The paper
Iterative Orthogonalization Scaling Laws · Read on arXiv
Devan Selvaraj
The muon optimizer has picked up much attention as of late as a possible replacement to the seemingly omnipresent Adam optimizer. Recently, care has been taken to document the scaling laws of hyper-parameters under muon such as weight decay and learning rate. However, at much larger scales the iterative orthogonalization procedure present in muon may suffer a possible issue as the singular values of random matrices shrink with scale. This paper shows this scaling behavior theoretically and empirically on random matrices but does not suggest what to do about it.
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: We're looking at the paper: "Iterative Orthogonalization Scaling Laws".
Jane: Here's the gist of the paper. The paper investigates the scaling behavior of hyper-parameters in the muon optimizer, specifically concerning weight decay and learning rate, and identifies a potential issue with its iterative orthogonalization procedure at larger scales <ref:2505.04005#pg2>.
Tom: This concern arises because the singular values of random matrices shrink with scale, which complicates the choice of coefficients for the polynomial used in muon's preconditioner <ref:2505.04005#pg3>.
Jane: Iterative Orthogonalization Mechanics Muon’s preconditioner employs a clever polynomial to iteratively orthogonalize the momentum before applying it to the model <ref:2505.04005#pg4>.
Tom: This procedure involves using odd-degree exponents of the matrix to directly apply a polynomial to the singular values of a matrix, resulting in an expression like U(a + bS2 + cS4)V TUSV T <ref:2505.04005#pg4>.
Paper summary: Tom: So to wrap up the "Iterative Orthogonalization Scaling Laws" paper by Devan Selvaraj, it looks like the main message is that we need to be careful when scaling Muon to really large models <ref:2505.04005#pg1>.
Jane: Right. The paper explores how the iterative orthogonalization procedure might suffer at much larger scales because the singular values of random matrices shrink with scale <ref:2505.04005#pg1>.
Lu: They show this scaling behavior both theoretically using the Marchenko-Pastur Theorem and empirically by looking at randomly initialized matrices from size one hundred twenty-eight up to eight thousand one hundred ninety-two <ref:2505.04005#pg3>.
Meng: The authors conclude that they don't suggest a specific fix for this issue, but they point out that either the slope of the polynomial at zero or the number of iterations should increase with scale <ref:2505.04005#pg5>.
Tom: It seems like the main implication is that for now, we should expect less than optimal performance if we push model sizes beyond eight thousand parameters <ref:2505.04005#pg3>.
Jane: So, the authors aren't saying this is a crisis right now, but they are definitely flagging that at scales over eight thousand parameters things will get worse <ref:2505.04005#pg3>.
Conclusion: Tom: So we've been looking at how Muon handles its math, and now we get to look at the scaling of that iterative process in this paper, "Iterative Orthogonalization Scaling Laws," by Devan Selvaraj and others.
Jane: That paper is really digging into whether this clever way Muon cleans up the momentum before it hits the model breaks down when you make things bigger.
Tom: Exactly. It looks at how weight decay and learning rate scale, but specifically focuses on that iterative orthogonalization part, which is basically Muon's secret sauce for getting good updates.
Lu: What they found is that because of how the singular values of random matrices shrink as the size increases, the math behind those polynomial coefficients gets tricky.
Meng: So what does that mean for us when we try to train massive AI models? Is this going to crash our hardware or something?
Jane: It suggests that if you keep pushing model sizes way past eight thousand parameters, you might not get optimal performance because the method can't capture the right directions anymore.
Tom: Right. The authors are saying that either we have to change how the polynomial behaves as it gets bigger, maybe make its slope steeper at zero, or just run more iterations to catch those less likely directions.
Lu: The big question they leave is whether capturing those less likely directions is actually something we want when we're training models this massive.
Jane: It’s a trade-off, isn't it? We get better accuracy on smaller models, but then the complexity and potential precision issues kick in at the very top end of model size.
More episodes
- 2603.28595-Optimistic Actor-Critic with Parametric Policies for Linear Markov Decision Processes
- 2603.29765-Training-Free Dynamic Upcycling of Expert Language Models
- 2605.00778-Observable Performance Does Not Fully Reflect Adaptive System Organization: A Multi-Level Analysis of Gait Dynamics Under Occlusal Constraint
- 2605.06906-TraXion: Rethinking Pre-training Frameworks for Mobility and Beyond
- 2605.09165-Sparse Layers are Critical to Scaling Looped Language Models
- 2605.09439-Inverse Design for Conditional Distribution Matching
- 2605.10313-Signature Approach for Contextual Bandits with Nonlinear and Path-dependent Rewards: Regret Bound and Statistical Inference
- 2605.10878-Minimum Neural Weight Norm = Kolmogorov Complexity
- 2605.11617-MIST: Reliable Streaming Decision Trees for Online Class-Incremental Learning via McDiarmid Bound
- 2605.15085-From Data to Action: Accelerating Refinery Optimization with AI