Iterative Orthogonalization Scaling Laws
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: We're looking at the paper: "Iterative Orthogonalization Scaling Laws".
Jane: Here's the gist of the paper. The paper investigates the scaling behavior of hyper-parameters in the muon optimizer, specifically concerning weight decay and learning rate, and identifies a potential issue with its iterative orthogonalization procedure at larger scales <ref:2505.04005#pg2>.
Tom: This concern arises because the singular values of random matrices shrink with scale, which complicates the choice of coefficients for the polynomial used in muon's preconditioner <ref:2505.04005#pg3>.
Jane: Iterative Orthogonalization Mechanics Muon’s preconditioner employs a clever polynomial to iteratively orthogonalize the momentum before applying it to the model <ref:2505.04005#pg4>.
Tom: This procedure involves using odd-degree exponents of the matrix to directly apply a polynomial to the singular values of a matrix, resulting in an expression like U(a + bS2 + cS4)V TUSV T <ref:2505.04005#pg4>.
Paper summary: Tom: So to wrap up the "Iterative Orthogonalization Scaling Laws" paper by Devan Selvaraj, it looks like the main message is that we need to be careful when scaling Muon to really large models <ref:2505.04005#pg1>.
Jane: Right. The paper explores how the iterative orthogonalization procedure might suffer at much larger scales because the singular values of random matrices shrink with scale <ref:2505.04005#pg1>.
Lu: They show this scaling behavior both theoretically using the Marchenko-Pastur Theorem and empirically by looking at randomly initialized matrices from size one hundred twenty-eight up to eight thousand one hundred ninety-two <ref:2505.04005#pg3>.
Meng: The authors conclude that they don't suggest a specific fix for this issue, but they point out that either the slope of the polynomial at zero or the number of iterations should increase with scale <ref:2505.04005#pg5>.
Tom: It seems like the main implication is that for now, we should expect less than optimal performance if we push model sizes beyond eight thousand parameters <ref:2505.04005#pg3>.
Jane: So, the authors aren't saying this is a crisis right now, but they are definitely flagging that at scales over eight thousand parameters things will get worse <ref:2505.04005#pg3>.
Conclusion: Tom: So we've been looking at how Muon handles its math, and now we get to look at the scaling of that iterative process in this paper, "Iterative Orthogonalization Scaling Laws," by Devan Selvaraj and others.
Jane: That paper is really digging into whether this clever way Muon cleans up the momentum before it hits the model breaks down when you make things bigger.
Tom: Exactly. It looks at how weight decay and learning rate scale, but specifically focuses on that iterative orthogonalization part, which is basically Muon's secret sauce for getting good updates.
Lu: What they found is that because of how the singular values of random matrices shrink as the size increases, the math behind those polynomial coefficients gets tricky.
Meng: So what does that mean for us when we try to train massive AI models? Is this going to crash our hardware or something?
Jane: It suggests that if you keep pushing model sizes way past eight thousand parameters, you might not get optimal performance because the method can't capture the right directions anymore.
Tom: Right. The authors are saying that either we have to change how the polynomial behaves as it gets bigger, maybe make its slope steeper at zero, or just run more iterations to catch those less likely directions.
Lu: The big question they leave is whether capturing those less likely directions is actually something we want when we're training models this massive.
Jane: It’s a trade-off, isn't it? We get better accuracy on smaller models, but then the complexity and potential precision issues kick in at the very top end of model size.
Devan Selvaraj
cs.LG
Submitted: 2025-05-06
Updated: 2026-11-10
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 72/100
The gist: The paper investigates the scaling behavior of hyper-parameters in the muon optimizer, specifically concerning weight decay and learning rate…
Key concepts
- Iterative Orthogonalization
- Muon uses a clever polynomial involving odd-degree matrix exponents to iteratively orthogonalize momentum. This process transforms a matrix operation into an expression like U(a+bS2+cS4)SV^T, aiming to improve the optimization path before applying it to the model.
- Singular Value Scaling Law
- This law mathematically describes how singular values behave in large random matrices. The paper proves that for a gradient normalized by its Frobenius norm, the singular values scale proportionally to 1/√d, where d is the matrix dimension. This scaling is crucial for understanding the behavior at different scales.
- Marchenko-Pastur Theorem
- This theorem provides a mathematical framework for understanding the distribution of eigenvalues (related to singular values) in matrices with i.i.d entries and a fixed aspect ratio. It allows researchers to calculate the exact scaling behavior of these values, confirming that singular values scale as 1/√in.
- Unlikely Directions
- These refer to specific directions within the parameter space that are less probable or less likely to be captured by the current polynomial approximation in Muon's preconditioner. The paper suggests that at larger scales, capturing these directions might be necessary for optimal performance.
Terminology
Summary
The paper investigates the scaling behavior of hyper-parameters in the muon optimizer, specifically concerning weight decay and learning rate, and identifies a potential issue with its iterative orthogonalization procedure at larger scales <ref:2505. p. 0 Recently,
care has been taken to document the scaling laws of hyper-parameters under muon such as weight decay and learning rate. However, at much larger
scales the iterative orthogonalization procedure present in muon may suffer a possible issue as the singular values of random matrices shrink with
scale.Where the paper says it
Iterative Orthogonalization Mechanics
Muon’s preconditioner employs a clever polynomial to iteratively orthogonalize the momentum before applying it to the model <ref:2505.04005#pg4>. This procedure involves using odd-degree exponents of the matrix to directly apply a polynomial to the singular values of a matrix, resulting in an expression like U(a + bS2 + cS4)V TUSV T <ref:2505. p. 1 2 Iterative Orthogonalization
Muon’s preconditioner uses a clever polynomial to iteratively orthgonalize the
momentum before applying it to the model. Using odd-degree exponents of the
matrix, you can directly apply a polynomial to the singular values of a matrix
as follows:
G
′ = aG + b(GGT
)G + c(GGT
)(GGT
)G
= (aI + b(GGT
) + c(GGT
)(GGT
))G
= (aUUT + bUS2U
T + cUS4U
T
)USV T
= U(a + bS2 + cS4
)U
TUSV T
= U(a + bS2 + cS4
)SV T
= U(aS + bS3 + cS5
)V
T
By normalizing the matrix with it’s Frobenius norm, it’s guaranteed that
the singular values will fall between [0, 1]. 2 Iterative Orthogonalization
Muon’s preconditioner uses a clever polynomial to iteratively orthgonalize the
momentum before applying it to the model. Using odd-degree exponents of the
matrix, you can directly apply a polynomial to the singular values of a matrix
as follows:
G
′ = aG + b(GGT
)G + c(GGT
)(GGT
)G
= (aI + b(GGT
) + c(GGT
)(GGT
))G
= (aUUT + bUS2U
T + cUS4U
T
)USV T
= U(a + bS2 + cS4
)U
TUSV T
= U(a + bS2 + cS4
)SV T
= U(aS + bS3 + cS5
)V
T
By normalizing the matrix with it’s Frobenius norm, it’s guaranteed that
the singular values will fall between [0, 1]. The procedure can then be applied iteratively to progressively orthogonalize the
matrix. 1 Muon Orthgonalization
In muon, the coefficients of the polynomial are chosen to increase steeply at 0 to
1 to minimize the number of steps taken.Where the paper says it
Where the paper says it
Where the paper says it
Scaling Behavior and Empirical Evidence
The expected distribution of singular values changes predictably with size, appearing to shrink with the square root of the smallest dimension <ref:2505.04005#pg4>. Empirically, this scaling behavior can be observed in randomly initialized matrices across 8-512 matrices of size 128 through 8192 <ref:2505. p. 2 3 Singular Value Scaling Law
Empirically, this scaling behavior can be observed in randomly initialized matrices. To show the distribution of singular values 8-512 matrices of size 128
through 8192 are randomly initialized N(0, 1), normalized and have the singular values computed. To show the distribution of singular values 8-512 matrices of size 128
through 8192 are randomly initialized N(0, 1), normalized and have the singular values computed. The distributions of the singular values of the largest and
smallest matrices are then plotted before and after applying the polynomial for
5 iterations. The distributions of the singular values of the largest and
smallest matrices are then plotted before and after applying the polynomial for
5 iterations. As can one can see, the higher matrix dimension has a heavier tail below
. As can one can see, the higher matrix dimension has a heavier tail below
. 6, showing the less likely directions are not being captured by the polynomial. p. 2 As can one can see, the higher matrix dimension has a heavier tail below
. 6, showing the less likely directions are not being captured by the polynomial.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Mathematical Scaling Proof
The exact scaling can be calculated mathematically using the Marchenko-Pastur Theorem <ref:2505.04005#pg7>. Theorem 1 (Normalized Singular Value Scaling Law) states that the singular values of a gradient normalized by its Frobenius norm scale 1/√d <ref:2505. p. 3 Theorem 1 (Normalized Singular Value Scaling Law). The singular values of
a gradient normalized by it’s Frobenius norm scale 1/
√
in d. The singular values of
a gradient normalized by it’s Frobenius norm scale 1/
√
in d. Assume the gradient entries are i. p. 3 The singular values of
a gradient normalized by it’s Frobenius norm scale 1/
√
in d. Assume the gradient entries are i. d
with mean 0, finite variance σ
2
, and a fixed aspect ratio γ = out d/in d the
singular values scale 1/
√
out d. Define initial matrix A
A ∼ N
in d×out d
(0, σ2
)
= UDV T
Define scaled matrix A¯
A¯ = A/√
out d
∼ N
in d×out d
(0, σ2
/out d)
= UDV¯ T
6 Theorem 1 (Normalized Singular Value Scaling Law). The singular values of
a gradient normalized by it’s Frobenius norm scale 1/
√
in d.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Implications for Large-Scale Training
The paper concludes that either the slope of the polynomial at 0 or the number of iterations should increase with scale to capture Either, the slope of the polynomial at 0 or the number of iterations should
increase with scale. However, it’s not clear if capturing these less likely directions
is actually desirable at larger scales. Kimi’s Moonlight 16B - the largest model trained with muon so far - only had
a model d of 2048, well within the range for Muon’s default of 5 iterations to
suffice. The issues presented in this paper will likely arise at model ds >8k, and will
at worse cause less than optimal performance. At the time of this paper, there have been no known model trains with Muon
at the model dimensions necessary to cause the issues predicted in this paper. Kimi’s Moonlight 16B - the largest model trained with muon so far - only had
a model d of 2048, well within the range for Muon’s default of 5 iterations to
suffice. Kimi’s Moonlight 16B - the largest model trained with muon so far - only had
a model d of 2048, well within the range for Muon’s default of 5 iterations to
suffice. The issues presented in this paper will likely arise at model ds >8k, and will
at worse cause less than optimal performance.unlikely directions
. It is not clear if capturing these less likely directions is desirable at larger scales. p. 3Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Appendix Details
Lemma 1 (Frobenius Norm Scaling Law) states that the Frobenius norm of a matrix scales √out d × in d assuming constant variance. Theorem 2 (Marchenko-Pastur Theorem) states that for a matrix with i. p. 5 ||W||F =
sX
i,j
W2
ij
≈
sX
i,j
N(0, σ2)
2
=
q
σ
2χ
2
in×out
∝ σ
√
out d × in d
Theorem 2 (Marchenko-Pastur Theorem). The Marchenko-Pastur Theorem[5]
states as in d → ∞ the empirical distribution of the singular values of a matrix with i. The Marchenko-Pastur Theorem[5]
states as in d → ∞ the empirical distribution of the singular values of a matrix with i. d entries with µ = 0, variance σ = ¯σ
2/out d, fixed aspect ratio
γ = out d/in d has singular values distributed:
ρ(s) =
p
((1 + √γ)
2 − s
2)(s
2 − |1 −
√γ|
2)
2πγs
Lemma 2 (Singular Value Scaling Law). The Marchenko-Pastur Theorem[5]
states as in d → ∞ the empirical distribution of the singular values of a matrix with i. d entries with µ = 0, variance σ = ¯σ
2/out d, fixed aspect ratio
γ = out d/in d has singular values distributed:
ρ(s) =
p
((1 + √γ)
2 − s
2)(s
2 − |1 −
√γ|
2)
2πγs
Lemma 2 (Singular Value Scaling Law). d
with mean 0, finite variance σ
2
, and a fixed aspect ratio γ = out d/in d the
singular values scale 1/
√
out d. Define initial matrix A
A ∼ N
in d×out d
(0, σ2
)
= UDV T
Define scaled matrix A¯
A¯ = A/√
out d
∼ N
in d×out d
(0, σ2
/out d)
= UDV¯ T
6 However, at much larger
scales the iterative orthogonalization procedure present in muon may suffer a possible issue as the singular values of random matrices shrink with
scale. This paper shows this scaling behavior theoretically and empirically
on random matrices but does not suggest what to do about it.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
References
[1] Essential AI,:, Ishaan Shah, Anthony M. Polloreno, Karl Stratos, Philip Monk, Adarsh Chaluvaraju, Andrew Hojel, Andrew Ma, Anil Thomas, Ashish Tanwer, Darsh J Shah, Khoi Nguyen, Kurt Smith, Michael Callahan,, Michael Pust,, Mohit Parmar,, Peter Rushton,, Platon Mazarakis,, Ritvik Kapila,, Saurabh Srivastava,, Somanshu Singla,, Tim Romanski,, Yash Vanjani,, and Ashish Vaswani. p. 4 References
[1] Essential AI, :, Ishaan Shah, Anthony M. Polloreno, Karl Stratos, Philip
Monk, Adarsh Chaluvaraju, Andrew Hojel, Andrew Ma, Anil Thomas,
Ashish Tanwer, Darsh J Shah, Khoi Nguyen, Kurt Smith, Michael Callahan, Michael Pust, Mohit Parmar, Peter Rushton, Platon Mazarakis, Ritvik
Kapila, Saurabh Srivastava, Somanshu Singla, Tim Romanski, Yash Vanjani, and Ashish Vaswani. Polloreno, Karl Stratos, Philip
Monk, Adarsh Chaluvaraju, Andrew Hojel, Andrew Ma, Anil Thomas,
Ashish Tanwer, Darsh J Shah, Khoi Nguyen, Kurt Smith, Michael Callahan, Michael Pust, Mohit Parmar, Peter Rushton, Platon Mazarakis, Ritvik
Kapila, Saurabh Srivastava, Somanshu Singla, Tim Romanski, Yash Vanjani, and Ashish Vaswani. Practical efficiency of muon for pretraining, 2025. Polloreno, Karl Stratos, Philip
Monk, Adarsh Chaluvaraju, Andrew Hojel, Andrew Ma, Anil Thomas,
Ashish Tanwer, Darsh J Shah, Khoi Nguyen, Kurt Smith, Michael Callahan, Michael Pust, Mohit Parmar, Peter Rushton, Platon Mazarakis, Ritvik
Kapila, Saurabh Srivastava, Somanshu Singla, Tim Romanski, Yash Vanjani, and Ashish Vaswani. Practical efficiency of muon for pretraining, 2025.Where the paper says it
Where the paper says it
Where the paper says it
[2] Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. p. 4 Practical efficiency of muon for pretraining, 2025. [2] Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. [2] Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. int8(): 8-bit matrix multiplication for transformers at scale, 2022. [2] Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. int8(): 8-bit matrix multiplication for transformers at scale, 2022.Where the paper says it
Where the paper says it
Where the paper says it
[3] Keller Jordan, Yuchen Jin, Vlado Boza,, Jiacheng You,, Franz Cesista,, Laker Newhouse,, and Jeremy Bernstein. p. 4 int8(): 8-bit matrix multiplication for transformers at scale, 2022. [3] Keller Jordan, Yuchen Jin, Vlado Boza, Jiacheng You, Franz Cesista, Laker
Newhouse, and Jeremy Bernstein. int8(): 8-bit matrix multiplication for transformers at scale, 2022. [3] Keller Jordan, Yuchen Jin, Vlado Boza, Jiacheng You, Franz Cesista, Laker
Newhouse, and Jeremy Bernstein. [3] Keller Jordan, Yuchen Jin, Vlado Boza, Jiacheng You, Franz Cesista, Laker
Newhouse, and Jeremy Bernstein. Muon: An optimizer for hidden layers in
neural networks, 2024. [3] Keller Jordan, Yuchen Jin, Vlado Boza, Jiacheng You, Franz Cesista, Laker
Newhouse, and Jeremy Bernstein. Muon: An optimizer for hidden layers in
neural networks, 2024.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
[4] Diederik P. p. 4 Muon: An optimizer for hidden layers in
neural networks, 2024. [4] Diederik P. Muon: An optimizer for hidden layers in
neural networks, 2024. [4] Diederik P. [4] Diederik P. Kingma and Jimmy Ba. [4] Diederik P. Kingma and Jimmy Ba. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
[5] V. A. Marˇcenko and L. p. 4 Adam: A method for stochastic optimization, 2017. Marˇcenko and L. Adam: A method for stochastic optimization, 2017. Marˇcenko and L. Distribution of eigenvalues for some sets
of random matrices. Matematicheskii Sbornik, 114(4):507–536, 1967.Where the paper says it
Where the paper says it
Where the paper says it
--- Page 1 ---
Iterative Orthogonalization Scaling Laws
Devan Selvaraj
research@ueaj.dev
May 2025
--- Page 2 ---
Muon’s preconditioner uses a clever polynomial to iteratively orthgonalize the momentum before applying it to the model. Using odd-degree exponents of the matrix, you can directly apply a polynomial to the singular values of a matrix as follows: G′ = aG + b(GGT)G + c(GGT)(GGT)G = (aI + b(GGT) + c(GGT)(GGT))G = (aUUT + bUS2U T + cUS4U T)USV T = U(a+bS2+cS4)SV T = U(aS + bS3 + cS5)V T <ref:2505. p. 1 2 Iterative Orthogonalization
Muon’s preconditioner uses a clever polynomial to iteratively orthgonalize the
momentum before applying it to the model. Using odd-degree exponents of the
matrix, you can directly apply a polynomial to the singular values of a matrix
as follows:
G
′ = aG + b(GGT
)G + c(GGT
)(GGT
)G
= (aI + b(GGT
) + c(GGT
)(GGT
))G
= (aUUT + bUS2U
T + cUS4U
T
)USV T
= U(a + bS2 + cS4
)U
TUSV T
= U(a + bS2 + cS4
)SV T
= U(aS + bS3 + cS5
)V
T
By normalizing the matrix with it’s Frobenius norm, it’s guaranteed that
the singular values will fall between [0, 1].Where the paper says it
--- Page 3 ---
Singular Value Scaling Law
Empirically, this scaling behavior can be observed in randomly initialized matrices. To show the distribution of singular values 8-512 matrices of size 128 through 8192 are randomly initialized N(0, 1), normalized and have the singular values computed <ref:2505. p. 2 3 Singular Value Scaling Law
Empirically, this scaling behavior can be observed in randomly initialized matrices. To show the distribution of singular values 8-512 matrices of size 128
through 8192 are randomly initialized N(0, 1), normalized and have the singular values computed. To show the distribution of singular values 8-512 matrices of size 128
through 8192 are randomly initialized N(0, 1), normalized and have the singular values computed. The distributions of the singular values of the largest and
smallest matrices are then plotted before and after applying the polynomial for
5 iterations. The distributions of the singular values of the largest and
smallest matrices are then plotted before and after applying the polynomial for
5 iterations. As can one can see, the higher matrix dimension has a heavier tail below
.Where the paper says it
Where the paper says it
Where the paper says it
--- Page 4 ---
3.1 Proof
The exact scaling can be calculated mathematically using the Marchenko-Pastur Theorem. Theorem 1 (Normalized Singular Value Scaling Law) The singular values of a gradient normalized by it’s Frobenius norm scale 1/√d. p. 3 Theorem 1 (Normalized Singular Value Scaling Law). The singular values of
a gradient normalized by it’s Frobenius norm scale 1/
√
in d. d
with mean 0, finite variance σ
2
, and a fixed aspect ratio γ = out d/in d the
singular values scale 1/
√
out d. Define initial matrix A
A ∼ N
in d×out d
(0, σ2
)
= UDV T
Define scaled matrix A¯
A¯ = A/√
out d
∼ N
in d×out d
(0, σ2
/out d)
= UDV¯ T
6 d with mean 0 finite variance σ
2
. Assume the in d/out d ratio remains constant with scale
Apply Frobenius Norm Scaling Law1 and ∇W as the gradient and SV DD
is the svd of the gradient returning only the diagonals. Assume the in d/out d ratio remains constant with scale
Apply Frobenius Norm Scaling Law1 and ∇W as the gradient and SV DD
is the svd of the gradient returning only the diagonals. SVDD(
∇W
||∇W||F
) ≈ SVDD(
Nin d×out d
(0, 1)
√
in d × out d
)
=
SVDD(Nin d×out d
(0, 1))
√
in d × out d
Apply Singular Value Scaling Law2
≈
ρ
√
out d
√
in d × out d
=
ρ
√
in d
4 Conclusion
As muon continues to gain traction and larger and larger experiments are conducted using the optimizer, it is important that any possible issues are foreseen in it’s scaling behavior prior to conducting experiments.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
--- Page 5 ---
Conclusion
As muon continues to gain traction and larger and larger experiments are conducted using the optimizer, it is important that any possible issues are foreseen in it’s scaling behavior prior to conducting experiments. The NS iteration method pioneered by muon shows promise but the current iteration requires tuning in order to capture the ”unlikely directions” that currently explain Muon’s success. p. 3 SVDD(
∇W
||∇W||F
) ≈ SVDD(
Nin d×out d
(0, 1)
√
in d × out d
)
=
SVDD(Nin d×out d
(0, 1))
√
in d × out d
Apply Singular Value Scaling Law2
≈
ρ
√
out d
√
in d × out d
=
ρ
√
in d
4 Conclusion
As muon continues to gain traction and larger and larger experiments are conducted using the optimizer, it is important that any possible issues are foreseen in it’s scaling behavior prior to conducting experiments. The NS iteration
method pioneered by muon shows promise but the current iteration requires tuning in order to capture the ”unlikely directions” that currently explain Muon’s
success. The NS iteration
method pioneered by muon shows promise but the current iteration requires tuning in order to capture the ”unlikely directions” that currently explain Muon’s
success. Either, the slope of the polynomial at 0 or the number of iterations should
increase with scale. Either, the slope of the polynomial at 0 or the number of iterations should
increase with scale. However, it’s not clear if capturing these less likely directions
is actually desirable at larger scales. However, it’s not clear if capturing these less likely directions
is actually desirable at larger scales. LLMs have been shown to pick up outlier
features on their own over around 7b parameters. LLMs have been shown to pick up outlier
features on their own over around 7b parameters. [2] Additionally, the higher
powers in the NS iterations may run into precision issues over a certain model
size, but this is likely way outside realistic model sizes. [2] Additionally, the higher
powers in the NS iterations may run into precision issues over a certain model
size, but this is likely way outside realistic model sizes. At the time of this paper, there have been no known model trains with Muon
at the model dimensions necessary to cause the issues predicted in this paper. At the time of this paper, there have been no known model trains with Muon
at the model dimensions necessary to cause the issues predicted in this paper. Kimi’s Moonlight 16B - the largest model trained with muon so far - only had
a model d of 2048, well within the range for Muon’s default of 5 iterations to
suffice. Kimi’s Moonlight 16B - the largest model trained with muon so far - only had
a model d of 2048, well within the range for Muon’s default of 5 iterations to
suffice. The issues presented in this paper will likely arise at model ds >8k, and will
at worse cause less than optimal performance. Kimi’s Moonlight 16B - the largest model trained with muon so far - only had
a model d of 2048, well within the range for Muon’s default of 5 iterations to
suffice. The issues presented in this paper will likely arise at model ds >8k, and will
at worse cause less than optimal performance.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
--- Page 6 ---
References
[1] Essential AI,:, Ishaan Shah, Anthony M. Polloreno, Karl Stratos,, Philip Monk,, Adarsh Chaluvaraju,, Andrew Hojel,, Andrew Ma,, Anil Thomas,, Ashish Tanwer,, Darsh J Shah,, Khoi Nguyen,, Kurt Smith, Michael Callahan,, Michael Pust,, Mohit Parmar,, Peter Rushton, Platon Mazarakis, Ritvik Kapila, Saurabh Srivastava,, Somanshu Singla,, Tim Romanski, Yash Vanjani,, and Ashish Vaswani. p. 4 References
[1] Essential AI, :, Ishaan Shah, Anthony M. Polloreno, Karl Stratos, Philip
Monk, Adarsh Chaluvaraju, Andrew Hojel, Andrew Ma, Anil Thomas,
Ashish Tanwer, Darsh J Shah, Khoi Nguyen, Kurt Smith, Michael Callahan, Michael Pust, Mohit Parmar, Peter Rushton, Platon Mazarakis, Ritvik
Kapila, Saurabh Srivastava, Somanshu Singla, Tim Romanski, Yash Vanjani, and Ashish Vaswani.Where the paper says it
[2] Tim Dettmers, Mike Lewis, Younes Belkada,, and Luke Zettlemoyer. p. 4 Practical efficiency of muon for pretraining, 2025. [2] Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer.Where the paper says it
[3] Keller Jordan, Yuchen Jin, Vlado Boza,, Jiacheng You,, Franz Cesista,, Laker Newhouse,, and Jeremy Bernstein. Muon: An optimizer for hidden layers in neural networks, 2024.
[4] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017.
[5] V. A. Marˇcenko and L. A. Pastur., Distribution of eigenvalues for some sets of random matrices, Matematicheskii Sbornik, 114(4):507–536, 1967. p. 4 Distribution of eigenvalues for some sets
of random matrices. Matematicheskii Sbornik, 114(4):507–536, 1967.Where the paper says it
--- Page 7 ---
5 Appendix
5.1 Theorems
Lemma 1 (Frobenius Norm Scaling Law) The Frobenius norm of a matrix scales √out d × in d assuming constant variance. Assume the elements of the gradient of a matrix are roughly normally distributed with mean 0 and σ2 variance. p. 5 Assume the elements of the gradient of a matrix are roughly normally distributed with mean 0 and σ
2
variance. ||W||F =
sX
i,j
W2
ij
≈
sX
i,j
N(0, σ2)
2
=
q
σ
2χ
2
in×out
∝ σ
√
out d × in d
Theorem 2 (Marchenko-Pastur Theorem). Assume the elements of the gradient of a matrix are roughly normally distributed with mean 0 and σ
2
variance. ||W||F =
sX
i,j
W2
ij
≈
sX
i,j
N(0, σ2)
2
=
q
σ
2χ
2
in×out
∝ σ
√
out d × in d
Theorem 2 (Marchenko-Pastur Theorem). Assume the elements of the gradient of a matrix are roughly normally distributed with mean 0 and σ
2
variance. ||W||F =
sX
i,j
W2
ij
≈
sX
i,j
N(0, σ2)
2
=
q
σ
2χ
2
in×out
∝ σ
√
out d × in d
Theorem 2 (Marchenko-Pastur Theorem). Assume the elements of the gradient of a matrix are roughly normally distributed with mean 0 and σ
2
variance. ||W||F =
sX
i,j
W2
ij
≈
sX
i,j
N(0, σ2)
2
=
q
σ
2χ
2
in×out
∝ σ
√
out d × in d
Theorem 2 (Marchenko-Pastur Theorem). Assume the elements of the gradient of a matrix are roughly normally distributed with mean 0 and σ
2
variance. ||W||F =
sX
i,j
W2
ij
≈
sX
i,j
N(0, σ2)
2
=
q
σ
2χ
2
in×out
∝ σ
√
out d × in d
Theorem 2 (Marchenko-Pastur Theorem). Assume the elements of the gradient of a matrix are roughly normally distributed with mean 0 and σ
2
variance. ||W||F =
sX
i,j
W2
ij
≈
sX
i,j
N(0, σ2)
2
=
q
σ
2χ
2
in×out
∝ σ
√
out d × in d
Theorem 2 (Marchenko-Pastur Theorem).Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
--- Page 8 ---
Lemma 1 (Frobenius Norm Scaling Law) The Frobenius norm of a matrix scales √out d × in d assuming constant variance. Assume the elements of the gradient of a matrix are roughly normally distributed with mean 0 and σ2 variance.
--- Page 9 ---
Lemma 1 (Frobenius Norm Scaling Law) The Frobenius norm of a matrix scales √out d × in d assuming constant variance. Assume the elements of the gradient of a matrix are roughly normally distributed with mean 0 and σ2 variance.
--- Page 10 ---
Lemma 1 (Frobenius Norm Scaling Law) The Frobenius norm of a matrix scales √out d × in d assuming constant variance. Assume the elements of the gradient of a matrix are roughly normally distributed with mean 0 and σ2 variance.
--- Page 11 ---
Lemma 1 (Frobenius Norm Scaling Law) The Frobenius norm of a matrix scales √out d × in d assuming constant variance. Assume the elements of the gradient of a matrix are roughly normally distributed with mean 0 and σ2 variance.
Pruning redundant reasoning tokens
Lemma 1 (Frobenius Norm Scaling Law) The Frobenius norm of a matrix scales √out d × in d assuming constant variance. Assume the elements of the gradient of a matrix are roughly normally distributed with mean 0 and σ2 variance.
How the training signal is built
The muon optimizer uses a specific mathematical formulation for its preconditioner. The procedure involves applying odd-degree exponents of the matrix to directly apply a polynomial to the singular values of a matrix. p. 1 2 Iterative Orthogonalization
Muon’s preconditioner uses a clever polynomial to iteratively orthgonalize the
momentum before applying it to the model. Using odd-degree exponents of the
matrix, you can directly apply a polynomial to the singular values of a matrix
as follows:
G
′ = aG + b(GGT
)G + c(GGT
)(GGT
)G
= (aI + b(GGT
) + c(GGT
)(GGT
))G
= (aUUT + bUS2U
T + cUS4U
T
)USV T
= U(a + bS2 + cS4
)U
TUSV T
= U(a + bS2 + cS4
)SV T
= U(aS + bS3 + cS5
)V
T
By normalizing the matrix with it’s Frobenius norm, it’s guaranteed that
the singular values will fall between [0, 1]. 2 Iterative Orthogonalization
Muon’s preconditioner uses a clever polynomial to iteratively orthgonalize the
momentum before applying it to the model. Using odd-degree exponents of the
matrix, you can directly apply a polynomial to the singular values of a matrix
as follows:
G
′ = aG + b(GGT
)G + c(GGT
)(GGT
)G
= (aI + b(GGT
) + c(GGT
)(GGT
))G
= (aUUT + bUS2U
T + cUS4U
T
)USV T
= U(a + bS2 + cS4
)U
TUSV T
= U(a + bS2 + cS4
)SV T
= U(aS + bS3 + cS5
)V
T
By normalizing the matrix with it’s Frobenius norm, it’s guaranteed that
the singular values will fall between [0, 1].Where the paper says it
Where the paper says it
Where the gains come from
The muon optimizer shows promise but the current iteration requires tuning in order to capture the ”unlikely directions” that currently explain Muon’s success. The issues presented in this paper will likely arise at model ds >8k, and will at worse cause less than optimal performance. The higher powers in the NS iterations may run into precision issues over a certain model size, but this is likely way outside realistic model sizes. p. 3 LLMs have been shown to pick up outlier
features on their own over around 7b parameters. [2] Additionally, the higher
powers in the NS iterations may run into precision issues over a certain model
size, but this is likely way outside realistic model sizes.Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Scale Muon parameters based on matrix size using a polynomial whose slope or number of iterations increases with scale, as suggested by
Either, the slope of the polynomial at 0 or the number of iterations should increase with scale.
This modification aims to reliably orthogonalize matrices at larger scales where the expected distribution of singular values shrinks. -
Implement a mechanism to dynamically adjust polynomial coefficients based on matrix dimensions to better approximate a function defined to be 1 within the interval, addressing the issue where
the expected distribution of singular values shrinks with the square root of the smallest dimension.
This will ensure that decisions on constants remain reliable as model sizes increase. -
Leverage the theoretical scaling law derived from Marchenko-Pastur Theorem to predict and manage potential performance degradation at large model dimensions, specifically anticipating issues when "model ds >8k
where current iteration limits may cause
less than optimal performance." This allows for proactive tuning before conducting expensive training experiments.
Abstract
The muon optimizer has picked up much attention as of late as a possible replacement to the seemingly omnipresent Adam optimizer. Recently, care has been taken to document the scaling laws of hyper-parameters under muon such as weight decay and learning rate. However, at much larger scales the iterative orthogonalization procedure present in muon may suffer a possible issue as the singular values of random matrices shrink with scale. This paper shows this scaling behavior theoretically and empirically on random matrices but does not suggest what to do about it.
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis