Iterative Orthogonalization Scaling Laws

arXiv:2505.04005 · cs.LG · Submitted 2025-05-06 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "Iterative Orthogonalization Scaling Laws".

Jane: Here's the gist of the paper. The paper investigates the scaling behavior of hyper-parameters in the muon optimizer, specifically concerning weight decay and learning rate, and identifies a potential issue with its iterative orthogonalization procedure at larger scales <ref:2505.04005#pg2>.

Tom: This concern arises because the singular values of random matrices shrink with scale, which complicates the choice of coefficients for the polynomial used in muon's preconditioner <ref:2505.04005#pg3>.

Jane: Iterative Orthogonalization Mechanics Muon’s preconditioner employs a clever polynomial to iteratively orthogonalize the momentum before applying it to the model <ref:2505.04005#pg4>.

Tom: This procedure involves using odd-degree exponents of the matrix to directly apply a polynomial to the singular values of a matrix, resulting in an expression like U(a + bS2 + cS4)V TUSV T <ref:2505.04005#pg4>.

Paper summary: Tom: So to wrap up the "Iterative Orthogonalization Scaling Laws" paper by Devan Selvaraj, it looks like the main message is that we need to be careful when scaling Muon to really large models <ref:2505.04005#pg1>.

Jane: Right. The paper explores how the iterative orthogonalization procedure might suffer at much larger scales because the singular values of random matrices shrink with scale <ref:2505.04005#pg1>.

Lu: They show this scaling behavior both theoretically using the Marchenko-Pastur Theorem and empirically by looking at randomly initialized matrices from size one hundred twenty-eight up to eight thousand one hundred ninety-two <ref:2505.04005#pg3>.

Meng: The authors conclude that they don't suggest a specific fix for this issue, but they point out that either the slope of the polynomial at zero or the number of iterations should increase with scale <ref:2505.04005#pg5>.

Tom: It seems like the main implication is that for now, we should expect less than optimal performance if we push model sizes beyond eight thousand parameters <ref:2505.04005#pg3>.

Jane: So, the authors aren't saying this is a crisis right now, but they are definitely flagging that at scales over eight thousand parameters things will get worse <ref:2505.04005#pg3>.

Conclusion: Tom: So we've been looking at how Muon handles its math, and now we get to look at the scaling of that iterative process in this paper, "Iterative Orthogonalization Scaling Laws," by Devan Selvaraj and others.

Jane: That paper is really digging into whether this clever way Muon cleans up the momentum before it hits the model breaks down when you make things bigger.

Tom: Exactly. It looks at how weight decay and learning rate scale, but specifically focuses on that iterative orthogonalization part, which is basically Muon's secret sauce for getting good updates.

Lu: What they found is that because of how the singular values of random matrices shrink as the size increases, the math behind those polynomial coefficients gets tricky.

Meng: So what does that mean for us when we try to train massive AI models? Is this going to crash our hardware or something?

Jane: It suggests that if you keep pushing model sizes way past eight thousand parameters, you might not get optimal performance because the method can't capture the right directions anymore.

Tom: Right. The authors are saying that either we have to change how the polynomial behaves as it gets bigger, maybe make its slope steeper at zero, or just run more iterations to catch those less likely directions.

Lu: The big question they leave is whether capturing those less likely directions is actually something we want when we're training models this massive.

Jane: It’s a trade-off, isn't it? We get better accuracy on smaller models, but then the complexity and potential precision issues kick in at the very top end of model size.

Devan Selvaraj

cs.LG

Submitted: 2025-05-06

Updated: 2026-11-10

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 72/100

The gist: The paper investigates the scaling behavior of hyper-parameters in the muon optimizer, specifically concerning weight decay and learning rate…

Key concepts

Iterative Orthogonalization
Muon uses a clever polynomial involving odd-degree matrix exponents to iteratively orthogonalize momentum. This process transforms a matrix operation into an expression like U(a+bS2+cS4)SV^T, aiming to improve the optimization path before applying it to the model.
Singular Value Scaling Law
This law mathematically describes how singular values behave in large random matrices. The paper proves that for a gradient normalized by its Frobenius norm, the singular values scale proportionally to 1/√d, where d is the matrix dimension. This scaling is crucial for understanding the behavior at different scales.
Marchenko-Pastur Theorem
This theorem provides a mathematical framework for understanding the distribution of eigenvalues (related to singular values) in matrices with i.i.d entries and a fixed aspect ratio. It allows researchers to calculate the exact scaling behavior of these values, confirming that singular values scale as 1/√in.
Unlikely Directions
These refer to specific directions within the parameter space that are less probable or less likely to be captured by the current polynomial approximation in Muon's preconditioner. The paper suggests that at larger scales, capturing these directions might be necessary for optimal performance.

Terminology

Summary

The paper investigates the scaling behavior of hyper-parameters in the muon optimizer, specifically concerning weight decay and learning rate, and identifies a potential issue with its iterative orthogonalization procedure at larger scales <ref:2505. p. 0

Where the paper says it

Recently, care has been taken to document the scaling laws of hyper-parameters under muon such as weight decay and learning rate. However, at much larger scales the iterative orthogonalization procedure present in muon may suffer a possible issue as the singular values of random matrices shrink with scale.

04005#pg2>. This concern arises because the singular values of random matrices shrink with scale, which complicates the choice of coefficients for the polynomial used in muon's preconditioner <ref:2505.04005#pg3>.

Iterative Orthogonalization Mechanics

Muon’s preconditioner employs a clever polynomial to iteratively orthogonalize the momentum before applying it to the model <ref:2505.04005#pg4>. This procedure involves using odd-degree exponents of the matrix to directly apply a polynomial to the singular values of a matrix, resulting in an expression like U(a + bS2 + cS4)V TUSV T <ref:2505. p. 1

Where the paper says it

2 Iterative Orthogonalization Muon’s preconditioner uses a clever polynomial to iteratively orthgonalize the momentum before applying it to the model. Using odd-degree exponents of the matrix, you can directly apply a polynomial to the singular values of a matrix as follows: G ′ = aG + b(GGT )G + c(GGT )(GGT )G = (aI + b(GGT ) + c(GGT )(GGT ))G = (aUUT + bUS2U T + cUS4U T )USV T = U(a + bS2 + cS4 )U TUSV T = U(a + bS2 + cS4 )SV T = U(aS + bS3 + cS5 )V T By normalizing the matrix with it’s Frobenius norm, it’s guaranteed that the singular values will fall between [0, 1].

04005#pg4>. By normalizing the matrix with its Frobenius norm, it is guaranteed that the singular values will fall between [0, 1] <ref:2505. p. 1
Where the paper says it

2 Iterative Orthogonalization Muon’s preconditioner uses a clever polynomial to iteratively orthgonalize the momentum before applying it to the model. Using odd-degree exponents of the matrix, you can directly apply a polynomial to the singular values of a matrix as follows: G ′ = aG + b(GGT )G + c(GGT )(GGT )G = (aI + b(GGT ) + c(GGT )(GGT ))G = (aUUT + bUS2U T + cUS4U T )USV T = U(a + bS2 + cS4 )U TUSV T = U(a + bS2 + cS4 )SV T = U(aS + bS3 + cS5 )V T By normalizing the matrix with it’s Frobenius norm, it’s guaranteed that the singular values will fall between [0, 1].

04005#pg4>. The coefficients of the polynomial are chosen to increase steeply at 0 to 1 in muon to minimize the number of steps taken <ref:2505. p. 1
Where the paper says it

The procedure can then be applied iteratively to progressively orthogonalize the matrix. 1 Muon Orthgonalization In muon, the coefficients of the polynomial are chosen to increase steeply at 0 to 1 to minimize the number of steps taken.

04005#pg4>.

Scaling Behavior and Empirical Evidence

The expected distribution of singular values changes predictably with size, appearing to shrink with the square root of the smallest dimension <ref:2505.04005#pg4>. Empirically, this scaling behavior can be observed in randomly initialized matrices across 8-512 matrices of size 128 through 8192 <ref:2505. p. 2

Where the paper says it

3 Singular Value Scaling Law Empirically, this scaling behavior can be observed in randomly initialized matrices. To show the distribution of singular values 8-512 matrices of size 128 through 8192 are randomly initialized N(0, 1), normalized and have the singular values computed.

04005#pg5>. The distribution of singular values for the largest and smallest matrices is plotted before and after applying the polynomial for 5 iterations <ref:2505. p. 2
Where the paper says it

To show the distribution of singular values 8-512 matrices of size 128 through 8192 are randomly initialized N(0, 1), normalized and have the singular values computed. The distributions of the singular values of the largest and smallest matrices are then plotted before and after applying the polynomial for 5 iterations.

04005#pg5>. Specifically, the higher matrix dimension has a heavier tail below. p. 2
Where the paper says it

The distributions of the singular values of the largest and smallest matrices are then plotted before and after applying the polynomial for 5 iterations. As can one can see, the higher matrix dimension has a heavier tail below .

6, indicating that less likely directions are not being captured by the polynomial <ref:2505. p. 2
Where the paper says it

As can one can see, the higher matrix dimension has a heavier tail below . 6, showing the less likely directions are not being captured by the polynomial. p. 2

Where the paper says it

As can one can see, the higher matrix dimension has a heavier tail below . 6, showing the less likely directions are not being captured by the polynomial.

04005#pg5>.

Mathematical Scaling Proof

The exact scaling can be calculated mathematically using the Marchenko-Pastur Theorem <ref:2505.04005#pg7>. Theorem 1 (Normalized Singular Value Scaling Law) states that the singular values of a gradient normalized by its Frobenius norm scale 1/√d <ref:2505. p. 3

Where the paper says it

Theorem 1 (Normalized Singular Value Scaling Law). The singular values of a gradient normalized by it’s Frobenius norm scale 1/ √ in d.

04005#pg7>. The proof assumes the gradient entries are i. p. 3
Where the paper says it

The singular values of a gradient normalized by it’s Frobenius norm scale 1/ √ in d. Assume the gradient entries are i. p. 3

Where the paper says it

The singular values of a gradient normalized by it’s Frobenius norm scale 1/ √ in d. Assume the gradient entries are i.

i.d with mean 0 finite variance σ2 <ref:2505. p. 5
Where the paper says it

d with mean 0, finite variance σ 2 , and a fixed aspect ratio γ = out d/in d the singular values scale 1/ √ out d. Define initial matrix A A ∼ N in d×out d (0, σ2 ) = UDV T Define scaled matrix A¯ A¯ = A/√ out d ∼ N in d×out d (0, σ2 /out d) = UDV¯ T 6

04005#pg7>. Applying the Frobenius Norm Scaling Law and then the Singular Value Scaling Law yields a scaling of 1/√in d <ref:2505. p. 3
Where the paper says it

Theorem 1 (Normalized Singular Value Scaling Law). The singular values of a gradient normalized by it’s Frobenius norm scale 1/ √ in d.

04005#pg7>.

Implications for Large-Scale Training

The paper concludes that either the slope of the polynomial at 0 or the number of iterations should increase with scale to capture unlikely directions. It is not clear if capturing these less likely directions is desirable at larger scales. p. 3

Where the paper says it

Either, the slope of the polynomial at 0 or the number of iterations should increase with scale. However, it’s not clear if capturing these less likely directions is actually desirable at larger scales.

The issues are predicted to arise at model sizes ds >8k and will cause less than optimal performance. p. 3
Where the paper says it

Kimi’s Moonlight 16B - the largest model trained with muon so far - only had a model d of 2048, well within the range for Muon’s default of 5 iterations to suffice. The issues presented in this paper will likely arise at model ds >8k, and will at worse cause less than optimal performance.

However, the largest model trained with muon so far, Kimi’s Moonlight 16B, only had a model d of 2048, which is within the range for muon’s default of 5 iterations to suffice. p. 3
Where the paper says it

At the time of this paper, there have been no known model trains with Muon at the model dimensions necessary to cause the issues predicted in this paper. Kimi’s Moonlight 16B - the largest model trained with muon so far - only had a model d of 2048, well within the range for Muon’s default of 5 iterations to suffice.

The issues presented in this paper will likely arise at model ds >8k. p. 3
Where the paper says it

Kimi’s Moonlight 16B - the largest model trained with muon so far - only had a model d of 2048, well within the range for Muon’s default of 5 iterations to suffice. The issues presented in this paper will likely arise at model ds >8k, and will at worse cause less than optimal performance.

Appendix Details

Lemma 1 (Frobenius Norm Scaling Law) states that the Frobenius norm of a matrix scales √out d × in d assuming constant variance. Theorem 2 (Marchenko-Pastur Theorem) states that for a matrix with i. p. 5

Where the paper says it

||W||F = sX i,j W2 ij ≈ sX i,j N(0, σ2) 2 = q σ 2χ 2 in×out ∝ σ √ out d × in d Theorem 2 (Marchenko-Pastur Theorem). The Marchenko-Pastur Theorem[5] states as in d → ∞ the empirical distribution of the singular values of a matrix with i.

i.d entries with µ = 0, variance σ = ¯σ2/out d, fixed aspect ratio γ = out d/in d has singular values distributed as ρ(s). p. 5
Where the paper says it

The Marchenko-Pastur Theorem[5] states as in d → ∞ the empirical distribution of the singular values of a matrix with i. d entries with µ = 0, variance σ = ¯σ 2/out d, fixed aspect ratio γ = out d/in d has singular values distributed: ρ(s) = p ((1 + √γ) 2 − s 2)(s 2 − |1 − √γ| 2) 2πγs Lemma 2 (Singular Value Scaling Law).

Lemma 2 (Singular Value Scaling Law) states that for a matrix whose entries are i. p. 5
Where the paper says it

The Marchenko-Pastur Theorem[5] states as in d → ∞ the empirical distribution of the singular values of a matrix with i. d entries with µ = 0, variance σ = ¯σ 2/out d, fixed aspect ratio γ = out d/in d has singular values distributed: ρ(s) = p ((1 + √γ) 2 − s 2)(s 2 − |1 − √γ| 2) 2πγs Lemma 2 (Singular Value Scaling Law).

i.d with mean 0, finite variance σ2, and a fixed aspect ratio γ = out d/in d the singular values scale 1/√out d. p. 5
Where the paper says it

d with mean 0, finite variance σ 2 , and a fixed aspect ratio γ = out d/in d the singular values scale 1/ √ out d. Define initial matrix A A ∼ N in d×out d (0, σ2 ) = UDV T Define scaled matrix A¯ A¯ = A/√ out d ∼ N in d×out d (0, σ2 /out d) = UDV¯ T 6

Lemma 2's proof shows that scaling the matrix by constant increasing its singular values linearly scales D with out d. The final result shows D ∼ ρ√out d. p. 0
Where the paper says it

However, at much larger scales the iterative orthogonalization procedure present in muon may suffer a possible issue as the singular values of random matrices shrink with scale. This paper shows this scaling behavior theoretically and empirically on random matrices but does not suggest what to do about it.

References

[1] Essential AI,:, Ishaan Shah, Anthony M. Polloreno, Karl Stratos, Philip Monk, Adarsh Chaluvaraju, Andrew Hojel, Andrew Ma, Anil Thomas, Ashish Tanwer, Darsh J Shah, Khoi Nguyen, Kurt Smith, Michael Callahan,, Michael Pust,, Mohit Parmar,, Peter Rushton,, Platon Mazarakis,, Ritvik Kapila,, Saurabh Srivastava,, Somanshu Singla,, Tim Romanski,, Yash Vanjani,, and Ashish Vaswani. p. 4

Where the paper says it

References [1] Essential AI, :, Ishaan Shah, Anthony M. Polloreno, Karl Stratos, Philip Monk, Adarsh Chaluvaraju, Andrew Hojel, Andrew Ma, Anil Thomas, Ashish Tanwer, Darsh J Shah, Khoi Nguyen, Kurt Smith, Michael Callahan, Michael Pust, Mohit Parmar, Peter Rushton, Platon Mazarakis, Ritvik Kapila, Saurabh Srivastava, Somanshu Singla, Tim Romanski, Yash Vanjani, and Ashish Vaswani.

Practical efficiency of muon for pretraining, 2025. p. 4
Where the paper says it

Polloreno, Karl Stratos, Philip Monk, Adarsh Chaluvaraju, Andrew Hojel, Andrew Ma, Anil Thomas, Ashish Tanwer, Darsh J Shah, Khoi Nguyen, Kurt Smith, Michael Callahan, Michael Pust, Mohit Parmar, Peter Rushton, Platon Mazarakis, Ritvik Kapila, Saurabh Srivastava, Somanshu Singla, Tim Romanski, Yash Vanjani, and Ashish Vaswani. Practical efficiency of muon for pretraining, 2025.

p. 4
Where the paper says it

Polloreno, Karl Stratos, Philip Monk, Adarsh Chaluvaraju, Andrew Hojel, Andrew Ma, Anil Thomas, Ashish Tanwer, Darsh J Shah, Khoi Nguyen, Kurt Smith, Michael Callahan, Michael Pust, Mohit Parmar, Peter Rushton, Platon Mazarakis, Ritvik Kapila, Saurabh Srivastava, Somanshu Singla, Tim Romanski, Yash Vanjani, and Ashish Vaswani. Practical efficiency of muon for pretraining, 2025.

[2] Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. p. 4

Where the paper says it

Practical efficiency of muon for pretraining, 2025. [2] Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer.

Llm.int8: 8-bit matrix multiplication for transformers at scale, 2022. p. 4
Where the paper says it

[2] Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. int8(): 8-bit matrix multiplication for transformers at scale, 2022.

p. 4
Where the paper says it

[2] Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. int8(): 8-bit matrix multiplication for transformers at scale, 2022.

[3] Keller Jordan, Yuchen Jin, Vlado Boza,, Jiacheng You,, Franz Cesista,, Laker Newhouse,, and Jeremy Bernstein. p. 4

Where the paper says it

int8(): 8-bit matrix multiplication for transformers at scale, 2022. [3] Keller Jordan, Yuchen Jin, Vlado Boza, Jiacheng You, Franz Cesista, Laker Newhouse, and Jeremy Bernstein.

p. 4
Where the paper says it

int8(): 8-bit matrix multiplication for transformers at scale, 2022. [3] Keller Jordan, Yuchen Jin, Vlado Boza, Jiacheng You, Franz Cesista, Laker Newhouse, and Jeremy Bernstein.

Muon: An optimizer for hidden layers in neural networks, 2024. p. 4
Where the paper says it

[3] Keller Jordan, Yuchen Jin, Vlado Boza, Jiacheng You, Franz Cesista, Laker Newhouse, and Jeremy Bernstein. Muon: An optimizer for hidden layers in neural networks, 2024.

p. 4
Where the paper says it

[3] Keller Jordan, Yuchen Jin, Vlado Boza, Jiacheng You, Franz Cesista, Laker Newhouse, and Jeremy Bernstein. Muon: An optimizer for hidden layers in neural networks, 2024.

[4] Diederik P. p. 4

Where the paper says it

Muon: An optimizer for hidden layers in neural networks, 2024. [4] Diederik P.

p. 4
Where the paper says it

Muon: An optimizer for hidden layers in neural networks, 2024. [4] Diederik P.

Kingma and Jimmy Ba. p. 4
Where the paper says it

[4] Diederik P. Kingma and Jimmy Ba.

p. 4
Where the paper says it

[4] Diederik P. Kingma and Jimmy Ba.

Adam: A method for stochastic optimization, 2017. p. 4
Where the paper says it

Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017.

p. 4
Where the paper says it

Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017.

[5] V. A. Marˇcenko and L. p. 4

Where the paper says it

Adam: A method for stochastic optimization, 2017. Marˇcenko and L.

p. 4
Where the paper says it

Adam: A method for stochastic optimization, 2017. Marˇcenko and L.

A. Pastur. Distribution of eigenvalues for some sets of random matrices, Matematicheskii Sbornik, 114(4):507–536, 1967. p. 4
Where the paper says it

Distribution of eigenvalues for some sets of random matrices. Matematicheskii Sbornik, 114(4):507–536, 1967.

--- Page 1 ---

Iterative Orthogonalization Scaling Laws

Devan Selvaraj

research@ueaj.dev

May 2025

--- Page 2 ---

Muon’s preconditioner uses a clever polynomial to iteratively orthgonalize the momentum before applying it to the model. Using odd-degree exponents of the matrix, you can directly apply a polynomial to the singular values of a matrix as follows: G′ = aG + b(GGT)G + c(GGT)(GGT)G = (aI + b(GGT) + c(GGT)(GGT))G = (aUUT + bUS2U T + cUS4U T)USV T = U(a+bS2+cS4)SV T = U(aS + bS3 + cS5)V T <ref:2505. p. 1

Where the paper says it

2 Iterative Orthogonalization Muon’s preconditioner uses a clever polynomial to iteratively orthgonalize the momentum before applying it to the model. Using odd-degree exponents of the matrix, you can directly apply a polynomial to the singular values of a matrix as follows: G ′ = aG + b(GGT )G + c(GGT )(GGT )G = (aI + b(GGT ) + c(GGT )(GGT ))G = (aUUT + bUS2U T + cUS4U T )USV T = U(a + bS2 + cS4 )U TUSV T = U(a + bS2 + cS4 )SV T = U(aS + bS3 + cS5 )V T By normalizing the matrix with it’s Frobenius norm, it’s guaranteed that the singular values will fall between [0, 1].

04005#pg3>

--- Page 3 ---

Singular Value Scaling Law

Empirically, this scaling behavior can be observed in randomly initialized matrices. To show the distribution of singular values 8-512 matrices of size 128 through 8192 are randomly initialized N(0, 1), normalized and have the singular values computed <ref:2505. p. 2

Where the paper says it

3 Singular Value Scaling Law Empirically, this scaling behavior can be observed in randomly initialized matrices. To show the distribution of singular values 8-512 matrices of size 128 through 8192 are randomly initialized N(0, 1), normalized and have the singular values computed.

04005#pg4>. The distributions of the singular values of the largest and smallest matrices are then plotted before and after applying the polynomial for 5 iterations. p. 2
Where the paper says it

To show the distribution of singular values 8-512 matrices of size 128 through 8192 are randomly initialized N(0, 1), normalized and have the singular values computed. The distributions of the singular values of the largest and smallest matrices are then plotted before and after applying the polynomial for 5 iterations.

As can one can see, the higher matrix dimension has a heavier tail below. p. 2
Where the paper says it

The distributions of the singular values of the largest and smallest matrices are then plotted before and after applying the polynomial for 5 iterations. As can one can see, the higher matrix dimension has a heavier tail below .

6, showing the less likely directions are not being captured by the polynomial.

--- Page 4 ---

3.1 Proof

The exact scaling can be calculated mathematically using the Marchenko-Pastur Theorem. Theorem 1 (Normalized Singular Value Scaling Law) The singular values of a gradient normalized by it’s Frobenius norm scale 1/√d. p. 3

Where the paper says it

Theorem 1 (Normalized Singular Value Scaling Law). The singular values of a gradient normalized by it’s Frobenius norm scale 1/ √ in d.

Proof. Assume the gradient entries are i.i.d with mean 0 finite variance σ2. p. 5
Where the paper says it

d with mean 0, finite variance σ 2 , and a fixed aspect ratio γ = out d/in d the singular values scale 1/ √ out d. Define initial matrix A A ∼ N in d×out d (0, σ2 ) = UDV T Define scaled matrix A¯ A¯ = A/√ out d ∼ N in d×out d (0, σ2 /out d) = UDV¯ T 6

Assume the in d/out d ratio remains constant with scale Apply Frobenius Norm Scaling Law1 and ∇W as the gradient and SV DD is the svd of the gradient returning only the diagonals. p. 3
Where the paper says it

d with mean 0 finite variance σ 2 . Assume the in d/out d ratio remains constant with scale Apply Frobenius Norm Scaling Law1 and ∇W as the gradient and SV DD is the svd of the gradient returning only the diagonals.

SVDD(∇W ∇WF) ≈ SVDD(Nin d×out d (0, 1/√in d × out d) = SVDD(Nin d×out d (0, 1/√in 2d)) = SVDD(Nin d×out 2d) Apply Singular Value Scaling Law2 ≈ ρ√out d / √(in d × out d) = ρ/√in. p. 3
Where the paper says it

Assume the in d/out d ratio remains constant with scale Apply Frobenius Norm Scaling Law1 and ∇W as the gradient and SV DD is the svd of the gradient returning only the diagonals. SVDD( ∇W ||∇W||F ) ≈ SVDD( Nin d×out d (0, 1) √ in d × out d ) = SVDD(Nin d×out d (0, 1)) √ in d × out d Apply Singular Value Scaling Law2 ≈ ρ √ out d √ in d × out d = ρ √ in d 4 Conclusion As muon continues to gain traction and larger and larger experiments are conducted using the optimizer, it is important that any possible issues are foreseen in it’s scaling behavior prior to conducting experiments.

--- Page 5 ---

Conclusion

As muon continues to gain traction and larger and larger experiments are conducted using the optimizer, it is important that any possible issues are foreseen in it’s scaling behavior prior to conducting experiments. The NS iteration method pioneered by muon shows promise but the current iteration requires tuning in order to capture the ”unlikely directions” that currently explain Muon’s success. p. 3

Where the paper says it

SVDD( ∇W ||∇W||F ) ≈ SVDD( Nin d×out d (0, 1) √ in d × out d ) = SVDD(Nin d×out d (0, 1)) √ in d × out d Apply Singular Value Scaling Law2 ≈ ρ √ out d √ in d × out d = ρ √ in d 4 Conclusion As muon continues to gain traction and larger and larger experiments are conducted using the optimizer, it is important that any possible issues are foreseen in it’s scaling behavior prior to conducting experiments. The NS iteration method pioneered by muon shows promise but the current iteration requires tuning in order to capture the ”unlikely directions” that currently explain Muon’s success.

Either, the slope of the polynomial at 0 or the number of iterations should increase with scale. p. 3
Where the paper says it

The NS iteration method pioneered by muon shows promise but the current iteration requires tuning in order to capture the ”unlikely directions” that currently explain Muon’s success. Either, the slope of the polynomial at 0 or the number of iterations should increase with scale.

However, it’s not clear if capturing these less likely directions is actually desirable at larger scales. p. 3
Where the paper says it

Either, the slope of the polynomial at 0 or the number of iterations should increase with scale. However, it’s not clear if capturing these less likely directions is actually desirable at larger scales.

LLMs have been shown to pick up outlier features on their own over around 7b parameters. p. 3
Where the paper says it

However, it’s not clear if capturing these less likely directions is actually desirable at larger scales. LLMs have been shown to pick up outlier features on their own over around 7b parameters.

Additionally, the higher powers in the NS iterations may run into precision issues over a certain model size, but this is likely way outside realistic model sizes. p. 3
Where the paper says it

LLMs have been shown to pick up outlier features on their own over around 7b parameters. [2] Additionally, the higher powers in the NS iterations may run into precision issues over a certain model size, but this is likely way outside realistic model sizes.

At the time of this paper, there have been no known model trains with Muon at the model dimensions necessary to cause the issues predicted in this paper. p. 3
Where the paper says it

[2] Additionally, the higher powers in the NS iterations may run into precision issues over a certain model size, but this is likely way outside realistic model sizes. At the time of this paper, there have been no known model trains with Muon at the model dimensions necessary to cause the issues predicted in this paper.

Kimi’s Moonlight 16B - the largest model trained with muon so far - only had a model d of 2048, well within the range for Muon’s default of 5 iterations to suffice. p. 3
Where the paper says it

At the time of this paper, there have been no known model trains with Muon at the model dimensions necessary to cause the issues predicted in this paper. Kimi’s Moonlight 16B - the largest model trained with muon so far - only had a model d of 2048, well within the range for Muon’s default of 5 iterations to suffice.

The issues presented in this paper will likely arise at model ds >8k, and will at worse cause less than optimal performance. p. 3
Where the paper says it

Kimi’s Moonlight 16B - the largest model trained with muon so far - only had a model d of 2048, well within the range for Muon’s default of 5 iterations to suffice. The issues presented in this paper will likely arise at model ds >8k, and will at worse cause less than optimal performance.

p. 3
Where the paper says it

Kimi’s Moonlight 16B - the largest model trained with muon so far - only had a model d of 2048, well within the range for Muon’s default of 5 iterations to suffice. The issues presented in this paper will likely arise at model ds >8k, and will at worse cause less than optimal performance.

--- Page 6 ---

References

[1] Essential AI,:, Ishaan Shah, Anthony M. Polloreno, Karl Stratos,, Philip Monk,, Adarsh Chaluvaraju,, Andrew Hojel,, Andrew Ma,, Anil Thomas,, Ashish Tanwer,, Darsh J Shah,, Khoi Nguyen,, Kurt Smith, Michael Callahan,, Michael Pust,, Mohit Parmar,, Peter Rushton, Platon Mazarakis, Ritvik Kapila, Saurabh Srivastava,, Somanshu Singla,, Tim Romanski, Yash Vanjani,, and Ashish Vaswani. p. 4

Where the paper says it

References [1] Essential AI, :, Ishaan Shah, Anthony M. Polloreno, Karl Stratos, Philip Monk, Adarsh Chaluvaraju, Andrew Hojel, Andrew Ma, Anil Thomas, Ashish Tanwer, Darsh J Shah, Khoi Nguyen, Kurt Smith, Michael Callahan, Michael Pust, Mohit Parmar, Peter Rushton, Platon Mazarakis, Ritvik Kapila, Saurabh Srivastava, Somanshu Singla, Tim Romanski, Yash Vanjani, and Ashish Vaswani.

Practical efficiency of muon for pretraining, 2025.

[2] Tim Dettmers, Mike Lewis, Younes Belkada,, and Luke Zettlemoyer. p. 4

Where the paper says it

Practical efficiency of muon for pretraining, 2025. [2] Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer.

Llm.int8: 8-bit matrix multiplication for transformers at scale, 2022.

[3] Keller Jordan, Yuchen Jin, Vlado Boza,, Jiacheng You,, Franz Cesista,, Laker Newhouse,, and Jeremy Bernstein. Muon: An optimizer for hidden layers in neural networks, 2024.

[4] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017.

[5] V. A. Marˇcenko and L. A. Pastur., Distribution of eigenvalues for some sets of random matrices, Matematicheskii Sbornik, 114(4):507–536, 1967. p. 4

Where the paper says it

Distribution of eigenvalues for some sets of random matrices. Matematicheskii Sbornik, 114(4):507–536, 1967.

--- Page 7 ---

5 Appendix

5.1 Theorems

Lemma 1 (Frobenius Norm Scaling Law) The Frobenius norm of a matrix scales √out d × in d assuming constant variance. Assume the elements of the gradient of a matrix are roughly normally distributed with mean 0 and σ2 variance. p. 5

Where the paper says it

Assume the elements of the gradient of a matrix are roughly normally distributed with mean 0 and σ 2 variance. ||W||F = sX i,j W2 ij ≈ sX i,j N(0, σ2) 2 = q σ 2χ 2 in×out ∝ σ √ out d × in d Theorem 2 (Marchenko-Pastur Theorem).

p. 5
Where the paper says it

Assume the elements of the gradient of a matrix are roughly normally distributed with mean 0 and σ 2 variance. ||W||F = sX i,j W2 ij ≈ sX i,j N(0, σ2) 2 = q σ 2χ 2 in×out ∝ σ √ out d × in d Theorem 2 (Marchenko-Pastur Theorem).

p. 5
Where the paper says it

Assume the elements of the gradient of a matrix are roughly normally distributed with mean 0 and σ 2 variance. ||W||F = sX i,j W2 ij ≈ sX i,j N(0, σ2) 2 = q σ 2χ 2 in×out ∝ σ √ out d × in d Theorem 2 (Marchenko-Pastur Theorem).

p. 5
Where the paper says it

Assume the elements of the gradient of a matrix are roughly normally distributed with mean 0 and σ 2 variance. ||W||F = sX i,j W2 ij ≈ sX i,j N(0, σ2) 2 = q σ 2χ 2 in×out ∝ σ √ out d × in d Theorem 2 (Marchenko-Pastur Theorem).

p. 5
Where the paper says it

Assume the elements of the gradient of a matrix are roughly normally distributed with mean 0 and σ 2 variance. ||W||F = sX i,j W2 ij ≈ sX i,j N(0, σ2) 2 = q σ 2χ 2 in×out ∝ σ √ out d × in d Theorem 2 (Marchenko-Pastur Theorem).

p. 5
Where the paper says it

Assume the elements of the gradient of a matrix are roughly normally distributed with mean 0 and σ 2 variance. ||W||F = sX i,j W2 ij ≈ sX i,j N(0, σ2) 2 = q σ 2χ 2 in×out ∝ σ √ out d × in d Theorem 2 (Marchenko-Pastur Theorem).

--- Page 8 ---

Lemma 1 (Frobenius Norm Scaling Law) The Frobenius norm of a matrix scales √out d × in d assuming constant variance. Assume the elements of the gradient of a matrix are roughly normally distributed with mean 0 and σ2 variance.

--- Page 9 ---

Lemma 1 (Frobenius Norm Scaling Law) The Frobenius norm of a matrix scales √out d × in d assuming constant variance. Assume the elements of the gradient of a matrix are roughly normally distributed with mean 0 and σ2 variance.

--- Page 10 ---

Lemma 1 (Frobenius Norm Scaling Law) The Frobenius norm of a matrix scales √out d × in d assuming constant variance. Assume the elements of the gradient of a matrix are roughly normally distributed with mean 0 and σ2 variance.

--- Page 11 ---

Lemma 1 (Frobenius Norm Scaling Law) The Frobenius norm of a matrix scales √out d × in d assuming constant variance. Assume the elements of the gradient of a matrix are roughly normally distributed with mean 0 and σ2 variance.

Pruning redundant reasoning tokens

Lemma 1 (Frobenius Norm Scaling Law) The Frobenius norm of a matrix scales √out d × in d assuming constant variance. Assume the elements of the gradient of a matrix are roughly normally distributed with mean 0 and σ2 variance.

How the training signal is built

The muon optimizer uses a specific mathematical formulation for its preconditioner. The procedure involves applying odd-degree exponents of the matrix to directly apply a polynomial to the singular values of a matrix. p. 1

Where the paper says it

2 Iterative Orthogonalization Muon’s preconditioner uses a clever polynomial to iteratively orthgonalize the momentum before applying it to the model. Using odd-degree exponents of the matrix, you can directly apply a polynomial to the singular values of a matrix as follows: G ′ = aG + b(GGT )G + c(GGT )(GGT )G = (aI + b(GGT ) + c(GGT )(GGT ))G = (aUUT + bUS2U T + cUS4U T )USV T = U(a + bS2 + cS4 )U TUSV T = U(a + bS2 + cS4 )SV T = U(aS + bS3 + cS5 )V T By normalizing the matrix with it’s Frobenius norm, it’s guaranteed that the singular values will fall between [0, 1].

This leads to an expression U(a+bS2+cS4)SV T. p. 1
Where the paper says it

2 Iterative Orthogonalization Muon’s preconditioner uses a clever polynomial to iteratively orthgonalize the momentum before applying it to the model. Using odd-degree exponents of the matrix, you can directly apply a polynomial to the singular values of a matrix as follows: G ′ = aG + b(GGT )G + c(GGT )(GGT )G = (aI + b(GGT ) + c(GGT )(GGT ))G = (aUUT + bUS2U T + cUS4U T )USV T = U(a + bS2 + cS4 )U TUSV T = U(a + bS2 + cS4 )SV T = U(aS + bS3 + cS5 )V T By normalizing the matrix with it’s Frobenius norm, it’s guaranteed that the singular values will fall between [0, 1].

Where the gains come from

The muon optimizer shows promise but the current iteration requires tuning in order to capture the ”unlikely directions” that currently explain Muon’s success. The issues presented in this paper will likely arise at model ds >8k, and will at worse cause less than optimal performance. The higher powers in the NS iterations may run into precision issues over a certain model size, but this is likely way outside realistic model sizes. p. 3

Where the paper says it

LLMs have been shown to pick up outlier features on their own over around 7b parameters. [2] Additionally, the higher powers in the NS iterations may run into precision issues over a certain model size, but this is likely way outside realistic model sizes.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Scale Muon parameters based on matrix size using a polynomial whose slope or number of iterations increases with scale, as suggested by Either, the slope of the polynomial at 0 or the number of iterations should increase with scale. This modification aims to reliably orthogonalize matrices at larger scales where the expected distribution of singular values shrinks.

  2. Implement a mechanism to dynamically adjust polynomial coefficients based on matrix dimensions to better approximate a function defined to be 1 within the interval, addressing the issue where the expected distribution of singular values shrinks with the square root of the smallest dimension. This will ensure that decisions on constants remain reliable as model sizes increase.

  3. Leverage the theoretical scaling law derived from Marchenko-Pastur Theorem to predict and manage potential performance degradation at large model dimensions, specifically anticipating issues when "model ds >8k where current iteration limits may cause less than optimal performance." This allows for proactive tuning before conducting expensive training experiments.

Abstract

The muon optimizer has picked up much attention as of late as a possible replacement to the seemingly omnipresent Adam optimizer. Recently, care has been taken to document the scaling laws of hyper-parameters under muon such as weight decay and learning rate. However, at much larger scales the iterative orthogonalization procedure present in muon may suffer a possible issue as the singular values of random matrices shrink with scale. This paper shows this scaling behavior theoretically and empirically on random matrices but does not suggest what to do about it.

Related papers