Spectrally Targeted Muon

arXiv:2610.10965 · cs.LG · Submitted 2026-10-07 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Spectrally Targeted Muon".

Jane: The gist The Muon optimizer orthogonalizes each update matrix, setting all of its singular values to one, and has proven highly effective for training large language models.

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So we're looking at this new paper called Spectrally Targeted Muon from Vishrut Goyal and Rohan Ramkumar. Basically, they take the Muon optimizer—which is known for orthogonalizing every update matrix so all its singular values become one—and they try to figure out if that success comes from boosting tiny directions that gradient descent usually ignores or if it's actually suppressing big directions that mess up training.

Jane: That's the core question there. They introduce Spectrally Targeted Muon, which only orthogonalizes the singular values above or below a certain threshold tau, so you can slide between standard SGD and Muon by changing that tau. They use projections calculated by Newton-Schulz iteration on a shifted Gram matrix to do this, which is neat because it avoids needing Singular Value Decomposition entirely.

Lu: It’s interesting how they framed the mechanism there. They're essentially isolating the relevant subspaces using those projections based on the threshold tau so they don't mess with everything uniformly. They test this on CIFAR-ten and NanoGPT speedruns, tracking effective rank of gradients and updates, and a new metric called alignment of updates with the tangent space of the weight matrix’s isospectral manifold <ref:2610.10965#pg1,alignment of updates with the tangent space of the weight matrix’s>.

Meng: Tracking those metrics sounds like a way to see if the optimizer is actually learning something useful structurally rather than just making small mathematical adjustments that don't help the loss. What does this mean for practical application in training really big language models?

Lalam: I think it suggests that we can tailor the orthogonalization to target specific parts of the update matrix, instead of applying a blanket rule to every single dimension. It makes sense because large language models have huge parameter spaces where some directions are clearly more important than others.

Tom: Exactly. And they found three main things about this targeted approach. First, they said that the small singular values of the momentum aren't just noise; actually, orthogonalizing everything except those few largest singular values nearly matches Muon but only touches a tiny fraction of it, whereas if you only target the top ones, it falls short even though it captures almost all of what's there.

Jane: That’s a big distinction. It means for these language models where every momentum singular value is far below one, targeted orthogonalization can only amplify directions that are too small to train on at their raw scale, and Muon wins because it makes those tiny directions trainable in their raw form.

Lu: And the second thing they pointed out is that shrinking the largest singular values doesn't actually keep the parameter spectrum flat; that's just cheap, and it’s not what drives the loss curve down. It’s something else entirely that dictates where you end up.

Meng: So if shrinking those big singular values isn't the driver for better performance, then we need to focus on how we handle those smaller components instead of trying to flatten the whole spectrum with massive orthogonalization. That shifts our engineering focus significantly, doesn't it?

Lalam: Right. And they also found that AdamW differs from Muon mainly in how slowly it builds structure, which explains why AdamW starts slower and why warmup helps AdamW but hurts Muon. This suggests the difference is about the timeline of structural building up in the model parameters.

Tom: It’s clear then that Spectrally Targeted Muon isn't just another optimizer with a different setting; it’s a way to selectively apply orthogonalization based on which parts of the update matrix are most important for learning, and it gives us insights into what those important parts actually are.

Jane: So, in conclusion, they show that focusing the orthogonalization allows us to see if we're training on directions too small for standard methods or if we're accidentally suppressing dominant ones by applying full Muon across the board.

Lu: The implications for AI research is that understanding which singular values matter—the top few versus the bottom many—is crucial for designing effective update mechanisms, especially when dealing with models that have such high-dimensional parameter spaces.

Meng: From an engineering standpoint, if we can use these projections to guide our optimization steps without needing expensive SVDs every time, that would make training much more efficient in practice on large systems.

Lalam: And for the culture of how we build these models, it shows that precision in applying mathematical tools can lead to a deeper understanding of the underlying learning process than just blindly following a single algorithm.

Tom: Yeah, so this paper really gives us a sharper lens into how optimizers interact with the geometry of high-dimensional parameter spaces. It moves beyond just comparing final accuracy numbers and starts looking at the structure of the learning itself.

Conclusion: Tom: So we’re wrapping up on Spectrally Targeted Muon from Goyal and Ramkumar by talking about what this paper actually means for us right now.

Jane: Basically, they took the standard Muon optimizer—which makes every update matrix singular values equal one—and they figured out how to target only the important ones.

Lu: They introduce this Spectrally Targeted Muon method that lets you pick a threshold tau, so you can interpolate between standard SGD and the full Muon approach.

Meng: It uses projections calculated by Newton-Schulz iteration on a shifted Gram matrix instead of needing to calculate a full singular value decomposition every time, which is pretty efficient for large models.

Lalam: The core idea they found is that those small singular values in the momentum aren't just noise; they actually represent directions that the model can train on once they’re amplified.

Tom: That’s a big distinction because it means Muon wins when you have those tiny directions, and targeted methods are better when you want to be precise about which parts of the update matrix matter.

Jane: They found that shrinking the largest singular values doesn't actually keep the parameter spectrum flat; that’s just cheap and it doesn't drive the loss down on its own.

Lu: The third thing they noticed is how AdamW builds structure—it does so much slower than Muon, which explains why warmup helps AdamW but hurts Muon.

Meng: From an engineering standpoint, this targeted approach gives us a way to control exactly how much structure we’re forcing into the update step based on what the paper shows is actually relevant.

Lalam: I see this as a way for future AI to be more efficient by not treating every single direction in the weight matrix equally during updates.

Tom: Yeah, it shifts the focus from just getting high accuracy numbers to understanding where those high accuracy steps are actually coming from structurally.

Jane: And that leads us into how they actually proved this works on CIFAR-ten and NanoGPT speedruns, looking at things like effective rank of gradients and updates.

Lu: The implication is that understanding the spectrum—the distribution of those singular values—is a more important tool for designing optimizers than just tweaking the learning rate schedule.

Vishrut Goyal, Rohan Ramkumar

Department of Computer Science, Stanford University · Department of Electrical Engineering, Stanford University

cs.LG

Submitted: 2026-10-07

Updated: 2026-10-07

Code: https://github.com/KellerJordan/modded-nanogpt

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 83/100

The gist: The gist The Muon optimizer orthogonalizes each update matrix, setting all of its singular values to one, and has proven highly effective for training large language models.

Key concepts

Muon Optimizer: This optimizer modifies update matrices to have all their singular values equal to one. It is highly effective for training large language models because it enforces a specific geometric constraint on the updates, leading to better training dynamics.
Spectrally Targeted Muon: This variant of Muon selectively orthogonalizes only the singular values above or below a threshold ($ au$). By varying $ au$, researchers can interpolate between standard normalized Stochastic Gradient Descent (SGD) and the full Muon optimizer, isolating relevant singular subspaces.
Effective Rank: This metric measures how complex or nonuniform the distribution of information is within a matrix. It is calculated by normalizing singular values to probabilities and using spectral entropy. A lower effective rank suggests a simpler, more uniform structure in the parameter updates.

Summary

The gist The Muon optimizer orthogonalizes each update matrix, setting all of its singular values to one, and has proven highly effective for training large language models.

How it works

The fundamental algorithm underlying Muon’s orthogonalization step is the Newton-Schulz algorithm for calculating the matrix sign function<ref:2610.10965#pg4>.
The standard method computes this via polynomial iteration, where odd polynomials like p(x) = 3x − 16/5 x3 + 6/5 x5 asymptotically converge to sign(x) on the interval [-1, 1]<ref:2610.10965#pg4>.
Extended to matrices, the polynomial acts like p(M) = 3M − 16/5 (MM⊤)M + 6/5 (MM⊤)2M, and iterating it converges to sign(M)<ref:2610.10965#pg4>.
To selectively orthogonalize singular values separated by a threshold τ > 0, the method isolates the relevant subspaces using projection matrices<ref:2610.10965#pg4>.
The shifted symmetric matrix A = M⊤M − τ2/I has eigenvalues that partition the singular values of M below and above τ<ref:2610.10965#pg4>.
By computing S = sign(A), projection matrices Pbot = 1/2 (I − S) and Ptop = 1/2 (I + S) are defined, allowing for selective application of Newton-Schulz orthogonalization<ref:2610.10965#pg4>.
The algorithm involves computing either sign(M)Ptop + MPbot (targeting top) or MPtop + sign(M)Pbot (targeting bottom)<ref:2610.10965#pg4>.

Evaluation Metrics

The paper evaluates the variants on CIFAR-10 and NanoGPT speedruns, tracking the effective rank of gradient, update, and weight matrices and a new metric called the alignment of updates with the tangent space of the weight matrix’s isospectral manifold<ref:2610.10965#pg4>.

The effective rank is defined as the exponent of spectral entropy; if singular values are normalized by Pn i=1 si to get probabilities pi, the effective rank is exp(-Pn i=1 pi log pi)<ref:2610.10965#pg4>.

Isospectral projection measures ρ(Z; W), the fraction of Z that lies in TWM, the tangent space of M at W<ref:2610.10965#pg4>. A value near one means the update mostly rotates singular bases while preserving its spectrum, and a smaller value means more of the update changes singular values<ref:2610.10965#pg4>.

Key Findings on Targeted Orthogonalization

The researchers found three main things regarding the targeted orthogonalization approach<ref:2610.10965#pg4>.

First, the small singular values of the momentum are not noise and Muon wins by making directions that are too small to train on at their raw scale trainable because Muon amplifies directions too small to train on at their raw scale<ref:2610.10965#pg4>.

Second, shrinking the largest singular values is what keeps the parameter spectrum flat; this is cheap, and it is not what drives the loss<ref:2610.10965#pg4>.

Third, AdamW differs from Muon mainly in how slowly it builds structure which explains AdamW’s slower start and why warmup helps AdamW but only hurts Muon<ref:2610.10965#pg4>.

Performance on CIFAR-10 and NanoGPT

For CIFAR-10 training, top targeted Muon with τ = 1 brought mean accuracy from 91.2% to 93.4%, while pure Muon left mean accuracy over 20 runs unchanged at 93.5%<ref:2610.10965#pg5>.

In the NanoGPT domain, the small singular values of the momentum are not noise but directions the model can feasibly train on once they are amplified<ref:2610.10965#pg5>.

The fraction of momentum orthogonalized in top arm holds 32–96% of the momentum in squared Frobenius norm, while at most 13% in the bottom arm<ref:2610.10965#pg6>.

Structural Insights

The effective rank of the gradient stays below about 280 for most of training, implying a low-entropy and highly nonuniform distribution within each matrix<ref:2610.10965#pg8>.

The bottom targeted variants show higher alignment than Muon, while the top targeted versions show lower alignment<ref:2610.10965#pg8>.

AdamW is slow to build structure because an elementwise normalized update must build a matrix out of many low rank steps, whereas an orthogonalized update writes a full-rank matrix in a single step<ref:2610.10965#pg8>.

The effective rank of the gradient becomes an effective tool to determine position in the loss landscape, showing the difference between optimizers in where the early steps take the parameters<ref:2610.10965#pg8>.

The bottom-targeted versions keep Muon’s rank without matching its loss, while full-spectrum updates keep the spectrum flat for free<ref:2610.10965#pg8>.

The top targeted variants fall the most in parameter effective rank, implying that the rich learning is a direct consequence of participation by the small singular values<ref:2610.10965#pg8>.

Improvements for AI systems

Bold Update Mechanism for Large Language Models: Implement Spectrally Targeted Muon to orthogonalize only singular values above or below a threshold τ, as this isolates the relevant singular subspaces with projections computed by NewtonSchulz iteration on a shifted Gram matrix, so no SVD is needed. This allows the system to amplify small but important directions with small singular values ignored by standard gradient descent, which Muon succeeds at because it makes directions that are too small to train on at their raw scale trainable.
Bold Parameter Spectrum Control: Utilize the threshold τ to control the parameter spectrum, as the paper finds that shrinking the largest singular values is what keeps the parameter spectrum flat; this is cheap, and it is not what drives the loss. This prevents dominant directions from misdirecting training, contrasting with methods like AdamW which collapses rank and still performs well but lacks Muon's structural benefits.
Bold Structure Building Efficiency: Exploit the difference in update mechanics by recognizing that an elementwise normalized update builds a matrix one bounded increment at a time, so how well it builds depends on how big the first steps are, and with warmup AdamW ends with higher parameter rank and lower loss compared to Muon's orthogonalized update [which] writes a full-rank matrix in a single step. This suggests that for structure building, an orthogonalized update is more efficient.
Bold Landscape Navigation via Effective Rank: Use the effective rank of the gradient and update matrix to help explain our optimizer performances, as it shows where the iterate sits in the valley, bouncing between the walls or settled on the floor. This allows for a dynamic understanding of whether an optimizer is navigating sharp loss landscape walls or has settled on a flatter region.
Bold Targeted Learning Rate Scaling: Dynamically adjust learning rates based on training progress by considering whether dynamically adapting τ might improve results, as we could normalize either small or large singular values depending on how far along we are in our training. This would allow the system to switch between amplifying small singular values early and focusing on large ones later.

Abstract

The Muon optimizer orthogonalizes each update matrix, setting all of its singular values to one, and has proven highly effective for training large language models. It remains unclear, however, whether this success comes from amplifying small singular directions that gradient descent neglects or from suppressing large, degenerate directions that disrupt training. We introduce Spectrally Targeted Muon, which orthogonalizes only the singular values above or below a threshold τ, so that varying τ interpolates between normalized SGD and Muon. It isolates the relevant singular subspaces with projections computed by Newton-Schulz iteration on a shifted Gram matrix, so no SVD is needed. We evaluate these variants on the CIFAR-10 and NanoGPT speedruns, tracking the effective rank of gradient, update, and weight matrices and a new metric, the alignment of updates with the tangent space of the weight matrix's isospectral manifold. We find three things. First, the small singular values of the momentum are not noise. Orthogonalizing everything except the few largest singular values of each matrix nearly matches Muon while touching only a small fraction of the momentum, whereas orthogonalizing only the top falls well short even though it holds almost all of it. On language models every momentum singular value is far below one, so targeted orthogonalization can only amplify, and Muon wins by making directions that are too small to train on at their raw scale trainable. Second, shrinking the largest singular values is what keeps the parameter spectrum flat; this is cheap, and it is not what drives the loss. Third, AdamW differs from Muon mainly in how slowly it builds structure, which explains its slower start and why warmup helps AdamW but only hurts Muon.