Spectrally Targeted Muon

Name: Spectrally Targeted Muon
Uploaded: 2026-10-15T06:01:19.279481+00:00
Duration: 421 s
Description: The Muon optimizer forces update matrices to have singular values of one, which works well for large models. The researchers introduced Spectrally Targeted Muon, which selectively orthogonalizes singular values based on a threshold ($ au$). This approach reveals that small singular values are crucial for training and that shrinking the largest ones flattens the parameter spectrum.

The Muon optimizer forces update matrices to have singular values of one, which works well for large models. The researchers introduced Spectrally Targeted Muon, which selectively orthogonalizes singular values based on a threshold ($ au$). This approach reveals that small singular values are crucial for training and that shrinking the largest ones flattens the parameter spectrum.

Open the full episode