From Parallel to Sequential: Small-Batch Noise Restores Simplicity Bias of Muon
The study investigates how adding small-batch noise fundamentally changes Muon's optimization behavior, which usually learns features in parallel. Stochastic smoothing of the spectral descent operator restores sequential feature learning by making updates proportional to actual gradients when noise is present. This mechanism introduces a depth-dependent implicit regularization that favors low-rank intermediate predictors.