Tight Complexity of Muon under Heavy-Tailed Noise
The study investigates optimizing neural networks using gradient-based methods under heavy-tailed noise and general norms. It establishes a dimension-dependent lower bound for required oracle calls and proposes improved optimization methods, like a transported Scion method, achieving better convergence rates for practical applications in training CNNs and transformers.