Monarch-Muon: Structure-Aware Muon for Fast and Memory-Efficient Pretraining
Monarch-Muon is a structured matrix-aware optimizer designed to make large language model training faster and more memory-efficient. It achieves this by parameterizing weight factors with 'Monarch' factors instead of dense updates, reducing storage for weights, gradients, and optimizer states. This results in significant memory savings while maintaining or improving optimization performance over existing methods.