Demystifying Manifold Constraints in LLM Pre-training
The study investigates how restricting LLM weights to specific geometric spaces (manifolds) affects training dynamics. The authors propose MACRO, a Riemannian optimizer, to solve this constrained optimization problem. Results show that these constraints offer a principled alternative to weight decay by regulating rotational dynamics and update ratios, leading to more stable and efficient training compared to existing methods.