Holistic Scaling Laws for Optimal Mixture-of-Experts Architecture Optimization
The framework optimizes Mixture-of-Experts (MoE) architectures by systematically studying their properties at scale. It addresses the inadequacy of using only FLOPs per token by employing a mathematical decoupling strategy to reduce the complex search space from O(n^16) to a manageable two-phase process of O(n^3) and O(n^2). This provides a reusable, cost-efficient method for finding optimal MoE designs.