Generalization and Scaling Laws for Mixture-of-Experts Transformers
The theory develops a framework for Mixture-of-Experts (MoE) Transformers by separating capacity from routing complexity. It derives generalization bounds showing that MoE scaling mirrors dense models when measured against active parameters, revealing neural scaling laws for sparse architectures. This provides a statistical reference point to distinguish between guaranteed performance and data-dependent routing effects.