Scaling Laws for Looped Mixture of Experts
cs.LG, cs.AI, cs.CL
Submitted: 2026-09-30
Updated: 2026-09-30
Comments: 19 pages
License: http://creativecommons.org/licenses/by/4.0/
The gist: Looped transformers and Mixture-of-Experts (MoE) offer complementary routes to efficient scaling: recurrence increases computational depth at fixed parameters, while MoE sparsity expands total
Terminology
Abstract
Looped transformers and Mixture-of-Experts (MoE) offer complementary routes to efficient scaling: recurrence increases computational depth at fixed parameters, while MoE sparsity expands total capacity at fixed active compute. Yet existing scaling laws model recurrence or sparsity in isolation. In this work, we introduce Loop Scaling Laws, the first scaling law to jointly model recurrence and sparsity alongside model size and data. At its core is a bounded, sparsity-conditional recurrence mapping that characterizes the effective-parameter gain from looping and how sparsity raises this gain. The laws predict the held-out loss of looped models more accurately than prior alternatives, and recover the standard dense and MoE scaling laws as special cases. Beyond prediction, the fitted laws provide a principled foundation for designing looped MoE models under compute and memory constraints. Downstream evaluations further demonstrate the complementary benefits of the two axes: sparsity delivers 3x active-parameter efficiency, recurrence yields 2x total-parameter efficiency on reasoning, and joint scaling further advances the performance frontier. As a practical extension, we show these gains hold at trillion-token scale: at matched training compute, a looped MoE with law-derived recurrence matches a 2x larger non-looped MoE on the reasoning benchmarks, while enabling test-time scaling through recurrence.
Sources
- Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of-Experts Language Models
- Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRA
- MobileMoE: Scaling On-Device Mixture of Experts
- Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
- Training Verifiers to Solve Math Word Problems
- DeepSeek-V3 Technical Report
- Universal Transformers
- Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach
- Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
- Looped Transformers as Programmable Computers
- ELT: Elastic Looped Transformers for Visual Generation
- Measuring Massive Multitask Language Understanding
- Training Compute-Optimal Large Language Models
- LoopFormer: Elastic-Depth Looped Transformers for Latent Reasoning via Shortcut Modulation
- Mixtral of Experts
- Scaling Laws for Neural Language Models
- Scaling Laws for Fine-Grained Mixture of Experts
- Sparse Layers are Critical to Scaling Looped Language Models
- Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficient
- Teaching Pretrained Language Models to Think Deeper with Retrofitted Recurrence
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks