Sparse Layers are Critical to Scaling Looped Language Models
cs.LG, cs.CL
Submitted: 2026-05-09
Updated: 2026-11-17
License: http://creativecommons.org/licenses/by/4.0/
The gist: Looped language models repeat a set of transformer layers through depth, reducing memory costs and providing natural early-exit points at loop boundaries.
Terminology
Abstract
Looped language models repeat a set of transformer layers through depth, reducing memory costs and providing natural early-exit points at loop boundaries. However, looped models do not scale as favorably as standard transformers with unique layers. We compare standard and Mixture-of-Experts (MoE) transformers, with and without looping, and find two main results. First, we find Looped-MoE models scale better than the standard baseline while dense looped models do not. We trace this to routing divergence between loops: in Looped-MoE models, different experts are activated on each pass through the same shared layers, recovering expressivity without additional parameters. Our second finding is that looped models have better compute-quality trade-offs with early exits than standard models. Because each loop ends with the same layers that produce the final output, loop boundaries are superior exit points, as confirmed by earlier output convergence at these points. In sum, we provide a clear direction for scaling looped models: a Looped-MoE model with early exits can not only beat standard transformers at scale, but also enable significant memory and inference savings with minimal degradation in quality.
Sources
- Universal Transformers
- Scaling Latent Reasoning via Looped Language Models
- Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach
- Scaling Laws for Neural Language Models
- Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
- DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference
- Confident Adaptive Language Modeling
- RoFormer: Enhanced Transformer with Rotary Position Embedding
- GLU Variants Improve Transformer
- ST-MoE: Designing Stable and Transferable Sparse Expert Models
- Mixtral of Experts
- Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer
- $\mu$-Parametrization for Mixture of Experts
- Training Compute-Optimal Large Language Models
- The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale
- MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies
- Training Dynamics of the Cooldown Stage in Warmup-Stable-Decay Learning Rate Scheduler
- OLMES: A Standard for Language Model Evaluations
- Approximating Two-Layer Feedforward Networks for Efficient Transformers
- SwitchHead: Accelerating Transformers with Mixture-of-Experts Attention
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis