How Much Is One Recurrence Worth? Iso-Depth Scaling Laws for Looped Language Models
cs.LG, cs.CL
Submitted: 2026-04-22
Updated: 2026-10-28
Comments: Accepted at NeurIPS 2026 as a Spotlight paper
Code: https://github.com/karpathy/nanochat
License: http://creativecommons.org/licenses/by/4.0/
The gist: We measure how much one recurrence is worth to a looped transformer in equivalent unique parameters.
Terminology
Abstract
We measure how much one recurrence is worth to a looped transformer in equivalent unique parameters. From an iso-depth pretraining sweep across recurrence counts r in 1, 2, 4, 8 spanning about 50 times in training compute, we fit a joint scaling law L = E + A,(N once + r φ N rec)-α + B,D-β and measure a recurrence-equivalence exponent φ= 0.46. Intuitively, φ tells us whether looping a block r times is equivalent in validation loss to r unique blocks of a non-looped model (full equivalence, φ = 1) or to a single block run repeatedly with no capacity gain (φ = 0). Our φ= 0.46 sits in between, so replacing unique blocks with shared recurrences increases validation loss at matched training compute. For example, at r = 4 a 410M looped model performs on par with a 580M non-looped model, but incurs the training cost of a 1B non-looped one. We demonstrate the utility of φ as a diagnostic tool on two case studies: commonly used truncated backpropagation lowers φ to 0.38, indicating that the loop mechanism is poorly trained under truncation, even though validation loss decreases. Conversely, hyperconnections raise φ to 0.65, a genuine capacity gain. Our method separates true loop improvements from training-side gains, a distinction raw validation loss cannot make. Code is available at https://github.com/kschwethelm/looped-lm-scaling.
Sources
- Think-at-Hard: Dynamic Looped Transformers for Improved Reasoning
- Teaching Pretrained Language Models to Think Deeper with Retrofitted Recurrence
- Encode, Think, Decode: Scaling test-time reasoning with recursive latent thoughts
- Parcae: Scaling Laws For Stable Looped Language Models
- Depth-Recurrent Attention Mixture (Dreamer): Giving Latent Reasoning the Attention it Deserves
- Hyperloop Transformers
- Scaling Laws for Neural Language Models
- Llama 2: Open Foundation and Fine-Tuned Chat Models
- Chinchilla Scaling: A replication attempt
- Test-Time Scaling Makes Overtraining Compute-Optimal
- Gemma 2: Improving Open Language Models at a Practical Size
- In-context Learning and Induction Heads
- Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis