How Much Is One Recurrence Worth? Iso-Depth Scaling Laws for Looped Language Models

arXiv:2604.21106 · cs.LG, cs.CL · Submitted 2026-04-22 · Read on arXiv

cs.LG, cs.CL

Submitted: 2026-04-22

Updated: 2026-10-28

Comments: Accepted at NeurIPS 2026 as a Spotlight paper

Code: https://github.com/karpathy/nanochat

License: http://creativecommons.org/licenses/by/4.0/

The gist: We measure how much one recurrence is worth to a looped transformer in equivalent unique parameters.

Terminology

Abstract

We measure how much one recurrence is worth to a looped transformer in equivalent unique parameters. From an iso-depth pretraining sweep across recurrence counts r in 1, 2, 4, 8 spanning about 50 times in training compute, we fit a joint scaling law L = E + A,(N once + r φ N rec)-α + B,D-β and measure a recurrence-equivalence exponent φ= 0.46. Intuitively, φ tells us whether looping a block r times is equivalent in validation loss to r unique blocks of a non-looped model (full equivalence, φ = 1) or to a single block run repeatedly with no capacity gain (φ = 0). Our φ= 0.46 sits in between, so replacing unique blocks with shared recurrences increases validation loss at matched training compute. For example, at r = 4 a 410M looped model performs on par with a 580M non-looped model, but incurs the training cost of a 1B non-looped one. We demonstrate the utility of φ as a diagnostic tool on two case studies: commonly used truncated backpropagation lowers φ to 0.38, indicating that the loop mechanism is poorly trained under truncation, even though validation loss decreases. Conversely, hyperconnections raise φ to 0.65, a genuine capacity gain. Our method separates true loop improvements from training-side gains, a distinction raw validation loss cannot make. Code is available at https://github.com/kschwethelm/looped-lm-scaling.

Sources

Related papers