Loop Dropout: Regularizing Shared Updates in Looped Language Models
cs.LG, cs.CL
Submitted: 2026-09-28
Updated: 2026-10-04
Code: https://github.com/EleutherAI/lm-evaluation-harness
License: http://creativecommons.org/licenses/by/4.0/
The gist: Looped language models separate computational depth from parameter count by repeatedly applying the same transformer block.
Terminology
Abstract
Looped language models separate computational depth from parameter count by repeatedly applying the same transformer block. Adapting these models requires a shared update that remains effective as hidden states evolve throughout the recurrent computation. Our empirical analysis reveals a pronounced late-loop bias in standard low-rank adaptation (LoRA): the shared update provides limited adaptation at early loop positions. This imbalance motivates training shared updates under varying combinations of their applications. Randomly omitting adapter applications alone, however, does not improve task performance; it reduces expected update strength during training while leaving inference unchanged. We introduce Loop Dropout, which couples stochastic masking of adapter applications with inverse-survival rescaling to preserve expected update strength and promote effective adaptation across loops. Extensive experiments demonstrate improved mathematical reasoning across model sizes, adapter ranks and training recipes, with benefits extending to general instruction tuning and code generation. Loop Dropout outperforms existing LoRA variants and adapter regularizers, while further analysis shows stronger early-loop adaptation. Every backbone loop remains active, and inference applies the adapter at all loops using standard LoRA without additional trainable parameters or inference computation. Code is available at https://github.com/NUS-HPC-AI-Lab/loop-dropout.
Sources
- Program Synthesis with Large Language Models
- Deep Equilibrium Models
- Evaluating Large Language Models Trained on Code
- Training Verifiers to Solve Math Word Problems
- Universal Transformers
- QLoRA: Efficient Finetuning of Quantized LLMs
- Think-at-Hard: Dynamic Looped Transformers for Improved Reasoning
- A Theoretically Grounded Application of Dropout in Recurrent Neural Networks
- Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach
- Adaptive Computation Time for Recurrent Neural Networks
- LoRA+: Efficient Low Rank Adaptation of Large Models
- Measuring Massive Multitask Language Understanding
- Measuring Mathematical Problem Solving With the MATH Dataset
- LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models
- Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2
- LoopFormer: Elastic-Depth Looped Transformers for Latent Reasoning via Shortcut Modulation
- Step-resolved data attribution for looped transformers
- A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA
- Large Language Models are Zero-Shot Reasoners
- Mixout: Effective Regularization to Finetune Large-scale Pretrained Language Models
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks