Recirculation
cs.LG
Submitted: 2026-08-18
Updated: 2026-09-28
Comments: v3: new experiments
License: http://creativecommons.org/licenses/by-nc-sa/4.0/
The gist: We describe an inference-time architectural enhancement for off-the-shelf foundation models that systematically reduces perplexity and boosts accuracy across generation and reasoning tasks.
Terminology
Abstract
We describe an inference-time architectural enhancement for off-the-shelf foundation models that systematically reduces perplexity and boosts accuracy across generation and reasoning tasks. Our approach incurs essentially no additional latency during generation, though it requires serial processing in the prefill phase. Motivated by the fundamental limitation that state updates in feedforward transformers are bounded by model depth, our technique, recirculation, introduces a specific form of recurrence that allows the model to act as a dynamical system and track belief states. We distinguish this technique from chain-of-thought computation---which is better reserved for complex inferences rather than basic state tracking---as well as from popular depth-recurrence techniques (looping) and the costly training of recurrent transformers. We also propose and evaluate an adaptive variant of recirculation which requires only light tuning of hyperparameters while freezing the original model weights. Relative to the off-the-shelf baseline, adaptive recirculation achieves surprising gains on the Gemma3 family, including a consistent reduction in perplexity on a suite of datasets, a 21% increase in relative accuracy on GSM8k, and reliable improvements in accuracy on other downstream tasks. Our training-free approach succeeds by leveraging the model itself to inform architectural modifications, suggesting a route to architectural evolution guided by a trained network's properties rather than forced, arbitrary design choices.
Sources
- Physics of Language Models: Part 4.1, Architecture Design and the Magic of Canon Layers
- Modeling Language as a Sequence of Thoughts
- Training-Free Looped Transformers
- Inner Thinking Transformer: Leveraging Dynamic Depth Scaling to Foster Adaptive Internal Thinking
- Adapting Language Models to Compress Contexts
- Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
- Training Verifiers to Solve Math Word Problems
- The Collaboration Gap: Exploration and Benchmarking of Open-World Agentic Cooperation
- Addressing Some Limitations of Transformers with Feedback Memory
- Catch Your Breath: Adaptive Computation for Self-Paced Sequence Production
- Gemma: Open Models Based on Gemini Research and Technology
- Semantic Tube Prediction: Beating LLM Data Efficiency with JEPA
- Scalable Adaptive Computation for Iterative Generation
- Less is More: Recursive Reasoning with Tiny Networks
- CooperBench: Why Coding Agents Cannot be Your Teammates Yet
- Peri-LN: Revisiting Normalization Layer in the Transformer Architecture
- Encode, Think, Decode: Scaling test-time reasoning with recursive latent thoughts
- LLMs Get Lost In Multi-Turn Conversation
- Skip a Layer or Loop it? Test-Time Depth Adaptation of Pretrained LLMs
- The Serial Scaling Hypothesis
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks