Stability and Generalization in Looped Transformers
cs.LG, cs.AI
Submitted: 2026-04-16
Updated: 2026-11-04
Comments: 10 main pages, 30 total. Accepted to NeurIPS 2026 Main Track
Code: https://github.com/ashlab11/generalization
Project page: https://t-dillon.github.io/tdoku
License: http://creativecommons.org/licenses/by/4.0/
The gist: Looped transformers promise test-time compute scaling by spending more iterations on harder problems, but it remains unclear which architectural choices let them extrapolate to harder problems at…
Terminology
Abstract
Looped transformers promise test-time compute scaling by spending more iterations on harder problems, but it remains unclear which architectural choices let them extrapolate to harder problems at test time rather than memorize training-specific solutions. We introduce a fixed-point based framework for analyzing looped architectures along three axes of stability -- local attractivity, input-dependence, and geometry -- and use it to characterize when fixed-point iteration yields meaningful predictions. Theoretically, we prove that looped networks without recall have countable fixed points and cannot achieve strong input-dependence at any spectral regime, while recall with outer normalization reliably produces a regime in which fixed points are simultaneously locally-attractive, locally smooth in the input, and supported by stable backpropagation. Empirically, we train small looped transformers on chess, sudoku, and prefix-sums and find that downstream performance tracks the framework's predictions across tasks and architectural configurations. We additionally introduce internal recall, a novel recall placement variant allowing an identity residual path, and exploit its contrasting fixed-point behavior to test our framework's predictions in a controlled setting.
Sources
- Path Independent Equilibrium Models Can Better Exploit Test-Time Computation
- PonderNet: Learning to Ponder
- End-to-end Algorithm Synthesis with Recurrent Networks: Logical Extrapolation Without Overthinking
- Universal Transformers
- Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach
- Query-Key Normalization for Transformers
- Less is More: Recursive Reasoning with Tiny Networks
- Normalization in Attention Dynamics
- Peri-LN: Revisiting Normalization Layer in the Transformer Architecture
- Exact Expressive Power of Transformers with Padding
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- Hierarchical Reasoning Model
- Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- On Layer Normalization in the Transformer Architecture
- Looped Transformers are Better at Learning Learning Algorithms
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis