LoopMTP: A looped transformer guided by latent multi-token prediction
Behzad Shomali, Markus Frey, David Berghaus, Joachim Koehler, Mehdi Ali
cs.CL
Submitted: 2026-08-04
License: http://creativecommons.org/licenses/by/4.0/
The gist: Looped transformers have emerged as a parameter-efficient alternative to scaling depth for strong reasoning.
Terminology
Abstract
Looped transformers have emerged as a parameter-efficient alternative to scaling depth for strong reasoning. By reusing one stack of layers across T iterations, they attain the effective depth and reasoning capabilities of larger models at a fixed parameter count. Yet existing approaches suffer from latent overthinking and undifferentiated computation, largely because intermediate representations receive no guidance across loops. Multi-token prediction (MTP) supplies exactly the dense, forward-looking supervision the loop is missing. We propose LoopMTP, which links the two through a structural correspondence in latent space: a model that loops T times can anticipate T future tokens. LoopMTP realizes this by softly aligning the hidden state of loop t with the embedding of the token t steps ahead, while a lightweight gate preserves useful information across iterations. LoopMTP improves average accuracy by up to 8.1% (relative) over the non-looped baseline, with training remaining stable for up to 15 loops.
Sources
- Program Synthesis with Large Language Models
- PonderNet: Learning to Ponder
- NVIDIA Nemotron Nano 2: An Accurate and Efficient Hybrid Mamba-Transformer Reasoning Model
- Thought Anchors: Which LLM Reasoning Steps Matter?
- The MiniMax-M2 Series: Mini Activations Unleashing Max Real-World Intelligence
- Evaluating Large Language Models Trained on Code
- Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
- Training Verifiers to Solve Math Word Problems
- Learning to Plan for Language Modeling from Unlabeled Data
- Training Large Language Models to Reason in a Continuous Latent Space
- Universal Transformers
- Adaptive Loops and Memory in Transformers: Think Harder or Know More?
- A Dual-Path Architecture for Scaling Compute and Capacity in LLMs
- LoopFormer: Elastic-Depth Looped Transformers for Latent Reasoning via Shortcut Modulation
- Simply Stabilizing the Loop via Fully Looped Transformer
- DeepSeek-V3 Technical Report
- Think-at-Hard: Dynamic Looped Transformers for Improved Reasoning
- Better & Faster Large Language Models via Multi-token Prediction
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- Muon is Scalable for LLM Training
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering