Depth-Recurrent Attention Mixture (Dreamer): Giving Latent Reasoning the Attention it Deserves
cs.AI, cs.CL, cs.LG
Submitted: 2026-01-29
Updated: 2026-11-05
Comments: NeurIPS 2026
Code: https://github.com/project-numina/aimo-progressprize
License: http://creativecommons.org/licenses/by/4.0/
The gist: Depth-recurrence facilitates latent reasoning by sharing parameters across depths.
Terminology
Abstract
Depth-recurrence facilitates latent reasoning by sharing parameters across depths. However, prior work lacks combined FLOP-, parameter-, and memory-matched baselines, underutilizes depth-recurrence due to partially fixed layer stacks, and ignores the bottleneck of constant hidden-sizes that restricts many-step latent reasoning. To address this, we introduce a modular framework of depth-recurrent attention mixtures (Dreamer), combining sequence attention, depth attention, and sparse expert attention. It alleviates the hidden-size bottleneck through attention along depth, decouples scaling dimensions, and allows depth-recurrent models to scale efficiently and effectively. Across language reasoning benchmarks, our models require 2 to 8x fewer training tokens for the same accuracy as FLOP-, parameter-, and memory-matched SOTA, and outperform ca. 2x larger SOTA models with the same training tokens. We further present insights into knowledge usage across depths, e.g., showing 2 to 11x larger expert selection diversity than SOTA MoEs.
Sources
- NVIDIA Nemotron Nano 2: An Accurate and Efficient Hybrid Mamba-Transformer Reasoning Model
- Adaptive Integrated Layered Attention (AILA)
- Training Verifiers to Solve Math Word Problems
- Continuous Thought Machines
- Universal Transformers
- Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach
- OpenThoughts: Data Recipes for Reasoning Models
- Training Large Language Models to Reason in a Continuous Latent Space
- DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoning
- Less is More: Recursive Reasoning with Tiny Networks
- Encode, Think, Decode: Scaling test-time reasoning with recursive latent thoughts
- Megrez2 Technical Report
- DeepSeek-V3 Technical Report
- Orca-Math: Unlocking the potential of SLMs in Grade School Math
- AIMO-2 Winning Solution: Building State-of-the-Art Mathematical Reasoning Models with OpenMathReasoning dataset
- GLU Variants Improve Transformer
- ReXMoE: Reusing Experts with Minimal Overhead in Mixture-of-Experts
- Hierarchical Reasoning Model
- Parallel Loop Transformer for Efficient Test-Time Computation Scaling
- NaturalReasoning: Reasoning in the Wild with 2.8M Challenging Questions
Related papers
- Multi-Objective Structured Pruning of LLMs for Latency and Model Size Optimization
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Embeddings for Preferences, Not Semantics
- Can We Trust Item Response Theory for AI Evaluation?
- Diagnosing Task Insensitivity in Language Agents