Full-bandwidth transformer
cs.AI
Submitted: 2026-08-09
Updated: 2026-10-01
License: http://creativecommons.org/licenses/by/4.0/
The gist: Autoregressive transformers compute along two axes: horizontally across generated tokens, and vertically through model depth.
Terminology
Abstract
Autoregressive transformers compute along two axes: horizontally across generated tokens, and vertically through model depth. Dense attention gives each token broad horizontal access to the past, but the vertical feedback channel between decoding steps remains narrow: only the sampled token returns to the bottom of the stack, while the top-layer hidden state is discarded. We introduce the full-bandwidth transformer, which widens this channel with latent feedback: at each decoding step, the previous top-layer hidden state is fused with the sampled token embedding through a gated linear unit and fed back as the next input. Latent feedback lets non-verbalized computation re-enter the stack with a renewed depth budget, while preserving the standard transformer architecture, KV cache, and language-modeling objective. To train full-bandwidth transformers without losing parallel teacher forcing, we use a scheduled multi-pass objective that introduces latent feedback late in pretraining and mixes a small fraction of deeper feedback passes for stability. We train 1B-parameter full-bandwidth transformers up to 400B tokens and find that latent feedback improves validation loss, 5-shot language-model evaluation, math and coding generation, and instruction-tuned performance. With negligible per-token decoding overhead, full-bandwidth transformers match or approach standard transformers trained with roughly 1.5 times more tokens, and manage to produce shorter reasoning traces at equal or better accuracy.
Sources
- Phi-4 Technical Report
- Efficient Joint Prediction of Multiple Future Tokens
- Program Synthesis with Large Language Models
- T^2MLR: Transformer with Temporal Middle-Layer Recurrence
- Training Verifiers to Solve Math Word Problems
- Context-level Language Modeling by Learning Predictive Context Embeddings
- ParaRNN: Unlocking Parallel Training of Nonlinear RNNs for Large Language Models
- Why Gradients Rapidly Increase Near the End of Training
- Universal Transformers
- Addressing Some Limitations of Transformers with Feedback Memory
- Bridging the Gap Between Latent and Explicit Reasoning with Looped Transformers
- Better & Faster Large Language Models via Multi-token Prediction
- Training Large Language Models to Reason in a Continuous Latent Space
- MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies
- Latent Recurrent Transformer: Architecture Exploration, Training Strategies, and Scaling Behavior
- Scaling Laws for Neural Language Models
- EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty
- Let's Verify Step by Step
- Next Concept Prediction in Discrete Latent Space Leads to Stronger Language Models
- q0: Primitives for Hyper-Epoch Pretraining
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems