Log-Depth Recurrent Language Modeling
cs.LG, cs.CL
Submitted: 2026-09-23
Updated: 2026-09-23
Comments: 5 pages, 3 figures
License: http://creativecommons.org/licenses/by/4.0/
The gist: Language modeling using Transformers has become commonplace despite their fixed computational depth and quadratic runtime with respect to input tokens.
Terminology
Abstract
Language modeling using Transformers has become commonplace despite their fixed computational depth and quadratic runtime with respect to input tokens. Recurrent models on the other hand offer linear depth but no parallel execution. In this work, we extend balanced-tree recursive operators from sequence encoding to autoregressive prediction, enabling all prefix representations to be computed with logarithmic depth and linear runtime. Our experiments provide an initial characterization of this model class, demonstrating robust length extrapolation and performance approaching that of ALiBi-based Transformers, highlighting its potential as an alternative architecture for language modeling.
Sources
- The Pile: An 800GB Dataset of Diverse Text for Language Modeling
- SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing
- Pointer Sentinel Mixture Models
- Length Generalization with Log-Depth Recurrent Units
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks