ParaTTT: Parallelizing Test-Time Training Layers with Deep Memories
cs.LG
Submitted: 2026-10-28
Updated: 2026-11-14
Comments: 34 pages, 7 figures. Project repository: https://github.com/ntumm120/paraTTT
Code: https://github.com/ntumm120/paraTTT
License: http://creativecommons.org/licenses/by/4.0/
The gist: Test-time training (TTT) layers replace the linear fast-weight memory of modern recurrent architectures with a small neural network trained online by gradient descent.
Abstract
Test-time training (TTT) layers replace the linear fast-weight memory of modern recurrent architectures with a small neural network trained online by gradient descent. This nonlinear memory is more expressive than the linear memories of Gated DeltaNet and Mamba-2, but is difficult to parallelize: existing implementations such as TTT-MLP rely on sequential, block-wise weight updates that run several times slower than chunkwise-parallel linear recurrences. Moreover, their block-wise execution only approximates the token-by-token online recurrence, whereas linear recurrences admit an exact chunkwise-parallel evaluation. We introduce ParaTTT (PTTT), an algorithm for parallelizing arbitrarily deep nonlinear memories by iteratively refining estimates of the inner-loop gradient trajectory. Starting from a guess of the hidden-layer gradients, each refinement consists of three chunkwise-parallel passes: a forward pass that evaluates the resulting feature trajectory, an exact gated-delta-rule pass that computes the readout trajectory and its feature gradients, and a backward pass that propagates these gradients into a new estimate. The exact online gradient descent recurrence is a fixed point of the iteration, while decoding requires no refinement iterations. We prove geometric convergence for a two-layer variant under sufficient contraction conditions. In practice, two iterations suffice. With two refinement steps at 4K context, PTTT evaluates the inner-loop trajectory 32 times faster than TTT-MLP with the block approximation at matched error, while the full 340M model trains 4.9 times faster. Models trained at 340M and 1.3B parameters match or exceed Gated DeltaNet and Mamba-2 and remain competitive with same-size Transformers on language-modeling perplexity and zero-shot accuracy, with better recall than the linear recurrences.
Sources
- Just read twice: closing the recall gap for recurrent language models
- ATLAS: Learning to Optimally Memorize the Context at Test Time
- T^2MLR: Transformer with Temporal Middle-Layer Recurrence
- Universal Transformers
- Towards Scalable and Stable Parallelization of Nonlinear RNNs
- Training Large Language Models to Reason in a Continuous Latent Space
- Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention
- Lattice: Learning to Efficiently Compress the Memory
- Kimi Linear: An Expressive, Efficient Attention Architecture
- Stability and Generalization in Looped Transformers
- M squared RNN: Non-Linear RNNs with Matrix-Valued States for Scalable Language Modeling
- Fixed-Point Reasoners: Stable and Adaptive Deep Looped Transformers
- Quantifying Memory Utilization with Effective State-Size
- End-to-End Test-Time Training for Long Context
- Test-time regression: a unifying framework for designing sequence models with associative memory
- Full-bandwidth transformer
- Parallelizing Linear Transformers with the Delta Rule over Sequence Length
- Scaling Latent Reasoning via Looped Language Models
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis