TextNCA: Neural Cellular Automata for Language Modeling via Hierarchical Local Attention
Avni Mittal, Avinash Anand, Ashutosh Kumar, Dikshant Kukreja, Kritarth Prasad, Sushane Dulloo, Erik Cambria, Timothy Liu, Zhengkui Wang, Rajiv Ratn Shah
cs.CL, cs.AI, cs.LG
Submitted: 2026-08-03
License: http://creativecommons.org/licenses/by/4.0/
The gist: Can a strictly local, iterated, weight-shared computation primitive support language modelling, and which of those three properties actually drives the model's behaviour? We define TextNCA, a 1D
Terminology
Abstract
Can a strictly local, iterated, weight-shared computation primitive support language modelling, and which of those three properties actually drives the model's behaviour? We define TextNCA, a 1D causal windowed-attention realisation of the Neural Cellular Automaton primitive, and study a hierarchical variant that cascades three stages with windows w in 8, 32, 128 and T s shared-weight iterations per stage, all on WikiText-103 at roughly 30M parameters and 60k training steps. The model does not match a parameter-matched Transformer at this scale (Hier-TextNCA 60.3 vs. Transformer-6L 52.8 and Transformer-12L 44.7 PPL), so we treat it as an analytical probe rather than a proposed alternative. The behaviour we observe is largely explained by the staged narrow-to-wide schedule: a non-iterating sliding-window Transformer that reuses the same schedule comes within +4.1 PPL of the iterated model, while reversing, flattening, or breaking the monotonic ordering of the schedule costs between +16.7 and +70.8 PPL. Iteration adds a smaller bounded benefit on top of the schedule, with a clear optimum at T s = 4 and a U-shaped degradation beyond it. The GRU gate and learned per-step embeddings are required for that benefit to appear, and training with random T s yields an inference-time iteration-count knob at the cost of substantially higher absolute PPL. We position the work as a controlled reading of which parts of NCA-style computation carry the weight in language modelling.
Sources
- Longformer: The Long-Document Transformer
- Training Deep Nets with Sublinear Memory Cost
- Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach
- Adaptive Computation Time for Recurrent Neural Networks
- Scaling Laws for Neural Language Models
- Training Language Models via Neural Cellular Automata
- Pointer Sentinel Mixture Models
- Texture Generation with Neural Cellular Automata
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering