Training Language Models via Neural Cellular Automata
cs.LG, cs.AI, cs.CL
Submitted: 2026-03-09
Updated: 2026-11-11
Code: https://github.com/danihyunlee/nca-pre-pretraining
Project page: http://skylion007.github.io/OpenWebTextCorpus
Terminology
Sources
- SemDeDup: Data-efficient learning at web-scale through semantic deduplication
- Measuring Gender and Racial Biases in Large Language Models
- Program Synthesis with Large Language Models
- Small Language Models are the Future of Agentic AI
- AutomataGPT: Forecasting and Ruleset Inference for Two-Dimensional Cellular Automata
- Universal pre-training by iterated random computation
- Language Models are Few-Shot Learners
- Sudden Drops in the Loss: Syntax Acquisition, Phase Transitions, and Simplicity Bias in MLMs
- Evaluating Large Language Models Trained on Code
- Pre-Training a Language Model Without Human Language
- Training Verifiers to Solve Math Word Problems
- Programming by Backprop: An Instruction is Worth 100 Examples When Finetuning LLMs
- Language Modeling Is Compression
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Language acquisition: do children and language models follow similar learning stages?
- From Entropy to Epiplexity: Rethinking Information for Computationally Bounded Intelligence
- General Intelligence Requires Reward-based Pretraining
- Emergence and Effectiveness of Task Vectors in In-Context Learning: An Encoder Decoder Perspective
- Between Circuits and Chomsky: Pre-pretraining on Formal Languages Imparts Linguistic Biases
- The Platonic Representation Hypothesis
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis