Towards Training Transformers as a Universal Computer

arXiv:2604.25166 · cs.AI · Submitted 2026-04-28 · Read on arXiv

cs.AI

Submitted: 2026-04-28

Updated: 2026-10-29

Comments: Findings of EMNLP 2026. 22 pages, 5 figures, 3 tables. Project page: https://www.micropy.org

Journal ref: Findings of the Association for Computational Linguistics: EMNLP 2026, pages 48933-48954

Code: https://github.com/karpathy/nanochat

License: http://creativecommons.org/licenses/by/4.0/

The gist: We demonstrate that a small transformer can learn to execute programs in MicroPy, a simplified yet computationally universal programming language.

Terminology

Abstract

We demonstrate that a small transformer can learn to execute programs in MicroPy, a simplified yet computationally universal programming language. Given procedure definitions together with an expression to evaluate, the transformer predicts small-step execution using PENCIL scaffolding for space-efficient execution within a bounded context window. After training on randomly generated, semantically arbitrary MicroPy programs, the learned transformer generalizes to held-out human-written programs including bit copying and flipping, binary addition and multiplication, and SAT verification and solving. The trained model exhibits out-of-distribution generalization: it correctly evaluates programs drawn from a substantially different distribution than the one used to generate its training data. Since MicroPy can express any computation, our results provide empirical evidence that a standard transformer can be trained to act as a universal computer when paired with PENCIL scaffolding. Project page: https://www.micropy.org

Sources

Related papers