Protoreasoning in Tiny Transformers

arXiv:2608.04980 · cs.CL, cs.AI, cs.LG · Submitted 2026-08-05 · Read on arXiv

Eduardo Valle, Fergal Reid

cs.CL, cs.AI, cs.LG

Submitted: 2026-08-05

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Terminology

• Trace — 84× in this paper · explained in 2 episodes
• Generalization — 37× in this paper · explained in 14 episodes
• Transformer — 19× in this paper · explained in 2 episodes
• Expressivity — 13× in this paper · explained in 2 episodes
• Transformers — 13× in this paper · explained in 2 episodes
• Accuracy — 9× in this paper · explained in 1 episode
• Reasoning Traces — 8× in this paper · explained in 4 episodes
• Augmentation — 7× in this paper · explained in 2 episodes
• Decoder — 5× in this paper · explained in 2 episodes
• Ablation — 4× in this paper · explained in 1 episode
• Data Augmentation — 4× in this paper · explained in 10 episodes
• Fine-Tuning — 3× in this paper · explained in 5 episodes
• International Conference on Learning Representations — 3× in this paper
• Chain-of-Thought Prompting — 2× in this paper · explained in 2 episodes
• Hyperparameters — 2× in this paper · explained in 2 episodes
• In-Context Learning — 2× in this paper · explained in 7 episodes
• Interpretability — 2× in this paper · explained in 15 episodes
• Papers — 2× in this paper · explained in 3 episodes
• Scaling Laws — 2× in this paper · explained in 4 episodes
• Weight Decay — 2× in this paper · explained in 2 episodes
• World Model — 2× in this paper · explained in 8 episodes
• Benchmark — 1× in this paper · explained in 2 episodes
• Compositionality — 1× in this paper · explained in 2 episodes
• Consistency — 1× in this paper · explained in 3 episodes
• EMMA — 1× in this paper · explained in 2 episodes
• Embeddings — 1× in this paper · explained in 2 episodes
• Faithfulness — 1× in this paper · explained in 2 episodes
• Foundation Model — 1× in this paper · explained in 3 episodes
• Foundation Models — 1× in this paper · explained in 8 episodes
• Generative Model — 1× in this paper · explained in 2 episodes
• Generator — 1× in this paper · explained in 2 episodes
• Grokking — 1× in this paper · explained in 2 episodes
• In-Context Learning (ICL) — 1× in this paper · explained in 11 episodes
• Language Models (LLMs) — 1× in this paper · explained in 2 episodes
• Large Language Models (LLMs) — 1× in this paper · explained in 9 episodes
• Mechanistic Interpretability — 1× in this paper · explained in 4 episodes
• Pruning — 1× in this paper · explained in 4 episodes
• Scaffold — 1× in this paper · explained in 2 episodes
• Self-Consistency — 1× in this paper · explained in 3 episodes
• Transformer Encoder — 1× in this paper · explained in 2 episodes
• Verifiability — 1× in this paper · explained in 3 episodes
• Zero-Shot Generalization — 1× in this paper · explained in 5 episodes

Sources

• A Comment On "The Illusion of Thinking": Reframing the Reasoning Cliff as an Agentic Gap
• Measuring Faithfulness in Chain-of-Thought Reasoning
• Diagnosing CFG Interpretation in LLMs
• An expressivity analysis of hierarchical modelling in deep transformers via bounded-depth grammars
• Comment on The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity
• The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity
• Rethinking the Illusion of Thinking
• Long Context Alignment with Short Instructions and Synthesized Positions

Related papers

• Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
• Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
• Subliminal Steering: Stronger Encoding of Hidden Signals
• MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
• The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
• Untangling the Mechanisms of Misleading Context in Medical Question Answering