Your Transformer Can Hold Two Thoughts at Once: Evidence of Linear Superposition in LLMs
cs.CL, cs.AI
Submitted: 2026-09-24
Updated: 2026-09-24
License: http://creativecommons.org/licenses/by-nc-nd/4.0/
The gist: While Large Language Models (LLMs) rely on highly non-linear components, in this work we demonstrate that they exhibit fundamental linearity: when inputs from distinct text streams are linearly
Terminology
Abstract
While Large Language Models (LLMs) rely on highly non-linear components, in this work we demonstrate that they exhibit fundamental linearity: when inputs from distinct text streams are linearly combined, the model outputs a superposition of the individual next-token distributions. We term this the Superposition Linearity Hypothesis. We provide evidence that superposition is an intrinsic property of the Transformer architecture rather than an emergent consequence of training; in fact, we observe that it tends to diminish as pretraining progresses. However, we demonstrate that linearity can be substantially restored through lightweight fine-tuning, significantly reducing the divergence between the predicted next-token distribution and the average of the individual next-token distributions. Finally, we introduce a guided decoding procedure that disentangles superposed outputs, enabling the simultaneous generation of two coherent continuations from a single forward pass.
Sources
- Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling
- TinyStories: How Small Can Language Models Be and Still Speak Coherent English?
- The Llama 3 Herd of Models
- OLMo: Accelerating the Science of Language Models
- MIMONets: Multiple-Input-Multiple-Output Neural Networks Exploiting Computation in Superposition
- DataMUX: Data Multiplexing for Neural Networks
- The LAMBADA dataset: Word prediction requiring a broad discourse context
- The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale
- Superposed Decoding: Multiple Generations from a Single Autoregressive Inference Pass
- Gemma 2: Improving Open Language Models at a Practical Size
- Everything Everywhere All at Once: LLMs can In-Context Learn Multiple Tasks in Superposition
- Qwen3 Technical Report
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering