Truth as a Compression Artifact in Language Model Training
cs.CL, cs.AI
Submitted: 2026-03-12
Updated: 2026-11-02
Comments: v4: 30 pages, 2 figures. Revised evaluation using probability mass over the full false support; added negative controls and learnable-selector experiments up to 3.08B parameters. Updated claims and discussion
Code: https://github.com/Rai220/compression-drives-truth
License: http://creativecommons.org/licenses/by/4.0/
The gist: When a transformer is trained from scratch on corpora that solve the same mathematical problem both correctly and incorrectly, which solution does it prefer? The usual test is pairwise…
Terminology
Abstract
When a transformer is trained from scratch on corpora that solve the same mathematical problem both correctly and incorrectly, which solution does it prefer? The usual test is pairwise, and it is not neutral: when the error can take several forms, a model that merely reproduces training frequencies wins it without preferring anything. We instead score the correct completion against every string the error generator can produce, a share on which frequency matching scores exactly 0.500. At 86M parameters the correct completion holds 59.0% +/- 0.8% of the mass against random errors (59.4% on an independent corpus), where the single realized error gives 77.6% and pair accuracy 87.4%, a three- to fourfold overstatement of the excess above chance; below 86M the primary corpus shows no preference. The coherent false rules of the main corpora are not filtered at any size (43.5--48.8%; the shortfall below 0.5 is a completion-length effect). Two experiments built to identify a compression mechanism raise the pairwise measures while the summed share stays flat or falls: both are negative controls, and compression remains motivation, not a measured mechanism. With the scored false strings held fixed, a false rule whose selector the model can learn moves the correct share from 5 points above the null to 7 below (418M--3.08B), though the correct completion still outweighs the selected string. What these models leave unfiltered is consistent or learnable false structure (one single-rule corruption excepted), not falsehood as such.
Sources
- Compression is all you need: Modeling Mathematics
- Predictable Compression Failures: Order Sensitivity and Information Budgeting for Evidence-Grounded Binary Adjudication
- Measuring Causal Effects of Data Statistics on Language Model's `Factual' Predictions
- Language Models (Mostly) Know What They Know
- Grokking as Compression: A Nonlinear Complexity Perspective
- A Compression Perspective on Simplicity Bias
- Understanding LLM Behaviors via Compression: Data Generation, Knowledge Acquisition and Scaling Laws
- The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale
- Deep Learning is Robust to Massive Label Noise
- Large Language Models as Computable Approximations to Solomonoff Induction
Related papers
- ALARM: Audio-Language Alignment for Reasoning Models
- ThinkSum: Probabilistic reasoning over sets using large language models
- Coherence boosting: When your pretrained language model is not paying enough attention
- From Facts to Insights: A Persona-Driven Dual Memory Framework and Dataset for Role-Playing Agents
- Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization
- The Art of Asking: Multilingual Prompt Optimization for Synthetic Data