Large Language Models are Shannon Lossy Compressors Not Solomonoff Induction Estimators: Self-improvement and Singularity Are Not Near Without Symbolic Model Synthesis

arXiv:2601.05280 · cs.IT, cs.AI, cs.LG, math.IT · Submitted 2026-01-05 · Read on arXiv

cs.IT, cs.AI, cs.LG, math.IT

Submitted: 2026-01-05

Updated: 2026-09-28

Comments: 39 pages. Major update

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Terminology

• Grounding — 25× in this paper · explained in 3 episodes
• Generative Model — 9× in this paper · explained in 2 episodes
• Machine Learning — 9× in this paper
• Generative Models — 8× in this paper · explained in 2 episodes
• Verification — 7× in this paper · explained in 2 episodes
• Benchmark — 6× in this paper · explained in 2 episodes
• Transformer — 6× in this paper · explained in 2 episodes
• Verifier — 6× in this paper · explained in 2 episodes
• Algorithmic Information Dynamics — 4× in this paper
• Conservation Laws — 3× in this paper · explained in 3 episodes
• Consistency — 3× in this paper · explained in 3 episodes
• Perplexity — 3× in this paper · explained in 3 episodes
• Block Decomposition Method — 2× in this paper
• Coding Theorem Method — 2× in this paper
• Distribution Matching — 2× in this paper · explained in 2 episodes
• Failure Modes — 2× in this paper · explained in 3 episodes
• Fidelity — 2× in this paper · explained in 2 episodes
• Generative AI — 2× in this paper · explained in 2 episodes
• Hybrid System — 2× in this paper · explained in 2 episodes
• Large Language Models (LLMs) — 2× in this paper · explained in 6 episodes
• Self-consistency — 2× in this paper · explained in 2 episodes
• Abstraction — 1× in this paper · explained in 1 episode
• Accuracy — 1× in this paper · explained in 1 episode
• Algorithmic Information Theory — 1× in this paper
• Alternation — 1× in this paper · explained in 1 episode
• Axioms — 1× in this paper · explained in 2 episodes
• Calibration — 1× in this paper · explained in 7 episodes
• Causal Discovery — 1× in this paper · explained in 2 episodes
• Causal Reasoning — 1× in this paper · explained in 2 episodes
• Degeneracy — 1× in this paper · explained in 3 episodes
• Diffusion Model — 1× in this paper · explained in 8 episodes
• Distribution Shift — 1× in this paper · explained in 6 episodes
• Elias — 1× in this paper · explained in 2 episodes
• Formal Verification — 1× in this paper · explained in 2 episodes
• Large Language Model (LLM) — 1× in this paper · explained in 3 episodes
• Loss Function — 1× in this paper · explained in 2 episodes
• Ontology — 1× in this paper · explained in 2 episodes
• Sample Efficiency — 1× in this paper · explained in 4 episodes
• Scaling Laws — 1× in this paper · explained in 4 episodes

Sources

• Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data
• Transformers As Approximations of Solomonoff Induction
• Large Language Models as Computable Approximations to Solomonoff Induction

Related papers

• Language Modeling is Monotone Compression
• Clipped Affine Policy: Low-Complexity Near-Optimal Online Power Control for Energy Harvesting Communications over Fading Channels
• Discrepancy for Random Linear Codes
• A New Approach to Code Smoothing Bounds
• Contextual Memory-Enhanced Source Coding for Low-SNR Communications
• Symmetry-Enforced Quadratic Approximate-Degradability Bounds for Noisy Landau-Streater Channels