No Model Required: Text Entropy Rate Filtering Mitigates Iterative Fine-Tuning Collapse
cs.CL, cs.AI, cs.IT, math.IT, physics.data-an, stat.ML
Submitted: 2026-10-01
Updated: 2026-10-01
Comments: 17 pages, 8 figures, NeurIPS 2026
Code: https://github.com/lewismath/no-model-required
License: http://creativecommons.org/licenses/by/4.0/
The gist: Iterative fine-tuning on synthetic data causes model collapse: output diversity narrows as rare patterns are progressively lost, a signature most visible as phrase-level repetition.
Terminology
Abstract
Iterative fine-tuning on synthetic data causes model collapse: output diversity narrows as rare patterns are progressively lost, a signature most visible as phrase-level repetition. Existing mitigations either require model log-probabilities, an external oracle, or continued access to real human data. Here we develop a new approach grounded in mathematical information theory: the non-parametric Kontoyiannis entropy rate estimator h k, computed entirely from raw text via match-length statistics, with no model of any kind. We show that this is in fact a superior training-data filter on text-diversity metrics in a fully-synthetic, single-lineage fine-tuning setting. In a six-generation QLoRA collapse experiment on Llama-3.1-8B, logprob-based filtering (the most established model-access-requiring baseline) provides no significant text-diversity benefit on any metric (p > 0.23), whereas h k-filtering yields +42% unique trigrams, +30% vocabulary, and-19% repetition (all p < 0.001). We validate h k as a cross-domain entropy proxy (β= 0.924, R squared = 0.746) and collapse detector (ρ= +0.454, p < 0.0001) across 4 domains, 2 temperatures, 2 generator--scorer model pairs, and 1, 520 generated documents. Our results demonstrate that information theoretic approaches to collapse mitigation are efficient, and suggest new approaches for maintaining multi-agent diversity.
Sources
- Large Language Models Suffer From Their Own Output: An Analysis of the Self-Consuming Training Loop
- Diversity Collapse in Multi-Agent LLM Systems: Structural Coupling and Collective Failure in Open-Ended Idea Generation
- The Llama 3 Herd of Models
- Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
- Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data
- Qwen2.5 Technical Report
- Escaping Model Collapse via Synthetic Data Verification: Near-term Improvements and Long-term Convergence
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering