Length Is Cheap, Competition Is Not: Attention Predicts Context Rot in Retrieval
cs.CL, cs.AI
Submitted: 2026-11-03
Updated: 2026-11-03
Comments: 15 pages, 2 figures, 5 tables. Code and trial-level data: https://github.com/yliusci/length-is-cheap
Code: https://github.com/yliusci/length-is-cheap
License: http://creativecommons.org/licenses/by/4.0/
The gist: Why do language models fail on long inputs? The common answer is length.
Abstract
Why do language models fail on long inputs? The common answer is length. We show that, for retrieval, length is cheap and competition is not. On five open models (0.5B to 3B parameters, four families), neutral filler costs nothing: accuracy stays at 0.97 to 1.00, up to 32K tokens on Qwen2.5-1.5B. Statements in the format of the evidence, placed at the same positions, drop it to 0.16 to 0.33. A single statement that gives the queried key a rival value multiplies the loss by 9 to 110, and every additional rival adds to it. Inside the model, one quantity tracks the failure: the attention mass left on the evidence. Across 126,473 trials it explains up to 58% of the variance in loss; log length explains at most 8%. The masked-filler protocol behind the claim that length alone hurts reproduces on MMLU (11 to 18 points lost at 30K tokens), but fails a visible control in retrieval: masking costs 45 accuracy points at 32K tokens, while the same filler left visible costs none. In retrieval, context rot is a competition for attention. We release the probe and all 126K trial-level measurements.
Sources
- Extending Context Window of Large Language Models via Positional Interpolation
- YaRN: Efficient Context Window Extension of Large Language Models
- LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens
- Landmark Attention: Random-Access Infinite Context Length for Transformers
- Memorizing Transformers
- The Llama 3 Herd of Models
- Qwen2.5 Technical Report
- RULER: What's the Real Context Size of Your Long-Context Language Models?
- Why Does the Effective Context Length of LLMs Fall Short?
- Insights into LLM Long-Context Failures: When Transformers Know but Don't Tell
- Lost in the Middle: How Language Models Use Long Contexts
- LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding
- $\infty$Bench: Extending Long Context Evaluation Beyond 100K Tokens
- BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack
- Same Task, More Tokens: the Impact of Input Length on the Reasoning Performance of Large Language Models
- Diagnosing and Mitigating Context Rot in Long-horizon Search
- Context Length Alone Hurts LLM Performance Despite Perfect Retrieval
- Attention Is All You Need
- L-Eval: Instituting Standardized Evaluation for Long Context Language Models
- ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understanding
Related papers
- ALARM: Audio-Language Alignment for Reasoning Models
- ThinkSum: Probabilistic reasoning over sets using large language models
- Coherence boosting: When your pretrained language model is not paying enough attention
- From Facts to Insights: A Persona-Driven Dual Memory Framework and Dataset for Role-Playing Agents
- Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization
- The Art of Asking: Multilingual Prompt Optimization for Synthetic Data