Learning When Not to Listen: Selective Anti-Interference Pretraining for Language Models
cs.CL
Submitted: 2026-08-22
Updated: 2026-08-22
License: http://creativecommons.org/licenses/by/4.0/
The gist: Language models can over-condition on irrelevant preceding text: predictions already supported by local context may still change when distant, unrelated prefix tokens are perturbed.
Terminology
Abstract
Language models can over-condition on irrelevant preceding text: predictions already supported by local context may still change when distant, unrelated prefix tokens are perturbed. This interference is especially consequential in long, packed, or distractor-heavy contexts, where useful evidence and irrelevant spans coexist. We propose Selective Prefix Anti-Interference Regularization (SPAR), a pretraining objective for selective anti-interference. SPAR runs the original sequence and a corrupt-prefix input in which only the far prefix is changed, then uses a short-context sufficiency gate and a gated KL objective to stabilize locally supported suffix predictions. The gate operationalizes a model-based estimate of whether the far prefix supplies additional information about the target token. Mechanism analyses show that the gate identifies locally sufficient tokens and sharply reduces prefix sensitivity on gate-selected suffix tokens. In continued training on pretrained base models, SPAR improves RULER across Qwen2.5-0.5B, Qwen2.5-3B, Llama-3.2-1B, Llama-3.1-8B, and GPT2-XL under equal counted training compute; pretraining experiments further show gains on both RULER and NoLiMa. These results show that selective anti-interference is an effective objective-level signal for robust context use.
Sources
- LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding
- Longformer: The Long-Document Transformer
- Extending Context Window of Large Language Models via Positional Interpolation
- LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models
- Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context
- LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens
- What is Wrong with Perplexity for Long-context Language Modeling?
- RULER: What's the Real Context Size of Your Long-Context Language Models?
- Reformer: The Efficient Transformer
- LooGLE: Can Long-Context Language Models Understand Long Contexts?
- NeedleBench: Evaluating LLM Retrieval and Reasoning Across Varying Information Densities
- Long-context LLMs Struggle with Long In-context Learning
- Lost in the Middle: How Language Models Use Long Contexts
- NoLiMa: Long-Context Evaluation Beyond Literal Matching
- YaRN: Efficient Context Window Extension of Large Language Models
- Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation
- RoFormer: Enhanced Transformer with Rotary Position Embedding
- Long Context Alignment with Short Instructions and Synthesized Positions
- HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughly
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering