Language Models Can Control Their Own Attention
cs.CL, cs.AI, cs.LG
Submitted: 2026-09-02
Updated: 2026-10-14
Code: https://github.com/openai/openai-openapi
License: http://creativecommons.org/licenses/by/4.0/
The gist: Language models spend most of their attention on a small fraction of context, yet they read the entire KV cache to find the few tokens that matter.
Terminology
Abstract
Language models spend most of their attention on a small fraction of context, yet they read the entire KV cache to find the few tokens that matter. If the user asks about a previous detail in a 1M-token conversation, global attention layers must scan the full context to generate each token of the reply. A prominent approach mitigates this cost by pre-selecting relevant tokens via lightweight proxy scores, but this extrinsic scoring still incurs O(N) per step. We take an intrinsic approach motivated by the simple question: wouldn't the model already know which parts of the context are relevant? To this end, we introduce Declarative Attention (DA), a protocol that elicits the model to declare where it needs to attend within its chain-of-thought, partitioning generation into three modes: <global> (full context), <focus> (a specific region), and <local> (recent output only). The inference engine parses these declarations like tool calls and skips most of the KV cache read. Under zero-shot evaluation across 15 long-context tasks, DA on off-the-shelf models (Gemma-4-31B, Qwen-3.6-27B) significantly reduces total attended tokens during decoding (52.0%, 31.1%) with modest accuracy drops (1.27pp, 2.75pp) that shrink with model scale. DA unlocks a new axis of sparse attention, with further potential under training-based methods that future work can explore.
Sources
- SpotAttention: Plug-In Block-Sparse Routing for Pretrained Long-Context Transformers
- Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection
- Longformer: The Long-Document Transformer
- Efficient Heterogeneous Large Language Model Decoding with Model-Attention Disaggregation
- Generating Long Sequences with Sparse Transformers
- PaLM: Scaling Language Modeling with Pathways
- DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model
- DeepSeek-V3 Technical Report
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models
- DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
- How Sparse Attention Approximates Exact Attention? Your Attention is Naturally $n^C$-Sparse
- Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs
- Gemma 4 Technical Report
- GLM-5: from Vibe Coding to Agentic Engineering
- Think before you speak: Training Language Models With Pause Tokens
- Self-Selected Attention Span for Accelerating Large Language Model Inference
- Kimi K3: Open Frontier Intelligence
- Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety
- MiniMax Sparse Attention
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering