Autonomy-of-Heads: Data-Free Sparse Attention from Frozen Query-Key Geometry
Yehan Yang, Junyuan Shang, Yang Li, Guanqun Zhao, Shuohuan Wang, Dianhai Yu
cs.CL, cs.AI
Submitted: 2026-08-07
Updated: 2026-08-10
License: http://creativecommons.org/licenses/by/4.0/
The gist: Long-context LLM inference is bottlenecked by quadratic attention computation and growing KV-cache costs.
Terminology
Abstract
Long-context LLM inference is bottlenecked by quadratic attention computation and growing KV-cache costs. Existing sparse attention and KV-compression methods typically decide which tokens or heads to preserve from runtime attention scores, observation windows, calibration prompts, or learned gates, making head diagnosis input-dependent and costly to deploy. We propose Autonomy-of-Heads (AoH), a data-free method that identifies retrieval and streaming heads from the spectral geometry of query-key projections. AoH defines the kernel attention operator M h = W K hW Q h and uses its effective-rank as a weight-space measure of head function: concentrated spectra indicate a small number of dominant query-key matching directions and are associated with retrieval heads, whereas diffuse spectra indicate the absence of a dominant global matching direction and are associated with streaming heads. We further derive an efficient d head-dimensional computation that avoids constructing the full d model times d model matrix. We conducted extensive experiments across models demonstrating that at 50% sparsity, AoH retains 96.5% of Full Attention performance on average while reducing prefill and decode latency by up to 41.4% and 66.0%, respectively, and KV-cache memory by 50.0% at 256K tokens.
Sources
- SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills
- IndexCache: Accelerating Sparse Attention via Cross-Layer Index Reuse
- The Llama 3 Herd of Models
- Generalizations of Sylvester's determinantal identity
- Longformer: The Long-Document Transformer
- LycheeDecode: Accelerating Long-Context LLM Inference via Hybrid-Head Sparse Decoding
- Multi-Head Attention: Collaborate Instead of Concatenate
- FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
- DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models
- Kascade: A Practical Sparse Attention Method for Long-Context LLM Inference
- Sliding Window Attention Training for Efficient Large Language Models
- Linear Predictability of Attention Heads in Large Language Models
- HySparse: A Hybrid Sparse Attention Architecture with Oracle Token Selection and KV Cache Sharing
- Qwen3 Technical Report
- Qwen2.5 Technical Report
- Kimi Linear: An Expressive, Efficient Attention Architecture
- D2O: Dynamic Discriminative Operations for Efficient Long-Context Inference of Large Language Models
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering