Attend at the Right Resolution: Head-Adaptive Sparse Attention via Query-Key Interaction Structure
cs.AI
Submitted: 2026-11-06
Updated: 2026-11-06
Terminology
Sources
- Longformer: The Long-Document Transformer
- LongBench Pro: A More Realistic and Comprehensive Bilingual Long-Context Evaluation Benchmark
- Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities
- UNIQUE: Universal Top-k Sparse Attention for Training-free Inference and Sparsity-aware Training
- The Llama 3 Herd of Models
- RULER: What's the Real Context Size of Your Long-Context Language Models?
- AB-Sparse: Sparse Attention with Adaptive Block Size for Accurate and Efficient Long-Context Inference
- GPT-4 Technical Report
Related papers
- Multi-Objective Structured Pruning of LLMs for Latency and Model Size Optimization
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Embeddings for Preferences, Not Semantics
- Can We Trust Item Response Theory for AI Evaluation?
- Diagnosing Task Insensitivity in Language Agents