Subquadratic Attention with a Hierarchical Tree-Structured Indexer
cs.CL, cs.AI
Submitted: 2026-10-30
Updated: 2026-11-02
Comments: 42 pages
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Sources
- LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks
- IndexCache: Accelerating Sparse Attention via Cross-Layer Index Reuse
- PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling
- DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- Compressed Convolutional Attention: Efficient Attention in a Compressed Latent Space
- Sliding Window Attention Training for Efficient Large Language Models
- The Zamba2 Suite: Technical Report
- Zamba: A Compact 7B SSM Hybrid Model
- Mamba: Linear-Time Sequence Modeling with Selective State Spaces
- RULER: What's the Real Context Size of Your Long-Context Language Models?
- MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies
- Hierarchical Sparse Attention Done Right: Toward Infinite Context Modeling
- DashAttention: Differentiable and Adaptive Sparse Hierarchical Attention
- Dynamic Chunking for End-to-End Hierarchical Sequence Modeling
- Fast KVzip: Efficient and Accurate LLM Inference with Gated KV Eviction
- Mamba-3: Improved Sequence Modeling using State Space Principles
- MiniMax Sparse Attention
- A Training-free Sub-quadratic Cost Transformer Model Serving Framework With Hierarchically Pruned Attention
- Jamba: A Hybrid Transformer-Mamba Language Model
Related papers
- ALARM: Audio-Language Alignment for Reasoning Models
- ThinkSum: Probabilistic reasoning over sets using large language models
- Coherence boosting: When your pretrained language model is not paying enough attention
- From Facts to Insights: A Persona-Driven Dual Memory Framework and Dataset for Role-Playing Agents
- Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization
- The Art of Asking: Multilingual Prompt Optimization for Synthetic Data