Kascade: A Practical Sparse Attention Method for Long-Context LLM Inference
cs.LG, cs.AI, cs.DC
Submitted: 2025-12-18
Updated: 2026-08-31
Comments: 11 pages, 8 figures, 3 tables and 1 algorithm
Code: https://github.com/microsoft/kascade
License: http://creativecommons.org/licenses/by/4.0/
The gist: Attention is the dominant source of latency during long-context LLM inference, an increasingly popular workload with reasoning models and RAG.
Terminology
Abstract
Attention is the dominant source of latency during long-context LLM inference, an increasingly popular workload with reasoning models and RAG. We propose Kascade, a training-free sparse attention method that leverages known observations such as 1) post-softmax attention is intrinsically sparse, and 2) the identity of high-weight keys is stable across nearby layers. Kascade computes exact Top-k indices in a small set of anchor layers, then reuses those indices in intermediate reuse layers. The anchor layers are selected algorithmically, via a dynamic-programming objective that maximizes cross-layer similarity over a development set, allowing easy deployment across models. The method incorporates efficient implementation constraints (e.g. tile-level operations), across both prefill and decode attention. The Top-k selection and reuse in Kascade is head-aware and we show in our experiments that this is critical for high accuracy. Kascade achieves up to 4.1x speedup in decode attention and 2.2x speedup in prefill attention over FlashAttention-3 baseline on H100 GPUs while closely matching dense attention accuracy on long-context benchmarks such as LongBench and AIME-24.
Sources
- gpt-oss-120b & gpt-oss-20b Model Card
- GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints
- Longformer: The Long-Document Transformer
- Leveraging redundancy in attention with Reuse Transformers
- SeerAttention: Learning Intrinsic Sparse Attention in Your LLMs
- SeerAttention-R: Sparse Attention Adaptation for Long Reasoning
- The Llama 3 Herd of Models
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- What Matters in Transformers? Not All Attention is Needed
- TurboRAG: Accelerating Retrieval-Augmented Generation with Precomputed KV Caches for Chunked Text
- Qwen3 Technical Report
- Less Is More: Fast and Accurate Reasoning with Cross-Head Unified Sparse Attention
- LServe: Efficient Long-sequence LLM Serving with Unified Sparse Attention
- Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference
- LazyFormer: Self Attention with Lazy Update
- Gemma: Open Models Based on Gemini Research and Technology
- TileLang: A Composable Tiled Programming Model for AI Systems
- Efficient Streaming Language Models with Attention Sinks
- Sharing Attention Weights for Fast Transformer
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks