Latent Sparse Attention: Towards Subquadratic Ultra-Long Context Attention
cs.AI
Submitted: 2026-10-30
Updated: 2026-11-03
Code: https://github.com/Prefrontal-AI/sglang
Terminology
Sources
- Longformer: The Long-Document Transformer
- Helix Parallelism: Rethinking Sharding Strategies for Interactive Multi-Million-Token LLM Decoding
- Striped Attention: Faster Ring Attention for Causal Transformers
- Generating Long Sequences with Sparse Transformers
- Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
- DeepSeek-V3 Technical Report
- DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models
- DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
- DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression
- Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
- MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies
- Hierarchical Sparse Attention Done Right: Toward Infinite Context Modeling
- The Common Pile v0.1: An 8TB Dataset of Public Domain and Openly Licensed Text
- MiniMax Sparse Attention
- InfiniteHiP: Extending Language Model Context Up to 3 Million Tokens on a Single GPU
- Simplified Sparse Attention via Gist Tokens
- MiniMax-01: Scaling Foundation Models with Lightning Attention
- On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability
- Block Sparse Attention with Log-Linear Complexity
- TileLang: A Composable Tiled Programming Model for AI Systems
Related papers
- Multi-Objective Structured Pruning of LLMs for Latency and Model Size Optimization
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Embeddings for Preferences, Not Semantics
- Can We Trust Item Response Theory for AI Evaluation?
- Diagnosing Task Insensitivity in Language Agents