Multi-layer Indexer: Overlapping Selection and Attention in DeepSeek Sparse Attention
cs.LG, cs.AI
Submitted: 2026-11-02
Updated: 2026-11-02
Code: https://github.com/ArjunKrishnak/dsa-multi-layer-indexer
Terminology
Sources
- IndexCache: Accelerating Sparse Attention via Cross-Layer Index Reuse
- Longformer: The Long-Document Transformer
- Generating Long Sequences with Sparse Transformers
- Kascade: A Practical Sparse Attention Method for Long-Context LLM Inference
- SeerAttention: Learning Intrinsic Sparse Attention in Your LLMs
- RULER: What's the Real Context Size of Your Long-Context Language Models?
- DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models
- HeadInfer: Memory-Efficient LLM Inference by Head-wise Offloading
- Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference
- GLM-5: from Vibe Coding to Agentic Engineering
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis