ADIndex: Layer-Adaptive Sparse Attention Indexing via Group-Level Distillation
cs.LG, cs.AI
Submitted: 2026-11-07
Updated: 2026-11-07
Terminology
Sources
- IndexCache: Accelerating Sparse Attention via Cross-Layer Index Reuse
- Longformer: The Long-Document Transformer
- Generating Long Sequences with Sparse Transformers
- DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models
- SeerAttention: Learning Intrinsic Sparse Attention in Your LLMs
- GLM-5: from Vibe Coding to Agentic Engineering
- PIVOT: Efficient Query-Group Indexing for Token-Level Sparse Attention
- On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability
- MISA: Mixture of Indexer Sparse Attention for Long-Context LLM Inference
- Full Attention Strikes Back: Transferring Full Attention into Sparse within Hundred Training Steps
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis