CISA: Retrofitting Dense Attention Models with Chunk-Indexed Sparse Attention
cs.LG
Submitted: 2026-11-02
Updated: 2026-11-02
Terminology
Sources
- LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks
- Longformer: The Long-Document Transformer
- FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
- DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models
- Better & Faster Large Language Models via Multi-token Prediction
- RULER: What's the Real Context Size of Your Long-Context Language Models?
- Efficient Memory Management for Large Language Model Serving with PagedAttention
- MiniMax Sparse Attention
- Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces
- GDPval: Evaluating AI Model Performance on Real-World Economically Valuable Tasks
- Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference
- Michelangelo: Long Context Evaluations Beyond Haystacks via Latent Structure Queries
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis