Accelerating Long-Context Decoding in Large-Scale Hybrid LLMs Without Retraining
cs.CL, cs.PF
Submitted: 2026-11-04
Updated: 2026-11-04
Code: https://github.com/jespb/CombinedSparseAttention
Terminology
Sources
- Longformer: The Long-Document Transformer
- Jet-Nemotron: Efficient Language Model with Post Neural Architecture Search
- Mistral 7B
- The Sparse Frontier: Sparse Attention Trade-offs in Transformer LLMs
- RazorAttention: Efficient KV Cache Compression Through Retrieval Heads
- Qwen3 Technical Report
Related papers
- ALARM: Audio-Language Alignment for Reasoning Models
- ThinkSum: Probabilistic reasoning over sets using large language models
- Coherence boosting: When your pretrained language model is not paying enough attention
- From Facts to Insights: A Persona-Driven Dual Memory Framework and Dataset for Role-Playing Agents
- Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization
- The Art of Asking: Multilingual Prompt Optimization for Synthetic Data