Recursive Sparse Attention: Million-Token Length Extrapolation at Logarithmic Cost
cs.CL, cs.LG
Submitted: 2026-11-04
Updated: 2026-11-04
Code: https://github.com/bojone/rerope
Terminology
Sources
- Program Synthesis with Large Language Models
- Evaluating Large Language Models Trained on Code
- Extending Context Window of Large Language Models via Positional Interpolation
- Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
- Training Verifiers to Solve Math Word Problems
- DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models
- DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
- Hierarchical Sparse Attention Done Right: Toward Infinite Context Modeling
- InfiniteHiP: Extending Language Model Context Up to 3 Million Tokens on a Single GPU
- Block Sparse Attention with Log-Linear Complexity
- Length Generalization of Causal Transformers without Position Encoding
- Qwen3 Technical Report
- Instruction-Following Evaluation for Large Language Models
Related papers
- ALARM: Audio-Language Alignment for Reasoning Models
- ThinkSum: Probabilistic reasoning over sets using large language models
- Coherence boosting: When your pretrained language model is not paying enough attention
- From Facts to Insights: A Persona-Driven Dual Memory Framework and Dataset for Role-Playing Agents
- Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization
- The Art of Asking: Multilingual Prompt Optimization for Synthetic Data