Windowed-MTP: Removing the Full-Context Draft-KV Tax at Million-Token Context
cs.LG, cs.CL, cs.PF
Submitted: 2026-07-23
Updated: 2026-10-30
Comments: 29 pages, 5 figures, 16 tables. v2: Windowed draft attention is merged upstream in SGLang (PR #32673). Code: https://github.com/avalliappan-nvidia/windowed-mtp-b200
Code: https://github.com/avalliappan-nvidia/windowed-mtp-b200
License: http://creativecommons.org/licenses/by/4.0/
The gist: Speculative decoding accelerates autoregressive generation by having a draft propose tokens that a target verifies in parallel.
Terminology
Abstract
Speculative decoding accelerates autoregressive generation by having a draft propose tokens that a target verifies in parallel. Frontier models increasingly ship a built-in Multi-Token-Prediction (MTP) draft head, assumed to be cheap beside verification. A shipped full-attention MTP head can become disproportionately expensive beside a hybrid target at long context. The head attends over the full KV cache for each of γ draft forwards per decode step, while only 9-25% of target layers retain full attention, so a single draft layer can issue context reads at a rate comparable to the whole target. On hard, low-acceptance workloads, deep native MTP becomes net-negative: slower than no speculation. We apply a StreamingLLM-style window plus attention sink to the draft only (Windowed-MTP); target verification remains full-context, so the target distribution is preserved. The intervention is training-free and drop-in. On three models from two hybrid families (Gated DeltaNet and Mamba2) at 1M context on one GPU in SGLang, it provides an acceptance-free 1.28-1.44x per-decode-step speedup over native MTP. The speedup grows with context; at matched acceptance, TPOT improves by the same factor. The window removes most of the draft's working set, and a compact ring buffer converts that reduction into physical capacity: it reclaims the otherwise dead draft pool, a measured 7.7-11.1% of total KV at 1M, making room for one additional concurrent 1M request in the Qwen-35B batch sweep.
Sources
- LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks
- SpecExtend: A Drop-in Enhancement for Speculative Decoding of Long Sequences
- Accelerating Large Language Model Decoding with Speculative Sampling
- MagicDec: Breaking the Latency-Throughput Tradeoff for Long Context Generation with Speculative Decoding
- DFlash: Block Diffusion for Flash Speculative Decoding
- Extending Context Window of Large Language Models via Positional Interpolation
- DeepSeek-V3 Technical Report
- Attention Drift: What Autoregressive Speculative Decoding Models Learn
- Mamba: Linear-Time Sequence Modeling with Selective State Spaces
- EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees
- YaRN: Efficient Context Window Extension of Large Language Models
- Qwen3 Technical Report
- LongSpec: Long-Context Lossless Speculative Decoding with Efficient Drafting and Verification
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis