A Sparse Glimpse of the Whole: Train-Free Self-Speculative Decoding

arXiv:2607.27735 · cs.CL · Submitted 2026-07-30 · Read on arXiv

Yuesong Liu, Yuan Zeng, Min Lyu, Ruilin Liu, Yu Guo, Yinlong Xu

cs.CL

Submitted: 2026-07-30

Comments: 9 pages, 4 figures, subbmited to AAAI 2027

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Terminology

• Verification — 34× in this paper · explained in 2 episodes
• Speculative Decoding — 31× in this paper · explained in 2 episodes
• KV cache — 14× in this paper · explained in 3 episodes
• Verifier — 4× in this paper · explained in 2 episodes
• Fidelity — 3× in this paper · explained in 2 episodes
• Generalization — 3× in this paper · explained in 13 episodes
• Sparse Attention — 3× in this paper · explained in 5 episodes
• Ablation — 2× in this paper · explained in 1 episode
• Adaptivity — 2× in this paper · explained in 1 episode
• Benchmark — 2× in this paper · explained in 2 episodes
• Decoupling — 2× in this paper · explained in 2 episodes
• KV Cache Compression — 2× in this paper · explained in 2 episodes
• Papers — 2× in this paper · explained in 3 episodes
• Pruning — 2× in this paper · explained in 4 episodes
• Transformer — 2× in this paper · explained in 2 episodes
• Transformers — 2× in this paper · explained in 2 episodes
• AUC — 1× in this paper · explained in 2 episodes
• Ablation Study — 1× in this paper · explained in 3 episodes
• Accuracy — 1× in this paper · explained in 1 episode
• Large Language Models (LLMs) — 1× in this paper · explained in 8 episodes
• Robustness — 1× in this paper · explained in 5 episodes
• Sparsity — 1× in this paper · explained in 4 episodes
• Trace — 1× in this paper · explained in 2 episodes

Sources

• AdaEDL: Early Draft Stopping for Speculative Decoding of Large Language Models via an Entropy-based Lower Bound on Token Acceptance Probability
• Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads
• PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling
• Accelerating Large Language Model Decoding with Speculative Sampling
• RAPID: Long-Context Inference with Retrieval-Augmented Speculative Decoding
• FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
• Better & Faster Large Language Models via Multi-token Prediction
• RULER: What's the Real Context Size of Your Long-Context Language Models?
• EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test
• Dynamic Speculation Lookahead Accelerates Speculative Decoding of Large Language Models
• MagicDec: Breaking the Latency-Throughput Tradeoff for Long Context Generation with Speculative Decoding
• TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decoding
• Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference
• Efficient Streaming Language Models with Attention Sinks
• Vegas: Self-Speculative Decoding with Verification-Guided Sparse Attention
• Accelerating Large-Scale Reasoning Model Inference with Sparse Self-Speculative Decoding

Related papers

• Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
• Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
• Subliminal Steering: Stronger Encoding of Hidden Signals
• MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
• The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
• Untangling the Mechanisms of Misleading Context in Medical Question Answering