SEED: Self-Speculative Decoding via Implicit Encoder-Decoder
cs.CL, cs.LG
Submitted: 2026-09-29
Updated: 2026-09-29
Comments: Accepted to NeurIPS 2026
Code: https://github.com/lhk2004/SEED
License: http://creativecommons.org/licenses/by/4.0/
The gist: Self-speculative decoding accelerates large language model (LLM) inference by drafting tokens from the target model itself, but faces a sharp tradeoff between the quality and cost of the draft.
Terminology
Abstract
Self-speculative decoding accelerates large language model (LLM) inference by drafting tokens from the target model itself, but faces a sharp tradeoff between the quality and cost of the draft. Early-exit methods produce drafts cheaply by terminating computation at intermediate layers, but forgo the deeper representations that later layers provide and thus suffer in draft quality. Multi-token prediction preserves draft quality by emitting from the model's final hidden states, but pays for a full forward pass to produce those states at every drafting step. We propose self-speculative encoder-decoder (SEED), a self-speculative method that obtains high-quality drafts cheaply by reusing the deep contextual representations already computed during verification. We reinterpret the standard decoder-only transformer as an implicit encoder-decoder: the first layers (encoder) build deep contextual representations, and the last few layers (decoder) emit tokens from them. Encoding and verification are merged into a single step: verification is performed by the full encoder-decoder, and the contextual representations of the verified prefix are cached for reuse during drafting. Drafting is therefore very fast: between verifications, the lightweight decoder drafts multiple tokens autoregressively, each conditioned on the cached representations and on preceding drafts. Experiments across multiple benchmarks show that SEED achieves up to 2.7 times average speedup on 4B-scale models, outperforming both early-exit and MTP-style self-speculative baselines and running 28% faster than the state-of-the-art EAGLE-3, while preserving or even improving the generation quality of standard autoregressive fine-tuning. Code is available at https://github.com/lhk2004/SEED.
Sources
- Speculative Streaming: Fast LLM Inference without Auxiliary Models
- FastMTP: Accelerating LLM Inference with Enhanced Multi-Token Prediction
- Accelerating Large Language Model Decoding with Speculative Sampling
- DFlash: Block Diffusion for Flash Speculative Decoding
- Training Verifiers to Solve Math Word Problems
- Parallel Token Prediction for Language Models
- Fast and Accurate Causal Parallel Decoding using Jacobi Forcing
- Multi-Token Prediction via Self-Distillation
- DeepSeek-V3 Technical Report
- TiDAR: Think in Diffusion, Talk in Autoregression
- On multi-token prediction for efficient LLM inference
- Your LLM Knows the Future: Uncovering Its Multi-Token Prediction Potential
- Qwen3 Technical Report
- DEL: Context-Aware Dynamic Exit Layer for Efficient Self-Speculative Decoding
- Self-Distillation for Multi-Token Prediction
- PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering