HyFlash: A Prefix-Memory Prior for On-Device Speculative Decoding
cs.CL
Submitted: 2026-11-05
Updated: 2026-11-05
License: http://creativecommons.org/licenses/by/4.0/
The gist: Parallel block drafting accelerates speculative decoding by predicting multiple future positions in a single backbone pass…
Terminology
Abstract
Parallel block drafting accelerates speculative decoding by predicting multiple future positions in a single backbone pass, but on-device deployment constrains both drafter capacity and context storage. Under tight memory budgets, drafters must model long-range dependencies and fine-grained draft-position interactions without the growing KV-cache overhead of full-context attention. We propose HyFlash, a compact parallel drafter that introduces a prefix-memory prior to separate long-range context modeling from fine-grained draft-position interactions with limited capacity. Local attention captures recent context and draft-position interactions, while recurrent memory compresses historical target features into fixed-size states. This inductive bias enables effective local and long-range modeling with bounded draft-side memory. Experiments show that HyFlash achieves higher acceptance than full-context attention with substantially less KV storage and with larger gains for shallow drafters and long contexts.
Sources
- Phi-4-reasoning Technical Report
- Program Synthesis with Large Language Models
- Layer Normalization
- DFlash: Block Diffusion for Flash Speculative Decoding
- Evaluating Large Language Models Trained on Code
- DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation
- Training Verifiers to Solve Math Word Problems
- Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models
- Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention
- H-Spec: Parallel Speculative Decoding Without a Drafter-Side KV Cache
- ReaDy: Recurrent States as Dynamic Context for Speculative Drafting in Hybrid Linear Attention Large Language Models
- The Perfect Blend: Redefining RLHF with Mixture of Judges
- Qwen3 Technical Report
- Strong Drafts Need Compact Memories: Long-Context Speculative Decoding with Compressed KV Cache
Related papers
- ALARM: Audio-Language Alignment for Reasoning Models
- ThinkSum: Probabilistic reasoning over sets using large language models
- Coherence boosting: When your pretrained language model is not paying enough attention
- From Facts to Insights: A Persona-Driven Dual Memory Framework and Dataset for Role-Playing Agents
- Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization
- The Art of Asking: Multilingual Prompt Optimization for Synthetic Data