Step-dLLM: Adaptive Step-aware Sparse Attention for Efficient Diffusion LLM Inference
cs.CL, cs.AI
Submitted: 2026-10-31
Updated: 2026-10-31
Terminology
Sources
- Longformer: The Long-Document Transformer
- LLaDA2.0: Scaling Up Diffusion Language Models to 100B
- Generating Long Sequences with Sparse Transformers
- SeerAttention: Learning Intrinsic Sparse Attention in Your LLMs
- MIDUS: Memory-Infused Depth Up-Scaling
- dInfer: An Efficient Inference Framework for Diffusion Language Models
- Large Language Diffusion Models
- Seed Diffusion: A Large-Scale Diffusion Language Model with High-Speed Inference
- Diffusion LLMs Can Do Faster-Than-AR Inference via Discrete Diffusion Forcing
- SparseD: Sparse Attention for Diffusion Language Models
- Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decoding
- RefreshKV: Updating Small KV Cache During Long-form Generation
- Dream 7B: Diffusion Large Language Models
- FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving
- LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuning
- Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention
- LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Models
- LLaDA-MoE: A Sparse MoE Diffusion Language Model
- Tactic: Adaptive Sparse Attention with Clustering and Distribution Fitting for Long-Context LLMs
Related papers
- ALARM: Audio-Language Alignment for Reasoning Models
- ThinkSum: Probabilistic reasoning over sets using large language models
- Coherence boosting: When your pretrained language model is not paying enough attention
- From Facts to Insights: A Persona-Driven Dual Memory Framework and Dataset for Role-Playing Agents
- Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization
- The Art of Asking: Multilingual Prompt Optimization for Synthetic Data