Decode Token Parallelism: Memory-Efficient Distributed Attention Parallelism for Scalable LLM Decoding
cs.DC, cs.CL, cs.LG
Submitted: 2026-11-09
Updated: 2026-11-09
Terminology
Sources
- No Request Left Behind: Tackling Heterogeneity in Long-Context LLM Inference with Medha
- L-Eval: Instituting Standardized Evaluation for Long Context Language Models
- LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding
- LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks
- Helix Parallelism: Rethinking Sharding Strategies for Interactive Multi-Million-Token LLM Decoding
- Accelerating Large Language Model Decoding with Speculative Sampling
- Efficient Heterogeneous Large Language Model Decoding with Model-Attention Disaggregation
- Token-Driven GammaTune: Adaptive Calibration for Enhanced Speculative Decoding
- The Llama 3 Herd of Models
- RULER: What's the Real Context Size of Your Long-Context Language Models?
- Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCache
- Ring Attention with Blockwise Transformers for Near-Infinite Context
- Beyond the Buzz: A Pragmatic Take on Inference Disaggregation
- Nemotron-4 340B Technical Report
- Splitwise: Efficient generative LLM inference using phase splitting
- FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision
- Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
- Polar Sparsity: High Throughput Batched LLM Inferencing with Scalable Contextual Sparsity
- Fast Distributed Inference Serving for Large Language Models
- KernelFlume: Elastic Core-Attention Scaling for Agentic Long-Context Decoding
Related papers
- Programmable Probabilistic Computer with 1,000,000 p-bits
- BatchGen: An Architecture for Scalable and Efficient Batch Inference
- Seamless Reconfiguration for DAG-BFT (Technical Report)
- ELDR: Expert-Locality-Aware Decode Routing for PD-Disaggregated MoE Serving
- iScheduler: Reinforcement Learning-Driven Continual Optimization for Large-Scale Resource Investment Problems
- TIRx: A Unified Tile Primitive ML Compiler Across Heterogeneous Backends