LiquidCP: Decoupling Attention Tasks from Communication and Kernel Boundaries in Context Parallelism
cs.DC
Submitted: 2026-11-03
Updated: 2026-11-03
Code: https://github.com/Dao-AILab/flash-attention
Terminology
Sources
- GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints
- Helix Parallelism: Rethinking Sharding Strategies for Interactive Multi-Million-Token LLM Decoding
- Striped Attention: Faster Ring Attention for Causal Transformers
- NanoCP: Taming Load Imbalance in Data-Expert Parallel Decoding under Variable-Length Workloads
- FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
- DeepSeek-V3 Technical Report
- USP: A Unified Sequence Parallelism Approach for Long Context Generative AI
- The Pile: An 800GB Dataset of Diverse Text for Language Modeling
- The Llama 3 Herd of Models
- LoongTrain: Efficient Training of Long-Sequence LLMs with Head-Context Parallelism
- DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models
- DISTFLASHATTN: Distributed Memory-efficient Attention for Long-context LLMs Training
- Ring Attention with Blockwise Transformers for Near-Infinite Context
- mKernel: Fast Multi-GPU, Multi-Node Fused Kernels
- ParallelKittens: Systematic and Practical Simplification of Multi-GPU AI Kernels
- Qwen3 Technical Report
- UniEP: Unified Expert-Parallel MoE MegaKernel for LLM Training
Related papers
- BatchGen: An Architecture for Scalable and Efficient Batch Inference
- Seamless Reconfiguration for DAG-BFT (Technical Report)
- ELDR: Expert-Locality-Aware Decode Routing for PD-Disaggregated MoE Serving
- iScheduler: Reinforcement Learning-Driven Continual Optimization for Large-Scale Resource Investment Problems
- TIRx: A Unified Tile Primitive ML Compiler Across Heterogeneous Backends
- AgentWeave: Efficient Distributed Agent Serving via Flow Decomposition