DynaFlow: Transparent and Flexible Intra-Device Parallelism via Programmable Operator Scheduling
cs.DC, cs.LG
Submitted: 2026-05-20
Updated: 2026-05-20
Code: https://github.com/uw-syfi/DynaFlow
Terminology
Sources
- Longformer: The Long-Document Transformer
- FLUX: Fast Software-based Communication Overlap On GPUs Through Kernel Fusion
- Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs
- TokenWeave: Efficient Compute-Communication Overlap for Distributed LLM Inference
- GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
- PyTorch Distributed: Experiences on Accelerating Data Parallel Training
- TorchTitan: One-stop PyTorch native solution for production ready LLM pre-training
- DeepSeek-V3 Technical Report
- YaRN: Efficient Context Window Extension of Large Language Models
- Horovod: fast and easy distributed deep learning in TensorFlow
- Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference
- LongCat-Flash Technical Report
- Efficient Streaming Language Models with Attention Sinks
- Comet: Fine-grained Computation-communication Overlapping for Mixture-of-Experts
- PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel
- Triton-distributed: Programming Overlapping Kernels on Distributed AI Systems with the Triton Compiler
- TileLink: Generating Efficient Compute-Communication Overlapping Kernels using Tile-Centric Primitives
Related papers
- iScheduler: Reinforcement Learning-Driven Continual Optimization for Large-Scale Resource Investment Problems
- SAMM: Sharded Automated Market Maker
- InferScale: GPU-Native KV Injection for Personalized LLM Serving
- Vigil: Accountable Liveness against Selective Silence
- Steelhead: Interleaving Partially Synchronous and Asynchronous Commit Rules on a Shared DAG
- Democratizing MoE inference on commodity GPUs with CoMoE