AutoParallel: Composable, PyTorch-Native Automatic Parallelism for Production Training
cs.DC
Submitted: 2026-11-05
Updated: 2026-11-05
Comments: 21 pages, 6 figures, 8 tables. Code: https://github.com/meta-pytorch/autoparallel
Code: https://github.com/meta-pytorch/autoparallel
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Terminology
Sources
- Training Deep Nets with Sublinear Memory Cost
- DeepSeek-V3 Technical Report
- DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models
- GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
- veScale: Consistent and Efficient Tensor Programming with Eager-Mode SPMD
- TorchTitan: One-stop PyTorch native solution for production ready LLM pre-training
- Ring Attention with Blockwise Transformers for Near-Infinite Context
- Colossal-Auto: Unified Automation of Parallelization and Activation Checkpoint for Large-scale Models
- The Llama 3 Herd of Models
- Qwen3 Technical Report
- Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
- GSPMD: General and Scalable Parallelization for ML Computation Graphs
- SimpleFSDP: Simpler Fully Sharded Data Parallel with torch.compile
Related papers
- Programmable Probabilistic Computer with 1,000,000 p-bits
- BatchGen: An Architecture for Scalable and Efficient Batch Inference
- Seamless Reconfiguration for DAG-BFT (Technical Report)
- ELDR: Expert-Locality-Aware Decode Routing for PD-Disaggregated MoE Serving
- iScheduler: Reinforcement Learning-Driven Continual Optimization for Large-Scale Resource Investment Problems
- TIRx: A Unified Tile Primitive ML Compiler Across Heterogeneous Backends