OOM: Offload-Optimized Full-Parameter MoE Training via Memory-Execution Co-Design
cs.DC, cs.LG
Submitted: 2026-11-08
Updated: 2026-11-08
Code: https://github.com/RegiaYoung/OOM-MoE
Terminology
Sources
- Training Deep Nets with Sublinear Memory Cost
- Gemma 4 Technical Report
- Liger Kernel: Efficient Triton Kernels for LLM Training
- Mixtral of Experts
- Adam: A Method for Stochastic Optimization
- Out of the Memory Barrier: A Highly Memory Efficient Training System for LLMs with Million-Token Contexts
- Mixture-of-Parallelisms: Towards Memory-Efficient Training Stack for Mixture-of-Experts Models
- Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
- LazyTrain: Limited-resource Allocation toward Zero-waste Yield Optimization in Large Language Model Training
- SlideDP: Scaling Host-Resident LLM Fine-Tuning Across Multiple GPUs
- MegaTrain: Full Precision Training of 100B+ Parameter Large Language Models on a Single GPU
- MemFine: Memory-Aware Fine-Grained Scheduling for MoE Training
Related papers
- Programmable Probabilistic Computer with 1,000,000 p-bits
- BatchGen: An Architecture for Scalable and Efficient Batch Inference
- Seamless Reconfiguration for DAG-BFT (Technical Report)
- ELDR: Expert-Locality-Aware Decode Routing for PD-Disaggregated MoE Serving
- iScheduler: Reinforcement Learning-Driven Continual Optimization for Large-Scale Resource Investment Problems
- TIRx: A Unified Tile Primitive ML Compiler Across Heterogeneous Backends