GC-MoE: Overlapping Expert-Parallel Token Dispatch with FP8/FP4 Grouped GEMM via CUDA Green Contexts
cs.DC, cs.AI, cs.AR, cs.LG, cs.PF
Submitted: 2026-11-03
Updated: 2026-11-03
Code: https://github.com/Bruce-Lee-LY/gc-moe
Terminology
Sources
- SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills
- FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
- DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model
- DeepSeek-V3 Technical Report
- SonicMoE: Accelerating MoE with IO and Tile-aware Optimizations
- Mixtral of Experts
- FP8 Quantization: The Power of the Exponent
- GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
- Performance Isolation for Inference Processes in Edge GPU Systems
- Pretraining Large Language Models with NVFP4
- FP8-LM: Training FP8 Large Language Models
- ECLIP: Energy-efficient and Practical Co-Location of ML Inference on Spatially Partitioned GPUs
- FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision
- Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
- Training LLMs with MXFP4
- AgentServe: Algorithm-System Co-Design for Efficient Agentic AI Serving on a Consumer-Grade GPU
Related papers
- Programmable Probabilistic Computer with 1,000,000 p-bits
- BatchGen: An Architecture for Scalable and Efficient Batch Inference
- Seamless Reconfiguration for DAG-BFT (Technical Report)
- ELDR: Expert-Locality-Aware Decode Routing for PD-Disaggregated MoE Serving
- iScheduler: Reinforcement Learning-Driven Continual Optimization for Large-Scale Resource Investment Problems
- TIRx: A Unified Tile Primitive ML Compiler Across Heterogeneous Backends