The GPU Needs an Operating System
cs.DC
Submitted: 2026-11-05
Updated: 2026-11-05
Comments: 9 pages, 6 figures. Measurements on NVIDIA RTX PRO 6000 Blackwell and AMD MI300X
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Sources
- ROSE: Rollout On Serving GPUs via Cooperative Elasticity for Agentic RL
- LeMix: Unified Scheduling for LLM Training and Inference on Multi-GPU Systems
- ConServe: Fine-Grained GPU Harvesting for LLM Online and Offline Co-Serving
- HyGen: Efficient LLM Serving via Elastic Online-Offline Request Co-location
- Inference First, Training Underneath: Layer-Granular LoRA Co-Serving with LayerLoom
- CrossPool: Efficient Multi-LLM Serving for Cold MoE Models through KV-Cache and Weight Disaggregation
- ECLIP: Energy-efficient and Practical Co-Location of ML Inference on Spatially Partitioned GPUs
- Boosting LLM Serving through Spatial-Temporal GPU Resource Sharing
- Towards High-Goodput LLM Serving with Prefill-decode Multiplexing
- Determinism-Preserving GPU Spatial Sharing with Vitamin-E
- MuxServe: Flexible Spatial-Temporal Multiplexing for Multiple LLM Serving
- Tally: Non-Intrusive Performance Isolation for Concurrent Deep Learning Workloads
- ExpertPlex: A High-Goodput Disaggregated Serving System for MoE LLMs with Adaptive Persistent Kernels
- Hummingbird: SLO-Oriented GPU Preemption at Microsecond-scale
- GPUOS: A GPU Operating System Primitive for Transparent Operation Fusion
Related papers
- BatchGen: An Architecture for Scalable and Efficient Batch Inference
- Seamless Reconfiguration for DAG-BFT (Technical Report)
- ELDR: Expert-Locality-Aware Decode Routing for PD-Disaggregated MoE Serving
- iScheduler: Reinforcement Learning-Driven Continual Optimization for Large-Scale Resource Investment Problems
- TIRx: A Unified Tile Primitive ML Compiler Across Heterogeneous Backends
- AgentWeave: Efficient Distributed Agent Serving via Flow Decomposition