Ripple: Deadline-Aware CPU-GPU Sharing for Embodied Inference
cs.DC, cs.AI, cs.OS, cs.PF
Submitted: 2026-10-31
Updated: 2026-10-31
License: http://creativecommons.org/licenses/by/4.0/
The gist: Unified-memory edge devices increasingly run LLM/VLM inference alongside time-critical CPU tasks for perception and control.
Terminology
Abstract
Unified-memory edge devices increasingly run LLM/VLM inference alongside time-critical CPU tasks for perception and control. GPU memory traffic can overwhelm these tasks, yet platforms such as DGX Spark expose neither bandwidth counters nor direct bandwidth regulation. We present Ripple, a CPU-GPU co-scheduler that combines model-derived GPU memory volumes with CPU completion feedback to regulate interference. Ripple divides inference into pausable, layer-based units and translates each CPU job's deadline slack into a GPU-byte budget, admitting work within the budgets of all overlapping jobs. Online updates track changing interference, while deadline-aware pacing and judicious SM footprint narrowing preserve inference progress. Implemented with SGLang engine hooks and a central scheduler, we evaluate Ripple on DGX Spark with four robotic CPU task mixes and LLM/VLM inference. It is the only evaluated approach that consistently meets the 2% CPU deadline-miss target and VLM deadlines across the tested deadline settings while retaining best-effort LLM throughput.
Sources
- Qwen3-VL Technical Report
- DuetServe: Harmonizing Prefill and Decode for LLM Serving via Adaptive GPU Multiplexing
- Valve: Production Online-Offline Inference Colocation with Jointly-Bounded Preemption Latency and Rate
- The Marathon 2: A Navigation System
- Real-time Neural-MPC: Deep Learning Model Predictive Control for Quadrotors and Agile Robotic Platforms
- Dissecting CPU-GPU Unified Physical Memory on AMD MI300A APUs
- Qwen3 Technical Report
- SGLang: Efficient Execution of Structured Language Model Programs
Related papers
- Programmable Probabilistic Computer with 1,000,000 p-bits
- BatchGen: An Architecture for Scalable and Efficient Batch Inference
- Seamless Reconfiguration for DAG-BFT (Technical Report)
- ELDR: Expert-Locality-Aware Decode Routing for PD-Disaggregated MoE Serving
- iScheduler: Reinforcement Learning-Driven Continual Optimization for Large-Scale Resource Investment Problems
- TIRx: A Unified Tile Primitive ML Compiler Across Heterogeneous Backends