Rose: Efficient Embodied RL through Graph Optimization and GPU Compute Resource Control
cs.DC
Submitted: 2026-11-02
Updated: 2026-11-02
License: http://creativecommons.org/licenses/by/4.0/
The gist: Embodied reinforcement learning (RL) coordinates policy inference, simulation, and model training with highly heterogeneous resource demands.
Terminology
Abstract
Embodied reinforcement learning (RL) coordinates policy inference, simulation, and model training with highly heterogeneous resource demands. Although individual components frequently underutilize GPU compute during execution, existing systems only overlap work at coarse component boundaries (e.g., pipelining inference and simulation across simulator groups), leaving substantial low-activity GPU windows within subcomponent phases and nested loops. Exploiting these fine-grained windows manually requires tedious dependency analysis and workload-specific coordination. Furthermore, even when independent work is scheduled concurrently, cross-process GPU time slicing and streaming multiprocessor (SM) contention can negate the benefits of overlap. We present Rose, a runtime-aware system that automates fine-grained overlap and GPU resource management for embodied RL while preserving synchronous training semantics. Through a lightweight programmable interface that exposes execution hierarchy and state accesses, Rose tracks runtime data readiness and GPU activity to perform a two-stage optimization. First, a window-guided graph optimizer identifies low-activity windows and applies graph transformations to overlap complementary work across components and loop levels. Second, leveraging NVIDIA Multi-Process Service (MPS) for cross-process GPU concurrency, a slack-aware resource manager tunes process-level SM quotas by exploiting branch synchronization slack to mitigate interference on the critical path. Both stages combine local profiling with end-to-end iteration validation to guarantee throughput gains with minimal tuning overhead. Across diverse policies, simulators, RL algorithms, and GPU platforms, Rose achieves 1.20--1.92 times training throughput improvement and increases average SM activity by 23.8--39.3% compared with the state-of-the-art systems.
Sources
- OpenRLHF: An Easy-to-use, Scalable and High-performance RLHF Framework
- SimpleVLA-RL: Scaling VLA Training via Reinforcement Learning
- EBRL: Asynchronous Embodied RL by Multi-Grained Resource Management
- GR00T N1: An Open Foundation Model for Generalist Humanoid Robots
- Proximal Policy Optimization Algorithms
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- RL-VLA cubed: A Flexible and Asynchronous Reinforcement Learning Framework for VLA Training
- robosuite: A Modular Simulation Framework and Benchmark for Robot Learning
Related papers
- Programmable Probabilistic Computer with 1,000,000 p-bits
- BatchGen: An Architecture for Scalable and Efficient Batch Inference
- Seamless Reconfiguration for DAG-BFT (Technical Report)
- ELDR: Expert-Locality-Aware Decode Routing for PD-Disaggregated MoE Serving
- iScheduler: Reinforcement Learning-Driven Continual Optimization for Large-Scale Resource Investment Problems
- TIRx: A Unified Tile Primitive ML Compiler Across Heterogeneous Backends