USER: A Unified and Extensible System for Real-World Online Policy Learning in Embodied AI

The gist: USER presents a unified and extensible system for real-world online policy learning that integrates hardware abstraction, adaptive communication, persistent data management, and asynchronous training into a single framework.

System Architecture

USER introduces a hardware abstraction layer (HAL) to treat physical robots at the same resource level as accelerators like GPUs and TPUs <ref:2602.07837#pg2>. This HAL provides unified interfaces for extending new types of hardware, automatically discovering available hardware resources, and scheduling hardware for different learning tasks <ref:2602.07837#pg4>. The system models deployment as a cluster of nodes where each node exports a set of hardware units <ref:2602.07837#pg5>. A key feature is the Hardware Registration and Discovery, which uses pluggable checkers to register and extend new hardware, isolating device-specific logic from the rest of the system <ref:2602.07837#pg6>. Hardware scheduling is managed through a single rank-based placement interface where components select node groups and resource ranks, allowing for heterogeneous placements in one job <ref:2602.07837#pg7>.

Adaptive Communication Plane

USER addresses cloud–edge deployment challenges with an adaptive communication plane designed to handle heterogeneous and isolated network domains <ref:2602.07837#pg10>. This plane enables cross-domain connectivity via tunneling-based cloud–edge networking for data exchange across network boundaries <ref:2602.07837#pg9>. To manage uneven bandwidth, USER employs a distributed data channel to localize traffic and reduce cross-domain transfer <ref:2602.07837#pg10>. Furthermore, it introduces streaming-multiprocessorbudgeted weight synchronization to regulate GPU communication overhead during training <ref:2602.07837#pg10>.

Learning Framework

The learning framework is organized as a fully asynchronous learning framework, in which data generation, training, data transmission, and weight synchronization proceed independently rather than through tightly synchronized stages <ref:2602.07837#pg2>. To support long-horizon learning, USER includes a persistent and cache-aware buffer that enables streaming trajectory ingestion, recovery, and reuse across training phases <ref:2602.07837#pg5>. This buffer persistently stores data, enabling long-running experiments to recover from network interruptions and temporary training pauses <ref:2602.07837#pg6>. Extensible abstractions are provided for policies, algorithms, and rewards, supporting everything from conventional CNN policies to flow-based generative policies and large VLA models <ref:2602.07837#pg2>.

Key Learning Components

USER supports diverse learning paradigms through its extensible components <ref:2602.07837#pg2>:

1. Policies range from lightweight policies include CNN-based models, such as ResNet-style visual policies [12], and expressive flow-matching policies [20] to larger scale, USER integrates VLA models (e.g., π0/π0.5 architectures [4, 15]) <ref:2602.07837#pg2>.

2. Algorithms include "off-policy RL such as Soft Actor-Critic (SAC) [11], sample-efficient RL for flow policies (e.g., SAC-Flow [32]), human-in-the-loop methods like RL with pretrained data (RLPD) [3], and imitationstyle updates for large models such as HG-DAgger [17]" <ref:2602.07837#pg2>.

3. Rewards can be specified via rules, human feedback, or learned reward models <ref:2602.07837#pg2>.

Performance and Results

Experiments demonstrate that USER supports multi-robot coordination, heterogeneous manipulators, cloud-edge training with large models, and longrunning asynchronous training <ref:2602.07837#pg2>. In the context of real-world manipulation tasks like Peg Insertion and Charger Plugging, USER achieves near-perfect success within 2000s with similar overall performance when compared against SERL <ref:2602.07837#pg7>. Furthermore, the persistent and cache-aware buffer mitigates forgetting in continual learning scenarios, as shown by maintaining success rates across multiple objects over several days of training <ref:2602.07837#pg6>. The fully asynchronous pipeline significantly improves training throughput, with CNN + SAC achieving a speed up of 1.55× and π0 achieving a speed up of 4.61× <ref:2602.07837#pg10>. The unified hardware layer enables training over multi-robot and heterogeneous robot fleets <ref:2602.07837#pg10>. USER's performance gains are evident in foundation VLA models, showing improvements after online training for tasks like Pick-and-Place, where success rates increased from 39/60 to 58/60 <ref:2602.07837#pg2>. The system provides an open-source implementation of USER that provides a reusable systems foundation for real-world online policy learning <ref:2602.07837#pg2>.

1. textbf Unified Hardware Abstraction Layer for Heterogeneous Systems Recovery: The HAL will automatically discover available hardware resources and manage heterogeneous robot fleets by treating each unit as a schedulable hardware unit, enabling flexible placement in complex deployments. This allows the system to maintain robust learning across different robot types and accelerators without platform-specific code modifications.

2. textbf Adaptive Communication Plane for Cross-Domain Scalability: The system will employ a flattened TCP/IP substrate based on the UDP tunneling technology to enable cross-domain connectivity via tunneling-based cloud–edge networking. This specifically addresses bandwidth asymmetry by using a distributed data channel that allows observations and actions to be localized, which reduces cross-domain transfer and can reduce episode generation time by up to 3× in cross-domain deployments.

3. textbf Persistent and Cache-Aware Buffer for Long-Horizon Continual Learning: The buffer will feature a design that decouples storage from memory by storing lightweight indices with metadata such as policy version, timestamps, and episode IDs, which allows for temporal- and policy-aware sampling over long horizons while balancing efficiency with capacity. This supports robust crash recovery and prevents the model from forgetting of the object trained on the first day.

4. textbf Fully Asynchronous Pipeline for High Throughput Training: The framework will organize learning as a fully asynchronous pipeline, in which data generation, training, data transmission, and weight synchronization proceed independently, which decouples stages to significantly improve both data collection and training throughput. This allows for faster convergence by leveraging pipeline overlapping and reduces overall training time from 8000+ seconds to approximately 1500 seconds for some models.

5. textbf Extensible Learning Abstractions for Diverse Paradigms: The system will expose unified interfaces that enable heterogeneous policies, optimizers, and reward mechanisms to share a single execution and data pipeline. This allows the AI system to seamlessly support diverse real-world learning setups—from RL to imitation and human-in-the-loop learning—without re-engineering deployment, data handling, or execution pipelines.

Online policy learning directly in the physical world is a promising yet challenging direction for embodied intelligence. Unlike simulation, real-world systems cannot be arbitrarily accelerated, cheaply reset, or massively replicated, suggesting that real-world policy learning is not merely an algorithmic problem, but inherently a systems problem. We present USER, a nified and extensible yst m for real-world online policy lea ning. On the systems side, USER introduces a hardware abstraction layer for unified robot management and an adaptive communication plane that enables efficient cloud-edge training. On the learning side, USER adopts a fully asynchronous training framework, designs a persistent and cache-aware replay buffer, and provides extensible abstractions for rewards, algorithms, and policies. Experiments in both simulation and the real world demonstrate that USER supports multi-robot coordination, heterogeneous manipulators, cloud-edge training with large models, and long-running asynchronous training. Together, these capabilities establish USER as a unified and extensible systems foundation for real-world online policy learning.