Leveraging Human-In-The-Loop Demonstrations in Reinforcement Learning for Digital Twin-Driven Robot Flexibility

arXiv:2610.12140 · cs.RO, cs.SY, eess.SY · Submitted 2026-10-08 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: I'm Rosa, and with me are Dev and Taro, guest researcher.

Dev: Today's paper: "Leveraging Human-In-The-Loop Demonstrations in Reinforcement Learning for Digital Twin-Driven Robot Flexibility".

Rosa: The gist The proposed framework combines a digital twin, reinforcement learning,

Dev: First, who's behind it and why it matters.

Paper summary: Rosa: So we're looking at this paper today, "Leveraging Human-In-The-Loop Demonstrations in Reinforcement Learning for Digital Twin-Driven Robot Flexibility." Basically, they’re talking about how to make collaborative robots actually work in messy, changing real worlds instead of just controlled lab settings.

Dev: That makes sense. The core idea seems to be combining a digital twin, reinforcement learning, and human demonstrations into one online training system that lets the robot adapt as things change in its physical workspace.

Rosa: Exactly. They’re not just making a simulation; they’re building a hardware-in-the-loop digital twin that stays synced with the actual robot on the factory floor and can actually resume learning when something unexpected happens, like an obstacle moves.

Taro: I'm interested in how this handles the world misbehaving. If you have a policy that works perfectly in one setup, but then someone puts a new object in the way, how does this system react?

Dev: Well, page two talks about their hardware-in-the-loop DT being synchronized with camera feeds so the virtual robot can update its observations and policy right from real-world feedback. If that synchronization detects a failure after a change, it resumes training in simulation.

Rosa: And they use this setup on the Ufactory Xarm5 robot, using a ZED 2i depth camera to map goals and obstacles into coordinates before feeding that information into their system <ref:2610.12140#pg1>.

Taro: That mapping process sounds crucial for how the system understands the physical space when it changes. What's interesting is how they structure the learning part of this framework, which they call the dual actor framework.

Dev: It uses a dual actor setup where demonstrations train one separate behavior-cloning actor and then guides the main reinforcement learning actor only through a critic target instead of adding a direct imitation loss to the RL actor.

Rosa: That distinction is important because it means that even if you have poor or non-optimal demonstrations, the RL agent still has room to explore actions beyond what those initial examples show. It lets it actually learn new things when things get tricky.

Taro: So, instead of just copying the human's exact moves, one actor proposes an alternative action based on the learned behavior cloning actor’s path and the critic target from the main RL actor.

Dev: Right, and they use these human demonstrations captured in virtual reality which are transferred to their DT, where it automatically evaluates each step using task rewards. This lets them use human input without having to manually label all those transitions with reward data.

Rosa: I saw some numbers that suggest this approach is quite robust even when the demonstrations aren't perfect, showing success rates over eighty percent in one test case.

Taro: They also tested it in three different scenes where the physical geometry was fixed but changed slightly, and they found that with DA-SACBC, which is their main proposed method, it was the only one to converge across all six seeds.

Paper summary: Dev: The results for non-optimal demonstrations were pretty strong too; DA-SACBC reached a mean deterministic evaluation success of eighty-three point three percent in one scenario and one hundred percent in another.

Rosa: That’s a big jump compared to some of the baseline imitation loss methods, which only got around seven point three percent and three point five percent. It really shows that this specific way of integrating the demonstrations helps the agent exploit that learned path while still exploring when it gets better than what the human showed.

Taro: Thinking about what this means for autonomy, if you’re deploying a robot in a dynamic environment, this framework suggests you don't need to perfectly pre-program every single contingency; you can let it learn on the fly by referencing human examples dynamically.

Dev: From an engineering standpoint, the challenge is keeping up with that real-time synchronization and managing the loop rate when those failures happen during operation. That’s a key part of how this system functions in practice.

Rosa: And they aren't just stopping at one setup; they tested moving goals too, showing that even with those changes, DA-SACBC still managed to achieve success rates around sixty-seven percent compared to other methods in those modified scenes.

Taro: The paper mentions some limitations regarding what the system currently assumes about the world. They flag that it assumes relevant workspace changes appear in the RL observations, like obstacle size.

Dev: And they also pointed out that if there are gaps in camera coverage or changes outside what's represented, adaptation might be limited. They also need to address obstacles that move while the task is executing.

Rosa: It sounds like the authors know this isn't a finished product yet; they’re pointing toward future work involving velocity estimates and control barrier functions to handle physical execution safety better.

Taro: That makes sense for real deployment because even if the digital twin thinks a move is safe, the physical robot might still hit something if it doesn't account for the exact timing or motion predictions properly.

Dev: So, to sum up this paper on "Leveraging Human-In-The-Loop Demonstrations in Reinforcement Learning for Digital Twin-Driven Robot Flexibility," it proposes a system where a digital twin and reinforcement learning work together online, using human demonstrations to guide the learning process without having to manually add imitation loss directly to the RL actor.

Rosa: It’s about creating an adaptable system that learns from real-world feedback in real time by constantly checking its performance against what it saw a human do.

Taro: The implication is that for complex collaborative tasks where environments shift, having this kind of continuous adaptation capability is much more useful than just having a static pre-trained model.

Dev: It’s about building flexibility into the training and operation loop itself, rather than trying to solve every single change with a completely new piece of code.

Rosa: That's the main thrust I got from this paper, moving away from rigid programming toward more adaptive learning systems for robotics in variable settings.

Conclusion: Rosa: So we've talked about this paper that’s titled "Leveraging Human-In-The-Loop Demonstrations in Reinforcement Learning for Digital Twin-Driven Robot Flexibility."

Dev: Yeah, it’s all about this system that lets a robot adapt online because it’s synced up with a digital twin and can keep learning when things change in the workspace.

Taro: It seems like they're focusing on making the robot flexible enough to handle unexpected situations on the fly, not just following a pre-set path.

Rosa: Exactly, and what’s really interesting is how they use human demonstrations inside that digital twin to train the learning process without having to write all that reward data by hand.

Dev: That dual actor framework is clever because it lets one part of the AI learn from those demonstrations while another part explores beyond them if it does better than what the human showed.

Taro: So, even if you give it imperfect instructions, it can still find a better way to move around when things get tricky.

Rosa: And they show some pretty solid results where this approach beats other methods when the robot has to deal with changes in its setup or goals.

Dev: The numbers are telling, showing that DA-SACBC reaches high success rates even with less than perfect examples compared to some of the imitation loss baselines.

Taro: It suggests that for real-world robotics, relying only on perfectly programmed scenarios isn't the answer; you need this kind of dynamic adaptation mechanism.

Rosa: So, what does this mean for us out here in the field, when we’re deploying these robots in actual factories or warehouses?

Dev: It means a robot could be much more resilient to changes—like a new tool being added or an obstacle shifting—because it’s constantly referencing its real-time view and learned behaviors.

Taro: It shifts the focus from building one perfect program to building a system that can keep learning and adjusting in response to continuous, messy interaction with the world.

Rosa: Right, so this paper is looking at how we can build robots that are less brittle when things go off-script.

Queen's University Belfast

cs.RO, cs.SY, eess.SY

Submitted: 2026-10-08

Updated: 2026-10-08

Code: https://github.com/ultralytics/ultralytics

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 83/100

The gist: The gist The proposed framework combines a digital twin, reinforcement learning, and human demonstrations to create an online training system for collaborative robots that can adapt to changes in

Key concepts

Digital Twin and Hardware-in-the-Loop System
This involves creating a virtual replica of the physical robot's workspace that stays synchronized with it using camera feeds. When the real robot encounters a change, the twin detects policy failure and restarts RL training in simulation to adapt to the new situation.
Dual Actor Framework for Demonstrations
This method uses two separate actors: one learns from human demonstrations (behavior-cloning actor) and another learns through reinforcement learning. The demonstration actor guides the RL agent, allowing it to explore beyond poor initial examples while exploiting successful demonstrated paths.
Human Effort Measurement
The study measured operator effort and workload during demonstrations. It found that complete demonstrations required the highest effort (90), followed by mental demand and performance, indicating the cognitive load involved in capturing expert robot actions.

Terminology

Summary

The gist The proposed framework combines a digital twin, reinforcement learning, and human demonstrations to create an online training system for collaborative robots that can adapt to changes in their physical workspace.

Digital Twin and Hardware-in-the-Loop System

The framework utilizes a hardware-in-the-loop DT that stays synchronized with the physical workspace and resumes RL training when a change causes the current policy to fail The digital twin is synchronized with the physical system in real time through camera feeds, allowing the virtual robot to update its observations and policy from real-world feedback If the twin detects that the current policy fails an obstacle-avoidance task after a workspace change, the RL agent resumes training in simulation The physical setup includes the Ufactory Xarm5 robot, a ZED 2i depth camera, physical obstacles and goal objects, and an HTC VIVE system Camera images update the DT in two stages: we detect goals and obstacles, then map their positions into PyBullet coordinates The linear form of Eq. (1) rests on the following assumptions: (i) the camera pose is fixed with respect to the workspace, and the perspective transformation removes perspective distortion, so pixel coordinates in the transformed view are proportional to metric coordinates on the table plane; (ii) all objects rest on the table plane, so the height hz fully determines the z coordinate of an object’s center; and (iii) residual lens distortion after calibration is negligible over the taped workspace area.

Dual Actor Framework for Demonstrations

The framework employs a dual actor framework in which demonstrations train a separate behaviour-cloning actor and guide the RL actor only through the critic target Unlike methods that add an imitation loss to the RL actor, this approach allows it to explore beyond poor demonstrations. The second actor learns only from VR demonstrations through behaviour cloning (BC) with objective JBC (ξ): JBC (ξ) = N X D i=1 πBC (si; ξ) − ai 2. The value function is then updated via: Vˆ (st+1) ← min i=1,2 Qθi(st+1, at+1). This approach helps the agent exploit the BC actor’s path more while exploring further when the agent’s performance surpasses that of the human.

Integration of Human Demonstrations

Human demonstrations are captured in virtual reality (VR) and transferred to the DT, where they guide the virtual robot. The DT evaluates each demonstrated step using the task reward and labels the resulting transitions automatically This allows an actor-critic learner to use human demonstrations without manual reward annotation. The dual actor framework adds no direct imitation loss to the RL actor. Instead, a separate behaviour-cloning (BC) actor proposes an alternative action.

Performance with Non-Optimal Demonstrations

With fixed non-optimal demonstrations, the proposed framework reached over 80% mean final training success, while two imitation-loss baselines rarely completed the task The results show that SACBC and SACBC + Q-Filter converge in none of the six seeds and end with training success rates of 7.3% and 3.5%, whereas DA-SACBC reaches 80.5% and 85.0% for the two target forms. With non-optimal demonstrations, the DASACBC policies reach a mean deterministic evaluation success of 83.3% and 100.0%, against 0.0% for SACBC and 16.7% for SACBC + Q-Filter.

Generalization to More Complex Scenarios

The framework was tested in three additional scenes derived from the nominal scene by one fixed geometric change in simulation. In all three scenes the existing policies fail, motivating retraining of the tested policies With the pillar (S1), DA-SACBC was the only method to converge in any seed and reached a mean deterministic evaluation success of 83%, against 50% for SACBC and 0% for SAC. With the moved goal (S3) and complete demonstrations, DA-SACBC and SAC without demonstrations reached the same mean deterministic evaluation success (67%), both above SACBC (33%), and DA-SACBC had the lowest mean number of sampled-contact episodes of the three.

Human Effort Measurement

Operator effort and workload were measured for one person in one session. For complete demonstrations, effort was highest (90), followed by mental demand and performance (75 each). The partial set took fewer attempts and less time, while unsuccessful trials in the DT posed no collision risk to the physical robot.

The framework currently assumes that relevant workspace changes appear in the RL observations, such as obstacle size; changes outside that representation and gaps in camera coverage may limit adaptation. Future work will address obstacles that move during task execution The state must include velocity estimates or short-horizon motion predictions, and the policy must be trained across a range of motions rather than retrained after every change. Because a trajectory judged safe in the DT may be unsafe by the time the physical robot executes it, contact-penalty rewards should be complemented by safe-RL mechanisms such as control barrier functions. Lidar could improve scene sensing, while force/torque sensing could help detect contact with coworkers. Moving obstacles also make demonstrations harder to time; we will integrate VR headsets into the DT so operators can see the virtual robot and obstacles from their own viewpoint.

The paper is supported by the European Commission through the Horizon Europe project HARMONICA under grant agreement No 101294331 The remainder of this paper is organized as follows. Section II reviews related work. Section III describes the DT, and Section IV details the dual actor framework. Section V presents the experiments and discussion; Section VI concludes the paper

The references section lists numerous prior works related to reinforcement learning, digital twins, and imitation learning in robotics. The authors compare their proposed framework against various baselines, including those that add an imitation loss to the actor objective. The comparison shows that the dual actor framework achieves a much higher final success rate than two methods that add an imitation loss to the actor. This work is supported by the European Commission through the Horizon Europe project HARMONICA under grant agreement No 101294331. The authors are with the School of Electronics, Electrical Engineering and Computer Science, Queen’s University Belfast, Northern Ireland, UK.

The paper is supported by the European Commission through the Horizon Europe project HARMONICA under grant agreement No 101294331. The authors are with the School of Electronics, Electrical Engineering and Computer Science, Queen’s University Belfast, Northern Ireland, UK.

The paper is supported by the European Commission through the Horizon Europe project HARMONICA under grant agreement No 101294331.

Improvements for AI systems

  1. Bold header: Hardware-in-the-loop DT for Resumed RL Training. This system can resume training after a change in the physical workspace by synchronizing a PyBullet digital twin with real-time camera feeds, allowing the RL agent to update its observations and policy from real-world feedback.

  2. Bold header: Dual Actor Framework for Robust Learning. The proposed framework utilizes a dual actor framework in which demonstrations train a separate behaviour-cloning actor and guide the RL actor only through the critic target, enabling the RL actor to explore beyond poor demonstrations without adding a direct imitation loss to its objective.

  3. Bold header: Real Human VR Demonstration Exploitation. The system can utilize real human VR demonstrations collected in virtual reality (VR) where the DT labels each demonstrated step using the task reward and labels the resulting transitions automatically, achieving high success rates even with demonstrations that never reach the goal.

  4. Bold header: Adaptive Policy Resumption During Workspace Change. The framework achieves adaptation when a change occurs by allowing RL training to resume, as it addresses the gap where traditional DTs may no longer reflect the physical scene and the model or policy may need to be updated.

  5. Bold header: Efficient Demonstration Guidance Without Direct Loss. By employing a dual actor structure, the system exploits demonstrations without adding a direct imitation loss to the RL actor, contrasting with methods that add an imitation loss on the RL actor, which can limit exploration of better solutions when using noisy, non-optimal demonstrations.

Sources

Related papers