Direct Dynamic Retargeting for Humanoid Imitation Learning from Videos
summary
The gist
Imitation Learning from monocular video demonstrations provides a scalable approach for teaching complex skills to humanoid robots <ref:2605.23762#pg2> However…
In short
Direct Dynamic Retargeting (DDR) is a new method for teaching robots complex skills from human videos. It generates high-quality, physically possible movement trajectories directly from videos, bypassing geometric limitations found in older methods. This results in better performance for robot learning and successful real-world deployment.
Key concepts
- Geometric Retargeting (GR)
- This is a standard method that attempts to map human motion onto a robot's body shape using geometric transformations. The issue is that the intermediate steps introduce a bias, which restricts the possible movement paths and often leads to less realistic or suboptimal dynamic behaviors for the robot.
- Direct Dynamic Retargeting (DDR)
- This novel framework generates trajectories directly from expert videos by formulating the problem in task space. It uses a sampling-based Model Predictive Control solver within a physics simulator, ensuring the resulting motions are both high-fidelity and physically feasible, thus avoiding geometric bias.
- Task Space Formulation
- Instead of relying solely on body geometry, DDR defines the motion problem in 'task space.' This approach allows the system to optimize for keypoints distance and shape matching metrics directly within a physics simulator, which helps mitigate input drift from noisy video data.
Terminology used across episodes
This episode discusses
- Direct Dynamic Retargeting for Humanoid Imitation Learning from Videos · Paper Radio
- Proximal Policy Optimization Algorithms
- Isaac Lab: A GPU-Accelerated Simulation Framework for Multi-Modal Robot Learning
The paper
Direct Dynamic Retargeting for Humanoid Imitation Learning from Videos · Read on arXiv
LAAS-CNRS, University of Toulouse, CNRS, Toulouse, France · IRT Saint-Exupéry, Toulouse, France · Artificial and Natural Intelligence Toulouse Institute (ANITI), Toulouse, France
Imitation Learning from monocular video demonstrations provides a scalable approach for teaching complex skills to humanoid robots. However, translating human motion to humanoids requires overcoming significant morphological mismatches. So far, dynamic retargeting methods obtain physically feasible references by tracking an intermediate geometric retargeting in simulation, usually through its joint angles. We show that this intermediate step biases the solution through two factors: the space in which the reference is tracked, joint angles rather than task-space keypoints, and the tracked reference itself, the keypoints of the geometric retargeting rather than those of the human demonstration. We propose Direct Dynamic Retargeting (DDR), which bypasses both: it tracks the human keypoints exclusively in task space with a sampling-based Model Predictive Control solver inside a physics simulator, without contact annotations. DDR is evaluated in simulation against joint-space dynamic retargeting under controlled conditions (identical solver, budget, and warm start), on three humanoids and on motions from monocular videos and motion capture. The tracking space explains most of the gain in end-effector accuracy and robustness, and tracking the human demonstration directly prevents the transmission of errors from the intermediate geometric reference. This advantage carries over to Reinforcement Learning imitation policies, which reproduce the human demonstration more faithfully than policies trained on joint-space dynamic references. These policies are deployed zero-shot on two physical humanoids of very different sizes, H1-2 and R1, including contact-rich motions. We release the source code and a dataset of dynamically feasible references covering all the flat-ground motions of AMASS on the three humanoids.
Transcript
Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.
Rosa: We're looking at the paper: "Direct Dynamic Retargeting for Humanoid Imitation Learning from Videos".
Title and authors: Rosa: So we're looking at this paper, "Direct Dynamic Retargeting for Humanoid Imitation Learning from Videos." It’s tackling that big problem of teaching complex skills to humanoid robots by using human video demonstrations.
Dev: Right. We know translating human motion to a robot is tough because there are these massive morphological mismatches between a person and a machine. This paper proposes something called Direct Dynamic Retargeting, or DDR, which aims to cut out some of the messy steps in the old methods.
Taro: I'm curious how it cuts things out. In previous work, they found that standard geometric retargeting and indirect dynamic retargeting introduce a geometric bias that really restricts where you can even look for a solution.
Rosa: Exactly. They suggest that these intermediate kinematic projections in those old methods create this bias, which limits the search space and ends up biasing the final feasible solution they get. Direct Dynamic Retargeting is supposed to bypass that entirely by generating trajectories directly from the expert videos using a specific Model Predictive Control framework called CEM-based MPC.
Dev: So instead of going through those intermediate projections, they are formulating the whole problem directly in task space and using a sampling-based solver inside a physics simulator. This means they're optimizing for physical consistency right from the start, which should help with demonstration tracking accuracy.
Taro: I'm thinking about what that means when we talk about the math here. They define the optimization as finding a trajectory Q in the feasible set Fq0 that minimizes the distance to the reference demonstration, where this distance function includes terms for both spatial tracking and relative shape matching.
Rosa: That's a key detail because it shows they aren't just optimizing for how close the robot is to a point; they are optimizing over something that accounts for both where things are in space and how well the body shapes match up. They call this augmented distance, composed of terms Ep and El.
Dev: And they show that this approach outperforms geometric retargeting and indirect dynamic retargeting baselines on diverse agile motions, specifically showing better demonstration tracking accuracy and physical consistency compared to those methods.
Taro: That leads us to the real-world tests. They validated this framework using the Unitree H1-two humanoid, and it successfully executed target motions without needing any task-specific manual tuning from the operators <ref:2605.23762#pg1>.
Rosa: That zero-shot sim-to-real transfer result is pretty significant because it shows that DDR can work directly on physical hardware without extensive fine-tuning on the robot itself. It’s a big validation for how robust these references are.
Dev: And what I find interesting from the experimental setup is that DDR maintains near perfect feasibility across most tasks, significantly outperforming GR and IDR in terms of both feasibility and contact sequence accuracy. That superior reference quality translates directly into higher reward for the downstream reinforcement learning agents they train next.
Taro: So when you look at it from a failure mode perspective, the paper points out that this enhanced reference quality actually makes the subsequent imitation learning policies converge faster and behave more robustly when dealing with diverse balancing tasks compared to policies trained on those other methods.
Rosa: It sounds like they're not just getting better references; they're setting up a much smoother training environment for the robots, which is crucial for complex skills. So, we’ve seen how this Direct Dynamic Retargeting framework works and where it stands against the existing techniques.
Dev: And we need to keep an eye on that limitation they mention: because the objective is defined purely in task space using that Laplacian graph distance, there’s a risk of getting ill-conditioned solutions or slower convergence if the initial setup isn't perfect.
Taro: That makes sense for a complex system; if the mathematical formulation itself can lead to slow convergence under certain conditions, we need to know what those conditions are before we deploy this broadly.
Rosa: So, to wrap up on this paper, Direct Dynamic Retargeting is presented as a novel framework that generates high-quality, dynamically feasible reference trajectories directly from expert videos by formulating the problem in task space using a Laplacian graph distance.
Dev: It achieves its gains by bypassing the intermediate geometric bias inherent in previous methods, leading to superior tracking accuracy and physical consistency across various motions.
Taro: The success on the Unitree H1-two with zero-shot sim-to-real transfer shows that this isn't just a theoretical improvement but something that translates well to real physical deployment <ref:2605.23762#pg1>.
Rosa: Ultimately, these higher fidelity references directly lead to downstream gains, where reinforcement learning agents trained on DDR trajectories converge faster and show more robust behavior across diverse balancing tasks.
Dev: That said, the authors flag that because the objective is defined purely in task space, it can sometimes lead to ill-conditioned solutions or slower convergence in those specific scenarios.
Taro: So for future work, they plan to scale this framework up to handle much larger and more diverse sets of motion variables coming from real-world web videos.
Rosa: That’s the direction they’re heading—moving from controlled expert data to handling the vast, messy reality of in-the-wild video demonstrations. We'll keep an eye on how that scaling goes.
The paper's summary: Rosa: So, to recap this paper, they’ve developed Direct Dynamic Retargeting or DDR, which is this new way to get high-quality movement references for teaching humanoid robots using just videos of people doing the task.
Dev: Right. The core idea is skipping those middle steps in older methods where you had to project the human motion into a robot shape, which was causing a kind of geometric bias that messed with what kind of movement the system could even search for.
Taro: It’s about formulating the whole problem right in the task space using something called a Laplacian graph distance, which means they’re optimizing based on how close you are to the goal in terms of task performance, not just how well your robot joints look compared to a person.
Rosa: Exactly. And they use this formulation inside a sampling-based solver that runs inside a physics simulator to make sure the resulting motion is actually physically possible, which is what Dev was worried about regarding stability and control loops.
Dev: It’s about ensuring the trajectory has physical viability by optimizing over that set of dynamically feasible motions while simultaneously minimizing the distance to that expert demonstration reference.
Taro: The way they build their optimization function includes terms for both tracking where things are in space and matching the relative shape between the human and the robot, which is what makes it more than just a simple spatial match.
Rosa: And when you look at what this means for real-world deployment, they’ve shown that DDR completely bypasses those geometric limitations we talked about earlier.
Dev: The result is that their references are much more physically consistent and accurate in tracking the actual demonstrations, which directly feeds into better learning outcomes for the robot.
Taro: For a researcher like me, what this means is that we can train downstream imitation learning agents on these references and they’ll converge faster because the training data itself is higher fidelity than what we could get from standard retargeting.
Rosa: That's a big deal for scaling up, because if we can generate better reference data this way, it makes teaching complex skills to robots much more efficient overall.
Dev: But there’s a caveat they mentioned, so keep an ear out: since the objective is defined purely in task space with that distance metric, there’s a risk that sometimes the mathematical setup can lead to solutions that are ill-conditioned or just take longer to converge than they should.
Taro: It means if you're trying to deploy this for something really high-stakes, you have to be careful because of those potential slowdowns under certain setups.
Rosa: So, it’s a powerful way to generate references by focusing on the task and physics simultaneously, but we still have that convergence speed issue to watch out for.
Dev: Exactly. But the real win is that they’ve proven this works outside the lab too with hardware like the Unitree H1-two without needing manual tuning, which shows it has practical utility.
Taro: That zero-shot sim to real transfer on physical hardware is a strong validation because it means we aren't just seeing theoretical improvements in simulation; they actually translate to doing things on the actual robot.
Rosa: And that’s what I find most interesting—when you combine that high-fidelity reference generation with the improved learning speed, it sets a much higher bar for what we expect from imitation learning setups.
Dev: So, the implication is that if we want robots to learn complex, dynamic tasks reliably from video demonstrations, focusing on this direct task-space formulation over geometric projections is a very promising path forward.
The paper's improvements: Taro: So, we’ve covered how they got their results by showing that Direct Dynamic Retargeting or DDR beats geometric retargeting and indirect dynamic retargeting on demonstration tracking accuracy and physical consistency.
Rosa: Right, but the authors are also pointing out what they can do next to make this even better.
Dev: They suggest that by formulating the problem directly in task space using that Laplacian graph distance, we can mitigate a lot of noise from the human data itself which is often messy in videos.
Taro: So they’re saying that if we use this task-space formulation, the resulting reference trajectories will be more stable and less sensitive to those noisy or drifting human motion data points extracted from a video.
Rosa: That means when an AI agent tries to learn from that reference, it’s going to have a cleaner signal to work with rather than having to fight through inconsistencies in the demonstration.
Dev: And another thing they touch on is that by using a sampling-based solver right inside the physics simulator, they are inherently optimizing over complex contact sequences while guaranteeing physical viability for every single motion generated.
Taro: That part is key because it means we don't have to worry as much about the resulting motion being physically impossible or unstable when we take that trajectory out of simulation and try to run it on actual hardware.
Rosa: It’s about building a system where the reference itself is both accurate to the human intent and guaranteed to be safe for the robot.
Dev: So, in terms of improvement, they aren't just changing how you retarget; they are changing the entire optimization process to prioritize physical feasibility alongside task tracking accuracy.
Taro: This moves away from just getting a geometrically close shape and toward generating a dynamically sound movement that respects the physics of the robot.
Rosa: It’s about making sure that the learning signal we give to our AI is actually high-quality, which should lead to those faster convergence speeds we talked about earlier.
Dev: That leads us into what they plan for next, since they admit there are still some limitations on how well this task-space definition works in certain edge cases where the setup might become ill-conditioned.
Conclusion: Rosa: So to wrap up, we’ve talked about how Direct Dynamic Retargeting for Humanoid Imitation Learning from Videos is changing how we get high-quality movement references by skipping those intermediate geometric steps and optimizing directly in task space using physics simulation.
Dev: It really comes down to bypassing that geometric bias that limits the search space in older methods, which means the resulting learning agents are trained on much more consistent and physically viable data.
Taro: From my side, this makes it easier for autonomy researchers because if the reference motion is physically sound and accurate, we get better downstream policies without having to spend a lot of time fixing weird physical glitches during training.
Rosa: And the practical validation was strong too, with zero-shot sim to real transfer on hardware like the Unitree H1-two showing it works outside the lab right away.
Dev: That’s the safety case we need for a controls engineer—if it transfers well to physical robots without constant manual tuning, that significantly lowers the barrier to deployment.
Taro: I still see a limitation though, which is that since they define everything in task space using that Laplacian graph distance, there are still some edge cases where the solution can become ill-conditioned or convergence can get slow.
Rosa: That’s fair; it’s a risk because we don't want our robots getting stuck in those tricky situations when the world misbehaves.
Dev: So, the main implication is that for complex skills, having high-fidelity references generated this way leads to faster learning and more robust behavior across different balancing tasks.
Taro: It’s about moving from just getting a shape that looks right to generating actual dynamic motions that respect the physics of walking or balancing.
Rosa: Exactly, so we’ve seen how this Direct Dynamic Retargeting for Humanoid Imitation Learning from Videos framework sets a new standard for generating reference data.
Dev: We gotta keep an eye on those convergence slowdowns when it comes to those specific task-space definitions, though that’s the main caveat.
Taro: Next up, we need to see how they scale this up when you move from just expert videos in a controlled setting to handling massive datasets from in-the-wild web videos.
More episodes
- 2608.13438-ContactGuard: Pre-Contact Execution Monitoring with Action-Conditioned Latent World Models
- 2610.34593-Future Prediction as Training-Only Video Supervision for Pretrained VLA Policies
- 2610.35385-EmbodiedLab: Beyond Manipulation, Benchmarking Embodied Agents That Act to Know
- 2610.36378-Data-Driven Hamiltonian for Neural Optimal Control
- 2610.37275-A Heuristic for Scheduling Autonomous Vehicles at Intersections
- 2610.37288-Dynamic Coordination of Autonomous Vehicles at Intersections
- 2610.37632-Real-Time Coordination of Autonomous Vehicles at Intersections
- 2611.13779-Softness is something you also need! Why compliance helps control: information metrics of a passive walker with compliant ankle
- 2611.13411-Residual-Corrected Physics-Informed Neural Network KKL Observers with Input-to-State Stability Guarantees
- 2611.14488-Is Your Robot's Memory Just a Clock? Temporal Shortcuts in Memory-Augmented VLA Policies