Direct Dynamic Retargeting for Humanoid Imitation Learning from Videos
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.
Rosa: We're looking at the paper: "Direct Dynamic Retargeting for Humanoid Imitation Learning from Videos".
Title and authors: Rosa: So we're looking at this paper, "Direct Dynamic Retargeting for Humanoid Imitation Learning from Videos." It’s tackling that big problem of teaching complex skills to humanoid robots by using human video demonstrations.
Dev: Right. We know translating human motion to a robot is tough because there are these massive morphological mismatches between a person and a machine. This paper proposes something called Direct Dynamic Retargeting, or DDR, which aims to cut out some of the messy steps in the old methods.
Taro: I'm curious how it cuts things out. In previous work, they found that standard geometric retargeting and indirect dynamic retargeting introduce a geometric bias that really restricts where you can even look for a solution.
Rosa: Exactly. They suggest that these intermediate kinematic projections in those old methods create this bias, which limits the search space and ends up biasing the final feasible solution they get. Direct Dynamic Retargeting is supposed to bypass that entirely by generating trajectories directly from the expert videos using a specific Model Predictive Control framework called CEM-based MPC.
Dev: So instead of going through those intermediate projections, they are formulating the whole problem directly in task space and using a sampling-based solver inside a physics simulator. This means they're optimizing for physical consistency right from the start, which should help with demonstration tracking accuracy.
Taro: I'm thinking about what that means when we talk about the math here. They define the optimization as finding a trajectory Q in the feasible set Fq0 that minimizes the distance to the reference demonstration, where this distance function includes terms for both spatial tracking and relative shape matching.
Rosa: That's a key detail because it shows they aren't just optimizing for how close the robot is to a point; they are optimizing over something that accounts for both where things are in space and how well the body shapes match up. They call this augmented distance, composed of terms Ep and El.
Dev: And they show that this approach outperforms geometric retargeting and indirect dynamic retargeting baselines on diverse agile motions, specifically showing better demonstration tracking accuracy and physical consistency compared to those methods.
Taro: That leads us to the real-world tests. They validated this framework using the Unitree H1-two humanoid, and it successfully executed target motions without needing any task-specific manual tuning from the operators <ref:2605.23762#pg1>.
Rosa: That zero-shot sim-to-real transfer result is pretty significant because it shows that DDR can work directly on physical hardware without extensive fine-tuning on the robot itself. It’s a big validation for how robust these references are.
Dev: And what I find interesting from the experimental setup is that DDR maintains near perfect feasibility across most tasks, significantly outperforming GR and IDR in terms of both feasibility and contact sequence accuracy. That superior reference quality translates directly into higher reward for the downstream reinforcement learning agents they train next.
Taro: So when you look at it from a failure mode perspective, the paper points out that this enhanced reference quality actually makes the subsequent imitation learning policies converge faster and behave more robustly when dealing with diverse balancing tasks compared to policies trained on those other methods.
Rosa: It sounds like they're not just getting better references; they're setting up a much smoother training environment for the robots, which is crucial for complex skills. So, we’ve seen how this Direct Dynamic Retargeting framework works and where it stands against the existing techniques.
Dev: And we need to keep an eye on that limitation they mention: because the objective is defined purely in task space using that Laplacian graph distance, there’s a risk of getting ill-conditioned solutions or slower convergence if the initial setup isn't perfect.
Taro: That makes sense for a complex system; if the mathematical formulation itself can lead to slow convergence under certain conditions, we need to know what those conditions are before we deploy this broadly.
Rosa: So, to wrap up on this paper, Direct Dynamic Retargeting is presented as a novel framework that generates high-quality, dynamically feasible reference trajectories directly from expert videos by formulating the problem in task space using a Laplacian graph distance.
Dev: It achieves its gains by bypassing the intermediate geometric bias inherent in previous methods, leading to superior tracking accuracy and physical consistency across various motions.
Taro: The success on the Unitree H1-two with zero-shot sim-to-real transfer shows that this isn't just a theoretical improvement but something that translates well to real physical deployment <ref:2605.23762#pg1>.
Rosa: Ultimately, these higher fidelity references directly lead to downstream gains, where reinforcement learning agents trained on DDR trajectories converge faster and show more robust behavior across diverse balancing tasks.
Dev: That said, the authors flag that because the objective is defined purely in task space, it can sometimes lead to ill-conditioned solutions or slower convergence in those specific scenarios.
Taro: So for future work, they plan to scale this framework up to handle much larger and more diverse sets of motion variables coming from real-world web videos.
Rosa: That’s the direction they’re heading—moving from controlled expert data to handling the vast, messy reality of in-the-wild video demonstrations. We'll keep an eye on how that scaling goes.
The paper's summary: Rosa: So, to recap this paper, they’ve developed Direct Dynamic Retargeting or DDR, which is this new way to get high-quality movement references for teaching humanoid robots using just videos of people doing the task.
Dev: Right. The core idea is skipping those middle steps in older methods where you had to project the human motion into a robot shape, which was causing a kind of geometric bias that messed with what kind of movement the system could even search for.
Taro: It’s about formulating the whole problem right in the task space using something called a Laplacian graph distance, which means they’re optimizing based on how close you are to the goal in terms of task performance, not just how well your robot joints look compared to a person.
Rosa: Exactly. And they use this formulation inside a sampling-based solver that runs inside a physics simulator to make sure the resulting motion is actually physically possible, which is what Dev was worried about regarding stability and control loops.
Dev: It’s about ensuring the trajectory has physical viability by optimizing over that set of dynamically feasible motions while simultaneously minimizing the distance to that expert demonstration reference.
Taro: The way they build their optimization function includes terms for both tracking where things are in space and matching the relative shape between the human and the robot, which is what makes it more than just a simple spatial match.
Rosa: And when you look at what this means for real-world deployment, they’ve shown that DDR completely bypasses those geometric limitations we talked about earlier.
Dev: The result is that their references are much more physically consistent and accurate in tracking the actual demonstrations, which directly feeds into better learning outcomes for the robot.
Taro: For a researcher like me, what this means is that we can train downstream imitation learning agents on these references and they’ll converge faster because the training data itself is higher fidelity than what we could get from standard retargeting.
Rosa: That's a big deal for scaling up, because if we can generate better reference data this way, it makes teaching complex skills to robots much more efficient overall.
Dev: But there’s a caveat they mentioned, so keep an ear out: since the objective is defined purely in task space with that distance metric, there’s a risk that sometimes the mathematical setup can lead to solutions that are ill-conditioned or just take longer to converge than they should.
Taro: It means if you're trying to deploy this for something really high-stakes, you have to be careful because of those potential slowdowns under certain setups.
Rosa: So, it’s a powerful way to generate references by focusing on the task and physics simultaneously, but we still have that convergence speed issue to watch out for.
Dev: Exactly. But the real win is that they’ve proven this works outside the lab too with hardware like the Unitree H1-two without needing manual tuning, which shows it has practical utility.
Taro: That zero-shot sim to real transfer on physical hardware is a strong validation because it means we aren't just seeing theoretical improvements in simulation; they actually translate to doing things on the actual robot.
Rosa: And that’s what I find most interesting—when you combine that high-fidelity reference generation with the improved learning speed, it sets a much higher bar for what we expect from imitation learning setups.
Dev: So, the implication is that if we want robots to learn complex, dynamic tasks reliably from video demonstrations, focusing on this direct task-space formulation over geometric projections is a very promising path forward.
The paper's improvements: Taro: So, we’ve covered how they got their results by showing that Direct Dynamic Retargeting or DDR beats geometric retargeting and indirect dynamic retargeting on demonstration tracking accuracy and physical consistency.
Rosa: Right, but the authors are also pointing out what they can do next to make this even better.
Dev: They suggest that by formulating the problem directly in task space using that Laplacian graph distance, we can mitigate a lot of noise from the human data itself which is often messy in videos.
Taro: So they’re saying that if we use this task-space formulation, the resulting reference trajectories will be more stable and less sensitive to those noisy or drifting human motion data points extracted from a video.
Rosa: That means when an AI agent tries to learn from that reference, it’s going to have a cleaner signal to work with rather than having to fight through inconsistencies in the demonstration.
Dev: And another thing they touch on is that by using a sampling-based solver right inside the physics simulator, they are inherently optimizing over complex contact sequences while guaranteeing physical viability for every single motion generated.
Taro: That part is key because it means we don't have to worry as much about the resulting motion being physically impossible or unstable when we take that trajectory out of simulation and try to run it on actual hardware.
Rosa: It’s about building a system where the reference itself is both accurate to the human intent and guaranteed to be safe for the robot.
Dev: So, in terms of improvement, they aren't just changing how you retarget; they are changing the entire optimization process to prioritize physical feasibility alongside task tracking accuracy.
Taro: This moves away from just getting a geometrically close shape and toward generating a dynamically sound movement that respects the physics of the robot.
Rosa: It’s about making sure that the learning signal we give to our AI is actually high-quality, which should lead to those faster convergence speeds we talked about earlier.
Dev: That leads us into what they plan for next, since they admit there are still some limitations on how well this task-space definition works in certain edge cases where the setup might become ill-conditioned.
Conclusion: Rosa: So to wrap up, we’ve talked about how Direct Dynamic Retargeting for Humanoid Imitation Learning from Videos is changing how we get high-quality movement references by skipping those intermediate geometric steps and optimizing directly in task space using physics simulation.
Dev: It really comes down to bypassing that geometric bias that limits the search space in older methods, which means the resulting learning agents are trained on much more consistent and physically viable data.
Taro: From my side, this makes it easier for autonomy researchers because if the reference motion is physically sound and accurate, we get better downstream policies without having to spend a lot of time fixing weird physical glitches during training.
Rosa: And the practical validation was strong too, with zero-shot sim to real transfer on hardware like the Unitree H1-two showing it works outside the lab right away.
Dev: That’s the safety case we need for a controls engineer—if it transfers well to physical robots without constant manual tuning, that significantly lowers the barrier to deployment.
Taro: I still see a limitation though, which is that since they define everything in task space using that Laplacian graph distance, there are still some edge cases where the solution can become ill-conditioned or convergence can get slow.
Rosa: That’s fair; it’s a risk because we don't want our robots getting stuck in those tricky situations when the world misbehaves.
Dev: So, the main implication is that for complex skills, having high-fidelity references generated this way leads to faster learning and more robust behavior across different balancing tasks.
Taro: It’s about moving from just getting a shape that looks right to generating actual dynamic motions that respect the physics of walking or balancing.
Rosa: Exactly, so we’ve seen how this Direct Dynamic Retargeting for Humanoid Imitation Learning from Videos framework sets a new standard for generating reference data.
Dev: We gotta keep an eye on those convergence slowdowns when it comes to those specific task-space definitions, though that’s the main caveat.
Taro: Next up, we need to see how they scale this up when you move from just expert videos in a controlled setting to handling massive datasets from in-the-wild web videos.
LAAS-CNRS, University of Toulouse, CNRS, Toulouse, France · IRT Saint-Exupéry, Toulouse, France · Artificial and Natural Intelligence Toulouse Institute (ANITI), Toulouse, France
cs.RO
Submitted: 2026-05-22
Updated: 2026-11-17
Comments: https://gepetto.github.io/robot-motion/ddr/
Code: https://github.com/stephane-caron/pink
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 87/100
The gist: Imitation Learning from monocular video demonstrations provides a scalable approach for teaching complex skills to humanoid robots <ref:2605.23762#pg2> However…
Key concepts
- Geometric Retargeting (GR)
- This is a standard method that attempts to map human motion onto a robot's body shape using geometric transformations. The issue is that the intermediate steps introduce a bias, which restricts the possible movement paths and often leads to less realistic or suboptimal dynamic behaviors for the robot.
- Direct Dynamic Retargeting (DDR)
- This novel framework generates trajectories directly from expert videos by formulating the problem in task space. It uses a sampling-based Model Predictive Control solver within a physics simulator, ensuring the resulting motions are both high-fidelity and physically feasible, thus avoiding geometric bias.
- Task Space Formulation
- Instead of relying solely on body geometry, DDR defines the motion problem in 'task space.' This approach allows the system to optimize for keypoints distance and shape matching metrics directly within a physics simulator, which helps mitigate input drift from noisy video data.
Terminology
Summary
Imitation Learning from monocular video demonstrations provides a scalable approach for teaching complex skills to humanoid robots <ref:2605. p. 0 Imitation policies obtained from these trajectories can then be ported on a real full-size humanoid robot. Abstract— Imitation Learning from monocular video demonstrations provides a scalable approach for teaching complex
skills to humanoid robots. Abstract— Imitation Learning from monocular video demonstrations provides a scalable approach for teaching complex
skills to humanoid robots. However, translating human motion
to humanoids requires overcoming significant morphological
mismatches. However, translating human motion
to humanoids requires overcoming significant morphological
mismatches. Standard approaches rely on Geometric Retargeting or Indirect Dynamic Retargeting pipelines. We identify that
these intermediate kinematic projections introduce a geometric
bias, restricting the search space and yielding suboptimal
dynamic behaviors. In this paper, we propose Direct Dynamic
Retargeting (DDR), a novel single-stage framework that generates high-fidelity, dynamically feasible trajectories directly from
expert videos.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Pruning redundant reasoning tokens
The core problem addressed is the translation of human motion to humanoid robots, which involves overcoming significant morphological mismatches <ref:2605.23762#pg2>. Standard methods like Geometric Retargeting (GR) introduce a geometric bias that restricts the search space and yields suboptimal dynamic behaviors <ref:2605. p. 0 Standard approaches rely on Geometric Retargeting or Indirect Dynamic Retargeting pipelines. We identify that
these intermediate kinematic projections introduce a geometric
bias, restricting the search space and yielding suboptimal
dynamic behaviors. We identify that
these intermediate kinematic projections introduce a geometric
bias, restricting the search space and yielding suboptimal
dynamic behaviors. In this paper, we propose Direct Dynamic
Retargeting (DDR), a novel single-stage framework that generates high-fidelity, dynamically feasible trajectories directly from
expert videos. We identify that
these intermediate kinematic projections introduce a geometric
bias, restricting the search space and yielding suboptimal
dynamic behaviors. In this paper, we propose Direct Dynamic
Retargeting (DDR), a novel single-stage framework that generates high-fidelity, dynamically feasible trajectories directly from
expert videos.Where the paper says it
Where the paper says it
Where the paper says it
How the training signal is built
The DDR method formulates the problem in the task space and leverages a sampling-based Model Predictive Control solver within a physics simulator <ref:2605.23762#pg2>. The optimization is defined as finding a trajectory Q in the feasible set Fq0 such that it minimizes the keypoints distance d to a reference demonstration xref <ref:2605. p. 2 The Geometric Retargeting approach searches for a pair
(QGR, xGR) ∈ Q × Q˜ that minimizes the distance to the
expert keypoints d(xGR, xref ) while minimizing the distance
to the feasible set Fq0
. The Indirect Dynamic Retargeting starts from the GR
result and refines the trajectory to find a feasible pair
(QIDR, xIDR) ∈ Fq0 × Feq0 minimizing the distance to
the GR keypoints d(xIDR, xGR) in a physically consistent
environment, such as a simulator. The alternative method proposed in this work, called Direct Dynamic Retargeting, directly searches for a pair
(QDDR, xDDR) ∈ Fq0 × Feq0
that minimizes the keypoints
distance d(xDDR, xref ) without relying on a geometric
intermediate. According to the definition above, the DDR method comes
to find a minimum of the reference discrepancy over the
dynamically feasible set Fq0
:
min
U∈U
d(FK(Sq0
(U)), xref ) = min
Q∈Fq0
d(FK(Q), xref) (2)
= d(xDDR, xref ) (3)
As QIDR ∈ Fq0
, it follows that:
d(xDDR, xref ) ≤ d(xIDR, xref ), (4)
where xDDR = FK(QDDR) ∈ Feq0
and xIDR =
FK(QIDR) ∈ Feq0
. Dynamically Feasible Trajectory
IDR and DDR methods introduced in Section III-A rely
on the minimization of the distance within the task space
R
3m×T
. In order to account for the the human-robot embodiment gap and avoid unnatural motion, we consider the
augmented distance introduced in [2], composed of two
terms, spatial tracking and relative shape-matching metric:
Ep(FK(Q), xref ) = X
T
i=1
∥f k(qi) − x˜i∥
2
(5)
El(FK(Q), xref ) = X
T
i=1
1
m
∥L(f k(qi) − x˜i)∥
2
(6)
where f k(·) is the classical forward kinematics, x˜ represents
either xref in DDR or xGR in IDR, Ep is the Euclidean
distance, and El
is a relative metric that uses the LaplacianWhere the paper says it
Where the paper says it
Where the paper says it
Where the gains come from
The primary gain is achieved by bypassing the intermediate geometric bias inherent in previous methods <ref:2605.23762#pg2>. The authors show that DDR outperforms GR and IDR baselines in terms of demonstration tracking accuracy, physical consistency, and learning efficiency <ref:2605. p. 1 of physical consistency, demonstration tracking accuracy, and
learning efficiency. The main contributions are as follows:
• Introduction of Direct Dynamic Retargeting, a novel
CEM-based MPC framework that generates dynamically feasible humanoid trajectories directly from
demonstrations. Table III summarizes the contact error rates
across all motions. DDR consistently yields the closest match
to the reference videos. Our evaluations demonstrate that DDR consistently outperforms state-of-the-art retargeting baselines, yielding superior tracking accuracy by eliminating the geometric bias inherent to previous methods. These higher-fidelity references
translate directly into downstream gains: Reinforcement
Learning agents trained on DDR trajectories converge faster
and exhibit more robust behavior across diverse balancing
tasks.Where the paper says it
Where the paper says it
Where the paper says it
Real World experiments
The framework has been validated through real-world experiments on the Unitree H1-2 humanoid <ref:2605.23762#pg2>. The robot successfully executes the target motions without any task-specific manual tuning, demonstrating zero-shot sim-to-real transfer <ref:2605. p. 6 As shown
in Fig. 8 and the supplementary video, the robot successfully
executes the target motions without any task-specific manual
tuning. We observe that IDR and DDR
significantly outperform GR in terms of feasibility. These results indicate that despite incorporating feasibility costs, GR
fails to adequately account for contact dynamics, whereas
our proposed DDR maintains near-perfect feasibility across
most tasks. Our evaluations demonstrate that DDR consistently outperforms state-of-the-art retargeting baselines, yielding superior tracking accuracy by eliminating the geometric bias inherent to previous methods. These higher-fidelity references
translate directly into downstream gains: Reinforcement
Learning agents trained on DDR trajectories converge faster
and exhibit more robust behavior across diverse balancing
tasks.Where the paper says it
Where the paper says it
Where the paper says it
Conclusion
In this work, we introduce Direct Dynamic Retargeting (DDR), a novel framework for generating high-quality, dynamically feasible reference trajectories for imitation learning <ref:2605.23762#pg2>. By formulating the reference generation problem directly in the task space using a Laplacian graph distance, our approach effectively mitigates the impact of noisy and drifting human data extracted from videos <ref:2605. p. 6 8: The Unitree robot H1-2 performing (a) a stable squat,
(b) a static kung fu pose, (c) an extended one-foot balance,
(d) a dynamic pistol squat, and (e) a balancing stick pose. By formulating the reference generation problem directly in the task space using a Laplacian graph distance,
our approach effectively mitigates the impact of noisy and
drifting human data extracted from videos. By formulating the reference generation problem directly in the task space using a Laplacian graph distance,
our approach effectively mitigates the impact of noisy and
drifting human data extracted from videos. Furthermore, by
employing a sampling-based solver directly within a physics
simulator, DDR inherently optimizes over complex contact
sequences while guaranteeing the physical viability of the
resulting motions. Furthermore, by
employing a sampling-based solver directly within a physics
simulator, DDR inherently optimizes over complex contact
sequences while guaranteeing the physical viability of the
resulting motions. Our evaluations demonstrate that DDR consistently outperforms state-of-the-art retargeting baselines, yielding superior tracking accuracy by eliminating the geometric bias inherent to previous methods. Our evaluations demonstrate that DDR consistently outperforms state-of-the-art retargeting baselines, yielding superior tracking accuracy by eliminating the geometric bias inherent to previous methods. These higher-fidelity references
translate directly into downstream gains: Reinforcement
Learning agents trained on DDR trajectories converge faster
and exhibit more robust behavior across diverse balancing
tasks. However, DDR has limitations. The objective is defined
purely in task space, which can lead to ill-conditioned solutions or slower convergence. Finally, performance remains
sensitive to task-space weighting and initialization, which
currently require manual tuning. Future work will scale this framework to handle large,
diverse sets of motion variables from in-the-wild web videos,
enabling the creation of a comprehensive dataset of physically viable humanoid reference motions for the community.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Direct Dynamic Retargeting (DDR) framework allows for generating
high-fidelity, dynamically feasible trajectories directly from expert videos,
which bypasses intermediate geometric bias that restricts search space in previous methods. -
The DDR method enables the generation of
physically consistent references
by formulating the problem in task space and leveraging a sampling-based Model Predictive Control solver within a physics simulator, thus mitigating input drift. -
Downstream Imitation Learning agents trained on DDR trajectories
converge faster and exhibit more robust behavior across diverse balancing tasks,
leading to improved final reward compared to kinematic approaches. -
The system can achieve
zero-shot sim-to-real transfer
on physical hardware like the Unitree H1-2 humanoid, validating the practical utility of the framework for real-world deployment.
Abstract
Imitation Learning from monocular video demonstrations provides a scalable approach for teaching complex skills to humanoid robots. However, translating human motion to humanoids requires overcoming significant morphological mismatches. So far, dynamic retargeting methods obtain physically feasible references by tracking an intermediate geometric retargeting in simulation, usually through its joint angles. We show that this intermediate step biases the solution through two factors: the space in which the reference is tracked, joint angles rather than task-space keypoints, and the tracked reference itself, the keypoints of the geometric retargeting rather than those of the human demonstration. We propose Direct Dynamic Retargeting (DDR), which bypasses both: it tracks the human keypoints exclusively in task space with a sampling-based Model Predictive Control solver inside a physics simulator, without contact annotations. DDR is evaluated in simulation against joint-space dynamic retargeting under controlled conditions (identical solver, budget, and warm start), on three humanoids and on motions from monocular videos and motion capture. The tracking space explains most of the gain in end-effector accuracy and robustness, and tracking the human demonstration directly prevents the transmission of errors from the intermediate geometric reference. This advantage carries over to Reinforcement Learning imitation policies, which reproduce the human demonstration more faithfully than policies trained on joint-space dynamic references. These policies are deployed zero-shot on two physical humanoids of very different sizes, H1-2 and R1, including contact-rich motions. We release the source code and a dataset of dynamically feasible references covering all the flat-ground motions of AMASS on the three humanoids.
Sources
- Proximal Policy Optimization Algorithms
- Isaac Lab: A GPU-Accelerated Simulation Framework for Multi-Modal Robot Learning
Related papers
- Rethinking Actions: Continuity-Aware Priors for World-Model Planning
- MOSAIC: Modular Obstacle and Spatial-Aware Framework for Manipulation In Cluttered Scenes
- SAGAS: Semantic-Aware Graph-Assisted Stitching for Offline Temporal Logic Planning
- DexSeed: Expanding Human Hand-Object Interactions into Robot Demonstrations for Dexterous Policy Learning
- LaST0.5: A Generalist World Action Model with Interaction-Centric Latent Reasoning
- CIALLO: Learning Shared Intent Representations for Cross-Embodiment Manipulation