Learning Generalizable Hand-Object Tracking from Synthetic Demonstrations
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.
Rosa: We're looking at the paper: "Learning Generalizable Hand-Object Tracking from Synthetic Demonstrations".
Paper summary: Rosa: So, we're looking at this paper, "Learning Generalizable Hand-Object Tracking from Synthetic Demonstrations." The core idea is that you can learn how to track hands and objects really well just by using synthetic data instead of needing a human to do it.
Dev: Right, so the thesis here is building a system that learns hand-object tracking controllers purely from synthetic data, without needing any human demonstrations at all. They claim this approach makes two main contributions: one is HOP, which they call a Hand-Object Planner, and another is HOT, which is the Hand-Object Tracker.
Taro: I'm interested in how that planner generates the data because acquiring diverse manipulation data from humans is really hard right now. If you can generate it synthetically, that removes a huge barrier for learning these skills.
Rosa: Exactly. HOP synthesizes diverse hand-object trajectories through this two-stage process: first, it uses force closure optimization and reinforcement learning to create a set of stable grasp configurations. Then, HOP anchors itself to those poses to synthesize trajectories for fundamental meta-skills like Grasping, Moving, Placing, and Rotating.
Dev: That sounds like they are decomposing the complex task into these eight decomposed meta-skills: Free Move, Grasp, Place, Move, Catch, Throw, Rotate—simplified—and Regrasp. That level of decomposition is key for a robust controller to handle different movements.
Taro: But if it’s just synthesized data from this planner and tracker combination that's the whole method? How does it actually generalize to things that weren't in the training data?
Rosa: The paper suggests that HOT bridges the gap between synthetic and physical worlds through reinforcement learning and interaction imitation learning. They extend this to handle diverse object shapes and hand morphologies, showing success in tracking challenging, long-horizon sequences like object re-arrangement or agile in-hand reorientation.
Dev: That's a strong claim about generalization because it implies the system isn't just memorizing specific movements but actually learning the underlying skill structure. They show this by evaluating different hand models, where Shadow and Allegro hands both achieved over eighty percent success rates for skills like Grasping, Moving, and Placing when trained with HOP data.
Taro: So it seems they’re tackling the difficulty of motion capture systems struggling with occlusion and depth ambiguity by focusing on learning from generated trajectories instead. That makes sense from a real-world failure mode standpoint.
Paper summary: Rosa: They also talk about how they refine this using a two-stage teacher student framework, where the second stage learns to assimilate all these skills at once using the first stage as guidance through behavior cloning and online data refinement.
Dev: From an engineering standpoint, that teacher student setup is interesting because it helps learn efficiently while ensuring you don't just get stuck optimizing one small skill at the expense of others. They also use domain randomization across three axes: object scale, initial state, and a curriculum on observation or action noise.
Taro: Domain randomization sounds like a good way to test robustness against real-world variations before deployment. What about the tracking mechanism itself? How does HOT actually output its actions for the hand and the wrist pose?
Dev: HOT is formulated as a policy parameterized by a multi-layer perceptron network that outputs actions for both the hand's native joints and the wrist's six degrees of freedom pose. Crucially, this policy learns residual adjustments for the wrist to make sure there’s consistent skill execution across different workspaces, leveraging translational invariance.
Rosa: The observation fed into this policy includes both what’s happening in the environment and a goal state for the next time step. This structure is designed to handle those long-horizon sequences they mentioned earlier.
Taro: I wonder how it handles situations where things go wrong, like when the object is suddenly thrown or if there’s unexpected contact during a complex reorientation sequence? How does it recover gracefully?
Dev: They use a unified imitation reward that combines different components multiplicatively to learn manipulation from demonstrations. They emphasize the contact reward for reliable grasping and releasing, and an interaction reward specifically highlighting hand-object relative motion.
Rosa: And they validate their control by showing that HOT can correct physically implausible artifacts; they measure this using object-hand penetration as a key metric when testing against real-world human-object interaction trajectories.
Taro: So the system is able to clean up unrealistic movements, which is important because you don't want a robot trying to do something physically impossible when it starts operating in the real world.
Dev: They also used an LLM to generate semantically annotated keyframes and then apply keyframe-guided interpolation in both position and orientation to ensure the generated trajectories maintain semantic coherence and dynamic feasibility. This is how they handle the high-level instruction part.
Paper summary: Rosa: That interpolation uses four special frame types to partition the LLM keypoints into three semantic segments: start to grasp, grasp to release, and release to end. Orientation is interpolated independently using spherical linear interpolation in each segment.
Taro: So when you’re following a language instruction for something like "catch that hammer," the system uses that LLM to guide the hand through these carefully constructed segments of movement?
Dev: Yes, the LLM prompt requires it to select one candidate grasp and align that generated grasp keypoint to a specific wrist pose. This pipeline ensures smooth motions inside each phase while strictly preserving those grasp and release boundaries.
Rosa: The paper also discusses how they use difficulty-prioritized sampling by calculating the mean reward of a trajectory to assess its learning difficulty, then assigning a sampling probability based on that metric for balanced training across challenging examples.
Taro: That sounds like a smart way to make sure the system doesn't just learn the easy parts and ignore the hard, long-horizon sequences. What about scaling this up?
Dev: They note that scaling to more complex skills and broader object categories might require large-scale, learning-based refinement of HOP data and hierarchical distillation strategies. Plus, they mention that real-world deployment faces the sim to real gap, which demands hardware with high control frequency and force sensing capabilities.
Rosa: So while this synthetic data approach is powerful for learning the core tracking skill, it points toward future work needing more refined planning and better bridging that gap when moving to actual physical robots.
Dev: Ultimately, the paper's contribution is showing a scalable framework where you can learn entirely from synthetic data while remaining compatible with human motion tracking and text-guided task execution. It breaks the data bottleneck that has constrained progress in dexterous manipulation for a long time.
Taro: So what this means for us outside of the lab is that we might see controllers that can track complex, dynamic sequences without needing thousands of hours of expensive human demonstration data.
Rosa: That’s right. It offers a flexible, scalable foundation for future work on general-purpose dexterous manipulation because it learns from synthetic demonstrations while still being compatible with real-world interaction data.
Conclusion: Rosa: So, this paper is about learning how to track hands and objects just by looking at fake data instead of needing people to show them how.
Dev: Yeah, the title is "Learning Generalizable Hand-Object Tracking from Synthetic Demonstrations," and the authors are presenting a system that does exactly that.
Rosa: Essentially, they built a way to teach a hand controller to follow an object using only computer-generated examples.
Dev: The main idea is two parts: one planner called HOP for making the fake data, and one tracker called HOT that learns how to use it.
Rosa: It’s about creating a foundation controller that can handle really varied objects and hands, even things it hasn't seen before.
Dev: They show this works for complex tasks like reorienting an object in mid-air or even moving things around on a table.
Rosa: It seems like they’re tackling the big problem of getting robots to learn from synthetic data, which is a massive hurdle in real manipulation.
Dev: The implication here is that we can get more general controllers that don't just memorize one specific movement but understand the underlying skills for grabbing, moving, and placing things.
Rosa: It’s a step toward making robotic manipulation much more flexible and less dependent on expensive human demonstrations.
Dev: But they also flag the sim-to-real gap as a big thing to solve later on when you actually put this in the physical world.
Rosa: So, they’ve made some strong claims about generalization and learning from synthetic data, but we still have to worry about how well it holds up outside of a perfect simulation.
Yinhuai Wang, Runyi Yu, Hok Wai Tsui, Xiaoyi Lin, Hui Zhang, Qihan Zhao, Ke Fan, Miao Li, Jie Song, Jingbo Wang
Hong Kong University of Science and Technology (HKUST) · Wuhan University of China · ETH Zurich
cs.RO, cs.GR
Submitted: 2025-12-22
Updated: 2026-11-17
License: http://creativecommons.org/licenses/by-nc-nd/4.0/
Importance score: 94/100
The gist: Learning Generalizable Hand-Object Tracking from Synthetic Demonstrations presents a system for learning generalizable hand-object tracking controllers purely from synthetic data…
Key concepts
- HOP (Hand-Object Planner)
- This is a framework designed to create varied hand-object movements. It first generates stable grasp configurations using optimization and reinforcement learning. Then, it uses these grasps as anchors to synthesize trajectories for fundamental skills like grasping, moving, and rotating objects.
- HOT (Hand-Object Tracker)
- HOT is the tracking controller that learns how to follow a trajectory from synthetic data. It is trained using a teacher-student framework and reinforcement learning. It outputs actions for both the hand's joints and the wrist pose, learning residual adjustments to ensure consistent skill execution.
- Domain Randomization
- This technique involves training the model across variations in object scale, starting positions, and noise levels in observations and actions. By varying these parameters extensively during training, the system learns to be robust and generalize well to unseen objects and environments.
- Imitation Reward
- The reward function is designed using a unified imitation approach where different rewards are combined multiplicatively. This includes a contact reward for reliable grasping/releasing and an interaction reward focusing on the relative motion between the hand and the object.
Terminology
Summary
Learning Generalizable Hand-Object Tracking from Synthetic Demonstrations presents a system for learning generalizable hand-object tracking controllers purely from synthetic data, without requiring any human demonstrations Our approach makes two key contributions: (1) HOP, a Hand-Object Planner, which can synthesize diverse hand-object trajectories; and (2) HOT, a Hand-Object Tracker that bridges synthetic-to-physical transfer through reinforcement learning and interaction imitation learning Our method extends to diverse object shapes and hand morphologies Through extensive evaluations, we show that our approach enables dexterous hands to track challenging, long-horizon sequences including object re-arrangement and agile in-hand reorientation These results represent a significant step toward scalable foundation controllers for manipulation that can learn entirely from synthetic data, breaking the data bottleneck that has long constrained progress in dexterous manipulation. p. 0 Through extensive
evaluations, we show that our approach enables dexterous hands to track
challenging, long-horizon sequences including object re-arrangement and
agile in-hand reorientation. These results represent a significant step toward scalable foundation controllers for manipulation that can learn entirely
from synthetic data, breaking the data bottleneck that has long constrained
progress in dexterous manipulation.Where the paper says it
System Architecture and Workflow
The system consists of two parts: HOP (Hand-Object Planner), a scalable data generation framework, and HOT (Hand-Object Tracker), a tracking-based controller trained on synthetic data In applications, our system achieves flexible tasks through a modular pipeline where a high-level directive is converted into sparse waypoints that are then fed to HOP to generate a dense hand-object trajectory, which HOT executes in the simulation. This standardized workflow allows for zero-shot adaptation to numerous downstream applications. p. 2 Finally, our
tracking controller, HOT, executes this trajectory in the simulation. This standardized workflow allows for zero-shot adaptation to numerous downstream applications.Where the paper says it
Data Synthesis and Trajectory Generation
HOP synthesizes meta-skill trajectories through a structured, two-stage process First, it employs force-closure optimization and reinforcement learning to generate and refine a diverse set of stable grasp configurations. Subsequently, HOP anchors itself to these grasp poses to synthesize trajectories for fundamental meta-skills—such as Grasping, Moving, Placing, and Rotating—as well as their combinations. p. 1 2024a]
to generate and refine a diverse set of stable grasp configurations. Subsequently, HOP anchors itself to these grasp poses to synthesize
trajectories for fundamental meta-skills—such as Grasping, Moving,
Placing, and Rotating—as well as their combinations. While these trajectories may be physically approximate,
they serve as tracking targets in reinforcement imitation learning,
enabling the policy to learn generalizable target-action mappings. For efficient synthesizing and learning, we decompose manipulation
into eight meta-skills: Free Move, Grasp, Place, Move, Catch, Throw,
Rotate, and Regrasp.Where the paper says it
Where the paper says it
Tracking Control Mechanism
HOT is formulated as a policy parameterized by an MLP network that outputs actions for both the hand’s native joints and the wrist’s 6-DoF pose The policy learns residual adjustments for the wrist to ensure consistent skill execution across the workspace, leveraging translational invariance. The observation of the HOT policy includes both the environmental observation and a goal state for the subsequent time step. p. 4 Policy Input. The observation of the HOT policy is
𝒉𝑡 = {𝒔𝑡
,𝒔ˆ
𝑔𝑜𝑎𝑙
𝑡+1
} (4)
where 𝒔𝑡 = {𝒔
ℎ𝑎𝑛𝑑
𝑡
,𝒔
𝑜𝑏𝑗
𝑡
} is the environmental observation, constitutes of current hand state 𝒔
ℎ𝑎𝑛𝑑
𝑡
and object state 𝒔
𝑜𝑏𝑗
𝑡
in the simulation environment.Where the paper says it
Training Strategy and Robustness
The training employs a two-stage teacher-student framework to enhance model capacity and learning efficiency The second stage introduces a unified student policy that learns to assimilate all skills simultaneously, using the teacher policies for both online data refinement and action guidance through behavior cloning. Domain randomization is applied along three main axes: object scale, initial state, and a curriculum on observation/action noise. p. 3 We reject and resample 𝒈
𝑒𝑛𝑑 if any hand keypoints
penetrate the ground plane; otherwise, keep it as the grasping frame. To generate the initial state 𝒈
𝑖𝑛𝑖𝑡 , we compute the vector v from
object center to the first joint of the index finger, randomly sample
the initial hand position within a cone centered along v, set all
hand joint angles to zero, while keeping other parameters identical
, Vol.Where the paper says it
Performance and Generalization
The system demonstrates remarkable generalization across the entire operational space when trained exclusively on synthetic data from HOP HOT robustly tracks long-horizon, dynamic sequences such as catching a thrown hammer or reorienting it in-air. Furthermore, the system enables zero-shot transfer to downstream applications when integrated with LLMs or VLMs to follow language instructions for goal-directed manipulation. p. 1 Notably, the system enables zero-shot transfer to downstream applications. When integrated with LLMs or VLMs [Achiam
et al. 6, we evaluate object generalization capability by comparing
HOT trained on 1, 5, and 15 objects, reporting the average tracking
success rates for Grasp, Move, and Place across 20 test objects. The
results demonstrate that our method converges stably under multiobject joint training and generalizes effectively to unseen objects,
with performance improving as training object diversity increases.Where the paper says it
Where the paper says it
Refinement and Evaluation
The unified imitation reward combines complementary components multiplicatively to learn manipulation from demonstrations The contact reward is crucial for reliable grasping and releasing operations, and the interaction reward highlights hand-object relative motion. The system's ability to correct physically implausible artifacts is validated when HOT tracks real-world HOI trajectories using object-hand penetration as a key metric. The final system shows effective zero-shot tracking on real hand-object interactions despite being trained purely on synthetic data. p. 7 Here, SR-grasp measures the success rate of achieving
a stable grasp, while SR evaluates the success rate of the complete
tracking. Notably, HOT achieves effective zero-shot tracking on real
hand-object interactions despite being trained purely on synthetic
data, demonstrating its potential to overcome the data bottleneck
in general dexterous manipulation.Where the paper says it
Limitations and Future Work
While HOT provides a general-purpose low-level control capability for trajectory tracking, high-level planners operate in an open-loop manner and do not yet support long-horizon, closed-loop task execution like an LLM-based agent. Scaling to more complex skills and broader object categories may require large-scale, learning-based refinement of HOP data and hierarchical distillation strategies. p. 5 While effective for individual skills, using a single policy to learn diverse skills or object
categories faces scalability limitations. The primary challenge arises
from physical implausibilities in synthetic data, which require extensive sampling to identify feasible solutions for each instance. Xue Bin Peng, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. Sim-to-real transfer of robotic control with dynamics randomization.Where the paper says it
Where the paper says it
Ablation Study Findings
The localization and residualization of HOT input-output representations prove crucial for generalization, allowing the model to achieve superior performance even with limited training data Data augmentation techniques consistently boost generalization performance across all skills, while increased training data leads to proportional performance improvements. The complete reward formulation yields the best overall results when analyzing the impact of key design choices. p. 7 As shown in Tab. 4,
each reward term contributes noticeably to performance, and the
complete reward formulation yields the best overall results. Ruihan Yang, Huazhe Xu, Yi Wu, and Xiaolong Wang. Multi-task reinforcement
learning with soft modularization. Notably, we also generate
Move trajectories starting from the final frame of Grasp demonstrations or ending at the initial frame of Place demonstrations to
ensure smooth cross-skill transitions. Rotate demonstrates in-hand manipulation where the
object is rotated while maintaining a fixed wrist pose.Where the paper says it
Where the paper says it
Where the paper says it
LLM Integration
We employ an LLM to generate semantically annotated keyframes and subsequently apply keyframe-guided interpolation in both position and orientation to ensure the generated trajectories maintain both semantic coherence and dynamic feasibility. The LLM prompt requires the model to select one candidate grasp and align the generated grasp keypoint to that wrist pose. p. 7 The typical
pipeline uses a Vision-Language Model (VLM) for high-level semantic planning. The VLM takes as input: the object’s initial 6D pose,
a set of feasible wrist poses corresponding to stable grasp configurations, the initial wrist pose, the target object position, an image
of current environment, and a natural-language task description. In both cases, we apply trajectory smoothing and Bayesian interpolation between keyframes
to ensure smooth motions. The resulting wrist path is combined
with the chosen grasp pose using the procedure in Sec.Where the paper says it
Where the paper says it
Trajectory Interpolation Details
The four special frame types partition the LLM keypoints into three semantic segments: start → grasp, grasp → release, and release → end Within each segment, all keypoints are used as interpolation anchors to construct a cubic Bezier curve for the wrist position. Orientation is interpolated independently using spherical linear interpolation (SLERP) within each segment. p. 4 To synthesize these demonstrations, we first randomly select a grasp configuration 𝒈 from
G. We then generate random initial and final object poses within
the operational space and create an object trajectory through interpolation.Where the paper says it
Evaluation on Different Hands
Experiments on different hand morphologies show that both Shadow and Allegro hands achieve over 80% average success rates when trained with HOP data for the Grasp, Move, and Place skills. MANO demonstrates the best performance among the tested hand models due to its highest degrees of freedom (DoF). p. 2 (1)
Taking the MANO hand model [Romero et al. 2017] as an example,
T
ℎ ∈ R
3
and R
ℎ ∈ 𝑆𝑂(3) represent the global wrist pose, while
𝜽 ∈ R
15×3 describes the joint angles of the five fingers (each finger
has three joints with three degrees of freedom).Where the paper says it
Penetration Correction
The original reference data contains noticeable penetrations, while the trajectories produced by HOT exhibit significantly reduced penetration, validating its capacity to correct physically implausible artifacts. The refined GRAB dataset serves as a high-quality benchmark for evaluation after retargeting and improved physical plausibility. p. 7 We further validate generalized tracking performance on realworld data using segmented trajectories from the GRAB dataset [Taheri
et al. 2020],where trajectories are refined by HOP for retargeting
and improved physical plausibility (See the appendix for details).Where the paper says it
Adaptive Sampling
We calculate the mean reward of a trajectory to assess its learning difficulty, assigning a sampling probability based on this metric to ensure balanced learning across challenging examples. This difficulty-prioritized sampling helps in acquiring stable tracking behavior and adapting to full diversity during training. p. 1 2025] significantly enhances the agility
and robustness of task execution. However, training a generalizable
hand-object tracking controller faces the fundamental challenge of
acquiring sufficient HOI data.Where the paper says it
Conclusion
Our approach consists of HOP, a scalable framework for synthesizing hand-object trajectories across diverse meta-skills, and HOT, a tracking policy trained via reinforcement imitation learning on such data The resulting controller demonstrates effective generalization to unseen objects, long-horizon tasks, and even real-world human-object motion data. By functioning effectively without human demonstrations while remaining compatible with them, our method offers a flexible, scalable, and powerful foundation for future work on general-purpose dexterous manipulation. p. 7 The resulting controller demonstrates effective generalization to unseen objects, long-horizon tasks, and even real-world
human-object motion data. By functioning effectively without human demonstrations while remaining compatible with them, our
method offers a flexible, scalable, and powerful foundation for future
work on general-purpose dexterous manipulation.Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Data Synthesis via HOP enables learning without human demonstrations by synthesizing
meta-skill trajectories through a structured, two-stage process.
This allows training onelemental skills like Grasp, Rotate, and Move,
bypassing the need for complex task-level synthesis. -
HOT (Hand-Object Tracker) can track
complex, highly dynamic, and long-sequence manipulation trajectories during inference
by being trained on synthetic data using a unified set of HOI imitation rewards. -
The system supports zero-shot transfer to downstream applications, enabling it to
follow language instructions to control both HOP-based data synthesis and HOT’s tracking for goal-directed manipulation.
-
The policy is formulated as a Gaussian distribution over actions, allowing the network to output
residual adjustments for the wrist
andtarget finger joint rotations,
which ensuresconsistent skill execution across the workspace.
-
A two-stage teacher-student training framework enhances learning efficiency by first training specialized teacher policies for different meta-skill categories before distilling expertise into a unified student policy.
-
Domain randomization techniques, including random object scale perturbation and initial state noise injection, enhance policy robustness across geometric variations and contact configurations.
-
The system can refine imperfect synthetic references through tracking;
the trajectories produced by HOT exhibit significantly reduced penetration,
correctingphysically implausible artifacts.
-
LLM-based trajectory planning allows for end-to-end execution of language-guided manipulation by converting high-level commands into sparse waypoints that HOP converts into dense trajectories for HOT to track.
Abstract
We present a system for learning generalizable hand-object tracking controllers purely from synthetic data, without requiring any human demonstrations. Our approach makes two key contributions: (1) HOG, a highly efficient Hand-Object data generator that rapidly synthesizes diverse, large-scale, and uniformly distributed trajectories; and (2) HOT, a Hand-Object Tracker that bridges synthetic-to-physical transfer through reinforcement learning and interaction imitation learning. Our method extends to diverse object shapes and hand morphologies. HOT also enables cross-simulator transfer. Through extensive evaluations, we show that our approach enables dexterous hands to track challenging, long-horizon sequences including object re-arrangement and agile in-hand reorientation. These results represent a significant step toward scalable foundation controllers for manipulation that can learn entirely from simple, scalable synthetic data, breaking the data bottleneck that has long constrained progress in dexterous manipulation.
Sources
- GPT-4 Technical Report
- Introducing HOT3D: An Egocentric Dataset for 3D Hand and Object Tracking
- Sequential Dexterity: Chaining Dexterous Policies for Long-Horizon Manipulation
- Object-Centric Dexterous Manipulation from Human Motion Data
- Open-TeleVision: Teleoperation with Immersive Active Visual Feedback
- HumanPlus: Humanoid Shadowing and Imitation from Humans
- Efficient Data Collection for Robotic Manipulation via Compositional Generalization
- SkillMimicGen: Automated Demonstration Generation for Efficient Skill Learning and Deployment
- OmniH2O: Universal and Dexterous Human-to-Humanoid Whole-Body Teleoperation and Learning
- Dynamic Handover: Throw and Catch with Bimanual Hands
- DexMimicGen: Automated Data Generation for Bimanual Dexterous Manipulation via Imitation Learning
- ParaHome: Parameterizing Everyday Home Activities Towards 3D Generative Modeling of Human-Object Interactions
- DexTrack: Towards Generalizable Neural Tracking Control for Dexterous Manipulation from Human References
- MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrations
- THE COLOSSEUM: A Benchmark for Evaluating Generalization for Robotic Manipulation
- Embodied Hands: Modeling and Capturing Hands and Bodies Together
- Proximal Policy Optimization Algorithms
- Learning High-DOF Reaching-and-Grasping via Dynamic Representation of Gripper-Object Interaction
- Gemini: A Family of Highly Capable Multimodal Models
- PhysHOI: Physics-Based Imitation of Dynamic Human-Object Interaction
Related papers
- Rethinking Actions: Continuity-Aware Priors for World-Model Planning
- MOSAIC: Modular Obstacle and Spatial-Aware Framework for Manipulation In Cluttered Scenes
- SAGAS: Semantic-Aware Graph-Assisted Stitching for Offline Temporal Logic Planning
- DexSeed: Expanding Human Hand-Object Interactions into Robot Demonstrations for Dexterous Policy Learning
- LaST0.5: A Generalist World Action Model with Interaction-Centric Latent Reasoning
- CIALLO: Learning Shared Intent Representations for Cross-Embodiment Manipulation