Learning Generalizable Hand-Object Tracking from Synthetic Demonstrations

arXiv:2512.19583 · cs.RO, cs.GR · Submitted 2025-12-22 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: We're looking at the paper: "Learning Generalizable Hand-Object Tracking from Synthetic Demonstrations".

Paper summary: Rosa: So, we're looking at this paper, "Learning Generalizable Hand-Object Tracking from Synthetic Demonstrations." The core idea is that you can learn how to track hands and objects really well just by using synthetic data instead of needing a human to do it.

Dev: Right, so the thesis here is building a system that learns hand-object tracking controllers purely from synthetic data, without needing any human demonstrations at all. They claim this approach makes two main contributions: one is HOP, which they call a Hand-Object Planner, and another is HOT, which is the Hand-Object Tracker.

Taro: I'm interested in how that planner generates the data because acquiring diverse manipulation data from humans is really hard right now. If you can generate it synthetically, that removes a huge barrier for learning these skills.

Rosa: Exactly. HOP synthesizes diverse hand-object trajectories through this two-stage process: first, it uses force closure optimization and reinforcement learning to create a set of stable grasp configurations. Then, HOP anchors itself to those poses to synthesize trajectories for fundamental meta-skills like Grasping, Moving, Placing, and Rotating.

Dev: That sounds like they are decomposing the complex task into these eight decomposed meta-skills: Free Move, Grasp, Place, Move, Catch, Throw, Rotate—simplified—and Regrasp. That level of decomposition is key for a robust controller to handle different movements.

Taro: But if it’s just synthesized data from this planner and tracker combination that's the whole method? How does it actually generalize to things that weren't in the training data?

Rosa: The paper suggests that HOT bridges the gap between synthetic and physical worlds through reinforcement learning and interaction imitation learning. They extend this to handle diverse object shapes and hand morphologies, showing success in tracking challenging, long-horizon sequences like object re-arrangement or agile in-hand reorientation.

Dev: That's a strong claim about generalization because it implies the system isn't just memorizing specific movements but actually learning the underlying skill structure. They show this by evaluating different hand models, where Shadow and Allegro hands both achieved over eighty percent success rates for skills like Grasping, Moving, and Placing when trained with HOP data.

Taro: So it seems they’re tackling the difficulty of motion capture systems struggling with occlusion and depth ambiguity by focusing on learning from generated trajectories instead. That makes sense from a real-world failure mode standpoint.

Paper summary: Rosa: They also talk about how they refine this using a two-stage teacher student framework, where the second stage learns to assimilate all these skills at once using the first stage as guidance through behavior cloning and online data refinement.

Dev: From an engineering standpoint, that teacher student setup is interesting because it helps learn efficiently while ensuring you don't just get stuck optimizing one small skill at the expense of others. They also use domain randomization across three axes: object scale, initial state, and a curriculum on observation or action noise.

Taro: Domain randomization sounds like a good way to test robustness against real-world variations before deployment. What about the tracking mechanism itself? How does HOT actually output its actions for the hand and the wrist pose?

Dev: HOT is formulated as a policy parameterized by a multi-layer perceptron network that outputs actions for both the hand's native joints and the wrist's six degrees of freedom pose. Crucially, this policy learns residual adjustments for the wrist to make sure there’s consistent skill execution across different workspaces, leveraging translational invariance.

Rosa: The observation fed into this policy includes both what’s happening in the environment and a goal state for the next time step. This structure is designed to handle those long-horizon sequences they mentioned earlier.

Taro: I wonder how it handles situations where things go wrong, like when the object is suddenly thrown or if there’s unexpected contact during a complex reorientation sequence? How does it recover gracefully?

Dev: They use a unified imitation reward that combines different components multiplicatively to learn manipulation from demonstrations. They emphasize the contact reward for reliable grasping and releasing, and an interaction reward specifically highlighting hand-object relative motion.

Rosa: And they validate their control by showing that HOT can correct physically implausible artifacts; they measure this using object-hand penetration as a key metric when testing against real-world human-object interaction trajectories.

Taro: So the system is able to clean up unrealistic movements, which is important because you don't want a robot trying to do something physically impossible when it starts operating in the real world.

Dev: They also used an LLM to generate semantically annotated keyframes and then apply keyframe-guided interpolation in both position and orientation to ensure the generated trajectories maintain semantic coherence and dynamic feasibility. This is how they handle the high-level instruction part.

Paper summary: Rosa: That interpolation uses four special frame types to partition the LLM keypoints into three semantic segments: start to grasp, grasp to release, and release to end. Orientation is interpolated independently using spherical linear interpolation in each segment.

Taro: So when you’re following a language instruction for something like "catch that hammer," the system uses that LLM to guide the hand through these carefully constructed segments of movement?

Dev: Yes, the LLM prompt requires it to select one candidate grasp and align that generated grasp keypoint to a specific wrist pose. This pipeline ensures smooth motions inside each phase while strictly preserving those grasp and release boundaries.

Rosa: The paper also discusses how they use difficulty-prioritized sampling by calculating the mean reward of a trajectory to assess its learning difficulty, then assigning a sampling probability based on that metric for balanced training across challenging examples.

Taro: That sounds like a smart way to make sure the system doesn't just learn the easy parts and ignore the hard, long-horizon sequences. What about scaling this up?

Dev: They note that scaling to more complex skills and broader object categories might require large-scale, learning-based refinement of HOP data and hierarchical distillation strategies. Plus, they mention that real-world deployment faces the sim to real gap, which demands hardware with high control frequency and force sensing capabilities.

Rosa: So while this synthetic data approach is powerful for learning the core tracking skill, it points toward future work needing more refined planning and better bridging that gap when moving to actual physical robots.

Dev: Ultimately, the paper's contribution is showing a scalable framework where you can learn entirely from synthetic data while remaining compatible with human motion tracking and text-guided task execution. It breaks the data bottleneck that has constrained progress in dexterous manipulation for a long time.

Taro: So what this means for us outside of the lab is that we might see controllers that can track complex, dynamic sequences without needing thousands of hours of expensive human demonstration data.

Rosa: That’s right. It offers a flexible, scalable foundation for future work on general-purpose dexterous manipulation because it learns from synthetic demonstrations while still being compatible with real-world interaction data.

Conclusion: Rosa: So, this paper is about learning how to track hands and objects just by looking at fake data instead of needing people to show them how.

Dev: Yeah, the title is "Learning Generalizable Hand-Object Tracking from Synthetic Demonstrations," and the authors are presenting a system that does exactly that.

Rosa: Essentially, they built a way to teach a hand controller to follow an object using only computer-generated examples.

Dev: The main idea is two parts: one planner called HOP for making the fake data, and one tracker called HOT that learns how to use it.

Rosa: It’s about creating a foundation controller that can handle really varied objects and hands, even things it hasn't seen before.

Dev: They show this works for complex tasks like reorienting an object in mid-air or even moving things around on a table.

Rosa: It seems like they’re tackling the big problem of getting robots to learn from synthetic data, which is a massive hurdle in real manipulation.

Dev: The implication here is that we can get more general controllers that don't just memorize one specific movement but understand the underlying skills for grabbing, moving, and placing things.

Rosa: It’s a step toward making robotic manipulation much more flexible and less dependent on expensive human demonstrations.

Dev: But they also flag the sim-to-real gap as a big thing to solve later on when you actually put this in the physical world.

Rosa: So, they’ve made some strong claims about generalization and learning from synthetic data, but we still have to worry about how well it holds up outside of a perfect simulation.

Yinhuai Wang, Runyi Yu, Hok Wai Tsui, Xiaoyi Lin, Hui Zhang, Qihan Zhao, Ke Fan, Miao Li, Jie Song, Jingbo Wang

Hong Kong University of Science and Technology (HKUST) · Wuhan University of China · ETH Zurich

cs.RO, cs.GR

Submitted: 2025-12-22

Updated: 2026-11-17

License: http://creativecommons.org/licenses/by-nc-nd/4.0/

Importance score: 94/100

The gist: Learning Generalizable Hand-Object Tracking from Synthetic Demonstrations presents a system for learning generalizable hand-object tracking controllers purely from synthetic data…

Key concepts

HOP (Hand-Object Planner)
This is a framework designed to create varied hand-object movements. It first generates stable grasp configurations using optimization and reinforcement learning. Then, it uses these grasps as anchors to synthesize trajectories for fundamental skills like grasping, moving, and rotating objects.
HOT (Hand-Object Tracker)
HOT is the tracking controller that learns how to follow a trajectory from synthetic data. It is trained using a teacher-student framework and reinforcement learning. It outputs actions for both the hand's joints and the wrist pose, learning residual adjustments to ensure consistent skill execution.
Domain Randomization
This technique involves training the model across variations in object scale, starting positions, and noise levels in observations and actions. By varying these parameters extensively during training, the system learns to be robust and generalize well to unseen objects and environments.
Imitation Reward
The reward function is designed using a unified imitation approach where different rewards are combined multiplicatively. This includes a contact reward for reliable grasping/releasing and an interaction reward focusing on the relative motion between the hand and the object.

Terminology

Summary

Learning Generalizable Hand-Object Tracking from Synthetic Demonstrations presents a system for learning generalizable hand-object tracking controllers purely from synthetic data, without requiring any human demonstrations Our approach makes two key contributions: (1) HOP, a Hand-Object Planner, which can synthesize diverse hand-object trajectories; and (2) HOT, a Hand-Object Tracker that bridges synthetic-to-physical transfer through reinforcement learning and interaction imitation learning Our method extends to diverse object shapes and hand morphologies Through extensive evaluations, we show that our approach enables dexterous hands to track challenging, long-horizon sequences including object re-arrangement and agile in-hand reorientation These results represent a significant step toward scalable foundation controllers for manipulation that can learn entirely from synthetic data, breaking the data bottleneck that has long constrained progress in dexterous manipulation. p. 0

Where the paper says it

Through extensive evaluations, we show that our approach enables dexterous hands to track challenging, long-horizon sequences including object re-arrangement and agile in-hand reorientation. These results represent a significant step toward scalable foundation controllers for manipulation that can learn entirely from synthetic data, breaking the data bottleneck that has long constrained progress in dexterous manipulation.

System Architecture and Workflow

The system consists of two parts: HOP (Hand-Object Planner), a scalable data generation framework, and HOT (Hand-Object Tracker), a tracking-based controller trained on synthetic data In applications, our system achieves flexible tasks through a modular pipeline where a high-level directive is converted into sparse waypoints that are then fed to HOP to generate a dense hand-object trajectory, which HOT executes in the simulation. This standardized workflow allows for zero-shot adaptation to numerous downstream applications. p. 2

Where the paper says it

Finally, our tracking controller, HOT, executes this trajectory in the simulation. This standardized workflow allows for zero-shot adaptation to numerous downstream applications.

Data Synthesis and Trajectory Generation

HOP synthesizes meta-skill trajectories through a structured, two-stage process First, it employs force-closure optimization and reinforcement learning to generate and refine a diverse set of stable grasp configurations. Subsequently, HOP anchors itself to these grasp poses to synthesize trajectories for fundamental meta-skills—such as Grasping, Moving, Placing, and Rotating—as well as their combinations. p. 1

Where the paper says it

2024a] to generate and refine a diverse set of stable grasp configurations. Subsequently, HOP anchors itself to these grasp poses to synthesize trajectories for fundamental meta-skills—such as Grasping, Moving, Placing, and Rotating—as well as their combinations.

The eight decomposed meta-skills include Free Move, Grasp, Place, Move, Catch, Throw, Rotate (simplified), and Regrasp. p. 2
Where the paper says it

While these trajectories may be physically approximate, they serve as tracking targets in reinforcement imitation learning, enabling the policy to learn generalizable target-action mappings. For efficient synthesizing and learning, we decompose manipulation into eight meta-skills: Free Move, Grasp, Place, Move, Catch, Throw, Rotate, and Regrasp.

Tracking Control Mechanism

HOT is formulated as a policy parameterized by an MLP network that outputs actions for both the hand’s native joints and the wrist’s 6-DoF pose The policy learns residual adjustments for the wrist to ensure consistent skill execution across the workspace, leveraging translational invariance. The observation of the HOT policy includes both the environmental observation and a goal state for the subsequent time step. p. 4

Where the paper says it

Policy Input. The observation of the HOT policy is 𝒉𝑡 = {𝒔𝑡 ,𝒔ˆ 𝑔𝑜𝑎𝑙 𝑡+1 } (4) where 𝒔𝑡 = {𝒔 ℎ𝑎𝑛𝑑 𝑡 ,𝒔 𝑜𝑏𝑗 𝑡 } is the environmental observation, constitutes of current hand state 𝒔 ℎ𝑎𝑛𝑑 𝑡 and object state 𝒔 𝑜𝑏𝑗 𝑡 in the simulation environment.

Training Strategy and Robustness

The training employs a two-stage teacher-student framework to enhance model capacity and learning efficiency The second stage introduces a unified student policy that learns to assimilate all skills simultaneously, using the teacher policies for both online data refinement and action guidance through behavior cloning. Domain randomization is applied along three main axes: object scale, initial state, and a curriculum on observation/action noise. p. 3

Where the paper says it

We reject and resample 𝒈 𝑒𝑛𝑑 if any hand keypoints penetrate the ground plane; otherwise, keep it as the grasping frame. To generate the initial state 𝒈 𝑖𝑛𝑖𝑡 , we compute the vector v from object center to the first joint of the index finger, randomly sample the initial hand position within a cone centered along v, set all hand joint angles to zero, while keeping other parameters identical , Vol.

Performance and Generalization

The system demonstrates remarkable generalization across the entire operational space when trained exclusively on synthetic data from HOP HOT robustly tracks long-horizon, dynamic sequences such as catching a thrown hammer or reorienting it in-air. Furthermore, the system enables zero-shot transfer to downstream applications when integrated with LLMs or VLMs to follow language instructions for goal-directed manipulation. p. 1

Where the paper says it

Notably, the system enables zero-shot transfer to downstream applications. When integrated with LLMs or VLMs [Achiam et al.

The performance improves as the number of training objects increases, confirming the benefit of multi-object training for generalization. p. 6
Where the paper says it

6, we evaluate object generalization capability by comparing HOT trained on 1, 5, and 15 objects, reporting the average tracking success rates for Grasp, Move, and Place across 20 test objects. The results demonstrate that our method converges stably under multiobject joint training and generalizes effectively to unseen objects, with performance improving as training object diversity increases.

Refinement and Evaluation

The unified imitation reward combines complementary components multiplicatively to learn manipulation from demonstrations The contact reward is crucial for reliable grasping and releasing operations, and the interaction reward highlights hand-object relative motion. The system's ability to correct physically implausible artifacts is validated when HOT tracks real-world HOI trajectories using object-hand penetration as a key metric. The final system shows effective zero-shot tracking on real hand-object interactions despite being trained purely on synthetic data. p. 7

Where the paper says it

Here, SR-grasp measures the success rate of achieving a stable grasp, while SR evaluates the success rate of the complete tracking. Notably, HOT achieves effective zero-shot tracking on real hand-object interactions despite being trained purely on synthetic data, demonstrating its potential to overcome the data bottleneck in general dexterous manipulation.

Limitations and Future Work

While HOT provides a general-purpose low-level control capability for trajectory tracking, high-level planners operate in an open-loop manner and do not yet support long-horizon, closed-loop task execution like an LLM-based agent. Scaling to more complex skills and broader object categories may require large-scale, learning-based refinement of HOP data and hierarchical distillation strategies. p. 5

Where the paper says it

While effective for individual skills, using a single policy to learn diverse skills or object categories faces scalability limitations. The primary challenge arises from physical implausibilities in synthetic data, which require extensive sampling to identify feasible solutions for each instance.

Real-world deployment faces the sim-to-real gap, demanding hardware with high control frequency and force-sensing capabilities. p. 8
Where the paper says it

Xue Bin Peng, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. Sim-to-real transfer of robotic control with dynamics randomization.

Ablation Study Findings

The localization and residualization of HOT input-output representations prove crucial for generalization, allowing the model to achieve superior performance even with limited training data Data augmentation techniques consistently boost generalization performance across all skills, while increased training data leads to proportional performance improvements. The complete reward formulation yields the best overall results when analyzing the impact of key design choices. p. 7

Where the paper says it

As shown in Tab. 4, each reward term contributes noticeably to performance, and the complete reward formulation yields the best overall results.

The distilled student policy significantly mitigates skill interference commonly observed in multi-task learning, yielding a 28. p. 8
Where the paper says it

Ruihan Yang, Huazhe Xu, Yi Wu, and Xiaolong Wang. Multi-task reinforcement learning with soft modularization.

5% improvement on the challenging Rotate skill. p. 4
Where the paper says it

Notably, we also generate Move trajectories starting from the final frame of Grasp demonstrations or ending at the initial frame of Place demonstrations to ensure smooth cross-skill transitions. Rotate demonstrates in-hand manipulation where the object is rotated while maintaining a fixed wrist pose.

LLM Integration

We employ an LLM to generate semantically annotated keyframes and subsequently apply keyframe-guided interpolation in both position and orientation to ensure the generated trajectories maintain both semantic coherence and dynamic feasibility. The LLM prompt requires the model to select one candidate grasp and align the generated grasp keypoint to that wrist pose. p. 7

Where the paper says it

The typical pipeline uses a Vision-Language Model (VLM) for high-level semantic planning. The VLM takes as input: the object’s initial 6D pose, a set of feasible wrist poses corresponding to stable grasp configurations, the initial wrist pose, the target object position, an image of current environment, and a natural-language task description.

This pipeline ensures smooth motions inside each phase while strictly preserving the grasp and release boundaries. p. 7
Where the paper says it

In both cases, we apply trajectory smoothing and Bayesian interpolation between keyframes to ensure smooth motions. The resulting wrist path is combined with the chosen grasp pose using the procedure in Sec.

Trajectory Interpolation Details

The four special frame types partition the LLM keypoints into three semantic segments: start → grasp, grasp → release, and release → end Within each segment, all keypoints are used as interpolation anchors to construct a cubic Bezier curve for the wrist position. Orientation is interpolated independently using spherical linear interpolation (SLERP) within each segment. p. 4

Where the paper says it

To synthesize these demonstrations, we first randomly select a grasp configuration 𝒈 from G. We then generate random initial and final object poses within the operational space and create an object trajectory through interpolation.

Evaluation on Different Hands

Experiments on different hand morphologies show that both Shadow and Allegro hands achieve over 80% average success rates when trained with HOP data for the Grasp, Move, and Place skills. MANO demonstrates the best performance among the tested hand models due to its highest degrees of freedom (DoF). p. 2

Where the paper says it

(1) Taking the MANO hand model [Romero et al. 2017] as an example, T ℎ ∈ R 3 and R ℎ ∈ 𝑆𝑂(3) represent the global wrist pose, while 𝜽 ∈ R 15×3 describes the joint angles of the five fingers (each finger has three joints with three degrees of freedom).

Penetration Correction

The original reference data contains noticeable penetrations, while the trajectories produced by HOT exhibit significantly reduced penetration, validating its capacity to correct physically implausible artifacts. The refined GRAB dataset serves as a high-quality benchmark for evaluation after retargeting and improved physical plausibility. p. 7

Where the paper says it

We further validate generalized tracking performance on realworld data using segmented trajectories from the GRAB dataset [Taheri et al. 2020],where trajectories are refined by HOP for retargeting and improved physical plausibility (See the appendix for details).

Adaptive Sampling

We calculate the mean reward of a trajectory to assess its learning difficulty, assigning a sampling probability based on this metric to ensure balanced learning across challenging examples. This difficulty-prioritized sampling helps in acquiring stable tracking behavior and adapting to full diversity during training. p. 1

Where the paper says it

2025] significantly enhances the agility and robustness of task execution. However, training a generalizable hand-object tracking controller faces the fundamental challenge of acquiring sufficient HOI data.

Conclusion

Our approach consists of HOP, a scalable framework for synthesizing hand-object trajectories across diverse meta-skills, and HOT, a tracking policy trained via reinforcement imitation learning on such data The resulting controller demonstrates effective generalization to unseen objects, long-horizon tasks, and even real-world human-object motion data. By functioning effectively without human demonstrations while remaining compatible with them, our method offers a flexible, scalable, and powerful foundation for future work on general-purpose dexterous manipulation. p. 7

Where the paper says it

The resulting controller demonstrates effective generalization to unseen objects, long-horizon tasks, and even real-world human-object motion data. By functioning effectively without human demonstrations while remaining compatible with them, our method offers a flexible, scalable, and powerful foundation for future work on general-purpose dexterous manipulation.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Data Synthesis via HOP enables learning without human demonstrations by synthesizing meta-skill trajectories through a structured, two-stage process. This allows training on elemental skills like Grasp, Rotate, and Move, bypassing the need for complex task-level synthesis.

  2. HOT (Hand-Object Tracker) can track complex, highly dynamic, and long-sequence manipulation trajectories during inference by being trained on synthetic data using a unified set of HOI imitation rewards.

  3. The system supports zero-shot transfer to downstream applications, enabling it to follow language instructions to control both HOP-based data synthesis and HOT’s tracking for goal-directed manipulation.

  4. The policy is formulated as a Gaussian distribution over actions, allowing the network to output residual adjustments for the wrist and target finger joint rotations, which ensures consistent skill execution across the workspace.

  5. A two-stage teacher-student training framework enhances learning efficiency by first training specialized teacher policies for different meta-skill categories before distilling expertise into a unified student policy.

  6. Domain randomization techniques, including random object scale perturbation and initial state noise injection, enhance policy robustness across geometric variations and contact configurations.

  7. The system can refine imperfect synthetic references through tracking; the trajectories produced by HOT exhibit significantly reduced penetration, correcting physically implausible artifacts.

  8. LLM-based trajectory planning allows for end-to-end execution of language-guided manipulation by converting high-level commands into sparse waypoints that HOP converts into dense trajectories for HOT to track.

Abstract

We present a system for learning generalizable hand-object tracking controllers purely from synthetic data, without requiring any human demonstrations. Our approach makes two key contributions: (1) HOG, a highly efficient Hand-Object data generator that rapidly synthesizes diverse, large-scale, and uniformly distributed trajectories; and (2) HOT, a Hand-Object Tracker that bridges synthetic-to-physical transfer through reinforcement learning and interaction imitation learning. Our method extends to diverse object shapes and hand morphologies. HOT also enables cross-simulator transfer. Through extensive evaluations, we show that our approach enables dexterous hands to track challenging, long-horizon sequences including object re-arrangement and agile in-hand reorientation. These results represent a significant step toward scalable foundation controllers for manipulation that can learn entirely from simple, scalable synthetic data, breaking the data bottleneck that has long constrained progress in dexterous manipulation.

Sources

Related papers