Hand-centric Human-to-Robot Trajectory Transfer from Video Demonstrations via Object Category-agnostic Temporal Localization
cs.RO
Submitted: 2026-06-09
Updated: 2026-09-19
Comments: under review
License: http://creativecommons.org/licenses/by/4.0/
The gist: Human videos provide a scalable source of demonstrations for robot imitation learning.
Terminology
Abstract
Human videos provide a scalable source of demonstrations for robot imitation learning. However, converting them into executable and semantically aligned robot trajectories requires determining when task-relevant interactions occur, how demonstrated human grasps should be retargeted across embodiments, and which post-contact motions should be reproduced. In this paper, we present HOCALT (Hand--Object Category--Agnostic Localization and Transfer), a hand-centric retargeting framework that unifies 3D hand motion reconstruction, temporal contact localization, and cross-embodiment trajectory generation. Given synchronized stereo videos, HOCALT identifies task-relevant contact intervals by jointly reasoning over reconstructed 3D hand articulation and category-agnostic object motion cues. These intervals serve as temporal anchors for initializing cross-embodiment transfer, where the demonstrated human grasps are retargeted into multi-modal robot grasp hypotheses. Within each interval, these hypotheses are then propagated along the demonstrated hand motion, yielding executable trajectories. Finally, we augment the transferred trajectories to generate diverse variants from a single demonstration. Across various tasks, HOCALT outperforms VLM-based temporal localization baselines, achieving higher replay success than existing retargeting approaches.
Sources
- Robot Learning from Human Videos: A Survey
- EasyMimic: A Low-Cost Framework for Robot Imitation Learning from Human Videos
- You Only Teach Once: Learn One-Shot Bimanual Robotic Manipulation from Video Demonstrations
- WARPED: Wrist-Aligned Rendering for Robot Policy Learning from Egocentric Human Demonstrations
- Learning to Imitate Object Interactions from Internet Videos
- Tool-as-Interface: Learning Robot Policies from Observing Human Tool Use
- Vision-based Manipulation from Single Human Video with Open-World Object Graphs
- X-Diffusion: Training Diffusion Policies on Cross-Embodiment Human Demonstrations
- EgoZero: Robot Learning from Smart Glasses
- RoboPCA: Pose-centered Affordance Learning from Human Demonstrations for Robot Manipulation
- HOGraspFlow: Taxonomy-Aware Hand-Object Retargeting for Multi-Modal SE(3) Grasp Generation
- EgoLoc: A Generalizable Solution for Temporal Interaction Localization in Egocentric Videos
- Embodied Hands: Modeling and Capturing Hands and Bodies Together
- Point Policy: Unifying Observations and Actions with Key Points for Robot Manipulation
- EgoAVFlow: Robot Policy Learning with Active Vision from Human Egocentric Videos via 3D Flow
- Affordance Transfer Across Object Instances via Semantically Anchored Functional Map
- SPIDER: Scalable Physics-Informed Dexterous Retargeting
- DexMachina: Functional Retargeting for Bimanual Dexterous Manipulation
- Depth Anything 3: Recovering the Visual Space from Any Views
- SAM 3: Segment Anything with Concepts
Related papers
- FMT x: An Efficient and Asymptotically Optimal Extension of the Fast Marching Tree for Dynamic Replanning
- MPCFormer: A physics-informed data-driven approach for explainable socially-aware autonomous driving
- RoboLab: A High-Fidelity Simulation Benchmark for Analysis of Task Generalist Policies
- ARC: A Reasoning Recipe for Robot Foundation Models
- HRDexDB: A 4D Dexterous Grasping Dataset Across Human and Multiple Robot Embodiments
- APT: Action Expert Pretraining Improves Instruction Generalization of Vision-Language-Action Policies