OmniHOI: Dexterous Hand-Object Interaction from Monocular Human Video

arXiv:2610.10855 · cs.RO · Submitted 2026-10-07 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: Today's paper: "OmniHOI: Dexterous Hand-Object Interaction from Monocular Human Video".

Dev: The gist: OmniHOI presents a pipeline that turns an RGB video of hand-object interaction into an interaction-faithful trajectory on dexterous hands by enforcing physical consistency across reconstruction, retargeting,

Rosa: First, who's behind it and why it matters.

Paper summary: Rosa: So we're looking at this paper called "OmniHOI: Dexterous Hand-Object Interaction from Monocular Human Video." Basically, they’ve built a pipeline to take a regular video of someone interacting with an object and turn it into a motion plan that works for a robot hand.

Dev: The core idea is that they aren't just doing one thing; they're enforcing physical consistency across the whole process, from how they reconstruct the scene to how they eventually refine the actual movement on a robot.

Taro: It sounds like it’s trying to solve those big problems where an image looks plausible but doesn't actually represent a physically possible grasp for a machine.

Rosa: Exactly. They claim this pipeline uses evidence at every stage: image data when they reconstruct things, contact geometry when they retarget the hand motion, and then dynamics during the physics-in-the-loop refinement phase.

Dev: That sequential error correction is what sets it apart from older methods that might just pass errors along to a learned policy later on.

Taro: If you look at how they do it, they have these three main stages: first, reconstructing the keyframes; second, an alternating image-contact refinement step; and finally, a finger-crossing repair stage.

Rosa: That sounds like a lot of moving parts just to get one smooth trajectory.

Dev: Right. In the reconstruction phase, they use SAM three for segmentation and MoGe-three to estimate camera intrinsics and depth as point maps on each frame, plus WiLoR reconstructs the hand mesh up to unknown scale and depth <ref:2610.10855#pg1>.

Taro: Then there's this alternating image-contact refinement where they correct two different error types using different evidence. Image alignment adjusts things like the hand articulation angles and shape parameters, while contact optimization adjusts those angles along with something called Th, which is related to the hand shape and its position relative to the object.

Rosa: And then there's this finger-crossing repair stage, which they use parity along each finger’s keypoint polyline to find the smallest pose change that gets rid of those crossings using a damped linearized update with line search.

Dev: The transfer phase takes those refined keyframes and turns them into robot motion through contact-aware retargeting, which initializes the configuration using a hand pose solver, and then it iteratively refines wrist and joint configurations to keep the contact information while getting rid of issues like morphology-induced penetration.

Taro: And after that, they refine everything in simulation using a CMA-ES optimizer to improve object tracking and grasp stability.

Rosa: They also do physics-in-the-loop refinement by interpolating those retargeted keyframes with shape-preserving cubic interpolation, then replaying the trajectory in MuJoCo to search for corrections by updating per-keyframe offsets.

Dev: That’s a lot of optimization happening in simulation, but the point is that this sequence aims to yield a task-effective trajectory.

Paper summary: Taro: The experimental results they share are pretty strong. Across one hundred fifty motion-capture trajectories transferred to five hands with six to twenty degrees of freedom, OmniHOI achieves success rates between thirty-nine and eighty-nine percent, which is much higher than the thirty-one percent seen in prior transfer methods.

Rosa: And on monocular video clips, they hit a fifty-three percent success rate compared to twenty-eight percent for the best prior pipeline.

Dev: They also outperformed baselines like iHOI and FollowMyHold on reconstruction metrics such as F5, F10, and CD when looking at how well they reconstruct the hand and object.

Taro: The limitation they point out is that their monocular perception accuracy is still an issue, especially with estimating metric scale. Plus, the physics-in-the-loop refinement can be affected by errors in the video-derived object geometry and estimated physical properties.

Rosa: So it stops working well when the underlying video data or physics assumptions are shaky?

Dev: Right. They also noted that real-world execution is currently open-loop, which means they haven't incorporated feedback control yet, so that’s a direction for future work to make it more robust.

Taro: The paper concludes that OmniHOI provides a training-free pipeline that enforces physical consistency in reconstruction, retargeting and physics-in-the-loop refinement by correcting each error where it arises during the process.

Rosa: So what does this mean for someone who only listens to this show? It means we might finally be able to use simple human videos to generate robot movements without needing tons of specific training data.

Dev: The implication is that we can bypass some of the heavy task-specific RL training that most people have to do just to get a robot moving correctly.

Taro: But they also admit that their failure modes show where the bottlenecks are, like upstream image-based geometry estimation being the biggest hurdle for TACO because of lens distortion or when a hand pushes an object aside instead of grasping it.

Rosa: So while the system is very consistent internally, it still struggles when the initial visual input is misleading about how things are positioned.

Dev: That’s a realistic assessment. The ablation studies show that adding those refinement steps one at a time lowers the intersection volume by about a third overall, and physics-in-the-loop refinement boosts success from between twenty-one and twenty-five percent up to eighty-five to eighty-nine percent on the four higher degree of freedom hands.

Taro: The final conclusion is that OmniHOI successfully turns a monocular RGB video into an interaction faithful trajectory on a dexterous hand.

Rosa: So, in short, this pipeline gives us a way to translate natural human movement videos into robot actions with much higher success rates than what was available before.

Conclusion: Rosa: So, to wrap up, OmniHOI is this pipeline that takes a regular video of someone touching something and makes it work for a robot hand.

Dev: Yeah, it’s all about making sure the reconstruction, the transfer to the robot motion, and then even the simulation refinement are all physically consistent with each other.

Taro: The authors are Mao and Shao and their team at Zhejiang University and Shanghai Jiao Tong University. They’ve put together a lot of work on how autonomy can handle real-world interaction from just a single video feed.

Rosa: What does this actually mean for us, you know, for someone who just needs to get a robot to pick something up?

Dev: It means they’re showing that you don't need huge datasets of robot demonstrations to teach the hand how to grasp things if you can capture the interaction from a video.

Taro: They claim it achieves pretty high success rates, like eighty-nine percent on some of those more complex hands with twenty degrees of freedom.

Rosa: Eighty-nine percent is a big jump when you’re talking about moving from just seeing something to actually making it work in the physical world, right?

Dev: It is a significant improvement over what most prior methods could do for transferring that motion from video to hardware.

Taro: The authors also pointed out some real bottlenecks, like how much they struggle with getting the scale of things right when you only have a single camera view.

Rosa: So even though it’s pretty impressive, it still has those limitations tied to the quality of the initial visual input, especially when determining exactly how big an object is.

Dev: Exactly. And there's that point about real-world execution being open-loop, meaning it doesn't have that feedback control loop yet to handle unexpected bumps or slips in practice.

Rosa: So what’s the next thing we should look at after seeing these results?

Zhejiang University · The University of Hong Kong

cs.RO

Submitted: 2026-10-07

Updated: 2026-10-14

Comments: The project page is available at https://hydrofoooil.github.io/OmniHOI/

Code: https://github.com/wuji-technology/wuji-retargeting

Project page: https://hydrofoooil.github.io/OmniHOI

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 92/100

The gist: The gist: OmniHOI presents a pipeline that turns an RGB video of hand-object interaction into an interaction-faithful trajectory on dexterous hands by enforcing physical consistency across

Key concepts

Keyframe Reconstruction
This initial stage uses AI models like SAM 3 and MoGe-3 to segment hands, objects, and the table from video frames. It estimates camera parameters and generates a point map representing depth for each frame. WiLoR then reconstructs the hand mesh at unknown depths and scales.
Alternating Image-Contact Refinement
This stage corrects errors by using different evidence for two types of mistakes. Image alignment adjusts hand articulation angles, while contact-penetration optimization adjusts joint angles simultaneously to ensure fingers correctly interact with the object's surface.
Physics-in-the-Loop Refinement
The final refinement step uses a MuJoCo simulator to test and correct the motion. It interpolates keyframes into a continuous path and replays the trajectory, using CMA-ES optimization to find pose offsets that improve object tracking, contact, and grasp stability.

Terminology

Summary

The gist: OmniHOI presents a pipeline that turns an RGB video of hand-object interaction into an interaction-faithful trajectory on dexterous hands by enforcing physical consistency across reconstruction, retargeting, and physics-in-the-loop refinement.

How it works

OmniHOI is a pipeline designed to resolve errors at each stage of the process before they propagate downstream or must be absorbed by a learned policy (Page 1). The key idea involves enforcing physical consistency using evidence available at each stage: image evidence during reconstruction, contact geometry during retargeting, and dynamics during physics-in-the-loop refinement (Page 1). Each stage optimizes the corresponding representation directly, correcting errors before they propagate downstream or must be absorbed by a learned policy (Page 2).

The pipeline consists of three main stages:

  1. Keyframe reconstruction: This stage uses SAM 3 (Carion et al., 2026) to segment hands, objects and table, and MoGe-3 to estimate camera intrinsics and each frame’s depth as point map (Page 4). It also uses WiLoR to reconstruct the hand mesh up to unknown depth and scale (Page 4).

  2. Alternating image-contact refinement: This stage alternates two error types corrected by different evidence and different variables (Page 4). Image alignment adjusts the articulation θ and hand shape β, while contact-penetration optimization adjusts θ together with Th = (Rh, th) (Page 4).

  3. Finger-crossing repair: This stage detects surface crossings using parity along each finger’s keypoint polyline and solves for the minimum pose change that removes the crossing residuals, using a damped linearized update with line search (Page 5).

Key components of the method

The reconstruction phase employs adaptive keyframe sampling where keyframes are placed at equal increments of At, which crowds where the hand moves fast and is sparse on steady stretches (Page 4). The image-alignment optimization penalizes violations of occlusion order by minimizing a loss term Lord (Page 4). The contact-penetration optimization attracts each finger’s palmar samples Pf to the object surface while repelling its full-surface samples Qf from the interior, using a loss Lgrasp (Page 4).

The transfer phase focuses on turning keyframe poses into executable robot motion through contact-aware retargeting and physics-in-the-loop refinement (Page 3). Contact-aware retargeting initializes the robot configuration q init with the keypoint-vector solver of WujiHand Retargeting, and iteratively refines wrist and joint configurations to preserve contact information while removing morphology-induced penetration and crossings (Page 3).

Refinement in simulation

The resulting keyframes are then refined in simulation with CMA-ES optimizer (Hansen & Ostermeier, 2001) to improve object tracking, contact, and grasp stability (Page 3). The physics-in-the-loop refinement involves interpolating the retargeted keyframes into a continuous trajectory by shape-preserving cubic interpolation (PCHIP), then replaying the trajectory in MuJoCo to search for corrections (Page 3). The optimization variables are per-keyframe offsets ∆qk, which are interpolated and added to the base trajectory (Page 3).

Experimental results

OmniHOI achieves superior performance compared to prior methods across reconstruction, motion transfer, and end-to-end video-to-robot execution (Page 2). Across 150 motion-capture trajectories transferred to five dexterous hands with 6 to 22 DoF, OmniHOI achieves 39–89% success, compared with at most 31% for prior transfer methods (Page 1). On 60 monocular video clips, the method achieves 53% success, compared with 28% for the best prior video-to-robot pipeline (Page 1).

In hand-object reconstruction, OmniHOI outperforms baselines like iHOI and FollowMyHold on object reconstruction metrics such as F5, F10, and CD (Page 7). For human-to-robot motion transfer, OmniHOI achieves the highest success rate on every hand, reaching 85–89% on the four higher-DoF hands where baselines reach at most 31% (Page 7).

Limitations

OmniHOI is currently limited by monocular perception accuracy, especially metric scale estimation (Page 9). Physics-in-the-loop refinement is also affected by errors in the video-derived object geometry and estimated physical properties, and could benefit from more accurate geometry and dynamics estimation from video (Page 9). Real-world execution is open-loop, so incorporating feedback control is a natural direction for improving robustness (Page 9).

The paper concludes that OmniHOI provides a training-free pipeline that enforces physical consistency in reconstruction, retargeting and physics-in-the-loop refinement, correcting each error in the stage where it arises (Page 10). Experiments show lower hand pose error and hand-object intersection in reconstruction, markedly higher success than prior motion-transfer methods on five hands with 6 to 22 DoF, and end-to-end success on videos where prior video-to-robot pipelines fail (Page 10).

The paper also details its failure modes, noting that upstream image-based geometry estimation is the largest bottleneck for TACO due to lens distortion and for OakInk2 because the hand pushes the object aside instead of grasping it (Page 16). OmniHOI fails on 28 clips, 14 in perception, 13 in execution and 1 in motion planning, most often because the object is reconstructed at the wrong size or rotates in the hand (Page 16).

The ablation studies show that adding refinement steps of Section 3.1 one at a time lowers the intersection volume by about a third overall, while PAMPVPE and PA-MPJPE rise by only 0.67 mm and remain below every baseline in Table 1 (Page 9). Furthermore, physics-in-the-loop refinement raises success from 9–29% to 85–89% on the four higher-DoF hands and from 21–25% to 36–39% on the Inspire Hand (Page 10). The contact refinement's open-loop losses are mostly false successes, as after physics-in-the-loop refinement, contact refinement also lowers the fraction of frames with hand-object contact from 41.4% to 35.1% (Page 9). The final conclusion is that OmniHOI successfully turns a monocular RGB video into an interaction-faithful trajectory on a dexterous hand (Page 10).

--- Page 3 ---

OmniHOI: Dexterous Hand-Object Interaction from Monocular Human Video

The paper is published as arXiv:2610.10855v1 [cs.RO] on Oct 7, 2026 (Page 1). The authors are Ting Mao, Yanming Shao, Ziheng Wang, Haoyu Liu, Yiqun Wang, Xuanye Wu and Yao Mu (Page 1). The affiliations include Zhejiang University 2The University of Hong Kong 3Shanghai Jiao Tong University and Shanghai AI Laboratory 4 (Page 1).

Improvements for AI systems

  1. Identify physical consistency across reconstruction, retargeting, and execution to enforce physical laws at each stage. This allows for a pipeline that corrects errors before they propagate downstream or must be absorbed by a learned policy.

  2. Develop a hand-object reconstruction method that specifically resolves geometric inconsistencies by alternating between image alignment adjusts the articulation θ and hand shape β and contact-penetration optimization adjusts θ together with Th = (Rh, th). This ensures lower hand pose error than prior reconstruction methods while maintaining low handobject intersection.

  3. Implement a contact-aware transfer method that uses iterative refinement to ensure contact is preserved during retargeting by requiring anchors to lie in a narrow band around Rf and jointly removing penetration and enclosure. This leads to higher success rates, achieving 39–89% success on transferred trajectories compared to prior methods.

  4. Employ a physics-in-the-loop refinement step using CMA-ES optimizer with a rolling horizon search to improve grasp stability in simulation by minimizing the loss function: L = λposEpos + λrotErot + λcontactEcontact + λoppEopp + λpenEpen + λprior∥∆qk∥2 + λtermEterm.

Sources

Related papers