AnyDexRT: Calibration-Efficient Dexterous Hand Retargeting with Few-Shot Human Guidance
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.
Rosa: We're looking at the paper: "AnyDexRT: Calibration-Efficient Dexterous Hand Retargeting with Few-Shot Human Guidance".
Dev: Here's the gist of the paper. AnyDexRT is a calibration-free retargeting method for intuitive dexterous teleoperation across different human-like dexterous hands by combining self-supervised shape matching with few-shot human guidance and contact classifier refinement Problem Statement Teleoperation's effectiveness largely depends on kinematic retargeting, which maps operator hand motions to feasible and intuitive robot hand motions.
Rosa: Existing methods often require hand-crafted objectives, precise calibration, or global shape matching between human and robot hand spaces, making them sensitive to hand-specific tuning and less reliable across different dexterous hands.
Dev: Traditional retargeting methods often rely on inverse kinematics with hand-crafted task vectors or keypoints, requiring careful calibration of coordinate frames, vector origins, scale factors, and objective weights.
Rosa: Neural variants remain tied to calibrated targets and hand-specific assumptions.
Paper summary: Rosa: So looking at the whole AnyDexRT paper, what it really means is that we can get much more intuitive dexterous teleoperation across different types of robot hands without having to spend hours calibrating each one individually. The authors are showing that this mapping works across seven different hands in simulation, where they improved the average local motion consistency from fifty-nine point eight percent up to ninety point two percent.
Dev: That jump in motion consistency is significant because it means the operator’s intent translates into robot movement much more reliably, which is crucial for real-time control loops where latency and failure modes are a major concern. They also showed that in actual teleoperation tests, AnyDexRT achieved the shortest completion time on all tasks and had the highest pinch success rate compared to other methods.
Taro: It suggests that if we want to build systems that can handle diverse human-like hands—not just one specific setup—we need methods that are fundamentally less dependent on hand-specific tuning, which is a huge win for building generalized dexterous manipulation skills.
Rosa: The title itself, AnyDexRT: Calibration-Efficient Dexterous Hand Retargeting with Few-Shot Human Guidance, points to the core of the work: it’s about being efficient without needing calibration and guided by minimal human input. It moves the focus away from hand-crafted objectives or global shape matching.
Dev: The implication for anyone building a teleoperation interface is that they might be able to achieve high-quality demonstrations for imitation learning much faster because the retargeting itself is more robust and less sensitive to initial setup errors.
Taro: What the authors flag as a limitation is that while it’s calibration-free, the method still relies on those few-shot anchors to resolve ambiguity, so if you don't have any good starting points, that guidance becomes crucial.
Rosa: So in short, AnyDexRT gives us a way to make robot hands feel more intuitive and controllable across different setups by using self-supervised learning anchored by just a little bit of human help.
Conclusion: Rosa: So, AnyDexRT is basically showing how you can get really good at making robot hands follow human movements without needing to spend tons of time calibrating every single hand individually.
Dev: That's right, it’s about calibration efficiency for dexterous teleoperation across different hands. It tackles the problem of mapping human motion to robot motion intuitively.
Taro: The authors are using a combination of self-supervised shape matching and some few-shot human guidance to anchor this mapping without needing huge datasets.
Rosa: They learned fingertip correspondence by matching shapes, but they used a specific loss called LP-Chamfer loss, which means it maps the human fingertip space into the robot space without trying to cover every single part of the robot hand.
Dev: And to fight that distortion from only mapping one way, they added a pairwise distance preservation objective to keep things geometrically sensible.
Taro: Plus, they have this local motion loss which tries to make sure the retargeting behaves well even if there are small errors in the calibration itself by preserving movement directions locally.
Rosa: Then they use few-shot human guidance, which means they only need a tiny bit of paired data from humans and robots to figure out where things should line up initially.
Dev: That alignment loss helps resolve the mapping ambiguity while still letting the self-supervised learning handle the rest, keeping it scalable.
Taro: They also refined pinch poses using a contact classifier, which means they trained a system to recognize when a pinch is happening to get better results there.
Rosa: The experiments show that across seven different dexterous hands in simulation, their local motion consistency jumped from nearly sixty percent up to over ninety percent.
Dev: In the real world, they found it actually got the task done faster than other methods and had a much higher success rate for pinching tasks.
Taro: But the catch is that you still need those few anchors to start, so if you don't have good starting points, that guidance becomes really important.
Rosa: The big picture is that this method could make teleoperation way more intuitive and easier to deploy across different robot setups without all the manual tuning headache.
Noematrix University of Shanghai Jiao Tong University, Shanghai Innovation Institute
cs.RO
Submitted: 2026-07-09
Updated: 2026-11-05
Comments: Correct parameter; simplify anchor collection. See Appendix E for details
Code: https://github.com/wujitechnology/wuji-retargeting
Project page: https://chenxi-wang.github.io/projects/anydexrt
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Importance score: 95/100
The gist: AnyDexRT is a calibration-free retargeting method for intuitive dexterous teleoperation across different human-like dexterous hands by combining self-supervised shape matching with few-shot human…
Key concepts
- Self-supervised Shape Matching
- This technique learns how the shape of a human fingertip relates to the corresponding robot fingertip without needing pre-existing data or calibration. It uses a partial Chamfer loss to map human fingertip space into feasible robot space, ensuring the mapping anchors itself in task-relevant areas rather than requiring perfect global alignment.
- Few-Shot Human Guidance
- Instead of needing extensive paired data for every hand, this method uses a small set of human-robot anchors to guide the correspondence learning process. This resolves mapping ambiguity by aligning the learned self-supervised features with known correct pairings, allowing the system to generalize effectively across different hands without extensive manual tuning.
- Contact Classifier Refinement
- A contact classifier (fc) is trained to recognize specific contact patterns, such as pinching. This classifier helps refine the retargeting behavior specifically for pinch-related poses. By recognizing these critical interaction types, the system improves control and success rates during tasks requiring precise gripping actions.
Terminology
Summary
AnyDexRT is a calibration-free retargeting method for intuitive dexterous teleoperation across different human-like dexterous hands by combining self-supervised shape matching with few-shot human guidance and contact classifier refinement
Problem Statement
Teleoperation's effectiveness largely depends on kinematic retargeting, which maps operator hand motions to feasible and intuitive robot hand motions. Existing methods often require hand-crafted objectives, precise calibration, or global shape matching between human and robot hand spaces, making them sensitive to hand-specific tuning and less reliable across different dexterous hands. p. 0 Traditional retargeting methods often rely on inverse kinematics with hand-crafted task vectors or keypoints, requiring careful calibration of coordinate frames, vector origins, scale factors, and objective weights. p. 1 Neural variants remain tied to calibrated targets and hand-specific assumptions. p. 1
Fingertip Correspondence Learning
AnyDexRT learns fingertip correspondence via self-supervised shape matching to anchor the mapping in task-relevant regions. The method employs a partial Chamfer loss, LP-Chamfer, which maps the human fingertip space into the feasible robot fingertip space without requiring full coverage of redundant robot regions. p. 3 To mitigate geometric distortion caused by this one-way mapping, a pairwise distance preservation objective, Ldist, is introduced to regularize the distribution of fm(C H) and alleviate geometric distortion. p. 3 Furthermore, a local motion loss, Lmotion, encourages retargeting behavior that is less sensitive to calibration errors by preserving motion directions in local coordinate frames. p. 4
Few-Shot Guidance and Refinement
The system uses few-shot human guidance to anchor the correspondence without requiring large-scale paired data, resolving mapping ambiguity. This alignment is achieved through an alignment loss, Lalign, which utilizes a small set of paired human-robot anchors to resolve ambiguity while preserving the scalability of self-supervised learning. p. 4 Additionally, pinch-related poses are refined using a contact classifier, fc, trained via binary cross entropy loss to recognize contact patterns such as pinching. p. 5
Experimental Validation
Experiments across diverse dexterous hands demonstrate that AnyDexRT improves retargeting quality and reduces manual tuning effort. In simulation, AnyDexRT improved the average local motion consistency from 59. p. 58% to 90.2% across seven dexterous hands. p. 5 Real-world teleoperation evaluations showed that AnyDexRT achieves the shortest completion time on all tasks and achieves the highest pinch success rate, significantly outperforming baselines. p. 7 The method maintains stable retargeting quality across different training initializations, demonstrating its calibration-free capability. p. 6
Conclusion and Impact
AnyDexRT provides a calibration-free retargeting method that enables intuitive dexterous teleoperation across different human-like dexterous hands. The system successfully reduces hand-specific calibration and tuning while maintaining an intuitive operator-to-robot mapping across different hands. p. 1 Real-world evaluations confirm its effectiveness in improving task efficiency and pinch control, supporting its applicability to dexterous teleoperation and data collection. p. 7 The authors note that future work could automate anchor selection or adapt the mapping online from operator feedback. p. 7
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Calibration-free retargeting across diverse hands: The system can achieve
high-quality retargeting across diverse human-like hands
without requiringprecise calibration or manually tuned hyperparameters such as scale factors, offsets, and task weights.
-
Improved mapping stability through geometric regularization: By employing the objective Ldist, the mapping is made less sensitive to calibration errors and more consistent with operator intent by preserving
the geometric structure of the retargeted space.
-
Enhanced robustness against redundant robot spaces: The partial Chamfer loss prevents
forcing fim(CH,i) to cover the entire CRi
which avoids distorting mappings caused byredundant robot-hand regions.
-
Reduced manual tuning effort: The system requires only three hyperparameters and runs at about 300Hz, allowing default settings to be reused across different hands, thus reducing
manual tuning effort.
-
Better pinch control in real-world scenarios: The refinement of pinch poses using a contact classifier allows the system to achieve the
highest pinch success rate,
even when measured data fail to capture fine contact patterns due to sensor limitations.
Sources
- AnyDexGrasp: General Dexterous Grasping for Different Hands with Human-level Learning Efficiency
- DEXOP: A Device for Robotic Transfer of Dexterous Human Manipulation
- HumDex: Humanoid Dexterous Manipulation Made Easy
- Cross-Hand Latent Representation for Vision-Language-Action Models
- Dexterous Teleoperation of 20-DoF ByteDexter Hand via Human Motion Retargeting
- UniDex: A Robot Foundation Suite for Universal Dexterous Hand Control from Egocentric Human Videos
Related papers
- Rethinking Actions: Continuity-Aware Priors for World-Model Planning
- MOSAIC: Modular Obstacle and Spatial-Aware Framework for Manipulation In Cluttered Scenes
- SAGAS: Semantic-Aware Graph-Assisted Stitching for Offline Temporal Logic Planning
- DexSeed: Expanding Human Hand-Object Interactions into Robot Demonstrations for Dexterous Policy Learning
- LaST0.5: A Generalist World Action Model with Interaction-Centric Latent Reasoning
- CIALLO: Learning Shared Intent Representations for Cross-Embodiment Manipulation