DexSeed: Expanding Human Hand-Object Interactions into Robot Demonstrations for Dexterous Policy Learning
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.
Rosa: We're looking at the paper: "DexSeed: Expanding Human Hand-Object Interactions into Robot Demonstrations for Dexterous Policy Learning".
Title and authors: Rosa: So we’re looking at DexSeed today. It’s about taking those human hand-object interactions and turning them into data for training robot hands and vision language models. The authors are Huaijin Pi, Zehong Shen, Yong Wang, Kechun Xu, Changpeng Wang Dingkun Wei, Xueyang Wang, Wenjia Wang, Xing Zhu, Yujun Shen.
Dev: Yeah it’s about taking those human hand-object interactions and turning them into data for training robot hands and vision language models. It sounds like they’re tackling that big problem of needing tons of expensive teleoperated examples for every new object or manipulation style.
Taro: I'm curious if this actually works when you take something from a human and try to make it work on a robot hand with totally different geometry. That transfer step seems tricky.
Rosa: The paper introduces DexSeed as a framework specifically for low-cost generation of dexterous robot data from human hand-object interactions. Their main idea is expanding those human sequences while keeping the original grasps intact and then realizing those motions under arm-tracking dynamics in physics simulation.
Dev: So it’s not just copying the motion; they are adding things to the sequence, like different object starting positions or movement paths, and making sure the robot arm follows those same general trajectories when simulating physics. That’s a big step toward making real robots move intelligently.
Taro: But if you expand the sequences—you vary where the hand starts or where it moves—how do you make sure that expansion doesn't break the core grasp relationship they are trying to preserve during that retargeting process?
Rosa: They organize human interactions by grasp similarity first, and then select representative modes, each one paired with a shared description of how to grasp. Then they use InteractionMesh based retargeting which adapts the sequence to the robot hand and compatible object geometries while preserving local hand-object relationships.
Dev: InteractionMesh sounds like they’re using some kind of mesh structure to map the human fingers and object anchors onto whatever robot hand or new object shape you're working with. How does that help preserve those local relationships?
Rosa: They tetrahedralize the hand landmarks and object anchors using three dimensional Delaunay triangulation to define a fixed neighborhood, we call it N, for each vertex throughout the sequence. This is what keeps those local relationships stable during the transfer process.
Taro: That sounds like a way to create a consistent spatial context for every point on the hand and object across the whole sequence, which is smart for ensuring consistency when you're expanding trajectories later. But what about when you’re doing that trajectory augmentation? How do they vary the motion in a way that still respects those initial grasp constraints?
Title and authors: Dev: They combine those hand references with whole-arm pick-and-place trajectories for different object starting positions and placement targets. So, they’re adding a whole new sequence of approach, grasping, transport, placement, and retreat steps to the original interaction seeds.
Rosa: That augmentation provides a wider variety of kinematic references for policy training compared to just replaying the initial retargeted motion alone. Then they realize these references under arm-tracking dynamics by training an RL finger-control policy in IsaacLab while the robot arm tracks precomputed joint trajectories.
Taro: Training the finger controller only when the robot arm is tracking a trajectory sounds like you’re trying to solve a complex control problem where you have to manage both the hand and the wrist movement simultaneously. What about those resulting wrist-tracking errors?
Dev: The policy learns finger corrections under those resulting wrist-tracking errors, which helps it learn how to adapt its grip when the arm moves unexpectedly during simulation. That’s what allows for more reliable whole-arm execution than just training on static references.
Rosa: They render observations from the policy’s successful rollouts and pair them with the recorded robot actions and corresponding grasp descriptions to form training demonstrations. These demonstrations are then used to fine-tune a vision language action model that jointly controls the arm and fingers to execute language-specified grasps.
Taro: So they generate these visual observations, the actions, and those grasp descriptions so that this vision language model can learn how to follow text instructions for complex physical tasks. What kind of input does that VLA need?
Dev: The VLA is fine-tuned on this multimodal dataset, which includes rendered observations from the policy rollouts, the recorded robot actions from those successful runs, and those corresponding grasp descriptions. It’s a comprehensive input set for learning coordinated control.
Rosa: They select eight representative grasp modes from the GRAB dataset and use nineteen source sequences from these modes to generate a robot-centric dataset of twenty-five thousand two hundred seventy demonstrations <ref:2611.14092#pg2,source sequences from these modes to generate a robot-centric dataset of>. That’s a massive dataset they generated from just nineteen human sequences.
Taro: twenty-five thousand demonstrations is quite substantial, but I wonder what the limitations are here. Does this data cover all possible object geometries or all possible manipulation locations?
Dev: The authors mention that they selected eight representative grasp modes from the GRAB dataset, which gives them a defined set of starting points. They also note that they use nineteen source sequences spanning these modes to generate the dataset, so it’s focused on coverage within those modes <ref:2611.14092#pg2>.
Title and authors: Rosa: They also use a small threshold to render observations and compute a one hundred seventy dimensional descriptor, which is a concatenation of eight feature blocks describing things like how much the fingers bend and other hand features. This gives them rich visual input for the model.
Taro: That descriptor sounds like it’s trying to capture a lot of subtle finger configuration details, which is good for fine motor control, but I wonder if that one hundred seventy dimensional descriptor is all that matters when the world misbehaves outside of simulation <ref:2611.14092#pg3>.
Dev: It's designed to be a detailed estimate of the grasp establishment, and it's paired with recorded actions and descriptions for the VLA training. The whole point is to get a model that can handle language instructions for dexterous manipulation in simulation first, which is much cheaper than real-world trials.
Rosa: The results show that this method improves contact fidelity in retargeting and achieves more reliable whole-arm execution than baselines. Qualitatively, they show better preservation of grasp configurations across different object geometries.
Taro: I read about the physics-based imitation part where they train the finger policy under arm-tracking dynamics. How much does that help when you move from simulation to a real robot? Does it prepare it for real-world tracking errors?
Dev: The paper shows that when coupled with the robot arm, this method achieves lower tracking errors, higher contact recall, and higher success under arm tracking than both baselines. That gap highlights how important it is to train the finger controller with those arm-tracking dynamics for reliable whole-arm execution.
Rosa: And finally, when they evaluate the VLA trained on DexSeed data in simulation, it yields higher task success than kinematic data and data generated using either imitation baseline across all evaluation settings. The poor performance of kinematic data highlights the importance of realizing references under contact dynamics before using them for policy learning.
Taro: So the paper’s main implication is that you can generate high-quality, diverse robot data from human demonstrations without needing expensive teleoperation, provided you do it in a simulation environment that accurately models arm dynamics.
Dev: Exactly. And they show that when you use this generated data to fine-tune the VLA, it performs better than using just kinematic data or even imitation baseline methods for learning coordinated arm and finger actions from visual observations and language instructions.
Title and authors: Rosa: The real-world deployment part is interesting too; they can fine-tune the VLA on simulation-generated demonstrations and deploy it on a real robot without collecting any new real-robot action demonstrations. They successfully execute all eight grasp modes on the Franka arm, though they noted that handle threading achieved the lowest success rate because of its tight requirements for precise finger object alignment.
Taro: So, if you look at what this paper does—DexSeed: Expanding Human Hand-Object Interactions into Robot Demonstrations for Dexterous Policy Learning—it’s about using human data to create a physics-based pipeline that produces robot demonstrations suitable for language-guided dexterity learning.
Dev: It transforms human hand-object interactions into diverse, physically executable robot data. The framework organizes interactions by grasp mode and expands sequences across object geometries and arm trajectories through interaction-preserving retargeting and physics-based imitation.
Rosa: The improvements they suggest are basically taking those steps: using InteractionMesh based retargeting to adapt sequences to different geometries while keeping hand-object relationships, augmenting these with whole-arm pick-and-place trajectories for variety, training the finger policy under arm tracking dynamics in IsaacLab, and finally fine-tuning a VLA on the resulting data so it can execute language instructions on real objects.
Taro: And they also added a specific reward function for the finger policy that balances object tracking, interaction preservation, and smooth finger control using terms like Rcontact and Rtip to make sure it’s not just tracking but actually maintaining contact.
Dev: That reward function adds another layer of constraint to the learning process, ensuring the policy learns physically meaningful behaviors beyond just reaching a target point. It also points out that they can deploy this VLA on a real robot without needing new real-robot action demonstrations at deployment.
Rosa: So, in conclusion, DexSeed shows that by organizing human interactions by grasp mode and expanding individual sequences across object geometries and arm trajectories through interaction-preserving retargeting and physics-based imitation, you get better contact fidelity and more reliable physical realization than the evaluated baselines.
Taro: It really confirms that realizing references under contact dynamics before using them for policy learning is crucial for getting good results when you move toward actual physical control.
Dev: And it shows that the generated data supports learning coordinated arm and finger actions from visual observations and language instructions, which is a big step for general dexterous manipulation.
Rosa: DexSeed transforms human hand-object interactions into diverse, physically executable robot data for language-guided dexterous manipulation. It’s a framework that organizes those interactions and expands them through retargeting and physics-based imitation to create rich training data.
The paper's summary: Rosa: So, DexSeed takes those human hand-object interactions and turns them into robot data for training vision language models to do dexterous manipulation without needing tons of expensive real-world examples for every single object or setup.
Dev: It’s about organizing those human sequences by the grasp they use, and then expanding each one by varying where the object is placed on a table and how you move it, all while keeping the original grasp description intact.
Taro: The way they do this expansion is key because it gives the finger policy a lot of different things to practice under arm tracking dynamics instead of just seeing one static motion.
Rosa: Exactly. And then they use interaction mesh retargeting to make sure that when you move from a human hand to a robot hand, the local relationships between the fingers and the object stay preserved, using something called tetrahedralization for that spatial mapping.
Dev: That sounds like they’re building a very consistent coordinate system for every point on both the human hand and the robot finger throughout the entire sequence, which helps keep things stable.
Taro: But what about when you realize these augmented sequences in IsaacLab? How does that physical realization help fix any errors when the arm itself is moving unexpectedly during training?
Dev: They use closed-loop physics-based imitation where the robot arm follows precomputed joint trajectories with a simple PD control, but only the fingers are controlled by a learned policy, and that policy learns to correct for those tracking errors under contact dynamics.
Rosa: That's what makes it reliable; the finger controller learns to adapt its grip when the arm doesn't move exactly as planned during simulation. They even added a reward function that balances object tracking with maintaining contact and smooth finger control using terms like Rcontact and Rtip.
Taro: So you’re not just training it to reach a point; you’re making it learn how to maintain the grip while the whole arm is moving, which is a much harder thing for real robots to do reliably.
Dev: And because they generate this data, which includes rendered observations and descriptions of those grasps, they can fine-tune a vision language action model that can execute language instructions on real objects without needing new real-robot actions at all during deployment.
Rosa: That means you could train the system in a simulator for months and then deploy it to a physical robot to perform complex tasks based purely on natural language commands.
Taro: It really makes the barrier for learning coordinated arm and finger actions from visual input much lower because you’re training on data that has already been vetted through physics simulation.
Dev: It confirms that realizing those references under contact dynamics before you use them for policy learning is crucial for getting good results when you move toward actual physical control, which is what this paper really emphasizes.
Rosa: DexSeed transforms human hand-object interactions into diverse, physically executable robot data for language-guided manipulation by organizing them and expanding them through retargeting and physics imitation.
Taro: It's a framework that shows you how to use human data to create rich training material that actually works when you try to make an AI control both the arm and the fingers simultaneously.
Dev: The implication is that we can generate high-quality, diverse robot data from human demonstrations without needing expensive teleoperation or collecting tons of new real-world action demonstrations for every single object geometry.
Rosa: That’s a big deal for accessibility in robotics; it moves the heavy lifting of motion planning and contact modeling into the simulation phase, which is much cheaper.
Taro: It also shows that this generated data supports learning coordinated arm and finger actions from visual observations and language instructions, which is a big step for general dexterous manipulation.
Dev: It's about building a pipeline where the simulation handles the complex physics and dynamics so the policy learns to handle the high-level reasoning, like following a verbal command.
Rosa: So, if you look at what this paper does—DexSeed: Expanding Human Hand-Object Interactions into Robot Demonstrations for Dexterous Policy Learning—it’s about using human data to create a physics-based pipeline that produces robot demonstrations suitable for language-guided dexterity learning.
The paper's improvements: Rosa: So, DexSeed suggests several ways to make this data generation process even better than what they already have by focusing on improving the transfer and training steps.
Dev: They focus on using InteractionMesh based retargeting more heavily to ensure that the robot hand really understands those local hand-object relationships when adapting sequences for different object geometries.
Taro: That seems like it addresses a weakness in just retargeting; it’s about ensuring the spatial context is preserved consistently across every vertex of the hand and object throughout the entire sequence.
Rosa: Right, because that consistency feeds into the whole augmentation process where you vary the starting points and movement paths to give those finger policies more diverse references to train on.
Dev: And they also emphasize adding a specific reward function for the finger policy that actively balances object tracking with interaction preservation and smooth control using terms like Rcontact and Rtip.
Taro: That reward function is important because it’s not just about getting the fingertips close; it forces the policy to maintain the intended contact during manipulation, which is a big step toward real-world robustness.
Rosa: So they're trying to create a learning signal that rewards both achieving the goal and keeping things physically stable and well-gripped, even when the arm is moving around.
Dev: It’s about making sure the learned finger controller isn't just tracking a path; it’s learning how to handle the physics of contact during that whole movement.
Taro: And this all feeds into the VLA training because you want those demonstrations to be physically executable and robust, not just visually plausible.
Rosa: Exactly. They are pushing for data that is not only rich visually but also inherently tied to physical constraints, which is what makes the resulting vision language model actually useful for real-world tasks.
Dev: This moves us toward a safety case where we can trust the VLA because it’s been trained on data generated by physics-based imitation that explicitly models contact dynamics.
Taro: So, to summarize, they’re refining the input data generation by making the retargeting more rigorous, adding richer augmentation for variety, and tuning the finger policy with a reward function focused on physical contact maintenance.
Rosa: It’s about taking those initial successful results and building a more constrained system that ensures the learned policy behaves reliably when it gets deployed on a real robot.
Dev: This refinement is what bridges the gap between simulation success and real-world execution, making sure the model learns to recover gracefully if things go slightly off script during operation.
Taro: So, we’re looking at a method that makes the generated data more physically meaningful, which should lead to better generalization when you eventually try to apply this vision language model outside of simulation.
Conclusion: Rosa: So, to wrap up, DexSeed shows how you can organize human hand-object interactions by grasp mode and then expand those sequences through interaction-preserving retargeting and physics-based imitation to create rich robot data for vision language models.
Dev: It’s a framework that takes messy human motion and turns it into diverse, physically executable robot demonstrations for language-guided dexterity learning, which is a big step because you don't need expensive teleoperation anymore.
Taro: I just think the real implication is that by realizing these references under contact dynamics before policy learning, you are giving the AI a much more robust foundation for handling uncertainty in physical tasks.
Rosa: That’s right, it really confirms that realizing references under contact dynamics before using them for policy learning is crucial for getting good results when you move toward actual physical control.
Dev: It shows that the generated data supports learning coordinated arm and finger actions from visual observations and language instructions, which is a big step for general dexterous manipulation.
Taro: I think that capability to generate this kind of physically grounded training data will open up a lot more possibilities for training complex robotic systems than just using standard kinematic motion replay.
Rosa: And the real-world deployment aspect is interesting too; they can fine-tune the vision language model on simulation-generated demonstrations and deploy it on a real robot without collecting any new real-robot action demonstrations at all.
Dev: That’s huge because it drastically lowers the barrier to entry for deploying sophisticated manipulation skills, as we don't have to rely solely on expensive real-robot action collection.
Taro: It means we can iterate much faster in the simulation phase, and then deploy a policy that has been rigorously tested under simulated physical constraints before risking hardware.
Rosa: So, in conclusion, DexSeed transforms human hand-object interactions into diverse, physically executable robot data for language-guided manipulation by organizing them and expanding them through retargeting and physics imitation to create rich training data.
Dev: It’s a framework that organizes those interactions and expands them through retargeting and physics-based imitation to create rich training data.
Taro: I think this approach of realizing references under contact dynamics before policy learning is crucial for getting good results when you move toward actual physical control, which is what the paper really emphasizes.
Rosa: DexSeed shows that by organizing human interactions by grasp mode and expanding individual sequences across object geometries and arm trajectories through interaction-preserving retargeting and physics-based imitation, you get better contact fidelity and more reliable physical realization than the evaluated baselines.
Dev: And it shows that when coupled with the robot arm, this method achieves lower tracking errors, higher contact recall, and higher success under arm tracking than both baselines.
Taro: It really confirms that realizing references under contact dynamics before using them for policy learning is crucial for getting good results when you move toward actual physical control.
Rosa: So, in conclusion, DexSeed transforms human hand-object interactions into diverse, physically executable robot data for language-guided manipulation.
Huaijin Pi, Zehong Shen, Yong Wang, Kechun Xu, Changpeng Wang, Dingkun Wei, Xueyang Wang, Wenjia Wang, Xing Zhu, Yujun Shen†, Taku KomuraΩ
University of Hong Kong · Robbyant Research Institute · Zhejiang University
cs.RO
Submitted: 2026-11-10
Updated: 2026-11-10
Comments: Project page: https://robbyant-research.github.io/DexSeed/
Code: https://github.com/kevinzakka/mink
License: http://creativecommons.org/licenses/by-nc-sa/4.0/
Importance score: 100/100
The gist: DexSeed is a framework for low-cost generation of dexterous robot data from human hand–object interactions.
Key concepts
- InteractionMesh retargeting
- This technique adapts human hand-object interactions to fit a robot hand and new object shapes. It uses 3D triangulation of landmarks and anchors to preserve the local relationships between the hand and object, allowing for diverse robot demonstrations without needing new data collection for every scenario.
- Whole-Arm Trajectory Augmentation
- This step creates many different manipulation sequences by varying where the hand starts, where the object is placed on a table, and where it needs to go. This variation generates a wide range of kinematic references, making the policy training more robust than just using single motion replays.
- Physics-Based Dexterous Imitation
- This involves training a finger policy in simulation while the robot arm follows precomputed joint trajectories using PD control. This teaches the fingers to learn necessary corrections to handle wrist tracking errors, resulting in more reliable whole-arm execution under realistic contact dynamics.
- VLA Learning in Simulation
- A Vision Language Action (VLA) model is fine-tuned on the generated robot data, including rendered views and grasp descriptions. This VLA can then execute language-specified grasps on real objects during deployment without requiring any additional real-robot action demonstrations.
Terminology
Summary
DexSeed is a framework for low-cost generation of dexterous robot data from human hand–object interactions. p. 0 Turning these interactions into VLA training data
further requires coverage of different objects and manipulation locations, together with executable arm and finger
actions, and visual observations. To achieve this, we present
DexSeed, a framework for low-cost generation of dexterous
robot data from human hand–object interactions. To achieve this, we present
DexSeed, a framework for low-cost generation of dexterous
robot data from human hand–object interactions. Our key
idea is to expand human sequences while preserving their
grasps and realizing the resulting motions under arm-tracking
dynamics in physics simulation. We render observations from the policy’s
successful rollouts and pair them with the recorded robot
actions and corresponding grasp descriptions to form training
demonstrations. The resulting data are used to fine-tune a
VLA that jointly controls the arm and fingers to execute
language-specified grasps.Where the paper says it
Where the paper says it
Where the paper says it
Framework Overview
DexSeed organizes human interactions by grasp mode and expands them into simulated robot data. The process involves several key steps, as illustrated in Figure 2, starting from input human–object interactions to Gaussian splatting rendering. p. 2 Human
demonstrations
01 Grasp bank construction
02 InteractionMesh retargeting
03 Trajectory augmentation
04 Physics-based realization
05 Gaussian splatting
rendering
Input human–object
interactions
Grasp mode 1 Grasp mode 2
Selected source sequence
Original Resized New geometry
Geometry-conditioned sequence
⋯ ⋯
Augmented trajectories
⋯ ⋯
Realized trajectories
3D interaction sequence
External view
Wrist view
Fig. 2: DexSeed Overview.Where the paper says it
The overall pipeline is structured as follows:
(01) Grasp bank construction
(02) InteractionMesh retargeting
(03) Trajectory augmentation and (04) Physics-based realization)
The framework groups human sequences by grasp similarity and selects representative modes, each paired with a shared description of how to grasp.
Data Generation Methodology
The method constructs a robot-centric dataset of 25,270 demonstrations from 19 human sequences spanning 8 representative grasp modes. This generation relies on three main components:
(B) Dexterous Robot Data Generation
The InteractionMesh-based Cross-Embodiment and CrossObject Retargeting adapts human interactions to the robot hand and new object geometries while preserving local hand–object relationships. This involves tetrahedralizing the hand landmarks and object anchors using 3D Delaunay triangulation to define a fixed neighborhood N (i) for each vertex throughout the sequence. p. 2 At a representative
frame within the interaction interval, we jointly tetrahedralize
the hand landmarks and object anchors using 3D Delaunay triangulation. The resulting connectivity defines a fixed
neighborhood N (i) for each vertex throughout the sequence.Where the paper says it
The Whole-Arm Trajectory Augmentation varies where the hand starts, where the object is initially placed, and where it is moved to create multiple tabletop manipulation sequences. This involves sampling an initial wrist pose, an object pose on the table, and a placement target to form a complete sequence covering approach, grasping, transport, placement, and retreat. p. 3 At the
placement target, we reuse the source motion to set down
and release the object, then withdraw the hand. This produces a hand–object trajectory covering approach, grasping,
transport, placement, and retreat.Where the paper says it
The Physics-Based Dexterous Imitation uses closed-loop physics-based imitation to obtain executable robot rollouts under contact dynamics. The finger policy is trained jointly on augmented references, where the robot arm tracks precomputed joint trajectories using PD control while the learned policy controls only the 20-DoF robot fingers. p. 3 For each source sequence, we
train a separate policy jointly on its augmented references,
covering variations in object geometry and whole-arm pickand-place trajectories. We separate whole-arm motion from dexterous finger
control: the robot arm tracks the precomputed joint-space
reference trajectories using PD control, while the learned
policy controls only the 20-DoF robot fingers.Where the paper says it
Performance and Evaluation
Experiments demonstrate that DexSeed improves contact fidelity in retargeting and achieves more reliable whole-arm execution than baselines. The results are validated through simulation evaluation and real-world deployment. p. 0 Experiments show
improved contact fidelity in retargeting and more reliable
whole-arm execution than the evaluated baselines. Simulation
evaluation and real-world deployment further demonstrate
that a VLA fine-tuned solely on the generated demonstrations
can execute language-specified grasps.Where the paper says it
Key findings from the evaluations include:
(B) Interaction Retargeting
The method outperforms baselines in contact accuracy and angular alignment, indicating better preservation of the demonstrated hand–object interactions. Qualitative comparisons show improved preservation of grasp configurations across different object geometries. p. 4 The qualitative comparison in Fig. 3(a) further
illustrates the improved preservation of the reference grasp
configurations.Where the paper says it
(C) Physics-Based Dexterous Imitation
The method achieves lower tracking errors, higher contact recall, and higher success under arm tracking than both baselines when coupled with the robot arm. This gap highlights the importance of training the finger controller with arm-tracking dynamics for reliable whole-arm execution. p. 5 The baselines and w/o Arm
ablation perform well with a floating hand but become less
reliable when coupled with the robot arm. This gap highlights
the importance of training the finger controller with armtracking dynamics for reliable whole-arm execution.Where the paper says it
(D) VLA Learning in Simulation
The VLA trained on DexSeed data yields higher task success than kinematic data and data generated using either imitation baseline across all evaluation settings. The poor performance of kinematic data highlights the importance of realizing references under contact dynamics before using them for policy learning. p. 5 As shown in Table IV, our demonstrations yield higher
task success than kinematic data and data generated using either imitation baseline across all evaluation settings. The poor performance of kinematic data highlights the
importance of realizing references under contact dynamics
before using them for policy learning.Where the paper says it
Real-World Deployment
The VLA fine-tuned on simulation-generated demonstrations can be deployed on a real robot without collecting real-robot action demonstrations. In real-world deployment, the VLA successfully executes all eight grasp modes on the Franka arm. p. 6 Real-World VLA Deployment
Finally, we evaluate the VLA trained on our data on the
real robot. We use a Franka arm equipped with a Wuji
dexterous hand and evaluate all eight grasp modes, using
one test object per mode and conducting ten trials for
each object–mode pair. As shown in Table VII, the VLA trained on our generated
demonstrations successfully executes all eight grasp modes
on the real robot. Success rates vary across modes, with handle threading achieving the lowest success, consistent with
its tighter requirements for precise finger–object alignment.Where the paper says it
Where the paper says it
Conclusion
DexSeed transforms human hand–object interactions into diverse, physically executable robot data for language-guided dexterous manipulation. The framework organizes human interactions by grasp mode and expands individual sequences across object geometries and arm trajectories through interaction-preserving retargeting and physics-based imitation. p. 1 • We present a framework that groups human interactions
by grasp mode and expands individual sequences into
diverse robot demonstrations paired with language descriptions of how to grasp. • We develop an InteractionMesh-based data generation
method that preserves hand–object interactions during
retargeting and transfer across different object geometries. The resulting demonstrations
are paired with robot-centric observations and shared descriptions of how to grasp. Experiments show improved
contact fidelity and more reliable physical realization than
the evaluated baselines. Experiments show improved
contact fidelity and more reliable physical realization than
the evaluated baselines. VLA evaluation in simulation and
deployment on a real robot further demonstrate that the
generated data support learning coordinated arm and finger
actions from visual observations and language instructions.Where the paper says it
Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Transfer human hand-object interactions to robots by using InteractionMesh based retargeting to adapt sequences to robot hands and different object geometries while preserving local hand-object relationships. This allows the system to generate diverse robot demonstrations without costly teleoperated examples or requiring new data collection for every object geometry.
-
Augment these sequences into multiple tabletop manipulation sequences by varying where the hand starts, where the object is initially placed, and where it is moved using an approach trajectory. This provides a wider variety of kinematic references for policy training compared to direct replay of the retargeted motion alone.
-
Train an RL finger control policy in IsaacLab while the robot arm tracks precomputed trajectories to produce executable rollouts under arm-tracking dynamics. This enables the finger controller to learn necessary corrections under wrist tracking errors, leading to more reliable whole-arm execution than baselines.
-
Fine-tune a Vision Language Action (VLA) model on these generated demonstrations using a multimodal dataset that includes rendered observations, recorded robot actions, and grasp descriptions. The resulting VLA can execute language-specified grasps on real objects without needing real-robot action demonstrations at deployment.
-
Implement a reward function for the finger policy that balances object tracking, interaction preservation, and smooth finger control using terms like Rcontact and Rtip. This ensures the learned policy not only tracks the object but also maintains the intended contact during manipulation.
Abstract
We study how to generate diverse, physically executable robot demonstrations for dexterous vision-language-action (VLA) learning. Existing demonstration-generation methods typically rely on costly teleoperated examples. Meanwhile, human hand--object datasets provide a reusable source, but existing transfer methods mainly reproduce given interactions. Turning these interactions into VLA training data further requires coverage of different objects and manipulation locations, together with executable arm and finger actions, and visual observations. To achieve this, we present DexSeed, a framework for low-cost generation of dexterous robot data from human hand--object interactions. Our key idea is to expand human sequences while preserving their grasps and realizing the resulting motions under arm-tracking dynamics in physics simulation. Specifically, we organize source sequences by grasp mode with shared language descriptions. InteractionMesh-based retargeting and arm trajectory augmentation adapt these sequences to the robot hand, different object geometries, and pick-and-place locations while preserving local hand--object relationships. An RL finger-control policy produces executable rollouts, while the arm tracks precomputed trajectories. Rendered observations and recorded actions from successful rollouts are paired with grasp descriptions to fine-tune a VLA. Experiments show improved contact fidelity in retargeting and more reliable whole-arm execution than the evaluated baselines. Simulation evaluation and real-world deployment further demonstrate that a VLA fine-tuned solely on the generated demonstrations can execute language-specified grasps. Project page: https://robbyant-research.github.io/DexSeed/.
Sources
- $\pi_0$: A Vision-Language-Action Flow Model for General Robot Control
- DemoGen: Synthetic Demonstration Generation for Data-Efficient Visuomotor Policy Learning
- SPIDER: Scalable Physics-Informed Dexterous Retargeting
- TopoRetarget: Interaction-Preserving Retargeting for Dexterous Manipulation
- C2Dex: Contact-Consistent Reconstruction and Retargeting for Dexterous Manipulation from Monocular Video
- A Minimalist Retargeting-Guided Reinforcement Learning Recipe for Dexterous Manipulation
- Do as I Do: Dexterous Manipulation Data from Everyday Human Videos
- Isaac Lab: A GPU-Accelerated Simulation Framework for Multi-Modal Robot Learning
- DexCap: Scalable and Portable Mocap Data Collection System for Dexterous Manipulation
- DexUMI: Using Human Hand as the Universal Manipulation Interface for Dexterous Manipulation
- DexTransfer: Real World Multi-fingered Dexterous Grasping with Minimal Human Demonstrations
- HERMES: Human-to-Robot Embodied Learning from Multi-Source Motion Data for Mobile Dexterous Manipulation
- DexImit: Learning Bimanual Dexterous Manipulation from Monocular Human Videos
- OmniRetarget: Interaction-Preserving Data Generation for Humanoid Whole-Body Loco-Manipulation and Scene Interaction
- 3D Diffusion Policy: Generalizable Visuomotor Policy Learning via Simple 3D Representations
- A Pragmatic VLA Foundation Model
- UniDex: A Robot Foundation Suite for Universal Dexterous Hand Control from Egocentric Human Videos
Related papers
- Rethinking Actions: Continuity-Aware Priors for World-Model Planning
- MOSAIC: Modular Obstacle and Spatial-Aware Framework for Manipulation In Cluttered Scenes
- SAGAS: Semantic-Aware Graph-Assisted Stitching for Offline Temporal Logic Planning
- LaST0.5: A Generalist World Action Model with Interaction-Centric Latent Reasoning
- CIALLO: Learning Shared Intent Representations for Cross-Embodiment Manipulation
- SSR: Scaling Surefooted and Symmetric Humanoid Traversal Toward the Open World