Dex-One2Many: Learning Dexterous Manipulation from a Single Human Demonstration

arXiv:2610.12470 · cs.RO, cs.CV · Submitted 2026-10-08 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: I'm Rosa, and with me are Dev and Taro, guest researcher.

Dev: Today's paper: "Dex-One2Many: Learning Dexterous Manipulation from a Single Human Demonstration".

Rosa: The gist Dex-One2Many presents a real-to-sim-to-real framework that learns a generalizable dexterous manipulation policy from a single human video by abstracting it into sequential scene graphs to guide reinforcement learning.

Dev: First, who's behind it and why it matters.

Paper summary: Dev: So, looking at "Dex-One2Many: Learning Dexterous Manipulation from a Single Human Demonstration," it's about learning how to do complex hand movements just from watching one human demonstration video and using scene graphs to guide reinforcement learning.

Taro: Right, so you've got a framework that takes a single video and turns it into a sequence of these structured graphs that the AI uses as its roadmap for figuring out how to grasp things.

Rosa: The core idea is abstracting the task into these sequential scene graphs, which lets the reinforcement learning part explore all those ways to do it efficiently instead of just mimicking every single tiny motion from the video.

Dev: And that abstraction lets them ground the policy in this graph structure—meaning when it makes a decision, its observation and its reward are built directly from what that graph tells it about the task at that moment.

Taro: When you look at how they define rewards, they aren't just looking at where things are; they’re using predicates from the scene graph edges—like distance metrics or grasp quality measures that check for force closure.

Rosa: They assemble a total task reward by adding up the rewards from every stage, so it keeps feeding the learning process feedback through each step of the manipulation process.

Dev: And they set up an initial state distribution based on those graph rules, using geometric predicates as placement rules to get things in the right starting position for each step.

Taro: The zero-shot sim-to-real transfer is pretty neat because they train it all in simulation but then test it on real robots with unseen object poses and goal configurations, showing decent success rates on those novel setups.

Rosa: So, what does this mean for the world? It means if we can reliably structure a task from just one human video, we might be able to bootstrap skills for a huge variety of manipulation problems without needing tons of specific demonstrations for every single new setup.

Dev: It shifts the focus from matching exact movements to learning the underlying logic of how objects interact in a sequence, which is pretty powerful if you can get that abstraction right.

Taro: But they also showed some limitations, like removing stage-wise resets tanked performance because starting from intermediate stages is really important for exploring multi-stage tasks.

Rosa: Yeah, so the authors are showing how to move past just copying execution details and instead learn the task structure itself, even when the world throws unexpected configurations at it.

Dev: The next thing we can look at is how they handle those real-world differences, like varying robot kinematics or adding noise to observations for robustness during training.

Conclusion: Rosa: So, Dex-One2Many is learning how to do dexterous manipulation from just one human video by turning it into sequential scene graphs that guide reinforcement learning.

Dev: Right, so they're taking a single video and breaking it down into those structured graphs that the AI uses as its roadmap for figuring out how to grasp things.

Taro: The core idea is abstracting the task into these sequential scene graphs, which lets the reinforcement learning part explore all those ways to do it efficiently instead of just mimicking every single tiny motion from the video.

Rosa: And that abstraction lets them ground the policy in this graph structure—meaning when it makes a decision, its observation and its reward are built directly from what that graph tells it about the task at that moment.

Dev: That's key because it gives the system a clear signal on what to look for, like object geometry or relative poses between things, which makes building the policy observation much more structured.

Taro: When you look at how they define rewards, they aren't just looking at where things are; they’re using predicates from the scene graph edges—like distance metrics or grasp quality measures that check for force closure.

Rosa: They assemble a total task reward by adding up the rewards from every stage, so it keeps feeding the learning process feedback through each step of the manipulation process.

Dev: And they set up an initial state distribution based on those graph rules, using geometric predicates as placement rules to get things in the right starting position for each step.

Taro: The zero-shot sim-to-real transfer is pretty neat because they train it all in simulation but then test it on real robots with unseen object poses and goal configurations, showing decent success rates on those novel setups.

Rosa: So, what does this mean for the world? It means if we can reliably structure a task from just one human video, we might be able to bootstrap skills for a huge variety of manipulation problems without needing tons of specific demonstrations for every single new setup.

Dev: It shifts the focus from matching exact movements to learning the underlying logic of how objects interact in a sequence, which is pretty powerful if you can get that abstraction right.

Taro: But they also showed some limitations, like removing stage-wise resets tanked performance because starting from intermediate stages is really important for exploring multi-stage tasks.

Rosa: Yeah, so the authors are showing how to move past just copying execution details and instead learn the task structure itself, even when the world throws unexpected configurations at it.

Dev: The next thing we can look at is how they handle those real-world differences, like varying robot kinematics or adding noise to observations for robustness during training.

Jusuk Lee, Sungha Kim, Yeonsoo Park, Jonguk Cheon, Yoonkyo Jung, Yongjun You, H. Jin Kim, Jia-Bin Huang, Furong Huang, +Youngseok Jang†, Seungjae Lee†

Seoul National University · University of Maryland, College Park · KAIST

cs.RO, cs.CV

Submitted: 2026-10-08

Updated: 2026-10-08

Comments: Project page: https://dex-one2many.github.io/

Code: https://github.com/teal024/FoundationPose-plus-plus

Project page: https://dex-one2many.github.io

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 91/100

The gist: The gist Dex-One2Many presents a real-to-sim-to-real framework that learns a generalizable dexterous manipulation policy from a single human video by abstracting it into sequential scene graphs to

Key concepts

Stage-Wise Scene Graph Abstraction
This process breaks down a video into a sequence of graphs that describe only the necessary relations for the task, such as holding an object or placing it in a container. It uses a vocabulary of task-independent predicates to encode these relations, focusing on 'what' needs to happen rather than exact poses.
Scene-Graph-Grounded Reinforcement Learning
The reinforcement learning policy is built upon the scene graph. Observations are derived from the graph's nodes (representing objects and hand parts), and rewards are directly tied to the predicates in the edges (like 'grasp' or 'contact'). This grounds the learning process in task structure.
Zero-shot Sim-to-Real Transfer
The learned policy is trained entirely in simulation using PPO, but it is designed to work on real robots. It uses domain randomization and a separate low-level controller to adapt constraints at deployment time, ensuring the policy generalizes well to unseen objects and environments.
Predicate-Derived Rewards
Rewards are calculated based on the specific relations encoded in the scene graph edges. For instance, a reward is given when an edge representing a 'grasp' relation is satisfied. This dense feedback guides exploration toward achieving the required task relations efficiently.

Terminology

Summary

The gist Dex-One2Many presents a real-to-sim-to-real framework that learns a generalizable dexterous manipulation policy from a single human video by abstracting it into sequential scene graphs to guide reinforcement learning.

Task Abstraction as a Stage-wise Scene Graph

The core insight of the framework is to abstract the video into sequential scene graphs that guide RL, enabling efficient exploration while preserving broad generalizability (pg2). This is achieved through two main techniques:

  1. Stage-Wise Scene Graph Abstraction: This process reduces the video to the relations the task requires, such as the hand holding an object or an object resting inside a container, and in what order (pg2). The video is encoded into a sequence of graphs G1:K where each graph describes a set of states that all preserve the task (pg2). This abstraction uses a predefined, task-independent predicate vocabulary to encode relations like grasp, contact, and inside, while leaving object poses, goal poses, and grasps free to vary (pg3). The VLM is used to recover these graphs from the video by identifying entities as nodes and inferring predicate relations over time (pg21).

Scene-Graph-Grounded Reinforcement Learning

The learned policy is formulated as a Markov decision process (MDP) grounded in this graph, where observations, rewards, and an initial-state distribution are derived from the graph (pg3).

  1. Observation Construction: The policy observation is constructed by concatenating features from scene-graph nodes and edges; object nodes encode object geometry and 6D pose, functional-part nodes localize regions like the grasp region, and edges capture the relative 6D pose between connected entities (pg3).

  2. Task Reward Definition: Rewards are derived directly from the predicates encoded in each scene-graph edge; geometric predicates use a distance metric in Table 3 as a dense progress signal, and for grasp, it uses CHORD’s force-closure reward augmented with a grasp-quality measure to ensure stability (pg6). The complete task reward is assembled by summing rewards from each stage, ensuring that rewards through the grasp stage are always available (pg29).

  3. Initial-State Distribution: The initial state distribution ρ0 is constructed as a generative constraint using the scene graph, where geometric predicates serve as placement rules to position nodes relative to one another (pg31). For the final stage, states are sampled near the goal configuration, which reduces the exploration burden of long-horizon tasks (pg31).

Zero-shot Sim-to-Real Transfer

The policy is trained entirely in simulation using PPO with an asymmetric actor–critic where the actor receives only the scene-graph-grounded observation available on the real robot (pg34). To ensure robustness, domain randomization is applied, varying physical properties and deliberately add noise to the actor’s observations to improve robustness to real-world differences (pg35). Furthermore, a crucial component is the separation of control: a QP low-level controller maps the policy’s palm and fingertip velocity commands to joint velocities subject to kinematic and safety constraints (pg33). This allows for deployment-time constraint adaptation, such as adding a hand–table clearance constraint without retraining the task-space policy (pg34).

Experimental Results and Generalization

Dex-One2Many demonstrates superior generalization across unseen configurations, achieving a 71%p improvement over baselines in unseen scenarios (pg1). On five tasks, it achieves 60–85% real-world success on configurations not shown in the video (pg3). The framework also shows robustness to diverse embodiments, with mean success rates ranging from 94.6% to 96.8% across four tested URDF and hand kinematics (pg43). Qualitative results show that different hands adopt different grasp types, and the same hand can change its grasp as the object configuration changes, indicating that the policy learns embodiment- and configuration-specific grasp strategies rather than a fixed one (pg41).

Critical Design Choices

Ablation studies highlight several critical components of Dex-One2Many:

  1. Removing stage-wise resets significantly drops success, showing that starting from intermediate stages is key to exploring multi-stage tasks (pg10).

  2. Removing predicate-derived rewards results in a success rate of 0% throughout training, proving that dense feedback toward the required task relations is critical for guiding exploration (pg10).

  3. Omitting the verified closing command lowers success to about 48% after 2K PPO updates, indicating that restoring active contact forces is vital for stable transitions across stages (pg10).

These results suggest that extracting what must be achieved from a human video, while allowing RL to discover how to achieve it, provides a promising route from a single demonstration to diverse dexterous behaviors (pg5). This work addresses the challenges of task abstraction and hard exploration by grounding the policy in task structure rather than imitating execution details (pg2). The framework successfully transfers zero-shot to a real multi-fingered hand across initial and goal object poses not shown in the video (pg3).

--- Page 1 ---

Dex-One2Many: Learning Dexterous Manipulation from a Single Human Demonstration

The gist Dex-One2Many presents a real-to-sim-to-real framework that learns a generalizable dexterous manipulation policy from a single human video by abstracting it into sequential scene graphs to guide reinforcement learning.

Critical Design Choices

Ablation studies highlight several critical components of Dex-One2Many:

Improvements for AI systems

  1. textbfThe Stage-Wise Scene Graph Abstraction enables task structure learning over motion imitation by extracting not the video’s motion but its task structure, a stage-wise scene graph specifying which hand–object and object–object relations must hold, and in what order. This allows the policy to learn the task structure rather than the recorded execution, enabling generalization beyond specific demonstrated trajectories.

  2. textbfThe Scene-Graph-Grounded RL facilitates efficient exploration by using each stage graph as a generative constraint, sampling reset states that satisfy the stage’s relations, with diverse multi-finger grasps, which keeps exploration short and guided. This mechanism allows the policy to practice the last stage before mastering the first, rather than reaching it only by chance.

  3. textbfThe Zero-shot Sim-to-Real Transfer capability ensures real-world applicability by training entirely in simulation and then deploying with domain randomization and a QP low-level controller that enforces kinematic and deployment-time safety constraints without retraining the policy. This allows the policy to transfer zero-shot to a real multi-fingered hand.

Abstract

While learning dexterous manipulation from a single human video offers a promising alternative to costly robot demonstrations, many recent methods predominantly imitate demonstrated motions. Such strict motion matching often limits generalization to initial object poses, goal poses, and grasps not shown in the video. Alternatively, discovering a policy via reinforcement learning (RL) allows for broad generalization, but without prior guidance, it struggles with high-dimensional exploration in complex, multi-stage tasks. To address these coupled generalization and exploration challenges, we present Dex-One2Many, a real-to-sim-to-real framework that learns a generalizable dexterous manipulation policy from a single human video. Our key insight is to abstract the video into sequential scene graphs that guide RL, enabling efficient exploration while preserving broad generalizability. The graphs serve as generative constraints for sampling diverse reset states and provide dense rewards for each stage. Because the graphs constrain relations rather than exact poses, these reset states cover object poses and grasps beyond the video, while initializing each stage from them with dense rewards keeps exploration short and guided. Trained entirely in simulation, Dex-One2Many transfers zero-shot to a real multi-fingered hand. Across five tool-use and manipulation tasks, Dex-One2Many exceeds baselines by 6.5% in seen configurations, while its robust generalization widens this gap to 71% in unseen scenarios.

Sources

Related papers