HuMI: Humanoid Whole-Body Manipulation from Humanoid-Free Demonstrations

arXiv:2602.06643 · cs.RO, cs.AI, cs.LG · Submitted 2026-02-06 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: We're looking at the paper: "HuMI: Humanoid Whole-Body Manipulation from Humanoid-Free Demonstrations".

Dev: Here's the gist of the paper. Humanoid Manipulation Interface (HuMI) is a portable and efficient framework for learning diverse whole-body manipulation tasks across various environments.

Rosa: This system enables robot-free data collection by capturing rich whole-body motion using portable hardware.

Dev: Core Framework and Data Collection HuMI consists of a robot-free demonstration system and a hierarchical policy learning framework.

Rosa: The demonstration system uses handheld sensorized grippers and basestation-free wearable pose trackers to capture whole-body trajectories.

Title and authors: Rosa: The title of this paper is HuMI: Humanoid Whole-Body Manipulation from Humanoid-Free Demonstrations, and the authors are Ruiqian Nai, Boyuan Zheng, Junming Zhao, Haodong Zhu, Sicong Dai, Zunhao Chen, Yihang Hu, Yingdong Hu.

Dev: It’s interesting to see this work because it directly addresses the hardware logistics and complex reward engineering issues that usually limit autonomous skills to controlled environments >

Taro: So the core idea is using a robot-free demonstration system, which uses handheld sensorized grippers and wearable pose trackers on the grippers, waist, and feet >

Rosa: Exactly. They eliminate the need to manage real-robot balance or manually compensate for controller tracking errors by having this portable setup fit into a single backpack >

Dev: And they’re reporting that this approach achieves a three times increase in data collection throughput compared to teleoperation, which is pretty significant for getting usable data quickly >

Taro: It’s about making the system accessible, but I wonder if it's robust enough when the world misbehaves or when we try to use it outside of perfect conditions >

The paper's summary: Rosa: To summarize HuMI, they present a framework that moves away from needing perfect physical interaction or massive amounts of labeled data for whole-body skills by collecting robot-free demonstrations >

Dev: They show that this system supports a wide range of humanoid manipulation tasks involving whole-body coordination, precise bimanual actions, dynamic motions, and base mobility >

Taro: The main thing I'm seeing is their generalization capability. They report that with diverse demonstrations collected across many environments, HuMI achieves a seventy percent success rate on unseen objects and environments >

Rosa: That means we don't have to spend months collecting new datasets for every single thing the robot wants to do if it sees something new >

Dev: It really changes how we think about training robots because you're moving away from needing perfect physical interaction or massive amounts of labeled data for whole-body skills >

Taro: I think the main limitation they point out is that relying on visual trackers means you need good lighting and texture in your environment, which limits where you can actually use this system right now >

The paper's improvements: Rosa: They suggest a few key improvements to make this better. One is addressing underspecified demonstrations by using inverse kinematics to augment the base and feet trajectories into full robot degrees of freedom >

Dev: That’s smart because relying solely on gripper poses under-specifies whole-body motion, leading to unnatural postures, so adding those other body parts makes it more realistic >

Taro: And they also introduced an online inverse kinematics preview interface that lets the human in the loop kinematically adapt movements in real time >

Rosa: That allows operators to intuitively adjust their movements while collecting data, ensuring the collected data is both task-compliant and executable >

Dev: On the control side, they use adaptive tracking rewards and variable-speed augmentation to maximize precision without sacrificing stability in the low-level controller >

Taro: And for the policy interface, they use the previous target pose as an action reference instead of the actual executed pose to prevent action chunk discontinuity and maintain a smooth trajectory >

Conclusion: Rosa: So we've covered HuMI, Humanoid Whole-Body Manipulation from Humanoid-Free Demonstrations, and how you can teach a robot complex whole-body stuff without needing a physical robot around during training >

Dev: Right. The efficiency gain is substantial because they achieved a three times increase in throughput compared to teleoperation and hit a seventy percent success rate on unseen objects and environments >

Taro: That generalization is what I'm most interested in. If the policy works on stuff it never saw before, then we don't have to spend months collecting new data every time we want the robot to do something different >

Taro: The main limitation they flag is that relying on visual trackers means you need good lighting and texture in your environment, which limits where you can actually use this system right now >

Dev: Exactly. Moving forward, I think we need to look at how these learned policies interact with things like real-time execution models and safety guarantees as we move toward deployment >

Rosa: This paper on HuMI: Humanoid Whole-Body Manipulation from Humanoid-Free Demonstrations shows a really promising direction for learning whole-body skills through data collected outside of a fully controlled robot environment, even if we still have some practical hurdles to overcome >

Ruiqian Nai, Boyuan Zheng, Junming Zhao, Haodong Zhu, Sicong Dai, †Zunhao Chen, †Yihang Hu, †Yingdong Hu, —Tong Zhang, 㹜huan Wen

Tsinghua University · Shanghai Qi Zhi Institute · Spirit.AI Innovation Program

cs.RO, cs.AI, cs.LG

Submitted: 2026-02-06

Updated: 2026-11-06

Comments: Website: https://humanoid-manipulation-interface.github.io

Code: https://github.com/thu-ml/RDT2

Project page: https://humanoid-manipulation-interface.github.io

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 85/100

The gist: Humanoid Manipulation Interface (HuMI) is a portable and efficient framework for learning diverse whole-body manipulation tasks across various environments.

Key concepts

Robot-free Demonstration System
This system captures whole-body motion without needing a physical robot. It uses handheld sensorized grippers and wearable pose trackers to record detailed trajectories. This allows for the collection of rich data from humans in various environments, bypassing the need for expensive or complex robotic setups.
Online IK Preview Interface
This feature allows human demonstrators to adjust their movements in real-time while recording data. It uses inverse kinematics (IK) to fill in missing body parts like the base and feet of the demonstration. This ensures that the collected demonstrations are kinematically feasible and executable by allowing intuitive, on-the-fly correction.
Adaptive Tracking Rewards
This mechanism refines the low-level controller for precision. It dynamically scales how much precision is required based on the speed of movement. This means the system maintains high accuracy during slow interactions while automatically becoming more flexible as the robot moves faster.
Continuous Action Chunking
To ensure smooth motion, this policy interface uses the previous target pose as a reference for the current action instead of just tracking where it is executed. This prevents abrupt changes in trajectory, maintaining momentum and avoiding sudden reversals that disrupt the learned movement.

Terminology

Summary

Humanoid Manipulation Interface (HuMI) is a portable and efficient framework for learning diverse whole-body manipulation tasks across various environments. p. 0

Where the paper says it

Consequently, demonstrated autonomous skills remain limited and are typically restricted to controlled environments. In this paper, we present the Humanoid Manipulation Interface (HuMI), a portable and efficient framework for learning diverse whole-body manipulation tasks across various environments.

This system enables robot-free data collection by capturing rich whole-body motion using portable hardware. p. 0
Where the paper says it

In this paper, we present the Humanoid Manipulation Interface (HuMI), a portable and efficient framework for learning diverse whole-body manipulation tasks across various environments. HuMI enables robot-free data collection by capturing rich whole-body motion using portable hardware.

Core Framework and Data Collection

HuMI consists of a robot-free demonstration system and a hierarchical policy learning framework. The demonstration system uses handheld sensorized grippers and basestation-free wearable pose trackers to capture whole-body trajectories. p. 0

Where the paper says it

To this end, we propose the Humanoid Manipulation Interface (HuMI) (Fig. 1), a data-collection and learning framework with the following advantages: Robot-free, portable, and efficient data collection: Our system requires only handheld sensorized grippers and basestation-free wearable pose trackers.

This data serves two purposes: visual observations and task-space SE(3) trajectories train the high-level policy, while whole-body IK solutions provide reference motions for the low-level controller. p. 2
Where the paper says it

A real-time IK preview interface enables human-in-the-loop kinematic adaptation. (c) Data Processing: Collected data serves two purposes: visual observations and task-space SE(3) trajectories train the high-level policy, while whole-body IK solutions provide reference motions for the low-level controller.

Addressing Embodiment Gaps

The framework systematically addresses key challenges in humanoid whole-body manipulation. The authors tackle underspecified demonstrations by recording trajectories for the base (pelvis) and feet and using inverse kinematics (IK) to augment these into full robot degrees of freedom. p. 1

Where the paper says it

To address this, we record trajectories not only for the grippers but also for the base (pelvis) and feet. We then use inverse kinematics (IK) to augment these trajectories into full robot degrees of freedom.

To ensure kinematic feasibility, an online IK preview interface allows human-in-the-loop adaptation, enabling demonstrators to adjust movements in real-time. p. 2
Where the paper says it

To ensure feasibility, we incorporate a human-in-the-loop adaptation mechanism via an online IK preview interface (see Fig. By visualizing the virtual robot’s kinematic motion in real-time, operators can adjust their demonstrations on the fly to satisfy both feasibility and task constraints.

Controller Refinements for Precision

The low-level whole-body controller is designed to maximize precision without sacrificing stability. This is achieved through two mechanisms: adaptive tracking rewards and variable-speed augmentation. p. 3

Where the paper says it

it counterproductive: over-prioritizing end-effector precision leads to neglecting whole-body coordination, which actually compromises stability and impairs overall task performance (see Appendix C). To maintain coordination while seeking high precision, we introduce two mechanisms: adaptive tracking rewards and variable-speed augmentation.

The adaptive end-effector reward dynamically scales precision tolerance based on reference end-effector speed, ensuring high accuracy during slow interactions while granting flexibility as velocity increases. p. 3
Where the paper says it

Inspired by this intuition, we dynamically scale precision tolerance for the end-effectors: requiring high accuracy during slow interactions but granting greater flexibility as velocity increases. The adaptive endeffector reward is defined as: rEE = I v ref base < δ · X χ∈{p,θ} r e¯ EE χ , σχ v ref EE , where the dynamic scaling term σχ(ˆvEE) is linearly interpolated between [σ min χ , σmax χ ] based on reference end-effector speed.

Policy Interface Robustness

To ensure robust whole-body execution, two modifications are introduced to the policy interface. First, using the previous target pose as the action reference instead of the actual executed pose prevents action chunk discontinuity by maintaining a smooth, continuous trajectory. p. 4

Where the paper says it

Naively anchoring the next action chunk to the current executed pose results in a sudden trajectory reversal (red line), disrupting momentum. By instead using the previous scheduled target as the reference frame, the policy produces a smooth, continuous trajectory (green line) that maintains the intended motion profile.

Second, tracking relative transforms for non-vision-grounded keypoints mitigates unrecoverable errors during inference by discarding absolute tracking in favor of relative transforms within the current action chunk. p. 4
Where the paper says it

Because the absolute height (left axis) is corrupted by cumulative error, we discard absolute tracking in favor of relative transforms within the chunk (right axis). Relative pose tracking for non-vision-grounded keypoints.

Empirical Validation and Efficiency

Extensive experiments across five whole-body tasks demonstrate HuMI’s high data-collection efficiency and strong task success rates. The system achieves a 3x increase in data-collection throughput compared to teleoperation and attains a 70% success rate in unseen environments. p. 0

Where the paper says it

This data drives a hierarchical learning pipeline that translates human motions into dexterous and feasible humanoid skills. Extensive experiments across five whole-body tasks—including kneeling, squatting, tossing, walking, and bimanual manipulation—demonstrate that HuMI achieves a 3x increase in data collection efficiency compared to teleoperation and attains a 70% success rate in unseen environments.

Furthermore, HuMI successfully collected 50 demonstrations within 15 minutes for the marriage proposal motion with a 100% acceptance rate. p. 7
Where the paper says it

As shown in Fig. 7 (bottom), HuMI successfully collected 50 demonstrations within 15 minutes with a 100% acceptance rate, averaging just 18 s per episode.

The robot-free workflow yields substantially higher throughput than the state-of-the-art humanoid teleoperation system TWIST2, collecting 62 episodes versus 28 with TWIST2. p. 7
Where the paper says it

7 (upper), HuMI yields substantially higher throughput, collecting 62 episodes versus 28 with TWIST2, while also achieving a higher acceptance rate (96. HuMI’s streamlined, robotfree workflow further reduces the average time per acceptable episode to 30.

Generalization Capabilities

HuMI demonstrates strong generalization ability by gathering robot-free demonstrations across diverse real-world scenes. The learned policy achieves a 70% success rate in unseen environments with unseen objects, successfully completing 14/20 episodes in four new scenes and six novel items. p. 7

Where the paper says it

We further test on six novel items, including bottles and bottle-like objects absent from the training set. Across all trials, HuMI successfully completes 14/20 = 70% episodes, maintaining reliable performance even in a dim stairwell and on out-of-distribution objects such as a vase whose shape and texture differ markedly from the training bottles.

This confirms that policies trained on in-the-wild data can genuinely generalize to unseen configurations. p. 6
Where the paper says it

In contrast, HuMI gathers robot-free demonstrations across diverse real-world scenes. We thus ask whether policies trained on such in-thewild data can genuinely generalize to unseen configurations.

Data Collection Throughput Comparison

The comparison between HuMI and TWIST2 workflows shows that HuMI yields substantially higher throughput, collecting 62 episodes versus 28 with TWIST2. HuMI’s streamlined, robot-free workflow further reduces the average time per acceptable episode to 30. p. 7

Where the paper says it

HuMI’s streamlined, robotfree workflow further reduces the average time per acceptable episode to 30. 0% of that of TWIST2, indicating that users can obtain dense, high-quality datasets much more quickly with our robot-free pipeline.

0% of that of TWIST2. p. 4
Where the paper says it

To study these questions, we design five representative whole-body manipulation tasks and evaluate HuMI under both in-domain and out-of-domain settings with respect to environments and objects. To assess data-collection efficiency, we further compare HuMI against the state-of-the-art humanoid teleoperation system TWIST2[54].

This highlights the advantage of HuMI’s robot-free data collection in capturing diverse, highly articulated wholebody motions that go beyond existing humanoid teleoperation setups. p. 7
Where the paper says it

Conversely, TWIST2 failed to produce any usable demonstrations, as the teleoperated humanoid cannot reliably realize the required deep kneeling and often lose stability. This underscores a key advantage of HuMI’s robot-free data collection: it can capture diverse, highly articulated wholebody motions that go beyond the control limitations of existing humanoid teleoperation setups, providing broad coverage of complex behaviors for downstream policy learning.

Controller Training Details

The low-level controller is trained in a teacher–student framework using the DAgger algorithm. The student policy operates on real-world states and keypoint trajectories, strictly aligning with the high-level policy output. p. 1

Where the paper says it

First, we collect human demonstration data in the form of whole-body trajectories and image observations. These data are used to train the high-level manipulation policy, in which a Diffusion Policy [7] maps image observations to actions represented as target keypoint trajectories.

Reward design decomposes the low-level reward into a tracking term and a regularization term. p. 3
Where the paper says it

The adaptive endeffector reward is defined as: rEE = I v ref base < δ · X χ∈{p,θ} r e¯ EE χ , σχ v ref EE , where the dynamic scaling term σχ(ˆvEE) is linearly interpolated between [σ min χ , σmax χ ] based on reference end-effector speed. This reward is further gated by I(·), which deactivates end-effector tracking when the reference base velocity exceeds δ to prioritize stability during rapid movement.

Domain randomization includes physical, velocity, reset, and speed randomization to expose the policy to out-of-distribution issues during deployment. p. 7
Where the paper says it

Prior research predominantly relies on sim-to-real RL [15, 25, 27, 46] or teleoperation [2, 4, 5, 9, 14, 21, 22, 52–54], yet both entail substantial overhead. Sim-to-real necessitates intricate reward shaping and domain randomization [30, 34, 40], while teleoperation imposes challenges for managing balance and compensating tracking errors.

High-Level Policy Architecture

The high-level manipulation policy is implemented using a Diffusion Policy [7] as the backbone for manipulation. This component conditions on both visual and proprioceptive observations, defining the desired positions and rotations of keypoints along with gripper widths. p. 1

Where the paper says it

3, by integrating the high-level policy with the low-level controller, the resulting system enables humanoid whole-body manipulation using observations from onboard sensors. In the following sections, we describe each component and their integration in detail.

The training loss for this policy is Flow Matching. p. 2
Where the paper says it

This approach further benefits downstream learning by representing demonstrations with full-body degrees of freedom, providing comprehensive supervision for low-level controller training (see Fig. Manipulation-Centric Whole-Body Controller To execute target trajectories from the high-level policy, we train a reinforcement learning controller in simulation to track whole-body reference motions.

Limitations

Despite its efficacy, limitations remain, including reliance on visual trackers which require sufficient environmental texture and lighting. Additionally, the low-level controllers are not yet generalpurpose, and evaluation was limited to a single platform. p. 8

Where the paper says it

Second, while training configurations are unified, our low-level controllers are not yet generalpurpose. Finally, evaluation was limited to a single platform [41]; however, we anticipate the framework can extend to other humanoids with minimal modification.

The framework is anticipated to extend to other humanoids with minimal modification. p. 8
Where the paper says it

Second, while training configurations are unified, our low-level controllers are not yet generalpurpose. Finally, evaluation was limited to a single platform [41]; however, we anticipate the framework can extend to other humanoids with minimal modification.

Acknowledgments

This work is supported by Shanghai Qi Zhi Institute & Spirit AI Innovation Program and the Tsinghua University Dushi Program. The authors thank Fanqi Lin and Gu Zhang for their contributions to the conceptual development. p. 8

Where the paper says it

ACKNOWLEDGMENTS This work is supported by Shanghai Qi Zhi Institute & Spirit AI Innovation Program and the Tsinghua University Dushi Program. We thank Fanqi Lin and Gu Zhang for their contributions to the conceptual development.

They also thank Chengbo Yuan and Rui Zhou for early discussions regarding the hardware setup, and Haoyang Weng for assistance with controller training. p. 8
Where the paper says it

We thank Fanqi Lin and Gu Zhang for their contributions to the conceptual development. We also thank Chengbo Yuan and Rui Zhou for early discussions regarding the hardware setup, and Haoyang Weng for assistance with controller training.

Finally, they are grateful to Jiacheng You for invaluable discussions concerning the experimental design. p. 8
Where the paper says it

We also thank Chengbo Yuan and Rui Zhou for early discussions regarding the hardware setup, and Haoyang Weng for assistance with controller training. Finally, we are grateful to Jiacheng You for invaluable discussions concerning the experimental design.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Whole-body Skill Transfer via HuMI framework: The improved system can learn diverse whole-body manipulation tasks across many environments by enabling robot-free data collection, which allows for teaching without the physical presence of a robot.

  2. Enhanced Data Collection Throughput: The system can acquire whole-body manipulation data efficiently, achieving a 3x increase in data-collection efficiency compared to teleoperation and reaching a 70% success rate in unseen environments.

  3. Kinematically Feasible Demonstration Generation: By incorporating an online IK preview interface, the system ensures that collected demonstrations are both task-compliant and executable by allowing operators to intuitively adjust their movements, ensuring the collected data is both task-compliant and executable.

  4. Robust Whole-Body Tracking Controller: The low-level controller can execute target trajectories with high precision by implementing a manipulation-centric whole-body controller designed to maximize precision without sacrificing stability, utilizing adaptive tracking rewards and variable speed augmentation.

  5. Continuous Action Chunking: The policy interface can maintain motion continuity during execution by using the previous target pose as the action reference instead of the actual executed pose, which prevents sudden trajectory reversal (red line), disrupting momentum.

  6. Flexible Keypoint Configuration: The system can handle diverse keypoint requirements by tracking relative transforms for blind points like the pelvis or feet, mitigating errors where absolute tracking becomes unreliable, and demonstrating robust performance even when scaled to five keypoints.

Abstract

Current approaches for humanoid whole-body manipulation, primarily relying on teleoperation or visual sim-to-real reinforcement learning, are hindered by hardware logistics and complex reward engineering. Consequently, demonstrated autonomous skills remain limited and are typically restricted to controlled environments. In this paper, we present the Humanoid Manipulation Interface (HuMI), a portable and efficient framework for learning diverse whole-body manipulation tasks across various environments. HuMI enables humanoid-free data collection by capturing rich whole-body motion using portable hardware. This data drives a hierarchical learning pipeline that translates human motions into dexterous and feasible humanoid skills. Extensive experiments across five whole-body tasks--including kneeling, squatting, tossing, walking, and bimanual manipulation--demonstrate that HuMI achieves a 3x increase in data collection efficiency compared to teleoperation and attains a 70% success rate in unseen environments.

Sources

Related papers