HuMI: Humanoid Whole-Body Manipulation from Humanoid-Free Demonstrations
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.
Rosa: We're looking at the paper: "HuMI: Humanoid Whole-Body Manipulation from Humanoid-Free Demonstrations".
Dev: Here's the gist of the paper. Humanoid Manipulation Interface (HuMI) is a portable and efficient framework for learning diverse whole-body manipulation tasks across various environments.
Rosa: This system enables robot-free data collection by capturing rich whole-body motion using portable hardware.
Dev: Core Framework and Data Collection HuMI consists of a robot-free demonstration system and a hierarchical policy learning framework.
Rosa: The demonstration system uses handheld sensorized grippers and basestation-free wearable pose trackers to capture whole-body trajectories.
Title and authors: Rosa: The title of this paper is HuMI: Humanoid Whole-Body Manipulation from Humanoid-Free Demonstrations, and the authors are Ruiqian Nai, Boyuan Zheng, Junming Zhao, Haodong Zhu, Sicong Dai, Zunhao Chen, Yihang Hu, Yingdong Hu.
Dev: It’s interesting to see this work because it directly addresses the hardware logistics and complex reward engineering issues that usually limit autonomous skills to controlled environments >
Taro: So the core idea is using a robot-free demonstration system, which uses handheld sensorized grippers and wearable pose trackers on the grippers, waist, and feet >
Rosa: Exactly. They eliminate the need to manage real-robot balance or manually compensate for controller tracking errors by having this portable setup fit into a single backpack >
Dev: And they’re reporting that this approach achieves a three times increase in data collection throughput compared to teleoperation, which is pretty significant for getting usable data quickly >
Taro: It’s about making the system accessible, but I wonder if it's robust enough when the world misbehaves or when we try to use it outside of perfect conditions >
The paper's summary: Rosa: To summarize HuMI, they present a framework that moves away from needing perfect physical interaction or massive amounts of labeled data for whole-body skills by collecting robot-free demonstrations >
Dev: They show that this system supports a wide range of humanoid manipulation tasks involving whole-body coordination, precise bimanual actions, dynamic motions, and base mobility >
Taro: The main thing I'm seeing is their generalization capability. They report that with diverse demonstrations collected across many environments, HuMI achieves a seventy percent success rate on unseen objects and environments >
Rosa: That means we don't have to spend months collecting new datasets for every single thing the robot wants to do if it sees something new >
Dev: It really changes how we think about training robots because you're moving away from needing perfect physical interaction or massive amounts of labeled data for whole-body skills >
Taro: I think the main limitation they point out is that relying on visual trackers means you need good lighting and texture in your environment, which limits where you can actually use this system right now >
The paper's improvements: Rosa: They suggest a few key improvements to make this better. One is addressing underspecified demonstrations by using inverse kinematics to augment the base and feet trajectories into full robot degrees of freedom >
Dev: That’s smart because relying solely on gripper poses under-specifies whole-body motion, leading to unnatural postures, so adding those other body parts makes it more realistic >
Taro: And they also introduced an online inverse kinematics preview interface that lets the human in the loop kinematically adapt movements in real time >
Rosa: That allows operators to intuitively adjust their movements while collecting data, ensuring the collected data is both task-compliant and executable >
Dev: On the control side, they use adaptive tracking rewards and variable-speed augmentation to maximize precision without sacrificing stability in the low-level controller >
Taro: And for the policy interface, they use the previous target pose as an action reference instead of the actual executed pose to prevent action chunk discontinuity and maintain a smooth trajectory >
Conclusion: Rosa: So we've covered HuMI, Humanoid Whole-Body Manipulation from Humanoid-Free Demonstrations, and how you can teach a robot complex whole-body stuff without needing a physical robot around during training >
Dev: Right. The efficiency gain is substantial because they achieved a three times increase in throughput compared to teleoperation and hit a seventy percent success rate on unseen objects and environments >
Taro: That generalization is what I'm most interested in. If the policy works on stuff it never saw before, then we don't have to spend months collecting new data every time we want the robot to do something different >
Taro: The main limitation they flag is that relying on visual trackers means you need good lighting and texture in your environment, which limits where you can actually use this system right now >
Dev: Exactly. Moving forward, I think we need to look at how these learned policies interact with things like real-time execution models and safety guarantees as we move toward deployment >
Rosa: This paper on HuMI: Humanoid Whole-Body Manipulation from Humanoid-Free Demonstrations shows a really promising direction for learning whole-body skills through data collected outside of a fully controlled robot environment, even if we still have some practical hurdles to overcome >
Ruiqian Nai, Boyuan Zheng, Junming Zhao, Haodong Zhu, Sicong Dai, †Zunhao Chen, †Yihang Hu, †Yingdong Hu, —Tong Zhang, 㹜huan Wen
Tsinghua University · Shanghai Qi Zhi Institute · Spirit.AI Innovation Program
cs.RO, cs.AI, cs.LG
Submitted: 2026-02-06
Updated: 2026-11-06
Comments: Website: https://humanoid-manipulation-interface.github.io
Code: https://github.com/thu-ml/RDT2
Project page: https://humanoid-manipulation-interface.github.io
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 85/100
The gist: Humanoid Manipulation Interface (HuMI) is a portable and efficient framework for learning diverse whole-body manipulation tasks across various environments.
Key concepts
- Robot-free Demonstration System
- This system captures whole-body motion without needing a physical robot. It uses handheld sensorized grippers and wearable pose trackers to record detailed trajectories. This allows for the collection of rich data from humans in various environments, bypassing the need for expensive or complex robotic setups.
- Online IK Preview Interface
- This feature allows human demonstrators to adjust their movements in real-time while recording data. It uses inverse kinematics (IK) to fill in missing body parts like the base and feet of the demonstration. This ensures that the collected demonstrations are kinematically feasible and executable by allowing intuitive, on-the-fly correction.
- Adaptive Tracking Rewards
- This mechanism refines the low-level controller for precision. It dynamically scales how much precision is required based on the speed of movement. This means the system maintains high accuracy during slow interactions while automatically becoming more flexible as the robot moves faster.
- Continuous Action Chunking
- To ensure smooth motion, this policy interface uses the previous target pose as a reference for the current action instead of just tracking where it is executed. This prevents abrupt changes in trajectory, maintaining momentum and avoiding sudden reversals that disrupt the learned movement.
Terminology
Summary
Humanoid Manipulation Interface (HuMI) is a portable and efficient framework for learning diverse whole-body manipulation tasks across various environments. p. 0 Consequently, demonstrated
autonomous skills remain limited and are typically restricted to
controlled environments. In this paper, we present the Humanoid
Manipulation Interface (HuMI), a portable and efficient framework for learning diverse whole-body manipulation tasks across
various environments. In this paper, we present the Humanoid
Manipulation Interface (HuMI), a portable and efficient framework for learning diverse whole-body manipulation tasks across
various environments. HuMI enables robot-free data collection
by capturing rich whole-body motion using portable hardware.Where the paper says it
Where the paper says it
Core Framework and Data Collection
HuMI consists of a robot-free demonstration system and a hierarchical policy learning framework. The demonstration system uses handheld sensorized grippers and basestation-free wearable pose trackers to capture whole-body trajectories. p. 0 To this end,
we propose the Humanoid Manipulation Interface (HuMI)
(Fig. 1), a data-collection and learning framework with the
following advantages:
Robot-free, portable, and efficient data collection: Our
system requires only handheld sensorized grippers and basestation-free wearable pose trackers. A real-time IK preview interface enables human-in-the-loop kinematic adaptation. (c) Data Processing: Collected
data serves two purposes: visual observations and task-space SE(3) trajectories train the high-level policy, while whole-body IK solutions
provide reference motions for the low-level controller.Where the paper says it
Where the paper says it
Addressing Embodiment Gaps
The framework systematically addresses key challenges in humanoid whole-body manipulation. The authors tackle underspecified demonstrations by recording trajectories for the base (pelvis) and feet and using inverse kinematics (IK) to augment these into full robot degrees of freedom. p. 1 To address this, we record
trajectories not only for the grippers but also for the base
(pelvis) and feet. We then use inverse kinematics (IK) to
augment these trajectories into full robot degrees of freedom. To ensure feasibility, we incorporate a
human-in-the-loop adaptation mechanism via an online IK
preview interface (see Fig. By visualizing the virtual
robot’s kinematic motion in real-time, operators can adjust
their demonstrations on the fly to satisfy both feasibility and
task constraints.Where the paper says it
Where the paper says it
Controller Refinements for Precision
The low-level whole-body controller is designed to maximize precision without sacrificing stability. This is achieved through two mechanisms: adaptive tracking rewards and variable-speed augmentation. p. 3 it counterproductive: over-prioritizing end-effector precision
leads to neglecting whole-body coordination, which actually
compromises stability and impairs overall task performance
(see Appendix C). To maintain coordination while seeking
high precision, we introduce two mechanisms: adaptive tracking rewards and variable-speed augmentation. Inspired by this intuition, we
dynamically scale precision tolerance for the end-effectors:
requiring high accuracy during slow interactions but granting
greater flexibility as velocity increases. The adaptive endeffector reward is defined as:
rEE = I
v
ref
base
< δ
·
X
χ∈{p,θ}
r
e¯
EE
χ , σχ
v
ref
EE ,
where the dynamic scaling term σχ(ˆvEE) is linearly interpolated between [σ
min
χ , σmax
χ ] based on reference end-effector
speed.Where the paper says it
Where the paper says it
Policy Interface Robustness
To ensure robust whole-body execution, two modifications are introduced to the policy interface. First, using the previous target pose as the action reference instead of the actual executed pose prevents action chunk discontinuity by maintaining a smooth, continuous trajectory. p. 4 Naively anchoring the
next action chunk to the current executed pose results in a sudden
trajectory reversal (red line), disrupting momentum. By instead using
the previous scheduled target as the reference frame, the policy
produces a smooth, continuous trajectory (green line) that maintains
the intended motion profile. Because the absolute
height (left axis) is corrupted by cumulative error, we discard absolute
tracking in favor of relative transforms within the chunk (right axis). Relative pose tracking for non-vision-grounded keypoints.Where the paper says it
Where the paper says it
Empirical Validation and Efficiency
Extensive experiments across five whole-body tasks demonstrate HuMI’s high data-collection efficiency and strong task success rates. The system achieves a 3x increase in data-collection throughput compared to teleoperation and attains a 70% success rate in unseen environments. p. 0 This data drives a hierarchical learning pipeline that translates
human motions into dexterous and feasible humanoid skills. Extensive experiments across five whole-body tasks—including
kneeling, squatting, tossing, walking, and bimanual manipulation—demonstrate that HuMI achieves a 3x increase in data
collection efficiency compared to teleoperation and attains a 70%
success rate in unseen environments. As shown in Fig. 7 (bottom), HuMI successfully collected 50 demonstrations within
15 minutes with a 100% acceptance rate, averaging just 18 s
per episode. 7 (upper), HuMI
yields substantially higher throughput, collecting 62 episodes
versus 28 with TWIST2, while also achieving a higher acceptance rate (96. HuMI’s streamlined, robotfree workflow further reduces the average time per acceptable
episode to 30.Where the paper says it
Where the paper says it
Where the paper says it
Generalization Capabilities
HuMI demonstrates strong generalization ability by gathering robot-free demonstrations across diverse real-world scenes. The learned policy achieves a 70% success rate in unseen environments with unseen objects, successfully completing 14/20 episodes in four new scenes and six novel items. p. 7 We further test on six novel
items, including bottles and bottle-like objects absent from
the training set. Across all trials, HuMI successfully completes
14/20 = 70% episodes, maintaining reliable performance even
in a dim stairwell and on out-of-distribution objects such as a
vase whose shape and texture differ markedly from the training
bottles. In contrast, HuMI
gathers robot-free demonstrations across diverse real-world
scenes. We thus ask whether policies trained on such in-thewild data can genuinely generalize to unseen configurations.Where the paper says it
Where the paper says it
Data Collection Throughput Comparison
The comparison between HuMI and TWIST2 workflows shows that HuMI yields substantially higher throughput, collecting 62 episodes versus 28 with TWIST2. HuMI’s streamlined, robot-free workflow further reduces the average time per acceptable episode to 30. p. 7 HuMI’s streamlined, robotfree workflow further reduces the average time per acceptable
episode to 30. 0% of that of TWIST2, indicating that users
can obtain dense, high-quality datasets much more quickly
with our robot-free pipeline. To study these questions, we design five representative
whole-body manipulation tasks and evaluate HuMI under both
in-domain and out-of-domain settings with respect to environments and objects. To assess data-collection efficiency, we
further compare HuMI against the state-of-the-art humanoid
teleoperation system TWIST2[54]. Conversely, TWIST2 failed to produce any usable
demonstrations, as the teleoperated humanoid cannot reliably
realize the required deep kneeling and often lose stability. This underscores a key advantage of HuMI’s robot-free data
collection: it can capture diverse, highly articulated wholebody motions that go beyond the control limitations of existing
humanoid teleoperation setups, providing broad coverage of
complex behaviors for downstream policy learning.Where the paper says it
Where the paper says it
Where the paper says it
Controller Training Details
The low-level controller is trained in a teacher–student framework using the DAgger algorithm. The student policy operates on real-world states and keypoint trajectories, strictly aligning with the high-level policy output. p. 1 First, we collect human demonstration data in
the form of whole-body trajectories and image observations. These data are used to train the high-level manipulation policy,
in which a Diffusion Policy [7] maps image observations
to actions represented as target keypoint trajectories. The adaptive endeffector reward is defined as:
rEE = I
v
ref
base
< δ
·
X
χ∈{p,θ}
r
e¯
EE
χ , σχ
v
ref
EE ,
where the dynamic scaling term σχ(ˆvEE) is linearly interpolated between [σ
min
χ , σmax
χ ] based on reference end-effector
speed. This reward is further gated by I(·), which deactivates
end-effector tracking when the reference base velocity exceeds
δ to prioritize stability during rapid movement. Prior research predominantly
relies on sim-to-real RL [15, 25, 27, 46] or teleoperation
[2, 4, 5, 9, 14, 21, 22, 52–54], yet both entail substantial
overhead. Sim-to-real necessitates intricate reward shaping
and domain randomization [30, 34, 40], while teleoperation
imposes challenges for managing balance and compensating
tracking errors.Where the paper says it
Where the paper says it
Where the paper says it
High-Level Policy Architecture
The high-level manipulation policy is implemented using a Diffusion Policy [7] as the backbone for manipulation. This component conditions on both visual and proprioceptive observations, defining the desired positions and rotations of keypoints along with gripper widths. p. 1 3, by
integrating the high-level policy with the low-level controller,
the resulting system enables humanoid whole-body manipulation using observations from onboard sensors. In the following
sections, we describe each component and their integration in
detail. This approach further
benefits downstream learning by representing demonstrations
with full-body degrees of freedom, providing comprehensive
supervision for low-level controller training (see Fig. Manipulation-Centric Whole-Body Controller
To execute target trajectories from the high-level policy, we
train a reinforcement learning controller in simulation to track
whole-body reference motions.Where the paper says it
Where the paper says it
Limitations
Despite its efficacy, limitations remain, including reliance on visual trackers which require sufficient environmental texture and lighting. Additionally, the low-level controllers are not yet generalpurpose, and evaluation was limited to a single platform. p. 8 Second, while training configurations are unified, our low-level controllers are not yet generalpurpose. Finally, evaluation was limited to a single platform
[41]; however, we anticipate the framework can extend to other
humanoids with minimal modification. Second, while training configurations are unified, our low-level controllers are not yet generalpurpose. Finally, evaluation was limited to a single platform
[41]; however, we anticipate the framework can extend to other
humanoids with minimal modification.Where the paper says it
Where the paper says it
Acknowledgments
This work is supported by Shanghai Qi Zhi Institute & Spirit AI Innovation Program and the Tsinghua University Dushi Program. The authors thank Fanqi Lin and Gu Zhang for their contributions to the conceptual development. p. 8 ACKNOWLEDGMENTS
This work is supported by Shanghai Qi Zhi Institute & Spirit
AI Innovation Program and the Tsinghua University Dushi
Program. We thank Fanqi Lin and Gu Zhang for their contributions
to the conceptual development. We thank Fanqi Lin and Gu Zhang for their contributions
to the conceptual development. We also thank Chengbo Yuan
and Rui Zhou for early discussions regarding the hardware
setup, and Haoyang Weng for assistance with controller training. We also thank Chengbo Yuan
and Rui Zhou for early discussions regarding the hardware
setup, and Haoyang Weng for assistance with controller training. Finally, we are grateful to Jiacheng You for invaluable
discussions concerning the experimental design.Where the paper says it
Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Whole-body Skill Transfer via HuMI framework: The improved system can learn diverse whole-body manipulation tasks across many environments by enabling robot-free data collection, which allows for
teaching without the physical presence of a robot.
-
Enhanced Data Collection Throughput: The system can acquire whole-body manipulation data efficiently, achieving a
3x increase in data-collection efficiency compared to teleoperation
and reaching a70% success rate in unseen environments.
-
Kinematically Feasible Demonstration Generation: By incorporating an online IK preview interface, the system ensures that collected demonstrations are both
task-compliant and executable
by allowing operators tointuitively adjust their movements, ensuring the collected data is both task-compliant and executable.
-
Robust Whole-Body Tracking Controller: The low-level controller can execute target trajectories with high precision by implementing a manipulation-centric whole-body controller designed to
maximize precision without sacrificing stability,
utilizing adaptive tracking rewards and variable speed augmentation. -
Continuous Action Chunking: The policy interface can maintain motion continuity during execution by using the
previous target pose as the action reference
instead of the actual executed pose, which preventssudden trajectory reversal (red line), disrupting momentum.
-
Flexible Keypoint Configuration: The system can handle diverse keypoint requirements by tracking relative transforms for
blind
points like the pelvis or feet, mitigating errors where absolute tracking becomes unreliable, and demonstrating robust performance even when scaled to five keypoints.
Abstract
Current approaches for humanoid whole-body manipulation, primarily relying on teleoperation or visual sim-to-real reinforcement learning, are hindered by hardware logistics and complex reward engineering. Consequently, demonstrated autonomous skills remain limited and are typically restricted to controlled environments. In this paper, we present the Humanoid Manipulation Interface (HuMI), a portable and efficient framework for learning diverse whole-body manipulation tasks across various environments. HuMI enables humanoid-free data collection by capturing rich whole-body motion using portable hardware. This data drives a hierarchical learning pipeline that translates human motions into dexterous and feasible humanoid skills. Extensive experiments across five whole-body tasks--including kneeling, squatting, tossing, walking, and bimanual manipulation--demonstrate that HuMI achieves a 3x increase in data collection efficiency compared to teleoperation and attains a 70% success rate in unseen environments.
Sources
- Retargeting Matters: General Motion Retargeting for Humanoid Motion Tracking
- HOMIE: Humanoid Loco-Manipulation with Isomorphic Exoskeleton Cockpit
- Expressive Whole-Body Control for Humanoid Robots
- Open-TeleVision: Teleoperation with Immersive Active Visual Feedback
- Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robots
- In-the-Wild Compliant Manipulation with UMI-FT
- HumanPlus: Humanoid Shadowing and Imitation from Humans
- UMI-on-Air: Embodiment-Aware Guidance for Embodiment-Agnostic Visuomotor Policies
- UMI on Legs: Making Manipulation Policies Mobile with Manipulation-Centric Whole-body Controllers
- OmniH2O: Universal and Dexterous Human-to-Humanoid Whole-Body Teleoperation and Learning
- VIRAL: Visual Sim-to-Real at Scale for Humanoid Loco-Manipulation
- Data Scaling Laws in Imitation Learning for Robotic Manipulation
- AMO: Adaptive Motion Optimization for Hyper-Dexterous Humanoid Whole-Body Control
- BeyondMimic: From Motion Tracking to Versatile Humanoid Control via Guided Diffusion
- OneTwoVLA: A Unified Vision-Language-Action Model with Adaptive Reasoning
- Sim-to-Real Reinforcement Learning for Vision-Based Dexterous Manipulation on Humanoids
- Emergent Active Perception and Dexterity of Simulated Humanoids from Visual Reinforcement Learning
- SONIC: Supersizing Motion Tracking for Natural Humanoid Whole-Body Control
- Humanoid Policy ~ Human Policy
- CAD2RL: Real Single-Image Flight without a Single Real Image
Related papers
- Rethinking Actions: Continuity-Aware Priors for World-Model Planning
- MOSAIC: Modular Obstacle and Spatial-Aware Framework for Manipulation In Cluttered Scenes
- SAGAS: Semantic-Aware Graph-Assisted Stitching for Offline Temporal Logic Planning
- DexSeed: Expanding Human Hand-Object Interactions into Robot Demonstrations for Dexterous Policy Learning
- LaST0.5: A Generalist World Action Model with Interaction-Centric Latent Reasoning
- CIALLO: Learning Shared Intent Representations for Cross-Embodiment Manipulation