HuMI: Humanoid Whole-Body Manipulation from Humanoid-Free Demonstrations
summary
The gist
Humanoid Manipulation Interface (HuMI) is a portable and efficient framework for learning diverse whole-body manipulation tasks across various environments.
In short
HuMI is a robot-free framework that learns complex whole-body manipulation skills by capturing human demonstrations using wearable sensors and handheld grippers. It overcomes embodiment gaps by using inverse kinematics to complete missing body parts and employs a robust low-level controller with adaptive rewards to ensure precise, stable execution of learned motions.
Key concepts
- Robot-free Demonstration System
- This system captures whole-body motion without needing a physical robot. It uses handheld sensorized grippers and wearable pose trackers to record detailed trajectories. This allows for the collection of rich data from humans in various environments, bypassing the need for expensive or complex robotic setups.
- Online IK Preview Interface
- This feature allows human demonstrators to adjust their movements in real-time while recording data. It uses inverse kinematics (IK) to fill in missing body parts like the base and feet of the demonstration. This ensures that the collected demonstrations are kinematically feasible and executable by allowing intuitive, on-the-fly correction.
- Adaptive Tracking Rewards
- This mechanism refines the low-level controller for precision. It dynamically scales how much precision is required based on the speed of movement. This means the system maintains high accuracy during slow interactions while automatically becoming more flexible as the robot moves faster.
- Continuous Action Chunking
- To ensure smooth motion, this policy interface uses the previous target pose as a reference for the current action instead of just tracking where it is executed. This prevents abrupt changes in trajectory, maintaining momentum and avoiding sudden reversals that disrupt the learned movement.
Terminology used across episodes
This episode discusses
- HuMI: Humanoid Whole-Body Manipulation from Humanoid-Free Demonstrations · Paper Radio
- Retargeting Matters: General Motion Retargeting for Humanoid Motion Tracking
- HOMIE: Humanoid Loco-Manipulation with Isomorphic Exoskeleton Cockpit
- Expressive Whole-Body Control for Humanoid Robots
- Open-TeleVision: Teleoperation with Immersive Active Visual Feedback
- Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robots
- In-the-Wild Compliant Manipulation with UMI-FT
- HumanPlus: Humanoid Shadowing and Imitation from Humans
- UMI-on-Air: Embodiment-Aware Guidance for Embodiment-Agnostic Visuomotor Policies
- UMI on Legs: Making Manipulation Policies Mobile with Manipulation-Centric Whole-body Controllers
- OmniH2O: Universal and Dexterous Human-to-Humanoid Whole-Body Teleoperation and Learning
- VIRAL: Visual Sim-to-Real at Scale for Humanoid Loco-Manipulation
- Data Scaling Laws in Imitation Learning for Robotic Manipulation
- AMO: Adaptive Motion Optimization for Hyper-Dexterous Humanoid Whole-Body Control
- BeyondMimic: From Motion Tracking to Versatile Humanoid Control via Guided Diffusion
- OneTwoVLA: A Unified Vision-Language-Action Model with Adaptive Reasoning
- Sim-to-Real Reinforcement Learning for Vision-Based Dexterous Manipulation on Humanoids
- Emergent Active Perception and Dexterity of Simulated Humanoids from Visual Reinforcement Learning
- SONIC: Supersizing Motion Tracking for Natural Humanoid Whole-Body Control
- Humanoid Policy Human Policy
- CAD2RL: Real Single-Image Flight without a Single Real Image
The paper
HuMI: Humanoid Whole-Body Manipulation from Humanoid-Free Demonstrations · Read on arXiv
Ruiqian Nai, Boyuan Zheng, Junming Zhao, Haodong Zhu, Sicong Dai, †Zunhao Chen, †Yihang Hu, †Yingdong Hu, —Tong Zhang, 㹜huan Wen
Tsinghua University · Shanghai Qi Zhi Institute · Spirit.AI Innovation Program
Current approaches for humanoid whole-body manipulation, primarily relying on teleoperation or visual sim-to-real reinforcement learning, are hindered by hardware logistics and complex reward engineering. Consequently, demonstrated autonomous skills remain limited and are typically restricted to controlled environments. In this paper, we present the Humanoid Manipulation Interface (HuMI), a portable and efficient framework for learning diverse whole-body manipulation tasks across various environments. HuMI enables humanoid-free data collection by capturing rich whole-body motion using portable hardware. This data drives a hierarchical learning pipeline that translates human motions into dexterous and feasible humanoid skills. Extensive experiments across five whole-body tasks--including kneeling, squatting, tossing, walking, and bimanual manipulation--demonstrate that HuMI achieves a 3x increase in data collection efficiency compared to teleoperation and attains a 70% success rate in unseen environments.
Transcript
Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.
Rosa: We're looking at the paper: "HuMI: Humanoid Whole-Body Manipulation from Humanoid-Free Demonstrations".
Dev: Here's the gist of the paper. Humanoid Manipulation Interface (HuMI) is a portable and efficient framework for learning diverse whole-body manipulation tasks across various environments.
Rosa: This system enables robot-free data collection by capturing rich whole-body motion using portable hardware.
Dev: Core Framework and Data Collection HuMI consists of a robot-free demonstration system and a hierarchical policy learning framework.
Rosa: The demonstration system uses handheld sensorized grippers and basestation-free wearable pose trackers to capture whole-body trajectories.
Title and authors: Rosa: The title of this paper is HuMI: Humanoid Whole-Body Manipulation from Humanoid-Free Demonstrations, and the authors are Ruiqian Nai, Boyuan Zheng, Junming Zhao, Haodong Zhu, Sicong Dai, Zunhao Chen, Yihang Hu, Yingdong Hu.
Dev: It’s interesting to see this work because it directly addresses the hardware logistics and complex reward engineering issues that usually limit autonomous skills to controlled environments >
Taro: So the core idea is using a robot-free demonstration system, which uses handheld sensorized grippers and wearable pose trackers on the grippers, waist, and feet >
Rosa: Exactly. They eliminate the need to manage real-robot balance or manually compensate for controller tracking errors by having this portable setup fit into a single backpack >
Dev: And they’re reporting that this approach achieves a three times increase in data collection throughput compared to teleoperation, which is pretty significant for getting usable data quickly >
Taro: It’s about making the system accessible, but I wonder if it's robust enough when the world misbehaves or when we try to use it outside of perfect conditions >
The paper's summary: Rosa: To summarize HuMI, they present a framework that moves away from needing perfect physical interaction or massive amounts of labeled data for whole-body skills by collecting robot-free demonstrations >
Dev: They show that this system supports a wide range of humanoid manipulation tasks involving whole-body coordination, precise bimanual actions, dynamic motions, and base mobility >
Taro: The main thing I'm seeing is their generalization capability. They report that with diverse demonstrations collected across many environments, HuMI achieves a seventy percent success rate on unseen objects and environments >
Rosa: That means we don't have to spend months collecting new datasets for every single thing the robot wants to do if it sees something new >
Dev: It really changes how we think about training robots because you're moving away from needing perfect physical interaction or massive amounts of labeled data for whole-body skills >
Taro: I think the main limitation they point out is that relying on visual trackers means you need good lighting and texture in your environment, which limits where you can actually use this system right now >
The paper's improvements: Rosa: They suggest a few key improvements to make this better. One is addressing underspecified demonstrations by using inverse kinematics to augment the base and feet trajectories into full robot degrees of freedom >
Dev: That’s smart because relying solely on gripper poses under-specifies whole-body motion, leading to unnatural postures, so adding those other body parts makes it more realistic >
Taro: And they also introduced an online inverse kinematics preview interface that lets the human in the loop kinematically adapt movements in real time >
Rosa: That allows operators to intuitively adjust their movements while collecting data, ensuring the collected data is both task-compliant and executable >
Dev: On the control side, they use adaptive tracking rewards and variable-speed augmentation to maximize precision without sacrificing stability in the low-level controller >
Taro: And for the policy interface, they use the previous target pose as an action reference instead of the actual executed pose to prevent action chunk discontinuity and maintain a smooth trajectory >
Conclusion: Rosa: So we've covered HuMI, Humanoid Whole-Body Manipulation from Humanoid-Free Demonstrations, and how you can teach a robot complex whole-body stuff without needing a physical robot around during training >
Dev: Right. The efficiency gain is substantial because they achieved a three times increase in throughput compared to teleoperation and hit a seventy percent success rate on unseen objects and environments >
Taro: That generalization is what I'm most interested in. If the policy works on stuff it never saw before, then we don't have to spend months collecting new data every time we want the robot to do something different >
Taro: The main limitation they flag is that relying on visual trackers means you need good lighting and texture in your environment, which limits where you can actually use this system right now >
Dev: Exactly. Moving forward, I think we need to look at how these learned policies interact with things like real-time execution models and safety guarantees as we move toward deployment >
Rosa: This paper on HuMI: Humanoid Whole-Body Manipulation from Humanoid-Free Demonstrations shows a really promising direction for learning whole-body skills through data collected outside of a fully controlled robot environment, even if we still have some practical hurdles to overcome >
More episodes
- 2608.13438-ContactGuard: Pre-Contact Execution Monitoring with Action-Conditioned Latent World Models
- 2610.34593-Future Prediction as Training-Only Video Supervision for Pretrained VLA Policies
- 2610.35385-EmbodiedLab: Beyond Manipulation, Benchmarking Embodied Agents That Act to Know
- 2610.36378-Data-Driven Hamiltonian for Neural Optimal Control
- 2610.37275-A Heuristic for Scheduling Autonomous Vehicles at Intersections
- 2610.37288-Dynamic Coordination of Autonomous Vehicles at Intersections
- 2610.37632-Real-Time Coordination of Autonomous Vehicles at Intersections
- 2611.13779-Softness is something you also need! Why compliance helps control: information metrics of a passive walker with compliant ankle
- 2611.13411-Residual-Corrected Physics-Informed Neural Network KKL Observers with Input-to-State Stability Guarantees
- 2611.14488-Is Your Robot's Memory Just a Clock? Temporal Shortcuts in Memory-Augmented VLA Policies