HumanX: Toward Agile and Generalizable Humanoid Interaction Skills from Human Videos

arXiv:2602.02473 · cs.RO, cs.LG · Submitted 2026-02-02 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: I'm Rosa, and with me are Dev and Taro, guest researcher.

Dev: Today's paper: "HumanX: Toward Agile and Generalizable Humanoid Interaction Skills from Human Videos".

Rosa: The gist The HumanX framework compiles human video into generalizable, real-world interaction skills for humanoids without task-specific rewards.

Dev: First, who's behind it and why it matters.

Title and authors: Rosa: So we're looking at the HumanX paper today, which is about building this full-stack framework to get agile and generalizable interaction skills for humanoids from just human videos, without needing task-specific rewards.

Dev: Exactly. The core idea here is that instead of relying on a ton of specific data or tedious reward tuning for every single skill, we compile all that video into general skills the robot can use anywhere.

Taro: I'm interested in how this moves beyond just mimicking a single action and gets into something more versatile, like handling things when the world throws us unexpected stuff.

Rosa: Well, HumanX uses two main parts to do this: XGen for generating data, and XMimic for learning the skills from that data.

Dev: Right, so XGen is this pipeline that takes a video and synthesizes physically plausible interaction data based on physics priors rather than just trying to perfectly reconstruct what happened in the video.

Taro: So if it’s not exact reconstruction, how does that help with things like when the robot bumps into something or grabs an object?

Rosa: XGen segments the video into contact and non-contact phases. During contact, it uses a predefined anchor point and then generates object trajectories by transforming the object's pose along an anchor trajectory derived from the robot’s motion, followed by force-closure optimization to keep it physically plausible during that touch.

Dev: And for those non-contact phases, XGen just simulates diverse and physically plausible object trajectories while also augmenting the data through things like scaling object geometry and varying trajectories to cover more ground.

Taro: That sounds like it tackles the problem of data scarcity by making sure the AI sees a wide variety of contact scenarios even if we only give it one video to start with.

Rosa: Right, and that’s where XMimic comes in. XMimic is this unified imitation learning framework that learns those generalizable interaction skills purely by mimicking the behaviors synthesized by XGen.

Dev: It uses a two-stage teacher-student approach to first master individual skills with some privilege information, and then consolidate them into one unified deployable policy for the student model.

Taro: What kind of reward system are they using when training this, since we're not using task-specific rewards?

Rosa: They use a composite reward function defined as rt = rbody t + robj t + rrel t + rc t + rreg t. This breaks down into body imitation rewards tracking position, rotation, and joint positions, an object reward for accurate state tracking, a relative motion reward encouraging correct spatial relationships between the body and object, a contact reward penalizing deviations from the reference contact graph, and regularization terms for stability.

Title and authors: Dev: That contact reward is pretty interesting because it’s computed based on the discrepancy between the simulated contact state and the reference contact state using a Contact Graph. It's like checking if the robot actually made physical sense when it touched something, not just if it looked right in a video.

Taro: So what does this mean for someone who only listens to this show? It means they aren't training the robot to do one specific basketball shot or one specific pickup; they’re training it to understand the general *concept* of interacting with an object.

Rosa: Right, and that generalization is bolstered by three mechanisms. First, you get diverse offline data from XGen covering a broad distribution of object states. Second, they use online augmentation through disturbed initialization during training to expand state coverage further. And third, they have interaction-aware termination which prioritizes interaction success and discourages overfitting just to the body motion alone.

Dev: They also mention applying domain randomization to various physical properties like object size and mass during training, which helps with that generalization issue when moving from simulation to the real world.

Taro: That sounds like a solid way to prevent it from getting stuck in one specific setup, even if the initial video was a bit limited.

Rosa: And for deployment, they have two practical schemes. There's a No External Perception mode that relies only on proprioception for dynamic interaction, and there's a MoCap-based mode that uses external object poses during training, which even includes realistic simulation of frame loss to enable zero-shot adaptation.

Dev: That MoCap integration with frame loss is smart because it simulates real-world sensor noise or dropouts that the robot might actually face when it’s deployed.

Taro: So, if a person just listens to this, they can imagine a robot learning complex things like basketball maneuvers or sustained human-robot passing sequences over ten consecutive cycles from just watching someone do them once.

Rosa: That’s exactly what the experiments show. They tested it across five domains: basketball, football, badminton, cargo pickup, and reactive fighting. On a Unitree G1 robot, they showed two things: first, perception-free execution of complex maneuvers like pump-fake turnaround fadeaways without any external sensing.

Dev: And second, with object sensing from a MoCap system in the MoCap mode, it achieved sustained closed-loop interactions over ten consecutive human-robot passes. Those are some pretty concrete results to look at.

Title and authors: Taro: What about the autonomy part? When the world misbehaves, how does this system react? Does it just fail or can it recover?

Rosa: The learned skills exhibit a high degree of autonomy and interesting emergent recovery behaviors, like distinguishing feints from real attacks during fighting. It's not just following a script; it’s showing some adaptive thinking.

Dev: The policy’s observation space even incorporates variables from the dynamics equation J TextFext = τ - Mq¨ + Cq˙ + G + τ f, meaning it's looking at all the relevant physical factors to make its decisions.

Taro: So, the naive question a listener would ask is how long can this system actually work reliably outside of a controlled lab setting before we have to worry about the data synthesis or the generalization breaking down?

Rosa: That’s a big one. They tested it on a Unitree G1 robot, and they showed performance in both perception-free and MoCap modes, suggesting these deployment schemes are viable ways to get it running in real environments.

Dev: But they did flag that the system relies heavily on the quality of the input video data for skill acquisition; if that video is poor, the resulting skills might not generalize well outside of that specific distribution.

Taro: And what about limitations? They mentioned that while it’s good at skill transfer, it doesn't inherently solve problems related to truly novel physical interactions that are completely outside the scope of the synthesized training data.

Rosa: That's a fair point. The system is built to compile and generalize skills from human video, so its strength lies in what’s in that video distribution. It doesn't magically invent entirely new interaction physics on its own.

Dev: So, to wrap up the HumanX paper, it’s a full-stack framework combining XGen for physically plausible data synthesis and XMimic for robust imitation learning from videos without task-specific rewards.

Taro: For someone listening to the show, this means we have a scalable pathway now for getting versatile, real-world robot interactive skills just by showing it what humans do in video.

Rosa: That’s the big picture we’re looking at today with HumanX. It shows that you can learn complex, task-agnostic interactions without needing a massive dataset for every single thing you want to teach a robot.

Dev: We'll keep an eye on how they tackle those generalization gaps in the next few papers.

Taro: Alright, I think that’s where we’ll leave it for today.

The paper's summary: Rosa: So, to wrap up what we just talked about, HumanX is this framework that takes raw human video and compiles it into these general interaction skills for robots without needing specific rewards for every single task.

Dev: Right, it’s all about making those skills versatile enough to be used in different situations.

Rosa: The paper shows they built two main parts for this, XGen which creates the training data, and XMimic which learns the actual policies from that data.

Dev: And what I find really interesting is how XGen doesn't just try to perfectly copy the video; it uses physics to create plausible interaction paths instead.

Rosa: Exactly. It focuses on generating diverse and physically sound robot interaction data, even when the video itself only shows one person doing one thing.

Dev: That’s a big deal for real-world robotics because in reality, things don't always look perfect or follow textbook physics exactly.

Rosa: And XMimic uses a teacher student setup to learn those general skills by mimicking what XGen produces, and they use this composite reward function to guide the learning process.

Dev: That reward function is pretty detailed, breaking down body movement, object tracking, and even how well the contact actually matches the reference graph.

Rosa: It’s a lot of math packed into that one equation to make sure the robot gets it right on a physical level.

Dev: And they show that this approach gives them over eight times better generalization success compared to other methods they looked at before.

Rosa: That means when you show the robot one basketball skill, it can actually perform a wide variety of basketball skills without needing new training for each one.

Dev: It’s moving away from teaching robots specific tricks and toward teaching them the underlying concept of interaction itself.

Rosa: And for deployment, they have options like a perception-free mode that only uses what the robot senses on its own, or a mode that uses external object information during training.

Dev: That's smart because it shows they've thought about how this system could actually be used in different kinds of real-world robot setups.

Rosa: So, the main thing this paper does is show a scalable way to get robots capable of doing complex, varied interactions just by watching people in videos.

Dev: It suggests that we don't always need massive datasets for every single task if we can synthesize the data smartly from video.

Rosa: And they’re testing it across five different areas, from basketball to fighting, showing a lot of versatility.

Dev: We should keep an eye on how they handle those generalization gaps when the robot runs into something completely new that wasn't in the training data.

The paper's improvements: Taro: So we just talked about how HumanX compiles video into general skills using XGen for data and XMimic for learning, but now let's talk about what they actually improved in this version.

Rosa: The paper highlights a few key upgrades to make the whole system more robust.

Dev: Yeah, they added domain randomization to the training process. That means they intentionally mess with physical properties like object mass and size during training so the robot isn't just trained on one specific setup.

Taro: Which is good because it helps prevent overfitting when you try to use the robot in a real environment where things are always slightly different.

Rosa: Exactly. They also introduced interaction-aware termination, which stops the learning episode if things aren't going well or if the robot isn't actually making progress in interacting with the object.

Dev: That’s important for stability and avoiding endless, useless training loops where the robot just keeps trying to do something that won't work.

Taro: And they also used an online augmentation technique, which is like throwing a little bit of noise into the initial setup while training to make sure the model learns from a wider range of starting conditions.

Rosa: Right, so they’re basically making the system smarter about how it learns and how it stops learning.

Dev: Plus, they refined their reward function even more by adding regularization terms to keep the motion stable during those interactions.

Taro: That extra stabilization helps when you’re dealing with complex physical contacts because it forces the policy to be smoother.

Rosa: And in terms of deployment, they clarified how it can run without external sensors, showing that the perception-free mode is actually quite capable of doing complex things like those fancy basketball maneuvers.

Dev: It means we could potentially use this in a lot of situations where we don't have high-end vision systems available on the robot itself.

Taro: But one thing they flagged, and I think it’s important, is that while the skill transfer is strong across ten different skills, it still relies heavily on the quality of that initial human video input.

Rosa: That’s true. If you feed it bad data, you get bad skills; so the system isn't a magic solution for every single interaction scenario ever conceived.

Dev: So, the next step for this research is likely to see if they can improve that dependency on the initial video by making XGen even better at synthesizing truly novel interactions.

Conclusion: Rosa: So we’re wrapping up on HumanX, which is this paper about building that full-stack framework to get agile interaction skills for humanoids from just human videos without needing task-specific rewards.

Dev: Yeah, it shows a way to move beyond just mimicking specific actions and toward general skills the robot can use anywhere.

Rosa: The main takeaway is that you can compile monocular video into versatile interaction skills by using XGen to generate plausible data and XMimic to learn the policies from it.

Dev: It suggests that we don't always need massive datasets for every single thing we want to teach a robot if we can synthesize the data smartly from video.

Rosa: And they showed that this method gets over eight times higher generalization success across ten different skills compared to previous work.

Dev: That’s a big number because it means the skills aren't just copied; they actually transfer well to new situations.

Taro: I think what stands out is how the system can handle unexpected things, like when you have to distinguish a feint from a real attack during fighting in those learned skills.

Rosa: That’s true, it’s not just following a script; it shows some emergent recovery behaviors when the world misbehaves.

Dev: From an engineering standpoint, the perception-free mode is pretty cool because it lets the robot operate without needing complex external sensors for every single move.

Taro: But that only works if the skill is learned well enough to handle things without seeing what's happening outside of its immediate physical space.

Rosa: The caveat they mentioned, though, is that while the skill transfer is strong across these ten skills, it still relies heavily on the quality of that initial human video input.

Dev: That’s a fair point; if the starting video is poor or limited in scope, the resulting skills won't generalize well beyond what’s in that specific training data distribution.

Taro: So it’s not a universal solution for every robot problem yet, but it’s definitely a strong pathway for acquiring versatile real-world interactive skills from human demonstration videos.

Rosa: Exactly. HumanX provides a scalable way to teach robots complex, task-agnostic interactions just by showing them what humans do in video without needing specific rewards for each one.

Dev: It's a full stack solution combining data synthesis and imitation learning that gives us tools to train policies that are more robust and adaptable.

Taro: I think the real future work is going to be about making XGen even better at synthesizing those truly novel physical interactions that aren't just simple contact sequences.

Rosa: Definitely. That’s where we’ll be looking next—how they handle those gaps when the robot runs into something completely new that wasn't in the training data.

The Hong Kong University of Science and Technology Shanghai AI Laboratory

cs.RO, cs.LG

Submitted: 2026-02-02

Updated: 2026-10-09

Project page: https://wyhuai.github.io/human-x

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 86/100

The gist: The gist The HumanX framework compiles human video into generalizable, real-world interaction skills for humanoids without task-specific rewards.

Key concepts

XGen Data Generation Pipeline
This component takes human video and synthesizes physically plausible robot interaction data. It extracts human motion and retargets it, then uses physics-based synthesis coupled with contact refinement to create realistic trajectories. Finally, it augments this data by scaling object geometry to ensure the training set is diverse and scalable for generalization.
XMimic Imitation Learning Framework
XMimic is a unified imitation learning system that learns interaction skills by mimicking the data generated by XGen. It uses a two-stage teacher-student approach: first, teachers master individual skills using privileged information, and then the student policy is trained on combined data using both PPO gradients and behavior cloning loss to consolidate knowledge into one deployable policy.
Unified Interaction Imitation Reward
Instead of task-specific rewards, HumanX uses a composite reward function (rt) that tracks multiple aspects of interaction: body movement (position, rotation, velocity), object state tracking, relative motion between the body and object, and contact adherence. This comprehensive reward guides the learning process toward physically accurate and successful interactions.
Zero-Shot Transfer
The framework's success is demonstrated by its ability to transfer skills zero-shot to a physical robot. This means that after learning 9 different skills from a single video demonstration, the robot can perform them on a new physical humanoid without any further task-specific training or reward engineering for those specific actions.

Terminology

Summary

The gist The HumanX framework compiles human video into generalizable, real-world interaction skills for humanoids without task-specific rewards.

HumanX Framework Overview

HumanX is a full-stack framework that integrates two co-designed components: XGen, a data generation pipeline that synthesizes diverse and physically plausible robot interaction data from video while supporting scalable data augmentation, and XMimic, a unified imitation learning framework that learns generalizable interaction skills<ref:2602.02473#pg2> The system enables autonomous interaction behaviors such as diverse basketball skills, consecutive football kicking, generalizable cargo pickup, and real-time counterattack against a human<ref:2602.02473#pg2>. It successfully acquires 10 different skills and transfers them zero-shot to a physical Unitree G1 humanoid<ref:2602.02473#pg2>.

XGen Data Generation Pipeline

XGen addresses the challenge of physically plausible interactions by synthesizing interaction trajectories governed by physical priors, rather than pursuing exact reconstruction<ref:2602.02473#pg4>. The pipeline operates in three stages: (1) extracting human motion and retargeting it to the robot; (2) physics-based synthesis of object trajectories coupled with contact-aware refinement; and (3) data augmentation through object geometry scaling and trajectory variation to maximize coverage for improved generalization<ref:2602.02473#pg4>. In the contact phase, the object trajectory is synthesized by propagating this relative pose along the anchor trajectory derived from the robot motion sequence, followed by force-closure optimization to ensure physical plausibility during contact<ref:2602.02473#pg5>.

XMimic Imitation Learning Framework

XMimic is a unified imitation learning framework that acquires interaction skills purely by mimicking the behaviors synthesized by XGen<ref:2602.02473#pg6>. Its training follows a two-stage teacher-student paradigm, first mastering individual skills with privilege information and then consolidating them into a unified deployable policy<ref:2602.02473#pg6>. The student policy is trained on the combined dataset D by extending the training objective to combine the PPO policy gradient term with a behavior cloning (BC) loss that distills knowledge from the pre-trained teachers<ref:2602.02473#pg6>.

Unified Interaction Imitation Reward

To enable accurate imitation, HumanX employs a composite reward function defined as rt = rbody t + robj t + rrel t + rc t + rreg t<ref:2602.02473#pg9>. The body imitation reward is decomposed into multiple tracking terms including body position, rotation, joint position (DoF), and velocities, along with an adversarial motion prior (AMP) term for naturalness<ref:2602.02473#pg9>. The contact imitation reward is computed based on the discrepancy between the simulated contact state and the reference contact state using a Contact Graph<ref:2602.02473#pg9>.

Generalization Mechanisms

Robust generalization is achieved through three complementary mechanisms: (1) Diverse offline data from XGen, which covers a broad distribution of object states; (2) Online augmentation via disturbed initialization during training, which further expands state coverage; and (3) Interaction-aware termination, which prioritizes interaction success and discourages overfitting to body motion alone<ref:2602.02473#pg9>. Furthermore, the system supports two practical deployment schemes: a No External Perception (NEP) mode that relies solely on proprioception for dynamic interaction, and a MoCap-based mode that uses external object poses, which includes realistic simulation of frame loss to enable zero-shot adaptation<ref:2602.02473#pg9>.

Experimental Results

Experiments show that HumanX achieves over 8× higher generalization success than prior methods<ref:2602.02473#pg9>. On a Unitree G1 robot, the system demonstrates two practical deployment ways: (1) Without any explicit external sensing, it executes basketball skills including dribbling, layups, and complex pump-fake turnaround fadeaways; and (2) With object sensing from a MoCap system, it achieves sustained closed-loop interactions<ref:2602.02473#pg9>. The learned capabilities include complex maneuvers such as pump-fake turnaround fadeaway jumpshots without any external perception, as well as interactive tasks like sustained human-robot passing sequences over 10 consecutive cycles—learned from a single video demonstration<ref:2602.02473#pg9>. This demonstrates a scalable and task-agnostic pathway for learning versatile, real-world robot interactive skills<ref:2602.02473#pg9>. The learned skills exhibit a high degree of autonomy and interesting emergent recovery behaviors, such as distinguishing feints from real attacks during fighting<ref:2602.02473#pg9>.

Conclusion

HumanX presents a full-stack framework that compiles monocular human video into agile and generalizable interaction skills for humanoids without task-specific rewards<ref:2602.02473#pg10>. HumanX integrates two synergistic components: XGen, which synthesizes and augments physically plausible interaction data from video, and XMimic, a unified imitation-learning framework that trains robust policies<ref:2602.02473#pg10>. Evaluated across 10 skills in five domains—from basketball to reactive fighting—our method achieves over 8× higher generalization success than prior approaches<ref:2602.02473#pg10>. On a Unitree G1 robot, HumanX enables both perception-free execution of complex maneuvers (e.g., pump-fake turnaround fadeaways) and sustained closed-loop interactions (e.g., over 10 consecutive human-robot passes), demonstrating a scalable, task-agnostic pathway for acquiring real-world interactive skills from human video<ref:2602.02473#pg10>. The system demonstrates that the skills acquired by HumanX extend far beyond simple motion replay<ref:2602.02473#pg10>.

How it works

  1. XGen Data Generation: XGen converts monocular human video into humanoid motion and synthesizes corresponding interaction under physical constraints<ref:2602.02473#pg4>. It extracts human motion using GVHMR and retargets it to the target humanoid pose sequence using GMR<ref:2602.02473#pg4>. The pipeline segments data into contact and non-contact phases, synthesizing object trajectories in the contact phase by propagating relative poses along the anchor trajectory derived from robot motion, followed by force-closure optimization<ref:2602.02473#pg5>.

  2. XMimic Training Architecture: XMimic employs a two-stage teacher-student paradigm to master individual skills and consolidate them into a unified deployable policy<ref:2602.02473#pg6>. The teacher policy is trained on each dataset Di using PPO to maximize the expected cumulative reward under privileged state observation st, which comprises proprioception ot, privileged body information opriv t, and object state sext t<ref:2602.02473#pg6>.

  3. Unified Reward Design: The composite reward function is defined as rt = rbody t + robj t + rrel t + rc t + rreg t<ref:2602.02473#pg9>. This includes body imitation rewards tracking position, rotation, and velocities, an object reward ensuring accurate state tracking, a relative motion reward encouraging correct spatial relationships between the body and object, a contact reward penalizing deviations from the reference contact graph, and regularization terms to improve motion stability<ref:2602.02473#pg9>.

  4. Generalization Enhancements: Robust generalization is achieved through three mechanisms: (1) Diverse offline data from XGen; (2) Online augmentation via disturbed initialization during training; and (3) Interaction-aware termination, which prioritizes interaction success and discourages overfitting to body motion alone<ref:2602.02473#pg10>. The system also applies domain randomization to various physical properties, including object size, mass, and coefficient of restitution<ref:2602.02473#pg9>.

Evaluation

The method is evaluated across five distinct domains—basketball, football, badminton, cargo pickup, and reactive fighting<ref:2602.02473#pg2>. Simulation results show that XMimic (Base) achieves 93.4% success rate in Basketball Catch-Shot and 97.1% in Cargo Pickup<ref:2602.02473#pg9>. Real robot experiments on the Unitree G1 demonstrate near-perfect SR in the base setting, alongside superior performance on other metrics, indicating that our reward design enables more accurate and robust interaction imitation<ref:2602.02473#pg9>. The results confirm that each skill is learned from a single video demonstration without any task-specific reward<ref:2602.02473#pg9>.

Perceiving External Force

The policy's observation space incorporates all relevant variables from the dynamics equation J TextFext = τ - Mq¨ + Cq˙ + G + τ f<ref:2602.02473#pg10>.

Improvements for AI systems

  1. textbf Synthesis of Generalizable Interaction Data via XGen + Augmentation for Task-Agnostic Skill Acquisition: The improved system can learn diverse basketball skills, consecutive football kicking, generalizable cargo pickup, and real-time counterattack against a human from a single video demonstration without task-specific rewards or external perception. This is achieved by synthesizing physically plausible robot interaction data from monocular video while enabling scalable augmentation.

  2. textbf Unified Interaction Policy Learning via XMimic's Teacher-Student Framework: The improved system can achieve generalization far beyond the original video demonstrations by distilling knowledge from teacher policies into a student policy that combines interaction imitation with behavior cloning. This results in policies that exhibit emergent, adaptive behaviors, such as autonomously regrabbing a removed object or distinguishing feints from real attacks.

  3. textbf Robust Deployment via Perception-Free and MoCap-Based Modes: The improved system can be deployed effectively without external sensors (No External Perception (NEP) mode) to execute complex maneuvers such as pump-fake turnaround fadeaways relying solely on proprioception, or with MoCap integration in MoCap mode to handle intermittent frame loss during training.

  4. textbf High-Fidelity Interaction Control via Composite Reward Function: The improved system ensures accurate imitation by employing the reward function rt = r body t + r obj t + r rel t + r c t + r reg t, which explicitly tracks body position, rotation, joint positions, and penalizes deviations from the reference contact graph to ensure precise contact timing and location.

  5. textbf Enhanced Robustness via Domain Randomization and Interaction Termination: The improved system achieves robust deployment by applying domain randomization (DR) to various physical properties like object size and mass, alongside implementing Interaction Termination (IT) which terminates episodes when the object's relative position error exceeds a threshold, preventing overfitting to restricted conditions.

Sources

Related papers