Social Humanoid: Socially Responsive Human-Humanoid Interaction
summary
The gist
Social Humanoid presents Social Humanoid, a hierarchical system that generates and executes socially responsive whole-body motion on a 29-DoF Unitree G1 conditioned on a human partner’s motion.
In short
Social Humanoid generates and executes coordinated whole-body motion on a Unitree G1 by conditioning it on a human partner's movement. It uses a shared representation between human and robot motions to learn socially responsive behavior. The system produces fluid, human-aligned interactions in simulation and the real world with low latency.
Key concepts
- High-Level Policy (DFoT)
- This is the brain of the system that generates what the humanoid should do next. It uses a Diffusion Forcing Transformer to autoregressively create motion sequences based on observed partner actions. It learns complex, non-linear timing dependencies to anticipate future movements, enabling fluid and reactive responses.
- Shared Codebook VQ-VAE
- This model compresses both human and robot motions into a single discrete vocabulary of tokens. This allows the system to reason jointly about both embodiments using one representation. It simplifies training because it assumes human and robot motions share a similar underlying structure, enabling translation between them.
- Adaptive-HHI Retargeting
- This procedure creates a benchmark dataset by retargeting one partner's motion onto the Unitree G1 while morphing the other partner's pose. This ensures that contact, reachability, and relative scale are preserved across different body shapes, providing physically meaningful training data.
- Low-Level Policy Execution
- This component takes the high-level generated motion and executes it on the physical robot using a tracking controller (NVIDIA SONIC). It ensures that the complex planned movements are translated into actual, stable whole-body actions on the G1, allowing for successful interaction without falling.
Terminology used across episodes
This episode discusses
- Social Humanoid: Socially Responsive Human-Humanoid Interaction · Paper Radio
- Synergy and Synchrony in Couple Dances
- Ready-to-React: Online Reaction Policy for Two-Character Interaction Generation
- Harmony4D: A Video Dataset for In-The-Wild Close Human Interactions
- Embody 3D: A Large-scale Multimodal Motion and Behavior Dataset
- SONIC: Supersizing Motion Tracking for Natural Humanoid Whole-Body Control
- Single-Shot Motion Completion with Transformer
- MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Model
- TWIST: Teleoperated Whole-Body Imitation System
- PASTA: Pretrained Action-State Transformer Agents
- Harmon: Whole-Body Motion Generation of Humanoid Robots from Language Descriptions
- Visual Imitation Enables Contextual Humanoid Control
- BeyondMimic: From Motion Tracking to Versatile Humanoid Control via Guided Diffusion
- OmniRetarget: Interaction-Preserving Data Generation for Humanoid Whole-Body Loco-Manipulation and Scene Interaction
- ARFlow: Human Action-Reaction Flow Matching with Physical Guidance
- Interactive Humanoid: Online Full-Body Motion Reaction Synthesis with Social Affordance Canonicalization and Forecasting
- Duolando: Follower GPT with Off-Policy Reinforcement Learning for Dance Accompaniment
- SoftMimic: Learning Compliant Whole-body Control from Examples
- GentleHumanoid: Learning Upper-body Compliance for Contact-rich Human and Object Interaction
- Learning Vision-based Pursuit-Evasion Robot Policies
- Denoising Diffusion Implicit Models
The paper
Social Humanoid: Socially Responsive Human-Humanoid Interaction · Read on arXiv
UC Berkeley Company Sony Group Corporation Meta
Transcript
Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.
Rosa: We're looking at the paper: "Social Humanoid: Socially Responsive Human-Humanoid Interaction".
Dev: Here's the gist of the paper. Social Humanoid presents Social Humanoid, a hierarchical system that generates and executes socially responsive whole-body motion on a 29-DoF Unitree G1 conditioned on a human partner’s motion.
Rosa: Its results demonstrate that Social Humanoid produces coordinated, human-aligned behavior across diverse interaction types in simulation and the real world.
Dev: System Overview The system is a hierarchical structure comprising a high-level policy and a low-level policy.
Rosa: The high-level policy generates reactive humanoid motion from the observed partner’s motion and anticipates the partner’s motion, even when they are occluded or human input is missing.
Title and authors: Rosa: So we're looking at this paper called "Social Humanoid: Socially Responsive Human-Humanoid Interaction," and it’s about building a system that lets robots actually coordinate with people physically, instead of just acting alone.
Dev: It focuses on using a hierarchical structure, so you have one high-level policy making the big plans and a low-level policy actually moving the physical robot, which is a Unitree G1 in this case.
Taro: What I find interesting about it is that it aims to solve the problem of humanoids being solitary agents; they can't anticipate or adjust to people naturally, which means they can't do real social interaction outside of a controlled lab setting.
Rosa: Exactly. The paper proposes this system generates and executes whole-body motion on the G1 conditioned on observing a human partner, even when that partner moves in an unpredictable way or is partially hidden behind something else.
Dev: And it does this by using a Diffusion Forcing Transformer policy that generates these responses over a shared vocabulary of motion tokens. This shared representation is key because it lets the system reason about both the human and the robot motions at once, which simplifies training.
Taro: That shared codebook VQ-VAE sounds like a neat way to bridge that gap between different embodiments by mapping them into one common discrete vocabulary. But does that sharing actually help when the physical shapes are so different?
Rosa: Well, they tackle that data gap using a two-phase Adaptive-HHI Retargeting procedure. They retarget one partner to the robot and morphs the other's body pose to make sure things like contact and reach are preserved, even across big differences in size or shape.
Dev: That two-pass process involves Pass I where they retarget the ego and optimize the humanoid configuration while keeping local shape and contact intact, then Pass II which co-morphing adapts the partner’s pose to restore that contact fidelity across large morphology gaps.
Taro: So, if they can create a new benchmark of two hundred sixty hours of human-humanoid interactions using that method, does it mean we have actual data for training these systems?
Title and authors: Rosa: It does. That dataset comes from applying this retargeting procedure across five different motion-capture datasets, giving them a solid set of data to train on. They also use this to create a benchmark for testing how well the system can handle these interactions.
Dev: The high-level policy itself is built using the Diffusion Forcing Transformer, which uses the denoising process from diffusion models combined with the temporal strengths of transformers to learn those non-Markovian dependencies across long sequences of motion.
Taro: That sounds like it gives them a good handle on long-horizon coherence, which is important for interactions that last more than just a quick greeting or a high-five.
Rosa: Right. And the training objective they use balances the diffusion accuracy with kinematic fidelity using latent, motion, and reconstruction losses to keep the motions both plausible and physically realistic.
Dev: When you actually run this on the real G1, their low-level tracking controller uses NVIDIA SONIC to execute those generated motions in simulation or in reality at a speed of fifty hertz.
Taro: That’s where I get concerned about real-world failure modes; if the world misbehaves during execution, how gracefully does this system recover?
Rosa: They found that on the real G1, the low-level policy reproduces nearly all the motions it was trained on with ninety to one hundred percent success per scenario in simulation and in reality.
Dev: And for deployment, they measured a median end-to-end latency from when you capture a camera frame to when the humanoid sends a command as three hundred eight milliseconds.
Taro: Three hundred eight milliseconds is pretty fast for physical interaction, but what about the limitations? I'm thinking about what it can't do.
Rosa: The paper points out that they don’t model things like hand articulation, facial expressions, or object pose very well right now. They also rely on SONIC and other off-the-shelf human pose estimation methods, meaning they inherit any failures those methods have in the real world deployment.
Dev: And they also noted that the low-level policy doesn't model interaction forces or safety constraints during close contact, which means they'd need separate safety policies for unconstrained environments.
Title and authors: Taro: So it’s not just a general social system; it’s specifically focused on dyadic human-humanoid interaction, and the lack of modeling for things like eye contact makes the resulting behavior sometimes feel awkward socially.
Rosa: True. They are focusing heavily on coordinated, interactive behavior in situations like a hug or a greeting followed by posing together for a photo, which they show works well in simulation and on the real G1.
Dev: But on the positive side, they also mentioned that this shared codebook VQ-VAE supports real-time shadow teleoperation where the humanoid can reproduce the whole-body motion of a human operator without needing any optimization-based retargeting at all.
Taro: That capability to translate human motion into humanoid motion in what they call a single feed forward pass is pretty powerful for quick control, but it relies on that shared representation being accurate.
Rosa: It seems like the main implication here is showing that coordinated, human-aligned behaviors can actually be produced when you condition the robot’s whole-body motion on observing a human partner.
Dev: And with this paper, we see practical feasibility because they achieved coordinated interaction in both simulation and on the real G1, with that three hundred eight millisecond latency being a tangible number for deployment.
Taro: I just hope they keep working on those limitations, like modeling hand articulation and ensuring safety constraints during actual physical contact so these systems are truly robust for anything outside of controlled demos.
Rosa: Yeah, that’s the path forward. So to wrap up, "Social Humanoid: Socially Responsive Human-Humanoid Interaction" shows a hierarchical system that can generate and execute socially responsive whole-body motion on a Unitree G1 conditioned on human partner’s motion through adaptive retargeting and a shared codebook VQ-VAE.
Dev: It proves they can produce coordinated, human-aligned behavior across diverse interaction types in simulation and the real world, with the low-level policy executing those motions reliably at fifty hertz.
Taro: The focus on generating motion conditioned on observation is what really matters for future autonomy when robots move into shared spaces.
Rosa: That’s what we’ve got from this paper. We'll be looking at other papers next week to see how these concepts apply to different kinds of robot coordination.
The paper's summary: Rosa: So we're looking at this paper called "Social Humanoid: Socially Responsive Human-Humanoid Interaction," and it’s about building a system that lets robots actually coordinate with people physically, instead of just acting alone.
Dev: It focuses on using a hierarchical structure, so you have one high-level policy making the big plans and a low-level policy actually moving the physical robot, which is a Unitree G1 in this case.
Taro: What I find interesting about it is that it aims to solve the problem of humanoids being solitary agents; they can't anticipate or adjust to people naturally, which means they can't do real social interaction outside of a controlled lab setting.
Rosa: Exactly. The paper proposes this system generates and executes whole-body motion on the G1 conditioned on observing a human partner, even when that partner moves in an unpredictable way or is partially hidden behind something else.
Dev: And it does this by using a Diffusion Forcing Transformer policy that generates these responses over a shared vocabulary of motion tokens. This shared representation is key because it lets the system reason about both the human and the robot motions at once, which simplifies training.
Taro: That shared codebook VQ-VAE sounds like a neat way to bridge that gap between different embodiments by mapping them into one common discrete vocabulary. But does that sharing actually help when the physical shapes are so different?
Rosa: Well, they tackle that data gap using a two-phase Adaptive-HHI Retargeting procedure. They retarget one partner to the robot and morphs the other's body pose to make sure things like contact and reach are preserved, even across big differences in size or shape.
Dev: That two-pass process involves Pass I where they retarget the ego and optimize the humanoid configuration while keeping local shape and contact intact, then Pass II which co-morphing adapts the partner’s pose to restore that contact fidelity across large morphology gaps.
Taro: So, if they can create a new benchmark of two hundred sixty hours of human-humanoid interactions using that method, does it mean we have actual data for training these systems?
The paper's summary: Rosa: It does. That dataset comes from applying this retargeting procedure across five different motion-capture datasets, giving them a solid set of data to train on. They also use this to create a benchmark for testing how well the system can handle these interactions.
Dev: The high-level policy itself is built using the Diffusion Forcing Transformer, which uses the denoising process from diffusion models combined with the temporal strengths of transformers to learn those non-Markovian dependencies across long sequences of motion.
Taro: That sounds like it gives them a good handle on long-horizon coherence, which is important for interactions that last more than just a quick greeting or a high-five.
Rosa: Right. And the training objective they use balances the diffusion accuracy with kinematic fidelity using latent, motion, and reconstruction losses to keep the motions both plausible and physically realistic.
Dev: When you actually run this on the real G1, their low-level tracking controller uses NVIDIA SONIC to execute those generated motions in simulation or in reality at a speed of fifty hertz.
Taro: That’s where I get concerned about real-world failure modes; if the world misbehaves during execution, how gracefully does this system recover?
Rosa: They found that on the real G1, the low-level policy reproduces nearly all the motions it was trained on with ninety to one hundred percent success per scenario in simulation and in reality.
Dev: And for deployment, they measured a median end-to-end latency from when you capture a camera frame to when the humanoid sends a command as three hundred eight milliseconds.
Taro: Three hundred eight milliseconds is pretty fast for physical interaction, but what about the limitations? I'm thinking about what it can't do.
Rosa: The paper points out that they don’t model things like hand articulation, facial expressions, or object pose very well right now. They also rely on SONIC and other off-the-shelf human pose estimation methods, meaning they inherit any failures those methods have in the real world deployment.
Dev: And they also noted that the low-level policy doesn't model interaction forces or safety constraints during close contact, which means they'd need separate safety policies for unconstrained environments.
The paper's summary: Taro: So it’s not just a general social system; it’s specifically focused on dyadic human-humanoid interaction, and the lack of modeling for things like eye contact makes the resulting behavior sometimes feel awkward socially.
Rosa: True. They are focusing heavily on coordinated, human-aligned behaviors in situations like a hug or a greeting followed by posing together for a photo, which they show works well in simulation and on the real G1.
Dev: But on the positive side, they also mentioned that this shared codebook VQ-VAE supports real-time shadow teleoperation where the humanoid can reproduce the whole-body motion of a human operator without needing any optimization-based retargeting at all.
Taro: That capability to translate human motion into humanoid motion in what they call a single feed forward pass is pretty powerful for quick control, but it relies on that shared representation being accurate.
Rosa: It seems like the main implication here is showing that coordinated, human-aligned behaviors can actually be produced when you condition the robot’s whole-body motion on observing a human partner.
Dev: And with this paper, we see practical feasibility because they achieved coordinated interaction in both simulation and on the real G1, with that three hundred eight millisecond latency being a tangible number for deployment.
Taro: I just hope they keep working on those limitations, like modeling hand articulation and ensuring safety constraints during actual physical contact so these systems are truly robust for anything outside of controlled demos.
Rosa: Yeah, that’s the path forward. So to wrap up, "Social Humanoid: Socially Responsive Human-Humanoid Interaction" shows a hierarchical system that can generate and execute socially responsive whole-body motion on a Unitree G1 conditioned on human partner’s motion through adaptive retargeting and a shared codebook VQ-VAE.
Dev: It proves they can produce coordinated, human-aligned behavior across diverse interaction types in simulation and the real world, with the low-level policy executing those motions reliably at fifty hertz.
Taro: The focus on generating motion conditioned on observation is what really matters for future autonomy when robots move into shared spaces.
The paper's improvements: Taro: We just talked about how they used that shared codebook VQ-VAE to map both human and robot motions into one common vocabulary, which lets them do real-time shadow teleoperation without redoing all that heavy optimization work.
Rosa: Yeah, and the authors also pointed out a couple of ways they could push this further in future work.
Dev: What are those? I'm thinking about what they said about the current limitations with hand articulation and facial expressions.
Taro: They mentioned that modeling those things is a big gap right now because it’s hard to capture those details reliably, so they say future work needs to include those.
Rosa: Right, because the paper's current system doesn't model what hands are doing or if someone has a smile.
Dev: And on the engineering side, they noted that while the low-level policy tracks motion well at fifty hertz, it doesn't actually model any physical forces or safety constraints during close contact.
Taro: That means if a human accidentally bumps into the robot while it’s hugging them, there isn't a built-in safety check for that.
Rosa: Exactly. They suggest that future work needs to include separate safety policies specifically for those unconstrained environments where physical interaction is happening.
Dev: So they aren't just looking at making the motion look good; they're thinking about how to make it safe when people are actually touching things.
Taro: And since the whole thing is dyadic, focused only on human-humanoid interactions, they also suggest that expanding beyond that specific kind of interaction is a future goal.
Rosa: That makes sense. They want this framework to be more general so it can handle different kinds of social situations or even other types of robot coordination down the road.
Dev: So the implication is that we move from just coordinated movements in a controlled setting to systems that are safer and more capable in messy, real-world physical contact scenarios.
Taro: It moves the goalposts from just making it look right to making it robust enough to survive unexpected physical events.
Conclusion: Rosa: So we’ve seen how the paper "Social Humanoid: Socially Responsive Human-Humanoid Interaction" uses this whole hierarchical setup to create human-aligned motion on a Unitree G1.
Dev: It boils down to using a shared codebook VQ-VAE and a Diffusion Forcing Transformer to generate these socially responsive responses based on observing a partner.
Taro: I think what this really changes for us is seeing how we can move beyond just making robots follow commands to actually producing coordinated, interactive behaviors in complex scenarios.
Rosa: Right, and the fact that they managed to get near one hundred percent success on the real G1 across different interaction types in simulation and reality is pretty telling.
Dev: The three hundred eight millisecond latency from camera frame to command shows it’s actually deployable for real physical tasks, which is a big deal for controls engineers like me.
Taro: But we can't forget the caveats they mentioned—it doesn't model hand articulation or facial expressions yet, and the safety policies are missing for close contact situations.
Rosa: True, so it’s a great start for dyadic interaction, but we still have a long way to go before these systems can handle truly messy human-robot physical interactions without crashing or looking awkward.
Dev: Exactly. The next step for the field is figuring out how to integrate those missing elements—like force modeling and more complex safety constraints—into this kind of generative architecture.
Taro: I think that’s where we need to look next, because if we can get the physical interaction right, then these systems become much more useful in shared spaces.
Rosa: Well, that’s where we’ll be looking at in our next session—we're going to see how these ideas apply to other kinds of robot coordination.
More episodes
- 2608.13438-ContactGuard: Pre-Contact Execution Monitoring with Action-Conditioned Latent World Models
- 2610.34593-Future Prediction as Training-Only Video Supervision for Pretrained VLA Policies
- 2610.35385-EmbodiedLab: Beyond Manipulation, Benchmarking Embodied Agents That Act to Know
- 2610.36378-Data-Driven Hamiltonian for Neural Optimal Control
- 2610.37275-A Heuristic for Scheduling Autonomous Vehicles at Intersections
- 2610.37288-Dynamic Coordination of Autonomous Vehicles at Intersections
- 2610.37632-Real-Time Coordination of Autonomous Vehicles at Intersections
- 2611.13779-Softness is something you also need! Why compliance helps control: information metrics of a passive walker with compliant ankle
- 2611.13411-Residual-Corrected Physics-Informed Neural Network KKL Observers with Input-to-State Stability Guarantees
- 2611.14488-Is Your Robot's Memory Just a Clock? Temporal Shortcuts in Memory-Augmented VLA Policies