Social Humanoid: Socially Responsive Human-Humanoid Interaction

arXiv:2611.08407 · cs.RO, cs.CV, cs.LG · Submitted 2026-11-05 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: We're looking at the paper: "Social Humanoid: Socially Responsive Human-Humanoid Interaction".

Dev: Here's the gist of the paper. Social Humanoid presents Social Humanoid, a hierarchical system that generates and executes socially responsive whole-body motion on a 29-DoF Unitree G1 conditioned on a human partner’s motion.

Rosa: Its results demonstrate that Social Humanoid produces coordinated, human-aligned behavior across diverse interaction types in simulation and the real world.

Dev: System Overview The system is a hierarchical structure comprising a high-level policy and a low-level policy.

Rosa: The high-level policy generates reactive humanoid motion from the observed partner’s motion and anticipates the partner’s motion, even when they are occluded or human input is missing.

Title and authors: Rosa: So we're looking at this paper called "Social Humanoid: Socially Responsive Human-Humanoid Interaction," and it’s about building a system that lets robots actually coordinate with people physically, instead of just acting alone.

Dev: It focuses on using a hierarchical structure, so you have one high-level policy making the big plans and a low-level policy actually moving the physical robot, which is a Unitree G1 in this case.

Taro: What I find interesting about it is that it aims to solve the problem of humanoids being solitary agents; they can't anticipate or adjust to people naturally, which means they can't do real social interaction outside of a controlled lab setting.

Rosa: Exactly. The paper proposes this system generates and executes whole-body motion on the G1 conditioned on observing a human partner, even when that partner moves in an unpredictable way or is partially hidden behind something else.

Dev: And it does this by using a Diffusion Forcing Transformer policy that generates these responses over a shared vocabulary of motion tokens. This shared representation is key because it lets the system reason about both the human and the robot motions at once, which simplifies training.

Taro: That shared codebook VQ-VAE sounds like a neat way to bridge that gap between different embodiments by mapping them into one common discrete vocabulary. But does that sharing actually help when the physical shapes are so different?

Rosa: Well, they tackle that data gap using a two-phase Adaptive-HHI Retargeting procedure. They retarget one partner to the robot and morphs the other's body pose to make sure things like contact and reach are preserved, even across big differences in size or shape.

Dev: That two-pass process involves Pass I where they retarget the ego and optimize the humanoid configuration while keeping local shape and contact intact, then Pass II which co-morphing adapts the partner’s pose to restore that contact fidelity across large morphology gaps.

Taro: So, if they can create a new benchmark of two hundred sixty hours of human-humanoid interactions using that method, does it mean we have actual data for training these systems?

Title and authors: Rosa: It does. That dataset comes from applying this retargeting procedure across five different motion-capture datasets, giving them a solid set of data to train on. They also use this to create a benchmark for testing how well the system can handle these interactions.

Dev: The high-level policy itself is built using the Diffusion Forcing Transformer, which uses the denoising process from diffusion models combined with the temporal strengths of transformers to learn those non-Markovian dependencies across long sequences of motion.

Taro: That sounds like it gives them a good handle on long-horizon coherence, which is important for interactions that last more than just a quick greeting or a high-five.

Rosa: Right. And the training objective they use balances the diffusion accuracy with kinematic fidelity using latent, motion, and reconstruction losses to keep the motions both plausible and physically realistic.

Dev: When you actually run this on the real G1, their low-level tracking controller uses NVIDIA SONIC to execute those generated motions in simulation or in reality at a speed of fifty hertz.

Taro: That’s where I get concerned about real-world failure modes; if the world misbehaves during execution, how gracefully does this system recover?

Rosa: They found that on the real G1, the low-level policy reproduces nearly all the motions it was trained on with ninety to one hundred percent success per scenario in simulation and in reality.

Dev: And for deployment, they measured a median end-to-end latency from when you capture a camera frame to when the humanoid sends a command as three hundred eight milliseconds.

Taro: Three hundred eight milliseconds is pretty fast for physical interaction, but what about the limitations? I'm thinking about what it can't do.

Rosa: The paper points out that they don’t model things like hand articulation, facial expressions, or object pose very well right now. They also rely on SONIC and other off-the-shelf human pose estimation methods, meaning they inherit any failures those methods have in the real world deployment.

Dev: And they also noted that the low-level policy doesn't model interaction forces or safety constraints during close contact, which means they'd need separate safety policies for unconstrained environments.

Title and authors: Taro: So it’s not just a general social system; it’s specifically focused on dyadic human-humanoid interaction, and the lack of modeling for things like eye contact makes the resulting behavior sometimes feel awkward socially.

Rosa: True. They are focusing heavily on coordinated, interactive behavior in situations like a hug or a greeting followed by posing together for a photo, which they show works well in simulation and on the real G1.

Dev: But on the positive side, they also mentioned that this shared codebook VQ-VAE supports real-time shadow teleoperation where the humanoid can reproduce the whole-body motion of a human operator without needing any optimization-based retargeting at all.

Taro: That capability to translate human motion into humanoid motion in what they call a single feed forward pass is pretty powerful for quick control, but it relies on that shared representation being accurate.

Rosa: It seems like the main implication here is showing that coordinated, human-aligned behaviors can actually be produced when you condition the robot’s whole-body motion on observing a human partner.

Dev: And with this paper, we see practical feasibility because they achieved coordinated interaction in both simulation and on the real G1, with that three hundred eight millisecond latency being a tangible number for deployment.

Taro: I just hope they keep working on those limitations, like modeling hand articulation and ensuring safety constraints during actual physical contact so these systems are truly robust for anything outside of controlled demos.

Rosa: Yeah, that’s the path forward. So to wrap up, "Social Humanoid: Socially Responsive Human-Humanoid Interaction" shows a hierarchical system that can generate and execute socially responsive whole-body motion on a Unitree G1 conditioned on human partner’s motion through adaptive retargeting and a shared codebook VQ-VAE.

Dev: It proves they can produce coordinated, human-aligned behavior across diverse interaction types in simulation and the real world, with the low-level policy executing those motions reliably at fifty hertz.

Taro: The focus on generating motion conditioned on observation is what really matters for future autonomy when robots move into shared spaces.

Rosa: That’s what we’ve got from this paper. We'll be looking at other papers next week to see how these concepts apply to different kinds of robot coordination.

The paper's summary: Rosa: So we're looking at this paper called "Social Humanoid: Socially Responsive Human-Humanoid Interaction," and it’s about building a system that lets robots actually coordinate with people physically, instead of just acting alone.

Dev: It focuses on using a hierarchical structure, so you have one high-level policy making the big plans and a low-level policy actually moving the physical robot, which is a Unitree G1 in this case.

Taro: What I find interesting about it is that it aims to solve the problem of humanoids being solitary agents; they can't anticipate or adjust to people naturally, which means they can't do real social interaction outside of a controlled lab setting.

Rosa: Exactly. The paper proposes this system generates and executes whole-body motion on the G1 conditioned on observing a human partner, even when that partner moves in an unpredictable way or is partially hidden behind something else.

Dev: And it does this by using a Diffusion Forcing Transformer policy that generates these responses over a shared vocabulary of motion tokens. This shared representation is key because it lets the system reason about both the human and the robot motions at once, which simplifies training.

Taro: That shared codebook VQ-VAE sounds like a neat way to bridge that gap between different embodiments by mapping them into one common discrete vocabulary. But does that sharing actually help when the physical shapes are so different?

Rosa: Well, they tackle that data gap using a two-phase Adaptive-HHI Retargeting procedure. They retarget one partner to the robot and morphs the other's body pose to make sure things like contact and reach are preserved, even across big differences in size or shape.

Dev: That two-pass process involves Pass I where they retarget the ego and optimize the humanoid configuration while keeping local shape and contact intact, then Pass II which co-morphing adapts the partner’s pose to restore that contact fidelity across large morphology gaps.

Taro: So, if they can create a new benchmark of two hundred sixty hours of human-humanoid interactions using that method, does it mean we have actual data for training these systems?

The paper's summary: Rosa: It does. That dataset comes from applying this retargeting procedure across five different motion-capture datasets, giving them a solid set of data to train on. They also use this to create a benchmark for testing how well the system can handle these interactions.

Dev: The high-level policy itself is built using the Diffusion Forcing Transformer, which uses the denoising process from diffusion models combined with the temporal strengths of transformers to learn those non-Markovian dependencies across long sequences of motion.

Taro: That sounds like it gives them a good handle on long-horizon coherence, which is important for interactions that last more than just a quick greeting or a high-five.

Rosa: Right. And the training objective they use balances the diffusion accuracy with kinematic fidelity using latent, motion, and reconstruction losses to keep the motions both plausible and physically realistic.

Dev: When you actually run this on the real G1, their low-level tracking controller uses NVIDIA SONIC to execute those generated motions in simulation or in reality at a speed of fifty hertz.

Taro: That’s where I get concerned about real-world failure modes; if the world misbehaves during execution, how gracefully does this system recover?

Rosa: They found that on the real G1, the low-level policy reproduces nearly all the motions it was trained on with ninety to one hundred percent success per scenario in simulation and in reality.

Dev: And for deployment, they measured a median end-to-end latency from when you capture a camera frame to when the humanoid sends a command as three hundred eight milliseconds.

Taro: Three hundred eight milliseconds is pretty fast for physical interaction, but what about the limitations? I'm thinking about what it can't do.

Rosa: The paper points out that they don’t model things like hand articulation, facial expressions, or object pose very well right now. They also rely on SONIC and other off-the-shelf human pose estimation methods, meaning they inherit any failures those methods have in the real world deployment.

Dev: And they also noted that the low-level policy doesn't model interaction forces or safety constraints during close contact, which means they'd need separate safety policies for unconstrained environments.

The paper's summary: Taro: So it’s not just a general social system; it’s specifically focused on dyadic human-humanoid interaction, and the lack of modeling for things like eye contact makes the resulting behavior sometimes feel awkward socially.

Rosa: True. They are focusing heavily on coordinated, human-aligned behaviors in situations like a hug or a greeting followed by posing together for a photo, which they show works well in simulation and on the real G1.

Dev: But on the positive side, they also mentioned that this shared codebook VQ-VAE supports real-time shadow teleoperation where the humanoid can reproduce the whole-body motion of a human operator without needing any optimization-based retargeting at all.

Taro: That capability to translate human motion into humanoid motion in what they call a single feed forward pass is pretty powerful for quick control, but it relies on that shared representation being accurate.

Rosa: It seems like the main implication here is showing that coordinated, human-aligned behaviors can actually be produced when you condition the robot’s whole-body motion on observing a human partner.

Dev: And with this paper, we see practical feasibility because they achieved coordinated interaction in both simulation and on the real G1, with that three hundred eight millisecond latency being a tangible number for deployment.

Taro: I just hope they keep working on those limitations, like modeling hand articulation and ensuring safety constraints during actual physical contact so these systems are truly robust for anything outside of controlled demos.

Rosa: Yeah, that’s the path forward. So to wrap up, "Social Humanoid: Socially Responsive Human-Humanoid Interaction" shows a hierarchical system that can generate and execute socially responsive whole-body motion on a Unitree G1 conditioned on human partner’s motion through adaptive retargeting and a shared codebook VQ-VAE.

Dev: It proves they can produce coordinated, human-aligned behavior across diverse interaction types in simulation and the real world, with the low-level policy executing those motions reliably at fifty hertz.

Taro: The focus on generating motion conditioned on observation is what really matters for future autonomy when robots move into shared spaces.

The paper's improvements: Taro: We just talked about how they used that shared codebook VQ-VAE to map both human and robot motions into one common vocabulary, which lets them do real-time shadow teleoperation without redoing all that heavy optimization work.

Rosa: Yeah, and the authors also pointed out a couple of ways they could push this further in future work.

Dev: What are those? I'm thinking about what they said about the current limitations with hand articulation and facial expressions.

Taro: They mentioned that modeling those things is a big gap right now because it’s hard to capture those details reliably, so they say future work needs to include those.

Rosa: Right, because the paper's current system doesn't model what hands are doing or if someone has a smile.

Dev: And on the engineering side, they noted that while the low-level policy tracks motion well at fifty hertz, it doesn't actually model any physical forces or safety constraints during close contact.

Taro: That means if a human accidentally bumps into the robot while it’s hugging them, there isn't a built-in safety check for that.

Rosa: Exactly. They suggest that future work needs to include separate safety policies specifically for those unconstrained environments where physical interaction is happening.

Dev: So they aren't just looking at making the motion look good; they're thinking about how to make it safe when people are actually touching things.

Taro: And since the whole thing is dyadic, focused only on human-humanoid interactions, they also suggest that expanding beyond that specific kind of interaction is a future goal.

Rosa: That makes sense. They want this framework to be more general so it can handle different kinds of social situations or even other types of robot coordination down the road.

Dev: So the implication is that we move from just coordinated movements in a controlled setting to systems that are safer and more capable in messy, real-world physical contact scenarios.

Taro: It moves the goalposts from just making it look right to making it robust enough to survive unexpected physical events.

Conclusion: Rosa: So we’ve seen how the paper "Social Humanoid: Socially Responsive Human-Humanoid Interaction" uses this whole hierarchical setup to create human-aligned motion on a Unitree G1.

Dev: It boils down to using a shared codebook VQ-VAE and a Diffusion Forcing Transformer to generate these socially responsive responses based on observing a partner.

Taro: I think what this really changes for us is seeing how we can move beyond just making robots follow commands to actually producing coordinated, interactive behaviors in complex scenarios.

Rosa: Right, and the fact that they managed to get near one hundred percent success on the real G1 across different interaction types in simulation and reality is pretty telling.

Dev: The three hundred eight millisecond latency from camera frame to command shows it’s actually deployable for real physical tasks, which is a big deal for controls engineers like me.

Taro: But we can't forget the caveats they mentioned—it doesn't model hand articulation or facial expressions yet, and the safety policies are missing for close contact situations.

Rosa: True, so it’s a great start for dyadic interaction, but we still have a long way to go before these systems can handle truly messy human-robot physical interactions without crashing or looking awkward.

Dev: Exactly. The next step for the field is figuring out how to integrate those missing elements—like force modeling and more complex safety constraints—into this kind of generative architecture.

Taro: I think that’s where we need to look next, because if we can get the physical interaction right, then these systems become much more useful in shared spaces.

Rosa: Well, that’s where we’ll be looking at in our next session—we're going to see how these ideas apply to other kinds of robot coordination.

UC Berkeley Company Sony Group Corporation Meta

cs.RO, cs.CV, cs.LG

Submitted: 2026-11-05

Updated: 2026-11-05

Comments: CoRL 2026, Project page: https://von31.github.io/social-humanoid/

Project page: https://von31.github.io/social-humanoid/Abstract

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 100/100

The gist: Social Humanoid presents Social Humanoid, a hierarchical system that generates and executes socially responsive whole-body motion on a 29-DoF Unitree G1 conditioned on a human partner’s motion.

Key concepts

High-Level Policy (DFoT)
This is the brain of the system that generates what the humanoid should do next. It uses a Diffusion Forcing Transformer to autoregressively create motion sequences based on observed partner actions. It learns complex, non-linear timing dependencies to anticipate future movements, enabling fluid and reactive responses.
Shared Codebook VQ-VAE
This model compresses both human and robot motions into a single discrete vocabulary of tokens. This allows the system to reason jointly about both embodiments using one representation. It simplifies training because it assumes human and robot motions share a similar underlying structure, enabling translation between them.
Adaptive-HHI Retargeting
This procedure creates a benchmark dataset by retargeting one partner's motion onto the Unitree G1 while morphing the other partner's pose. This ensures that contact, reachability, and relative scale are preserved across different body shapes, providing physically meaningful training data.
Low-Level Policy Execution
This component takes the high-level generated motion and executes it on the physical robot using a tracking controller (NVIDIA SONIC). It ensures that the complex planned movements are translated into actual, stable whole-body actions on the G1, allowing for successful interaction without falling.

Terminology

Summary

Social Humanoid presents Social Humanoid, a hierarchical system that generates and executes socially responsive whole-body motion on a 29-DoF Unitree G1 conditioned on a human partner’s motion. p. 0 Its results demonstrate that Social Humanoid produces coordinated, human-aligned behavior across diverse interaction types in simulation and the real world. p. 0

System Overview

The system is a hierarchical structure comprising a high-level policy and a low-level policy. The high-level policy generates reactive humanoid motion from the observed partner’s motion and anticipates the partner’s motion, even when they are occluded or human input is missing. p. 1 The low-level policy then executes the generated motion on a real Unitree G1 humanoid. p. 1 This architecture enables continuous interaction by generating and executing socially responsive whole-body motion conditioned on human partner’s motion. p. 9

Data Generation and Retargeting

The system addresses the data gap by introducing a two-phase Adaptive-HHI Retargeting procedure that retargets one partner to the 29-DoF Unitree G1 and morphs the other’s body pose to preserve contact, reach, and relative scale. This procedure was applied across five motion-capture datasets to produce a new Human-Humanoid Interaction (HHI) benchmark of 260 hours of human-humanoid interactions. p. 0 The two passes involve Pass I, which retargets the ego and optimizes the humanoid configuration while preserving local shape and contact, and Pass II, which co-morphing adapts the partner’s pose to the retargeted humanoid to restore contact fidelity across large morphology gaps. p. 6

Shared Representation Modeling

To bridge the representation gap, a shared motion representation is used where a shared-codebook VQ-VAE tokenizes human and humanoid motion into a single discrete vocabulary. This shared representation simplifies the training of the subsequent diffusion model because human and humanoid motions may share a similar latent structure. p. 3 The VQ-VAE uses embodiment-specific encoders and decoders that map motion features into a shared codebook C, which defines a common discrete motion vocabulary across the human–humanoid embodiment gap. p. 3 Crucially, the alignment terms Lalign and Lagree encourage cross-embodiment code prediction and tie the codebook to corresponding human–robot poses rather than letting each encoder claim separate regions of C. p. 4

High-Level Policy Generation

The high-level policy is a Diffusion Forcing Transformer (DFoT) that autoregressively generates humanoid responses over this shared vocabulary. DFoT integrates the denoising process of DDPMs with the temporal strengths of transformers, allowing it to learn non-Markovian temporal dependencies across entire sequences using adaptive noise levels. p. 4 The training objective combines latent, motion, and reconstruction losses to balance diffusion accuracy with kinematic fidelity. p. 4 The final inference uses autoregressive DDIM sampling with a sampling strategy matrix K to control per-token noise levels across denoising steps. p. 4

Performance and Deployment

The system was evaluated on three tasks: generative modeling, retargeting, and execution on the real humanoid. In simulation and on the real G1, the low-level policy reproduces nearly all generated motions with 90–100% success per scenario. p. 8 The median end-to-end latency from camera frame to humanoid command is 308 ms. p. 1 Furthermore, the shared codebook VQ-VAE supports real-time shadow teleoperation, where the humanoid reproduces the whole-body motion of a human operator without optimization-based retargeting. p. 1 The system's success in simulation and on the real G1 confirms that coordinated, interactive, and human-aligned behaviors are produced.

Limitations

Limitations include the lack of modeling for hand articulation, facial expressions, or object pose. The system relies on SONIC and off-the-shelf human pose estimation, inheriting their failure modes in real-world deployment. p. 9 Furthermore, the low-level policy does not model interaction forces or safety constraints during close contact, requiring additional safety policies for unconstrained environments. p. 9 The focus is primarily on dyadic human–humanoid interaction. p. 2 The lack of facial expressions or eye contact makes the humanoid's behavior difficult to interpret, leading to awkward social interactions.

Acknowledgments

The authors thank Zeldah Maluleke, Kaito Suzuki, Marsalis Gibson and Sharmita Dey for serving as the human partner/operator in their real-world demonstrations and assisting with recording the demonstrations. This project was funded in part by Sony Corporation, Bakar Fellows, NSF CAREER award no 2442491, the Humanoid Intelligence Center program, ONR MURI N00014-21-1-280, and MURI N000142612024. p. 0

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Adaptive-HHI Retargeting procedure produces a new Human-Humanoid Interaction benchmark of 260 hours of human-humanoid interactions, allowing for training systems on physically meaningful, contact-preserving data. This addresses the data gap by retargeting one partner to the robot while morphing the other partner to preserve contact dynamics, reachability, and relative body scale.

  2. Shared-Codebook VQ-VAE enables a single generative model to reason jointly about both embodiments through a shared tokenized representation, which supports real-time shadow teleoperation of the G1 without optimization-based retargeting. This allows the system to translate human motion into humanoid motion in "a single feed-forward pass (Eh -> C -> Dr)."

  3. Diffusion Forcing Transformer autoregressively generates humanoid responses over this shared vocabulary, enabling fast, reactive motion generation at inference time and supporting fluid, long-horizon interaction due to its ability to learn non-Markovian temporal dependencies. This high-level policy predicts future latents for both agents conditioned on observed history.

  4. The system can perform end-to-end deployment on a Unitree G1 with a median end-to-end latency of 308 ms from camera frame to humanoid command, achieving coordinated, interactive, and human-aligned behaviors in both simulation and the real world. This demonstrates practical feasibility for physical interaction tasks.

  5. The system can execute complex social interactions across diverse interaction types in simulation and the real world by producing coordinated, human-aligned behavior. This is validated by successfully reproducing nearly all generated motions in simulation (90–100%) and on the real G1 across scenarios like high-five, hug, and bend.

  6. The low-level whole-body tracking controller executes the generated motion on the robot via NVIDIA SONIC [13], which tracks decoded motion at 50 Hz, ensuring that The low-level policy can execute the appropriate reaction and complete the interaction without falling or operator intervention in real deployment trials.

Sources

Related papers