HABIT: Human-Aware Behavior and Interaction Training Dataset for Robot Manipulation

summary

The gist

Large-scale demonstration datasets have been central to recent progress in generalpurpose robot policies However, existing datasets are collected in human-absent settings…

In short

HABIT is a new dataset featuring robot demonstrations in human-present environments to train robots with human-aware behaviors. It organizes tasks into three roles: Collaborator, Coworker, and Supervisor. Training on this data allows robots to develop specific skills like spatiotemporal synchronization and proactive yielding, leading to safer and more coordinated interactions than robot-only data.

Key concepts

Dataset Structure
HABIT organizes robot tasks into three interaction roles: Collaborator (working together), Coworker (working separately in shared space), and Supervisor (human directing the robot). This structure allows policies to learn distinct social behaviors tailored to different types of human-robot interaction.
Human-Aware Behaviors
The dataset is designed to elicit specific human-aware actions, such as yielding when a collision risk exists or aligning movement speed with a human partner. These behaviors are learned directly from the data, enabling robots to perform complex coordination that robot-only datasets cannot teach.
Mid-training Efficiency
Training policies on HABIT during the middle stages significantly improves learning efficiency. Mid-training with just 100 demonstrations can outperform direct fine-tuning for tasks like waste sorting, showing the dataset provides a strong prior that allows for rapid acquisition of new interaction skills.
Robustness to Shifts
HABIT policies show stability when faced with changes in human appearance or environment (out-of-distribution shifts). Performance degradation is mostly limited to manipulation errors, indicating that the learned role-specific social behaviors remain robust even when the human context changes slightly.

Terminology used across episodes

This episode discusses

The paper

HABIT: Human-Aware Behavior and Interaction Training Dataset for Robot Manipulation · Read on arXiv

KAIST

Large-scale demonstration datasets have been central to recent progress in general-purpose robot policies. However, existing datasets are collected in human-absent settings, and policies trained on such data may perform tasks competently in isolation but fail to exhibit human-aware behaviors. To address this gap, we introduce HABIT, a large-scale robot demonstration dataset for human-present environments. We organize tasks into three roles capturing distinct modes of human-robot interaction: Collaborator, where human and robot jointly accomplish a task; Coworker, where they pursue separate tasks in a shared space; and Supervisor, where the human directs the robot. The dataset comprises over 10K episodes and over 160 hours across 60 tasks. Our experiments show that training on human-present data elicits human-aware behaviors that robot-only data fails to produce: spatiotemporal synchronization in Collaborator tasks, yielding in Coworker tasks, and gesture grounding in Supervisor tasks. Moreover, training on HABIT enables rapid adaptation to new human-robot interaction tasks. By introducing human presence as a new axis of dataset diversity, HABIT extends robot policies to environments shared with humans.

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: We're looking at the paper: "HABIT: Human-Aware Behavior and Interaction Training Dataset for Robot Manipulation".

Dev: Here's the gist of the paper. Large-scale demonstration datasets have been central to recent progress in generalpurpose robot policies However, existing datasets are collected in human-absent settings, and policies trained on such data may perform tasks competently in isolation but fail to exhibit human-aware behaviors To address this gap, HABIT introduces a large-scale robot demonstration dataset for human-present environments.

Rosa: Dataset Structure and Roles HABIT organizes tasks into three roles capturing distinct modes of humanrobot interaction: Collaborator, where human and robot jointly accomplish a task; Coworker, where they pursue separate tasks in a shared space; and Supervisor, where the human directs the robot.

Dev: The dataset comprises over 10K episodes and over 160 hours across 60 tasks.

Rosa: Data Collection Protocol The collection protocol is structured to elicit specific human-aware behaviors, varying other conditions to support generalization.

Title and authors: Rosa: So we're talking about this paper called "HABIT: Human-Aware Behavior and Interaction Training Dataset for Robot Manipulation". It’s about how to get robots to actually behave like they are working alongside a person instead of just doing whatever the training data says.

Dev: Right, Rosa. The main point here is that existing robot datasets are usually collected when there's no human around, and that means the robots can be competent in isolation but they totally miss how to interact with people in real life.

Taro: So, if a robot only sees instructions without a human partner, it won't know how to handle unexpected movements or social cues when someone is actually there.

Rosa: Exactly. HABIT introduces a new dataset specifically designed for human-present environments, and it organizes tasks into three interaction roles: Collaborator where you work together, Coworker where you share space doing separate things, and Supervisor where the human directs the robot.

Dev: And this dataset is pretty substantial; it’s got over ten thousand episodes and more than one hundred sixty hours of demonstration time across sixty different tasks. That's a lot of data to work with for training policies.

Taro: I see those roles, but what does that actually mean in practice when the robot is trying to learn? Like, what's the difference between just being in a shared space and actually coordinating with someone who might be bumping into you?

Rosa: Well, the paper shows they designed their collection protocol to elicit specific human-aware behaviors. They varied things like making operators only react after seeing what their partner does, or forcing the robot to yield when it gets close to a human or something they are holding.

Dev: That yielding under safety priority is interesting from a control loop standpoint; you’re essentially baking safety constraints directly into the learned behavior rather than just having them as an external check.

Taro: And they also did temporal adaptation, where the human operator's speed changes across episodes so the policy has to match that tempo, which seems like a good way to force better synchronization.

Rosa: Plus, they have gesture grounding where the robot has to pay attention to specific gestures by varying how long the human waits before pointing from certain spots. That forces policies to attend to those signals instead of just focusing on the physical object manipulation.

Title and authors: Dev: So, when you look at their evaluation framework, they aren't just looking at if the robot can move a cup; they’re measuring manipulation performance alongside workflow compliance and safety during interaction.

Taro: What did the experiments show about what kind of behaviors these policies actually learned compared to robot-only data? Was it really different stuff?

Rosa: The results show that training on this human-present data elicits specific human-aware behaviors that robot-only data just can't produce. They specifically point to spatiotemporal synchronization in Collaborator tasks, gesture grounding in Supervisor tasks, and proactive yielding under the Coworker role.

Dev: That suggests the key gain is achieving things like anticipating human motions or reacting safely when things are close by, which are behaviors that emerge directly from the data itself.

Taro: When we look at failure analysis, they found failures weren't just simple manipulation errors; they were distinct modes of failure tied to each role. For instance, in Collaborator tasks, you could see precondition violations and in Coworker tasks, collisions were dominating.

Rosa: And the key finding is that HABIT substantially reduces these role-specific failures across all three roles because it teaches those human-aware behaviors that conventional demonstration data simply cannot teach.

Dev: And they also showed something about efficiency; mid-training on this dataset improves both sample efficiency and final performance on downstream interaction tasks. That’s significant for real deployment because you don't need a massive amount of data upfront to get good results.

Taro: So, if I have limited demonstrations, like just a few hours of interaction data, training mid-way through this HABIT dataset gives the policy a really strong prior that lets it quickly acquire new tasks.

Rosa: It basically means we can rapidly adapt to new human-robot interaction tasks using only limited demonstrations because the foundation is already human-aware.

Dev: And they also looked at how robust the performance was when things got slightly different, especially under out of distribution conditions where the human-centric distribution shifts, like if you change what clothing people wear.

Taro: The results showed that performance degradation under those shifts is pretty modest and that role-specific behaviors tend to stay largely stable even when the environment changes.

Title and authors: Rosa: So, it seems this dataset establishes human presence as a learnable and impactful axis for diversity in robot learning, showing that demonstrations collected with a co-present human carry a signal that data without humans just can't provide.

Dev: And for future work, they suggest incorporating both the egocentric view and the exocentric view of the human into policy training to further improve that understanding.

Taro: I think incorporating both views would be really helpful for getting a complete picture of what’s happening in the shared workspace.

Rosa: It definitely sounds like a direction for future research, but what we have here is this dataset proving that training on human-present data generates behaviors—like spatiotemporal synchronization or proactive yielding—that robot-only data just cannot produce.

Dev: That's the core message: the quality of the interaction behavior you get from your robot policy depends entirely on whether you train it with data that shows humans actually interacting with robots in complex ways.

Taro: It’s less about teaching a robot to move an object and more about teaching it how to navigate social dynamics in a shared space.

Rosa: And the implications are huge for moving robots from controlled labs out into messy, real-world environments where people are constantly around.

Dev: We need to keep an eye on this because if we want robots that can safely and effectively work with people, we have to start thinking about how to collect data like this more systematically.

Taro: It sets a new baseline for what a robot should know about human interaction before it even starts learning the physical task itself.

Rosa: So that's what HABIT is: a large-scale dataset that shows us exactly what we need to teach robots about being part of a human world.

Dev: And it’s been fascinating seeing how mid-training on this data boosts efficiency so much when you only have limited examples to begin with.

Taro: It’s a great way to get a robot up and running quickly without needing hours of perfect, pre-recorded interaction data for every single task.

Rosa: That's what we're hearing about the HABIT dataset today, showing how it teaches robots coordination and safety through human-aware demonstrations.

Dev: We’ll keep an eye on how researchers incorporate those egocentric and exocentric views in future policy training.

The paper's summary: Rosa: So, we've talked about how HABIT is a big dataset for teaching robots how to work with people, and now we look at what they actually built in this paper.

Dev: The authors are presenting this dataset, HABIT, which collects demonstrations of robots interacting with humans in real-world settings where the human is present.

Rosa: It's not just a collection of videos; they've organized these interactions into three distinct roles: Collaborator, Coworker, and Supervisor.

Dev: The core idea here is that you get different kinds of data depending on how the human and robot are supposed to be interacting—whether they are working together, sharing a space, or if the human is directing everything.

Rosa: They collected over ten thousand episodes and one hundred sixty hours of interaction time across sixty different tasks for these three roles. That’s a lot of raw data to feed into a policy system.

Dev: What they're showing is how they structured the collection protocol to specifically pull out those human-aware behaviors we talked about earlier, like spatiotemporal synchronization or proactive yielding.

Rosa: The researchers varied things like making the robot only react after seeing what the human does, or forcing it to yield when it gets close to a person holding something.

Dev: And they also messed with temporal adaptation, changing how fast the human moves across different episodes so the policy has to learn how to match that tempo.

Rosa: Plus, they did gesture grounding by making the robot wait before pointing based on where the human is pointing from. This forces the AI to pay attention to those specific social cues rather than just looking at objects.

Dev: The evaluation framework they used isn't just measuring how well the robot can pick up a cup; it assesses manipulation performance alongside workflow compliance and safety during that interaction.

Rosa: And what they found is that training on this HABIT data gets policies to learn behaviors that robot-only data simply cannot produce. It shows specific coordination like anticipating human motions or reacting safely when things are close by.

Dev: This means the system isn't just following programmed steps; it’s learning social skills directly from the demonstrations.

Rosa: And they also looked at how this data helps with efficiency, showing that mid-training on HABIT improves both how quickly a robot learns and how well it performs on those interaction tasks.

Dev: It suggests that you don't need an enormous amount of perfect, pre-recorded interaction data to get a good result when you use this prior knowledge from HABIT.

Rosa: They also looked at robustness, seeing if the robot still does well even when the environment changes slightly, like if the clothes people are wearing are different colors or silhouettes.

Dev: The results showed that performance degradation under those shifts is mostly limited to basic manipulation failures, not to those specific human-aware behaviors like yielding or synchronization.

Rosa: It seems HABIT establishes a really strong baseline for what a robot needs to know about being part of a human world before it even starts learning the physical task itself.

Dev: It sets a new standard for how we can gather data that actually teaches robots social coordination and safety in shared spaces.

Rosa: So, the big picture here is that this dataset proves human presence is a learnable and important part of robot diversity, showing what kind of data we need to collect.

The paper's improvements: Dev: So, we've covered how HABIT collects data focused on specific social behaviors during robot interactions, and now we look at what improvements they propose to make this dataset even better for training policies.

Rosa: The authors suggest that incorporating human egocentric view and exocentric view into the policy training would be a big step forward for understanding these interactions.

Dev: That means the system wouldn't just see what the robot is doing, but it would also understand what the human is seeing and how they are perceiving things from their perspective.

Rosa: Thinking about that, it’s like giving a robot two sets of eyes—one focused on its own actions and one focused on where you are looking.

Dev: That should help the AI learn more complex social dynamics because it gets a richer understanding of the entire shared scene.

Rosa: It seems like this would give policies better context, which is crucial when dealing with those nuanced interactions we saw in the Coworker role, like avoiding collisions.

Dev: And for me, that means better real-time control because the system can anticipate human movements more accurately instead of just reacting to them after the fact.

Rosa: The paper implies that this dual view approach could lead to much more stable and predictable behavior when things get messy in a real environment.

Dev: It would address some of those failure modes we saw before, like those precondition violations in the Collaborator tasks, by giving the policy more complete information.

Rosa: It changes things for someone who only listens to the show because it moves us closer to robots that can handle unpredictable human behavior with more grace and safety.

Dev: The implication is that we might move beyond just teaching robots how to manipulate objects and start teaching them how to navigate social situations effectively.

Rosa: Exactly, it’s about making the learning process itself more human-aware from the start, rather than trying to patch in those social skills later.

Dev: And this leads us into a bigger question: if we can get this level of understanding from data augmentation, how does that fit into larger system designs for things like 6G networks or multi-agent systems?

Conclusion: Rosa: So we're wrapping up this look at HABIT, which is really about how to build a dataset that teaches robots how to behave like they’re working with people in shared spaces.

Dev: It establishes human presence as a learnable axis for data diversity, showing that demonstrations with humans carry signal that robot-only data just can't provide.

Rosa: The main implication is that we finally have a way to generate policies that exhibit specific coordination, like spatiotemporal synchronization or proactive yielding in different roles.

Dev: It moves us away from robots just executing commands and toward systems that can handle complex, dynamic social interactions safely.

Rosa: It’s about moving robots from controlled lab settings into messy, real-world environments where people are constantly around without breaking things or causing accidents.

Dev: That's the practical side of it—we need these kinds of demonstrations to build a solid safety case for real deployment.

Taro: From an autonomy research standpoint, I think this is huge because it gives us a way to test failure modes that are inherently social, not just physical ones.

Rosa: Right, Taro? Because when you can teach the system how to yield or ground gestures, you're testing its ability to handle unexpected human stuff in a very realistic way.

Dev: It means we can use mid-training on this data to get rapid adaptation for new interaction tasks because the policy already has that human-aware foundation.

Taro: I agree, that rapid adaptation is key; if it learns social rules quickly, it can recover more gracefully when the world misbehaves in a novel way.

Rosa: So, while this paper focuses on the dataset itself and its structure—the three roles and the elicitation choices like varying human speed—the real impact is on what we can learn from it.

Dev: It changes how we think about data collection for robotics; it shows that collecting data with a co-present human is a much more valuable signal than just recording physical manipulation.

Taro: It sets a new baseline for what we expect robots to know about human interaction before they even start learning the physical task itself.

Rosa: We’ve seen how HABIT helps us understand the social dynamics, but next up, we're going to look at how those ideas translate into policy training using both egocentric and exocentric views of the human.

More episodes

← Home