M2HRI: An LLM-Driven Multimodal Multi-Agent Framework for Personalized Human-Robot Interaction
summary
The gist
Multi-robot systems hold significant promise for social environments such as homes and hospitals, yet existing multi-robot systems often treat robots as functionally interchangeable…
In short
M2HRI introduces a framework where robots are modeled as distinct agents with personalities and long-term memories. It uses a multimodal perception system and a contextualized coordination mechanism to manage interactions. The research shows that agent identity significantly impacts how users perceive the interaction, with personality traits like Openness strongly influencing trait recognition accuracy.
Key concepts
- Agent Identity Profile ($\phi_i$)
- This defines each robot's core identity, consisting of its memory ($M_i$) and its personality representation ($B_i$). This profile is crucial because it dictates how the robot perceives itself and others, shaping its unique behavior in social environments.
- Cognitive Loop ($\Lambda_i$)
- This describes the internal operational process of each agent. It consists of three parts: perception ($P_i$), reasoning/planning ($R_i$), and execution ($E_i$). This loop allows each robot to independently process information, plan actions, and carry out those plans.
- Personality Modeling (FFM)
- Agents are modeled using the Five-Factor Model (Openness, Conscientiousness, Extraversion, Agreeableness, Neuroticism). This five-dimensional vector explicitly shapes the agent's personality and consequently influences its reasoning and how it interacts with users.
- Contextualized Coordination Mechanism
- This mechanism regulates how multiple agents interact while keeping their individual cognition decentralized. A shared coordinator assigns a participation-suitability score ($r_{i,t}$) based on the current context ($Z_t$), managing turn-taking and agent involvement.
Terminology used across episodes
This episode discusses
- M2HRI: An LLM-Driven Multimodal Multi-Agent Framework for Personalized Human-Robot Interaction · Paper Radio
- Improving Human Motion Prediction Through Continual Learning
The paper
M2HRI: An LLM-Driven Multimodal Multi-Agent Framework for Personalized Human-Robot Interaction · Read on arXiv
Shaid Hasan, Breenice Lee, Sujan Sarker, Tariq Iqbal
University of Virginia
Multi-robot systems hold significant promise for social environments such as homes and hospitals, yet existing multi-robot systems often treat robots as functionally interchangeable, overlooking how distinct agent identities shape user perception and how such individuality changes the coordination requirements of multi-robot interaction. To address this, we introduce M2HRI, a multimodal multi-agent framework that models each robot as an identity-bearing agent through personality and long-term memory, together with a contextualized coordination mechanism that regulates agent participation. In a controlled user study (n = 105) in a multi-agent human-robot interaction (HRI) scenario, we found that most personality contrasts were distinguishable and consistently expressed. Long-term memory improved preference awareness and interaction naturalness, while contextualized coordination improved conversational flow, response appropriateness, and overlap avoidance. Together, these findings show that agent individuality and contextualized participation coordination play complementary roles in supporting coherent and socially appropriate multi-agent HRI. Project website available at https://project-m2hri.github.io/.
DOI: 10.1109/RO-MAN72286.2026.11714798
Transcript
Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.
Rosa: I'm Rosa.
Dev: And I'm Dev, here with Taro.
Rosa: We're looking at the paper: "M2HRI: An LLM-Driven Multimodal Multi-Agent Framework for Personalized Human-Robot Interaction".
Paper summary: Dev: So to wrap up, the paper "M2HRI: An LLM-Driven Multimodal Multi-Agent Framework for Personalized Human-Robot Interaction" introduces a way to give robots identity by modeling personality and memory, and then uses a contextualized coordination mechanism to manage how they participate in an interaction.
Taro: The main implication I see is that it moves the goal past just building functional robots to building social agents that users can actually connect with on a deeper level because their distinct personalities are being reflected in the interaction.
Rosa: Right, and it shows that this isn't just about adding more sensors; it’s about using Large Language Models to process multimodal input—speech and vision—to build a rich, personalized context for each robot.
Dev: The authors claim that this approach improves things like conversational flow and response appropriateness by making the coordination dynamic instead of relying on static rules for who talks next. They use a controlled user study with one hundred five participants to back up these claims.
Taro: If you only listen to this, think about how much more natural a conversation feels when the robot seems to actually *know* you from previous interactions, not just what you said in the last three seconds.
Rosa: That’s right. The framework suggests that if we want robots in homes and hospitals to be truly useful social partners, they need that layer of distinct identity and contextual awareness woven into their core architecture.
Conclusion: Rosa: So we're wrapping up on M2HRI, which is this framework where robots get personalities and memory so they can interact with people more like actual individuals instead of just being interchangeable machines.
Dev: Yeah, the authors are really pushing the idea that giving each robot a distinct personality profile and a long-term memory changes how they coordinate their actions in a group.
Rosa: It seems like the big thing is how this structure lets you move past just having robots do tasks and toward having them build some kind of social relationship with users based on those stored traits.
Dev: Exactly, it’s not just about making the robots smarter in terms of what they can do, but making them more aware of who they are interacting with contextually.
Rosa: The study shows that this approach really helps improve how well the agents remember things and how naturally the interactions feel when you have these distinct identities involved.
Dev: I see it as a safety thing too, because if the coordination is based on those personalities, there’s a way to reason about what happens when one robot acts in a way that clashes with another's established personality.
Rosa: Right, and it opens up some big questions about how much we need to model these human-like traits versus just making the coordination rules really complex.
Dev: And it makes you wonder what the real limits are on how long this memory can be useful before things get too messy in a busy environment like a hospital or a home.
More episodes
- 2608.13438-ContactGuard: Pre-Contact Execution Monitoring with Action-Conditioned Latent World Models
- 2610.34593-Future Prediction as Training-Only Video Supervision for Pretrained VLA Policies
- 2610.35385-EmbodiedLab: Beyond Manipulation, Benchmarking Embodied Agents That Act to Know
- 2610.36378-Data-Driven Hamiltonian for Neural Optimal Control
- 2610.37275-A Heuristic for Scheduling Autonomous Vehicles at Intersections
- 2610.37288-Dynamic Coordination of Autonomous Vehicles at Intersections
- 2610.37632-Real-Time Coordination of Autonomous Vehicles at Intersections
- 2611.13779-Softness is something you also need! Why compliance helps control: information metrics of a passive walker with compliant ankle
- 2611.13411-Residual-Corrected Physics-Informed Neural Network KKL Observers with Input-to-State Stability Guarantees
- 2611.14488-Is Your Robot's Memory Just a Clock? Temporal Shortcuts in Memory-Augmented VLA Policies