M2HRI: An LLM-Driven Multimodal Multi-Agent Framework for Personalized Human-Robot Interaction

arXiv:2604.11975 · cs.RO · Submitted 2026-04-13 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: I'm Rosa.

Dev: And I'm Dev, here with Taro.

Rosa: We're looking at the paper: "M2HRI: An LLM-Driven Multimodal Multi-Agent Framework for Personalized Human-Robot Interaction".

Paper summary: Dev: So to wrap up, the paper "M2HRI: An LLM-Driven Multimodal Multi-Agent Framework for Personalized Human-Robot Interaction" introduces a way to give robots identity by modeling personality and memory, and then uses a contextualized coordination mechanism to manage how they participate in an interaction.

Taro: The main implication I see is that it moves the goal past just building functional robots to building social agents that users can actually connect with on a deeper level because their distinct personalities are being reflected in the interaction.

Rosa: Right, and it shows that this isn't just about adding more sensors; it’s about using Large Language Models to process multimodal input—speech and vision—to build a rich, personalized context for each robot.

Dev: The authors claim that this approach improves things like conversational flow and response appropriateness by making the coordination dynamic instead of relying on static rules for who talks next. They use a controlled user study with one hundred five participants to back up these claims.

Taro: If you only listen to this, think about how much more natural a conversation feels when the robot seems to actually *know* you from previous interactions, not just what you said in the last three seconds.

Rosa: That’s right. The framework suggests that if we want robots in homes and hospitals to be truly useful social partners, they need that layer of distinct identity and contextual awareness woven into their core architecture.

Conclusion: Rosa: So we're wrapping up on M2HRI, which is this framework where robots get personalities and memory so they can interact with people more like actual individuals instead of just being interchangeable machines.

Dev: Yeah, the authors are really pushing the idea that giving each robot a distinct personality profile and a long-term memory changes how they coordinate their actions in a group.

Rosa: It seems like the big thing is how this structure lets you move past just having robots do tasks and toward having them build some kind of social relationship with users based on those stored traits.

Dev: Exactly, it’s not just about making the robots smarter in terms of what they can do, but making them more aware of who they are interacting with contextually.

Rosa: The study shows that this approach really helps improve how well the agents remember things and how naturally the interactions feel when you have these distinct identities involved.

Dev: I see it as a safety thing too, because if the coordination is based on those personalities, there’s a way to reason about what happens when one robot acts in a way that clashes with another's established personality.

Rosa: Right, and it opens up some big questions about how much we need to model these human-like traits versus just making the coordination rules really complex.

Dev: And it makes you wonder what the real limits are on how long this memory can be useful before things get too messy in a busy environment like a hospital or a home.

Shaid Hasan, Breenice Lee, Sujan Sarker, Tariq Iqbal

University of Virginia

cs.RO

Submitted: 2026-04-13

Updated: 2026-06-21

Journal ref: 2026 IEEE 35th International Conference on Robot and Human Interactive Communication (RO-MAN), pp. 213-220 (2026)

DOI: 10.1109/RO-MAN72286.2026.11714798

Project page: https://project-m2hri.github.io

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 89/100

The gist: Multi-robot systems hold significant promise for social environments such as homes and hospitals, yet existing multi-robot systems often treat robots as functionally interchangeable…

Key concepts

Agent Identity Profile ($\phi_i$)
This defines each robot's core identity, consisting of its memory ($M_i$) and its personality representation ($B_i$). This profile is crucial because it dictates how the robot perceives itself and others, shaping its unique behavior in social environments.
Cognitive Loop ($\Lambda_i$)
This describes the internal operational process of each agent. It consists of three parts: perception ($P_i$), reasoning/planning ($R_i$), and execution ($E_i$). This loop allows each robot to independently process information, plan actions, and carry out those plans.
Personality Modeling (FFM)
Agents are modeled using the Five-Factor Model (Openness, Conscientiousness, Extraversion, Agreeableness, Neuroticism). This five-dimensional vector explicitly shapes the agent's personality and consequently influences its reasoning and how it interacts with users.
Contextualized Coordination Mechanism
This mechanism regulates how multiple agents interact while keeping their individual cognition decentralized. A shared coordinator assigns a participation-suitability score ($r_{i,t}$) based on the current context ($Z_t$), managing turn-taking and agent involvement.

Terminology

Summary

Multi-robot systems hold significant promise for social environments such as homes and hospitals, yet existing multi-robot systems often treat robots as functionally interchangeable, overlooking how distinct agent identities shape user perception and how such individuality changes the coordination requirements of multi-robot interaction

Core Framework Introduction

M2HRI introduces a multimodal multi-agent framework that models each robot as an identity-bearing agent through personality and long-term memory, together with a contextualized coordination mechanism that regulates agent participation

Agent Architecture Details

The framework models each robot agent ai as ai = ⟨ϕi,Λi, Γi⟩ where ϕi denotes the agent identity profile, Γi its action capability set and Λi denotes the cognitive loop

The agent identity profile is defined as ϕi = ⟨Mi, Bi⟩ where Mi denotes the agent’s memory, and Bi its personality representation

The cognitive loop is defined as Λi = ⟨Pi, Ri, Ei⟩ where Pi denotes the perception module, Ri the reasoning (planning) module, and Ei the execution module

Perceptual Processing and Memory

The Perception module Pi is implemented using a vision language model (VLM) that jointly processes speech and visual input to produce a coherent natural-language description of the current interaction context

The Memory Module provides each agent with persistent and contextual information processing capabilities, enabling both long-term knowledge retention and short-term context awareness

The memory module is organized into three distinct submodules: Working memory, which maintains a short-term interaction context consisting of recent conversational history and the current interaction state

Personality Modeling and Reasoning

The Personality Module shapes each agent’s reasoning and interaction behavior through an explicit and persistent personality representation

Each agent’s personality is modeled using the Five-Factor Model (FFM) [32], also known as the Big Five and represented as a five-dimensional vector Bai = (O, C, E, A, N)

Contextualized Coordination Mechanism

M2HRI adopts a contextualized coordination mechanism (Figure 2b) to regulate interaction among multiple agents while preserving decentralized cognition

Each agent independently maintains its perception-planning-execution loop, while a shared coordinator governs interaction turn-taking and agent participation

The coordinator then jointly reasons over Zt to assign a participation-suitability score to each agent: ri,t = fcoord(Zt) i, ri,t ∈ [0, 1]

The study found that Openness and Neuroticism produced strong perceptual separation, whereas Conscientiousness was less salient

Trait recognition accuracy was highest for Openness and Extraversion (93.3%), followed by Agreeableness and Neuroticism (86. p. 5

Where the paper says it

Accuracy was highest for Openness and Extraversion (93. 3%), followed by Agreeableness and Neuroticism (86.

7%)

Experimental Findings on Memory

Long-term memory significantly improves recall accuracy, preference awareness, and interaction naturalness

Recall accuracy showed a strong effect (t(14) = 6.52, p <.001, 95% CI [1. p. 4

Where the paper says it

** *** O C E A N (d) Trait Recognition 0% 20% 40% 60% 80% 100% Recognition accuracy (%) *** ** *** *** *** Significant (*** p<. 001, ** p<.

70, 3.37]), as did awareness of user preferences (t(14) = 12. p. 5
Where the paper says it

Memory (RQ2) We evaluated the effect of long-term memory on user perception by comparing with-memory and without-memory conditions across three measures (Fig. 4a): recall accuracy, awareness of user preferences, and interaction naturalness.

65, p <.001, 95% CI [2. p. 4
Where the paper says it

** *** O C E A N (d) Trait Recognition 0% 20% 40% 60% 80% 100% Recognition accuracy (%) *** ** *** *** *** Significant (*** p<. 001, ** p<.

p. 4
Where the paper says it

** *** O C E A N (d) Trait Recognition 0% 20% 40% 60% 80% 100% Recognition accuracy (%) *** ** *** *** *** Significant (*** p<. 001, ** p<.

22, 3.12])

Overlap avoidance demonstrated the strongest effect (t(14) = 7.89, p <.001, 95% CI [2.09, 3.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Agent individuality modeling: M2HRI models each robot as an identity-bearing agent through personality and long-term memory, allowing users to perceive robots as distinct social agents rather than identical machines operating side by side.

  2. Contextualized coordination mechanism: The framework uses a contextualized coordination mechanism that regulates agent participation by calculating a participation-suitability score for each agent based on its observation, personality, and shared context.

  3. Enhanced personalization through memory: Agents maintain an agent-specific long-term memory store Mai, enabling them to improve recall accuracy and preference awareness, which leads to a perception of personalized and contextually aware interaction.

  4. Improved conversational flow: Contextualized coordination directly addresses multi-agent challenges by improving conversational flow, response appropriateness, and overlap avoidance by selecting the most suitable agent for the turn.

Abstract

Multi-robot systems hold significant promise for social environments such as homes and hospitals, yet existing multi-robot systems often treat robots as functionally interchangeable, overlooking how distinct agent identities shape user perception and how such individuality changes the coordination requirements of multi-robot interaction. To address this, we introduce M2HRI, a multimodal multi-agent framework that models each robot as an identity-bearing agent through personality and long-term memory, together with a contextualized coordination mechanism that regulates agent participation. In a controlled user study (n = 105) in a multi-agent human-robot interaction (HRI) scenario, we found that most personality contrasts were distinguishable and consistently expressed. Long-term memory improved preference awareness and interaction naturalness, while contextualized coordination improved conversational flow, response appropriateness, and overlap avoidance. Together, these findings show that agent individuality and contextualized participation coordination play complementary roles in supporting coherent and socially appropriate multi-agent HRI. Project website available at https://project-m2hri.github.io/.

Sources

Related papers