Improving Human-Robot Teamwork in Urban Search and Rescue Through Episodic Memory of Prior Collaboration
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: We're looking at the paper: "Improving Human-Robot Teamwork in Urban Search and Rescue Through Episodic Memory of Prior Collaboration".
Jane: Here's the gist of the paper. Effective human-robot teamwork requires robots to adapt to partners, situations, and task dynamics from the start of an interaction <ref:2606.18836#pg2> The strongest gains appear at the beginning of interaction, suggesting that reusable episodic memory can help robots enter collaboration with more effective task knowledge and support smoother early teamwork <ref:2606.18836#pg8> This study investigates whether a robot can use prior team experience to become a better teammate in future interactions <ref:2606.18836#pg2> Episodic Team Memory Representation The authors represent collaboration patterns (CPs) from prior human-robot interactions as knowledge-graph episodic memories <ref:2606.18836#pg6> This representation preserves "situational context, action structure, and task outcomes in a form that can be reused across episodes" <ref:2606.18836#pg2> They define six entity types for their ontology, including "robot, participant, cp, situation, robot action, and human action" Relationships are captured through four edge families: "has cp, has situation, has robot action, and has human action" They make a fine-grained distinction for graph learning by treating indexed action relations as different relation labels to capture action order How the training signal is built The authors use a Relational Graph Convolutional Network (RGCN)
23: to learn meaningful representations of the CPs The model uses a node-classification objective with a single linear classifier on top of the final RGCN layer Node properties are transformed into fixed-length vectors, where cp nodes use six numerical features concatenated and normalized For situation, human action, and robot action nodes, string attributes are encoded using Sentence-BERT to obtain numerical vector representations The final graph-level embeddings are obtained by mean pooling the final node embeddings from the RGCN Where the gains come from The performance improvement is significant, raising rescue success from 25.7% to 41.3% and reducing average task time by 283 seconds <ref:2606.18836#pg2> The strongest gains appear at the beginning of interaction, suggesting the memory helps robots enter collaboration with "more effective task knowledge and support smoother early teamwork" The selected CP is a "minimal yet effective interaction pattern, where the robot operates with greater autonomy while maintaining efficiency" Representative CP Selection The authors cluster CPs using K-means on the learned graph representations to identify representative patterns Cluster 1 exhibited a higher success rate, suggesting that participants using CPs from this cluster were "more likely to rescue the victim" They heuristically selected Cluster 1 as the cluster from which to draw a reusable CP because it "represents a minimal yet effective interaction pattern, where the robot operates with greater autonomy while maintaining efficiency" The final choice was the CP closest to the cluster center by Euclidean distance, which consists of "only two consecutive robot actions without any human actions" Human-Subject Evaluation The memory-initialized robot raised victim-rescue success from 25.7% to 41.
Title and authors: Tom: Let's talk about who wrote this and what this title is really saying, Jane. It’s about using episodic memory to improve human-robot teamwork in urban search and rescue.
Jane: The authors are Taewoon Kim, Emma van Zoelen, and Mark Neerincx. They are tackling the problem of how robots can learn from their past collaborations so they can perform better in new rescue tasks.
Lu: What’s interesting is that they frame collaboration patterns as knowledge-graph episodic memories, which means they're not just saving a list of actions but capturing the whole situation and context.
Meng: So, for someone like me who has to deploy models, what does representing a pattern as a "knowledge-graph episodic memory" actually mean in practice? Is it just storing logs?
Lalam: It’s more than logs. They are defining specific entity types—robot, participant, situation—and relationships between them like robot action and human action, which gives the system a rich structure to work with.
Tom: Right. And they use this structure for graph learning to find the most useful pattern for the robot to adopt before starting a new mission.
Jane: So, it’s not just remembering what happened; it’s turning those memories into something that can be used mathematically to guide future robot behavior in a collaborative setting.
Meng: If I were deploying this, I'd want to know what the cost of building and maintaining these memory graphs is at scale. Does storing all these patterns become computationally expensive quickly?
Lu: They do tackle the representation part by using a Relational Graph Convolutional Network, or RGCN, to learn meaningful representations of these collaboration patterns.
Tom: The model uses node classification with a single linear classifier on top of the final layer to figure out which pattern is good for reuse. That seems like a solid mechanical approach.
Jane: And they use sentence-BERT to turn the descriptive text about situations and actions into numerical vectors, which lets the graph learning part actually understand what those actions mean contextually.
Meng: So, we're feeding it text descriptions, running a graph model on top of that, and then picking one pattern to load for a new task. That’s a pretty complex pipeline.
Lalam: It’s designed to be reusable across episodes by treating indexed action relations as different relation labels specifically for the graph learning part, which helps capture action order without changing the core ontology structure.
The paper's summary: Tom: So, what’s the main takeaway from this paper? They show that initializing the robot with a selected prior collaboration pattern significantly boosts its performance in search and rescue.
Jane: They found that this method increases rescue success rate from twenty-five point seven percent to forty-one point three percent and cuts the average task time by two hundred eighty-three seconds across twenty participants and one hundred sixty rounds of observation <ref:2606.18836#pg1,from 25.7% to 41.3% and>.
Lu: The key finding is that the gains are strongest right at the beginning of an interaction, which suggests that this memory helps robots enter collaboration with more effective task knowledge and support smoother early teamwork.
Meng: That makes sense, but I wonder about the setup. They chose one specific pattern to initialize with; how much effort does it take for a human to define that initial pattern before the robot can even start using it?
Jane: The paper suggests they use K-means clustering on the learned graph representations to identify representative patterns, and they selected Cluster one because it represents a "minimal yet effective interaction pattern where the robot operates with greater autonomy while maintaining efficiency <ref:2606.18836#pg2>."
Tom: So, they didn't just pick any random pattern; they used their own learning process to select one that seems inherently efficient for the robot.
Lu: That selection process involves clustering and then picking the CP closest to that cluster center by Euclidean distance, which is what resulted in a specific pattern: "only two consecutive robot actions without any human actions."
Meng: Wait, only two consecutive robot actions? Is that really all it takes to define a useful prior experience for a complex rescue mission? That seems very simple.
Jane: It’s about finding that minimal yet effective interaction pattern, and they tested the memory-initialized robot against a baseline to show the performance lift.
Tom: The experiment included one practice round followed by eight real rounds—five easier ones and three harder ones introducing a brown rock that could make rescue impossible if handled poorly.
Lu: That testing setup is good because it shows the system handling both routine tasks and genuinely difficult, high-stakes scenarios where teamwork matters most.
The paper's improvements: Tom: The main improvement they propose is initializing the robot with this automatically selected prior collaboration pattern instead of starting with an empty memory state.
Jane: This initialization increases rescue success from twenty-five point seven percent to forty-one point three percent, which is a sixty point seven percent relative improvement, and it reduces the average task time by two hundred eighty-three seconds in those tests <ref:2606.18836#pg1,increases rescue success from 25.7% to 41.3>.
Meng: So, the direct impact on efficiency is measurable in time saved and success rate gained; that’s concrete data we can use for deployment metrics.
Lu: The representation itself is a major improvement because they moved away from fixed-length Euclidean vectors for collaboration patterns and instead used property graphs to retain full relational context, which they call knowledge-graph episodic memories.
Tom: They defined six entity types—robot, participant, cp, situation, robot action, and human action—and captured relationships through four edge families like has cp or has situation.
Jane: This structure is what allows the memory to preserve "situational context, action structure and task outcomes in a form that can be reused across episodes," which is crucial for long-term learning.
Meng: I’m still curious about the training signal they built for this representation. How did they train that RGCN to recognize these meaningful patterns in the first place?
Lu: They use a node-classification objective with a single linear classifier on top of the final RGCN layer, transforming node properties into fixed-length vectors and using Sentence-BERT for encoding text attributes like situation descriptions.
Tom: And they get those final graph-level embeddings by mean pooling the embeddings from the RGCN across all nodes in the graph. That’s how they distill the whole experience down to a single representation.
Jane: The real improvement here is that this entire process—from defining entities to learning representations and selecting a pattern—is automated, allowing the robot to become a better teammate without needing explicit reprogramming for every new task.
Meng: If I were shipping this, I’d need assurance that the system doesn't just memorize noise from training data; they have to show robustness when encountering novel situations not seen in those prior episodes.
Lu: The method is designed to be general by treating indexed action relations as different relation labels for the graph learning part, which lets them capture action order without rigidly fixing the ontology structure itself.
Conclusion: Tom: So, to wrap up this paper on "Improving Human-Robot Teamwork in Urban Search and Rescue Through Episodic Memory of Prior Collaboration," they've shown that using prior experience can substantially improve robot performance.
Jane: They demonstrated that initializing the robot with a selected prior collaboration pattern leads to a rescue success rate increase from twenty-five point seven percent to forty-one point three percent and a task time reduction of two hundred eighty-three seconds, with the largest gains appearing at the very start of interaction <ref:2606.18836#pg1,from 25.7% to 41.3% and>.
Lu: The implication is that robots can carry forward useful team experience into new interactions by representing historical collaboration patterns as knowledge-graph episodic memories, which is a flexible way to store context.
Meng: For us in production, this means we’re moving towards systems that proactively select the best prior experience to guide their behavior before they even start the work.
Lalam: The system uses graph representation learning and node-type supervision to organize those patterns, ensuring the memory it initializes with is structurally similar and effective for use.
Tom: So, this paper shows that by letting robots learn from their past interactions through these structured memories, we can see a measurable boost in how well they collaborate with humans in difficult tasks.
Jane: It’s a step toward making robot teammates more adaptive and less reliant on purely reactive responses when facing the unpredictable dynamics of an environment like urban search and rescue.
Lu: The limitation they mention is that the representation relies heavily on the initial graph structure, so if you feed it completely alien data, its performance might drop significantly because it’s trying to fit a pre-existing learned context.
Meng: And for deployment, I’d need to know how much computational overhead that knowledge graph and RGCN part adds compared to just using a simpler task planner.
Lalam: The core idea is that this prior team experience becomes an effective form of task knowledge that the robot can use to support smoother early teamwork.
Taewoon Kim, Emma van Zoelen, Mark Neerincx
HumemAI · Vrije Universiteit Amsterdam · TNO
cs.HC, cs.AI
Submitted: 2026-06-17
Updated: 2026-10-29
Comments: Accepted at IEEE RO-MAN 2026
DOI: 10.1109/RO-MAN72286.2026.11714607
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Importance score: 75/100
The gist: Effective human-robot teamwork requires robots to adapt to partners, situations, and task dynamics from the start of an interaction <ref:2606.18836#pg2> The strongest gains appear at the beginning of…
Key concepts
- Episodic Team Memory Representation
- This involves storing past human-robot collaborations as structured knowledge graphs. It captures the context, action structure, and outcomes of previous teamwork episodes in a way that can be easily reused for future interactions.
- Relational Graph Convolutional Network (RGCN)
- This is a machine learning model used to learn meaningful representations of collaboration patterns from the knowledge graph. It processes the relationships between different entities like robots, humans, and actions to understand how these patterns evolve over time.
- Representative CP Selection
- The system uses clustering on learned graph representations to find the most effective past collaboration pattern. This representative pattern is then selected as a starting point for the robot's memory, ensuring it carries useful experience into new tasks.
Terminology
Summary
Effective human-robot teamwork requires robots to adapt to partners, situations, and task dynamics from the start of an interaction <ref:2606. p. 018836#pg2> The strongest gains appear at the beginning of interaction, suggesting that reusable episodic memory can help robots enter collaboration with more effective task knowledge and support smoother early teamwork <ref:2606. p. 018836#pg8> This study investigates whether a robot can use prior team experience to become a better teammate in future interactions <ref:2606. p. 018836#pg2>
Episodic Team Memory Representation
The authors represent collaboration patterns (CPs) from prior human-robot interactions as knowledge-graph episodic memories <ref:2606.18836#pg6> This representation preserves situational context, action structure, and task outcomes in a form that can be reused across episodes
<ref:2606.18836#pg2> They define six entity types for their ontology, including robot, participant, cp, situation, robot action, and human action
Relationships are captured through four edge families: has cp, has situation, has robot action, and has human action
They make a fine-grained distinction for graph learning by treating indexed action relations as different relation labels to capture action order
How the training signal is built
The authors use a Relational Graph Convolutional Network (RGCN) [23] to learn meaningful representations of the CPs The model uses a node-classification objective with a single linear classifier on top of the final RGCN layer Node properties are transformed into fixed-length vectors, where cp nodes use six numerical features concatenated and normalized For situation, human action, and robot action nodes, string attributes are encoded using Sentence-BERT to obtain numerical vector representations The final graph-level embeddings are obtained by mean pooling the final node embeddings from the RGCN
Where the gains come from
The performance improvement is significant, raising rescue success from 25.7% to 41.3% and reducing average task time by 283 seconds <ref:2606. p. 018836#pg2> The strongest gains appear at the beginning of interaction, suggesting the memory helps robots enter collaboration with more effective task knowledge and support smoother early teamwork
The selected CP is a minimal yet effective interaction pattern, where the robot operates with greater autonomy while maintaining efficiency
Representative CP Selection
The authors cluster CPs using K-means on the learned graph representations to identify representative patterns Cluster 1 exhibited a higher success rate, suggesting that participants using CPs from this cluster were more likely to rescue the victim
They heuristically selected Cluster 1 as the cluster from which to draw a reusable CP because it represents a minimal yet effective interaction pattern, where the robot operates with greater autonomy while maintaining efficiency
The final choice was the CP closest to the cluster center by Euclidean distance, which consists of only two consecutive robot actions without any human actions
Human-Subject Evaluation
The memory-initialized robot raised victim-rescue success from 25.7% to 41.
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Reusable Episodic Memory Initialization: The improved system initializes the robot with a selected prior collaboration pattern instead of an empty memory state before a new collaboration episode begins, which
increases rescue success from 25.7% to 41.3% (a 60.7% relative improvement)
andreduces average task time by 283 seconds.
-
Knowledge-Graph Episodic Memory Representation: The system represents collaboration patterns as knowledge-graph episodic memories, preserving
situational context, action structure, and task outcomes in a form that can be reused across episodes,
allowing the robot to carry forward auseful team experience into a new interaction.
-
Representative CP Selection via Graph Learning: The system uses graph representation learning with node-type supervision to organize historical collaboration patterns and identify a
representative prior collaboration pattern to initialize the robot before a new interaction begins,
ensuring the selected memory is structurally similar and effective. -
Early Interaction Performance Boost: The system demonstrates that the memory augmentation yields
the largest gains at the beginning of collaboration, when teams are still orienting to the task,
providinga more coordinated starting point
for humans and robots in unfamiliar environments.
Related papers
- EmoPhone: A Multi-Wave Dataset for In-the-Wild Mobile and Wearable Affect Sensing
- Tasteply: Externalizing Stable and Situational Preference as Manipulable Spatial Objects
- Shielded Trajectory Sampling for Personalized Relational Safety in Conversational AI
- From Tasks to Turns: Measuring Human Labor Contributions in Human-Agent Joint Production
- Reconstruction and Reflection of Positive Experiences through Resurfacing Laughter-indexed Everyday Moments
- LLM-Assisted Reranking to Operationalize Nuanced Objectives in Recommender Systems