NestRL: A Nested Training Regime for Mutual Adaptation in Human-AI Teaming
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.
Rosa: I'm Rosa.
Dev: And I'm Dev, here with Taro.
Rosa: We're looking at the paper: "NestRL: A Nested Training Regime for Mutual Adaptation in Human-AI Teaming".
Paper summary: Rosa: Thinking about this paper, "NestRL: A Nested Training Regime for Mutual Adaptation in Human-AI Teaming," it really boils down to how you handle the complexity of mutual adaptation between a human and an AI.
Dev: The authors argue that instead of just trying to diversify training partners with static ones, which usually fails because the partners don't adapt, this nested structure forces the agent to learn about multiple levels of partner beliefs and behaviors.
Taro: It moves beyond just modeling what a partner *might* do and actually models how that partner adapts in response to the AI agent itself through a recursive training setup.
Rosa: And the implication is that we can get structured co-adaptation, which is consistent with how people actually coordinate in real human-robot interactions, rather than falling into those brittle conventions where agents only work with their training partners.
Dev: So for someone just listening to this show, it means we need to think about layering our training environments so the AI learns not just one way to work, but a hierarchy of ways to respond based on who the human is at that moment.
Conclusion: Rosa: So we’ve been looking at NestRL, this paper by Gmytrasiewicz and Doshi. Basically, they're tackling how humans and AI can actually adapt to each other in a complex way.
Dev: Right, it proposes a specific training setup called nested I-POMDPs to model that mutual adjustment between the human and the AI agent.
Taro: The core idea is that instead of just training one agent against fixed partners, you train it at multiple levels against agents from the level below.
Rosa: So, what does this actually mean for real-world teaming when things get messy and unpredictable?
Dev: It prevents that common problem where the AI just settles on one simple way to react because it's only ever seen one type of human partner.
Taro: The authors show that this nested structure forces the agent to consider multiple ways its partner might behave at different levels, which stops it from collapsing into a single, brittle strategy.
Rosa: So when you look at the results, they claim better performance against both unseen partners and even real human teammates.
Dev: They did validate this in an Overcooked cooking environment and showed NestRL agents hitting success rates of eighty-one point three percent with held-out partners.
Taro: And what’s interesting is that the qualitative analysis showed Level one agents learn to wait when paired with a fixed partner, which makes sense for human coordination.
Rosa: That waiting behavior seems complementary when it pairs up with the second level of agent, which learns to take initiative earlier than the first level.
Dev: So it suggests a structured way for agents to coordinate—like one person waits while the other starts moving toward a goal.
Taro: It’s about learning those layered coordination strategies that we see in actual human-robot interactions, not just random successful actions.
Rosa: This paper sets up a principled foundation for how we can train AI to handle diverse partners without them just getting stuck in rigid routines.
Dev: But there are still questions about extending this to situations where the agents don't even share the same reward function or goals.
Upasana Biswas, Durgesh Kalwar, Subbarao Kambhampati, Sarath Sreedharan
School of Computing and AI, Arizona State University · Department of Computer Science, Colorado State University
cs.RO, cs.LG, cs.MA
Submitted: 2026-02-18
Updated: 2026-11-16
Code: https://github.com/ikostrikov/pytorch-a2c-ppo-acktr-gail
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 83/100
The gist: Mutual adaptation is a central challenge in human-AI teaming, as humans naturally adjust their strategies in response to an AI agent’s behavior The proposed NestRL learns the solution to a…
Key concepts
- I-POMDP
- Interactive Partially Observable Markov Decision Process. This mathematical framework models human-AI teaming where the state depends not only on the environment but also on the models of other agents, like a human's strategy or a robot's policy. It extends standard POMDPs to handle these interactions.
- Nested Training Regime
- A training method where learning happens in layers. Level-1 agents adapt to fixed Level-0 policies, and then Level-2 agents learn against these adaptive Level-1 partners. This structure forces agents to reason about multiple levels of partner behavior, preventing them from collapsing into simple conventions.
- Mutual Adaptation
- The core challenge in human-AI teaming where both parties change their strategies based on the other's actions. NestRL addresses this by structuring the learning process so that each agent adapts to a finite set of lower-level policies, exposing them to diverse, adaptive behaviors without creating unpredictable coordination.
- Level-l I-POMDP
- A specific mathematical formulation used in NestRL where the state includes the environment and models of other agents at different levels. For instance, a Level-1 I-POMDP considers the environment plus possible models of agent j at level l-1, encoding conventions and adaptation rules.
Terminology
Summary
Mutual adaptation is a central challenge in human-AI teaming, as humans naturally adjust their strategies in response to an AI agent’s behavior The proposed NestRL learns the solution to a finite-level I-POMDP by training agents at each level against adaptive agents from the level below This approach exposes agents to adaptive behavior while preventing emergence of opaque coordination strategies The method is validated empirically in the Overcooked domain against state-of-the-art baselines, achieving higher task performance with both unseen adaptive agents and real human teammates
Problem Formulation
The problem is formulated as an Interactive Partially Observable Markov Decision Process (I-POMDP) to model human-AI teaming This formulation extends a standard POMDP by incorporating explicit models of other agents into the state space The level-l I-POMDP is defined as Mi,l = ⟨ISi,l, Ai, Ti, Ri, omegai, Oi, γ⟩ The interactive state ISi,l = S × Mj,l−1 where S is the environment state and Mj,l−1 is the set of possible models of agent j at level l − 1 This model encodes both the convention the human follows and the human’s adaptation rule in response to the robot
How the training signal is built
NestRL employs a nested training regime to learn level-2 interactive reasoning through this structure Level-1 Training involves training a set of Level-1 partner policies against a subset of fixed Level-0 policies, where the Level-1 agent must adapt to the behavior of these fixed policies This produces adaptive partner policies that act as proxies for adaptive human behaviors at Level-1 reasoning Level-2 Training then trains the robot policy against this set of adaptive partner policies, implementing a Level-2 I-POMDP approximation This corresponds to second-order reasoning where the robot reasons about a Level-1 partner that itself reasons about Level-0 behavior
Where the gains come from
The nested training regime systematically prevents collapse to a single convention, as shown by Theorem 1 This is achieved because the level-(n + 1) agent trains against a finite set of lower-level policies and must encode distinct best responses to each The training exposes the agent to multiple conventions, preventing it from collapsing to a single joint solution in Π∗ Qualitative analysis shows that L1 learns to wait under uncertainty when paired with a fixed L0 partner, which corresponds to waiting for a team member to start acting Level-2 agents learn to take initiative by initiating goal-directed behavior earlier than the L1 agent, leading to complementary leader–follower coordination
Empirical Validation
Experimental results show that NestRL agents achieve an average success rate of 81.3% with held-out Level-1 partners, substantially outperforming baselines like LILI and PACE Furthermore, NestRL exhibits substantially lower variability (CV = 0. p. 6 NestRL exhibits substantially lower variability
(CV = 0. 108) compared to LILI (0. Nested training thus induces complementary strategies: L1’s waiting creates an opportunity for
symmetry breaking, and L2 exploits it by taking initiative—mirroring co-adaptation patterns in
human-robot interaction [van Zoelen et al. 4 User Study
1 2 3 4 5
40
60
80
100
Episode within round
Delivery rate (%)
PACE
NestRL (ours)
Figure 5: Delivery rate improves across episodes
within a round as the robot adapts to the partner.Where the paper says it
Where the paper says it
Conclusion
NestRL provides a principled foundation for developing agents that collaborate effectively with diverse, adaptive partners by grounding the learning process in the reasoning structure of finitely nested I-POMDPs The framework successfully models how each agent’s behavior influences its partner’s adaptive responses, enabling reasoning over multiple levels of partner beliefs Through simulation and a user study, the paper demonstrates that NestRL agents consistently outperform existing adaptive baselines Future work includes extending NestRL to settings where agents do not share the same reward function The framework provides evidence that nested training induces the structural mechanisms of co-adaptation in the trained agents The authors conclude that this approach yields structured co-adaptation consistent with established human–robot interaction patterns The framework offers a principled way to handle mutual adaptation by bounding the reasoning depth and restricting each level to best-respond only to fixed lower-level policies
References
The paper cites numerous related works on population-based training, opponent modeling, and I-POMDPs as context for the proposed method The framework is grounded in the I-POMDP formalism introduced by Gmytrasiewicz and Doshi The authors compare their approach against baselines such as LIAM, LILI, PACE, and Generalist to benchmark performance The study is conducted in the Overcooked collaborative cooking environment with a multi-recipe variant requiring required coordination The qualitative analysis grounds its findings in the behavioral taxonomy of co-adaptation proposed by van Zoelen et al The training objective is optimized using Proximal Policy Optimization (PPO) with a recurrent network implemented as a single-layer GRU The user study evaluates perceived partner quality through the NASA Task Load Index and specific questions on proactiveness and adaptiveness The paper acknowledges support from ONR, DARPA, DoD RAI, NSF grant 2303019, and a gift from Qualcomm The references section lists works covering various aspects of MARL and human-AI teaming research The paper includes detailed experimental setup descriptions for the Overcooked environment, peer pool generation, and belief estimation techniques The training details specify hyperparameters for all algorithms in the Overcooked environment The paper provides a comprehensive discussion on convention collapse and coordination strategies emerging from nested adaptation The final conclusion reiterates that NestRL agents learn coordination strategies observed previously in realistic human-robot settings
A Related Work
Population-Based and Opponent Modeling Approaches expose agents to diverse but stationary partners LIAM, LILI, and PACE adapt online to partner behavior via latent inference or meta-learning I-POMDPs model beliefs over other agents’ goals, policies, and beliefs but struggle to scale to high-dimensional environments The authors evaluate their approach in a modified Overcooked variant with distinct coordination equilibria that require adaptive reasoning The paper notes that standard Overcooked provides limited equilibrium diversity, necessitating the use of a multi-recipe version The study compares NestRL against baselines trained only against fixed partners and those trained to adapt to diverse partners The authors address the challenge of mutual adaptation by modeling the human’s adaptation in response to the agent The paper discusses how recursive reasoning is a well-established model of human strategic behavior and forms the foundation of theory of mind literature The authors note that higher-order partners (level-3 and beyond) are not modeled yet in this framework The paper discusses how agents co-learn often develop brittle conventions, which is a limitation of some co-learning agents The authors note that the optimal action at time t depends on the full history ht = (oτ, aτ) which determines beliefs over the partner’s model
B Experimental Setup
The environment is the Overcooked collaborative cooking environment Carroll et al. [2020], adopted in a multi-recipe, required-coordination variant In this setting, players must complete and serve one of 3 pre-defined recipes as quickly as possible The observation space for each agent is a 105-dimensional vector encoding features such as agent position, orientation, held objects, and visibility flags The reward structure includes Interaction Reward (0.5 points), Progress Reward (1. p. 2 2 Problem Formulation
Cooperative Markov Game. We model each episode as an m-player, finite-horizon cooperative
Markov game ⟨S, A, T , H, R⟩, where S is the state space, A = A1 × · · · × Am is the joint action
space, T is the transition function, H is the horizon, and R defines a shared reward r(s
h
, ah
) for
cooperative settings.Where the paper says it
C Qualitative Analysis of Adaptive Behavior
Level-1 coordination strategy involves waiting under uncertainty when paired with a fixed Level-0 agent, which corresponds to waiting for a team member to start acting L1 systematically delays goal-directed actions until L0 reveals its recipe type, and episodes in which L1 waited achieved a success rate of 54.9% Level-2 coordination strategy involves taking initiative by initiating goal-directed behavior earlier than the L1 agent, leading to complementary leader–follower coordination The results validate that L2 has learned to respond to adaptive dynamics of its partner, yielding structured co-adaptation consistent with established human–robot interaction patterns Baselines fail to adapt when paired with an adaptive partner because they are trained only against fixed partners
D User Study
The user study recruits N=25 participants and evaluates performance using the NASA Task Load Index and questions on partner proactiveness and adaptiveness NestRL achieves a higher episode delivery rate (79.5%) compared to PACE (51. p. 6 108) compared to LILI (0. 198), PACE (0. • We show that NestRL improves coordination with adaptive partners, prevents collapse to both
task-specific and task-irrelevant conventions. NestRL agents also learn coordination behaviors such
as waiting and initiative-taking, which facilitate mutual adaptation when paired with previously
unseen adaptive agents. 009, d = 0. NestRL receives lower or equal NASA-TLX ratings on all six
dimensions, with the strongest effect on temporal demand (M=4.Where the paper says it
Where the paper says it
Where the paper says it
E User Study Results
The user study data shows that NestRL receives lower or equal NASA-TLX ratings on all six workload dimensions compared to PACE The strongest signal is temporal demand, where participants report significantly lower time pressure with NestRL (M=4.91 vs. 5.64; t(21) = 3.84, p=.001) NestRL is also rated as significantly more adaptive (M=4. p. 9 001,
d=−0. It is also rated significantly more adaptive (M=4. Figure 1: Multi-recipe Overcooked domain. Each agent selects one of three
recipes (PotatoBroccoliSalad, LettuceOnionSalad, or TomatoCarrotSalad) per
episode. 54) and somewhat more proactive (no significant difference). The results provide converging
evidence across all three hypotheses.Where the paper says it
Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
NestRL improves coordination with unseen adaptive partners by training agents at each level against adaptive agents from the level below, which
exposes agents to adaptive behavior while preventing emergence of opaque coordination strategies.
This leads to a policy thatremains compatible with multiple conventions
and preventscollapse to both task-specific and task-irrelevant conventions.
-
The improved system learns emergent coordination strategies observed in human-robot interaction, such as waiting for a team member to act and taking initiative. Specifically, Level 1 agents learn
to wait under uncertainty
by delaying goal-directed actions until the partner reveals its type, while Level 2 agentslearn to take initiative by committing to Tomato–Carrot at t=1.
-
The system achieves better subjective user ratings regarding workload and perceived partnership quality. The NestRL agent receives lower or equal NASA-TLX ratings across all six dimensions compared to baselines, and participants rate it
significantly more adaptive
with a mean score of 4.41 versus 3.55 for PACE.
Abstract
Mutual adaptation is a central challenge in human-AI teaming, as humans naturally adjust their strategies in response to an AI agent's behavior. Existing approaches attempt to approximate human behavior by diversifying training partners; however, these partners are typically static and fail to capture the adaptive nature of human teammates. When agents are trained jointly in standard multi-agent settings, they often converge to opaque coordination strategies that work only with their co-trained partners, leading to poor generalization. To model adaptive human behavior, we formulate human-AI teaming as an Interactive Partially Observable Markov Decision Process (I-POMDP). We propose NestRL, a nested training regime that learns the solution to a finite-level I-POMDP by training agents at each level against adaptive agents from the level below. This exposes agents to adaptive behavior while preventing emergence of opaque coordination strategies. We provide theoretical analysis showing that NestRL agents avoid convergence to partner-specific strategies, and validate this empirically in the Overcooked domain against state-of-the-art baselines. NestRL achieves higher task performance with both unseen adaptive agents and real human teammates, while exhibiting significantly greater adaptability over the course of interaction.
Sources
- Reasoning about Hypothetical Agent Behaviours and their Parameters
- Who is Helping Whom? Analyzing Inter-dependencies to Evaluate Cooperation in Human-AI Teaming
- On the Utility of Learning about Humans for Human-AI Coordination
- Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge?
- Learning with Opponent-Learning Awareness
- Interactive POMDP Lite: Towards Practical Planning to Predict and Exploit Intentions for Interacting with Self-Interested Agents
- "Other-Play" for Zero-Shot Coordination
- Cross-environment Cooperation Enables Zero-shot Multi-agent Coordination
- On the Impossibility of Learning to Cooperate with Adaptive Partner Strategies in Repeated Games
- On the Pitfalls of Measuring Emergent Communication
- Fast Peer Adaptation with Context-aware Exploration
- A Survey of Ad Hoc Teamwork Research
- NASA-TLX Web App: An Online Tool to Analyse Subjective Workload
- Proximal Policy Optimization Algorithms
- ToM2C: Target-oriented Multi-agent Communication and Cooperation with Theory of Mind
- Model-Based Opponent Modeling
- Maximum Entropy Population-Based Training for Zero-Shot Human-AI Coordination
Related papers
- Rethinking Actions: Continuity-Aware Priors for World-Model Planning
- MOSAIC: Modular Obstacle and Spatial-Aware Framework for Manipulation In Cluttered Scenes
- SAGAS: Semantic-Aware Graph-Assisted Stitching for Offline Temporal Logic Planning
- DexSeed: Expanding Human Hand-Object Interactions into Robot Demonstrations for Dexterous Policy Learning
- LaST0.5: A Generalist World Action Model with Interaction-Centric Latent Reasoning
- CIALLO: Learning Shared Intent Representations for Cross-Embodiment Manipulation