World Models as Adversaries: Multi-Agent Self-Play Fine-Tuning for Robust Motion Planning

arXiv:2607.10630 · cs.RO, cs.AI · Submitted 2026-07-12 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: We're looking at the paper: "World Models as Adversaries".

Dev: Here's the gist of the paper. Robust motion planning in dense traffic requires autonomous vehicles to interact in rare and safety-critical scenarios that are underrepresented in naturalistic driving data <ref:2607.10630#pg2> The paper proposes Adversarial World Modeling (AWM), a multi-agent self-play fine-tuning framework that casts robust autoregressive motion planning as a constrained min-max game <ref:2607.10630#pg3> This method introduces a principled decoupled solver to handle the intractability of the exact game, utilizing scene-adaptive coalition learning and regret-aware robust best response optimization <ref:2607.10630#pg4>.

Rosa: The Core Framework The framework formulates adversarial training as a constrained two-player zero-sum game where the ego planner maximizes utility against an adversary that minimizes it <ref:2607.10630#pg5> The ideal robust objective is formulated as max πplan min πawm, s.t.

Dev: As ≤ Kmax, As ⊆ Vs, ∀s (Attack Budget) and a trust region constraint Ex∼D Dtrust πplan(· x)∥ πnom plan(· x) ≤ ϵ (Trust Region Constraint) <ref:2607.10630#pg6> This structure decouples the problem into two stages: Stage A, where the planner is frozen and the AWM searches for sparse adversarial coalitions, and Stage B, where the ego planner optimizes as a constrained robust best response against this frozen adversary <ref:2607.10630#pg7>.

Rosa: How the training signal is built Stage A involves role-conditioned adversarial generation where non-ego agents receive a latent role assignment ρi ∈ bg, atk, allowing the AWM to learn sparse, scene-adaptive attack coalitions via counterfactual credit assignment <ref:2607.10630#pg8> The coalition learning proceeds through a sequence: proposal probing, primary selection based on fused features xi = hprobe i, I(i ∈ Ps), rank(i) Bprop, qi, q¯i, ψS i, mi (21), conditional pair ranking using the relational feature ϕ(i⋆s, j) to predict marginal coalition gain gˆs(i⋆s, j) (23), and a scene-adaptive calibrator qη that decides admission probability ps based on the pair gain Gpair s = Us(i⋆s) − Us(i⋆s, ˆjs) (70)<ref:2607.10630#pg9> Where the gains come from The planner-side optimization in Stage B utilizes a regret-aware objective that prioritizes informative tail vulnerabilities, coupled with reference-anchored trust-region regularization to preserve nominal driving behavior <ref:2607.10630#pg10> Specifically, the objective is Lrob(θ) = Es∼D w top1 s(τ)l top1 s(θ) + w pair s(τ)l pair s(θ), where weights are determined by the closed-form CVaR allocation based on regret ∆b s (30) This ensures that the planner's gradients are guided by the scene-adaptive vulnerabilities discovered by the AWM, focusing on rare but critical failures Verification and Analysis Empirical results demonstrate that AWM-Planner improves five of the six nuPlan settings, with the largest gains on Test14-hard NR and R (+2.60 and +1.28) Furthermore, AWM as a Transferable Adversarial Simulator reduces the sim-agent closed-loop score for every planner, indicating that the learned adversary captures planner-agnostic interaction stress The analysis of the decoupled solver shows that freezing Stage A results in an error bounded by εA s + δcov s(πref) + 2ΛH,γδϵ This confirms that the structured class induced by AWM is a local Stackelberg approximation whose error is controlled by interpretable quantities The final safety-constrained dual update parameterization ensures that the planner avoids exploiting the objective by violating traffic rules The paper introduces AWM, a multi-agent self-play fine-tuning framework leveraging predictive world models as structured adversaries for robust motion planning By integrating scene-adaptive coalition learning with counterfactual credit assignment, AWM efficiently exposes sparse, synergistic interaction failures within the planner’s inherent autoregressive interface Subsequently, a regret-aware robust best response, with reference-anchored trust region constraints, fine-tunes the planner against the induced tail risk Theoretical analysis and empirical evaluations demonstrate that our framework significantly improves planner robustness in rare, complex interactions while maintaining nominal performance However, AWM remains limited by the fidelity of the tokenized world model and lacks exhaustive scenario coverage or formal closed-loop safety guarantees.

Paper summary: Dev: To wrap up, "World Models as Adversaries: Multi-Agent Self-Play Fine-Tuning for Robust Motion Planning" proposes AWM as a multi agent self play fine tuning framework leveraging predictive world models as structured adversaries for robust motion planning.

Rosa: By integrating scene adaptive coalition learning with counterfactual credit assignment, AWM efficiently exposes sparse synergistic interaction failures within the planner’s inherent autoregressive interface.

Taro: Subsequently, a regret aware robust best response, with reference anchored trust region constraints, fine tunes the planner against the induced tail risk.

Dev: The paper shows that this approach significantly improves planner robustness in rare complex interactions while maintaining nominal performance.

Rosa: It’s a lot of work to put together, and it moves the focus from just external scenario generation to endogenous solutions within the world model itself.

Taro: I think the main implication is using the simulator itself as a structured adversary, which could be useful for training systems in environments where real world data is genuinely sparse.

Dev: That’s right, but we still have that limitation they mentioned about the fidelity of the tokenized world model and the lack of exhaustive scenario coverage or formal closed loop safety guarantees.

Rosa: So it’s a strong tool for finding those rare failures, but it isn't a complete safety solution on its own yet.

Conclusion: Rosa: So, we’ve been looking at this paper called "World Models as Adversaries: Multi-Agent Self-Play Fine-Tuning for Robust Motion Planning." It basically takes a world model, which is like an AI that predicts what will happen next in a scene, and uses it to train the motion planner to be tougher.

Dev: Tougher how? We’ve talked about adversarial training before, but this one seems different because they use self-play where the world model itself acts as the opponent.

Taro: Yeah, they frame it like a game where one player is the ego planner trying to drive well, and the other is an AI that tries to trick it by behaving badly.

Rosa: They set up this constrained two-player zero-sum game where the ego planner tries to maximize its own progress while the world model tries to minimize it, all while keeping a budget for how much trouble can happen.

Dev: The math on that is structured into two parts, Stage A and Stage B. It sounds like they decouple learning how to create an attack from learning how to defend against it.

Taro: In Stage A, the world model learns who should attack whom based on what kind of role each agent plays in the scene. They use this coalition learning idea to find those specific, sneaky ways agents can mess up the plan.

Rosa: And in Stage B, the ego planner gets fine-tuned against those discovered threats using a regret aware objective. That means it focuses its training on those rare failures that actually matter most for safety.

Dev: The results show that this method makes the planner much better at handling rare but dangerous situations, like when things go totally wrong in traffic, without losing its normal driving skills.

Taro: It really shows how you can use the simulator's internal logic to find weaknesses that a standard test set might never even see.

Rosa: But they do admit the world model itself has limits because it’s built on tokenized data, so there are still gaps in what it can predict accurately.

Dev: Exactly. It’s not a complete safety guarantee yet, but for finding those specific failure modes in complex interactions, this framework is pretty solid.

Taro: So the next thing we need to look at is how much of that world model fidelity they actually need for these kinds of robust results to hold up in real life.

Hong Kong Polytechnic University · Tongji University · McGill University · Mila-Quebec AI Institute

cs.RO, cs.AI

Submitted: 2026-07-12

Updated: 2026-11-05

Comments: Accepted at NeurIPS 2026

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 95/100

The gist: Robust motion planning in dense traffic requires autonomous vehicles to interact in rare and safety-critical scenarios that are underrepresented in naturalistic driving data <ref:2607.10630#pg2> The…

Key concepts

Adversarial World Modeling (AWM)
AWM is a multi-agent self-play method where non-ego agents learn to form attack coalitions against the ego vehicle. This is done by assigning roles (background or attacker) and using counterfactual credit assignment to discover which combinations of agents pose the greatest threat to the planner's performance.
Constrained Min-Max Game
The paper frames robust planning as a two-player zero-sum game: the ego planner tries to maximize utility while an adversary tries to minimize it. This is mathematically constrained by an attack budget (Kmax) and a trust region constraint, ensuring the resulting robust plan remains close to the nominal driving behavior.
Regret-Aware Objective
The planner's optimization uses a loss function that prioritizes 'tail vulnerabilities'—failures that are rare but severe. This is achieved by weighting losses based on regret, which measures how much the current policy deviates from the best possible performance in specific scenarios, guiding the planner toward fixing these critical weaknesses.
Decoupled Solver
The framework separates the complex game into two stages: Stage A learns which agents attack (the adversary), and Stage B optimizes the ego planner's response against that learned adversary. This decoupling simplifies training by allowing each stage to focus on a specific part of the interaction without solving the entire game at once.

Terminology

Summary

Robust motion planning in dense traffic requires autonomous vehicles to interact in rare and safety-critical scenarios that are underrepresented in naturalistic driving data <ref:2607. p. 010630#pg2> The paper proposes Adversarial World Modeling (AWM), a multi-agent self-play fine-tuning framework that casts robust autoregressive motion planning as a constrained min-max game <ref:2607. p. 910630#pg3> This method introduces a principled decoupled solver to handle the intractability of the exact game, utilizing scene-adaptive coalition learning and regret-aware robust best response optimization <ref:2607. p. 010630#pg4>.

The Core Framework

The framework formulates adversarial training as a constrained two-player zero-sum game where the ego planner maximizes utility against an adversary that minimizes it <ref:2607.10630#pg5> The ideal robust objective is formulated as max πplan min πawm, s. p. 2t. As ≤ Kmax, As ⊆ Vs, ∀s (Attack Budget) and a trust region constraint Ex∼D Dtrust πplan(· x)∥ πnom plan(· x) ≤ ϵ (Trust Region Constraint) <ref:2607. p. 210630#pg6> This structure decouples the problem into two stages: Stage A, where the planner is frozen and the AWM searches for sparse adversarial coalitions, and Stage B, where the ego planner optimizes as a constrained robust best response against this frozen adversary <ref:2607. p. 410630#pg7>.

How the training signal is built

Stage A involves role-conditioned adversarial generation where non-ego agents receive a latent role assignment ρi ∈ bg, atk, allowing the AWM to learn sparse, scene-adaptive attack coalitions via counterfactual credit assignment <ref:2607.10630#pg8> The coalition learning proceeds through a sequence: proposal probing, primary selection based on fused features xi = hprobe i, I(i ∈ Ps), rank(i) Bprop, qi, q¯i, ψS i, mi (21), conditional pair ranking using the relational feature ϕ(i⋆s, j) to predict marginal coalition gain gˆs(i⋆s, j) (23), and a scene-adaptive calibrator qη that decides admission probability ps based on the pair gain Gpair s = Us(i⋆s) − Us(i⋆s, ˆjs) (70)<ref:2607. p. 410630#pg9>

Where the gains come from

The planner-side optimization in Stage B utilizes a regret-aware objective that prioritizes informative tail vulnerabilities, coupled with reference-anchored trust-region regularization to preserve nominal driving behavior <ref:2607.10630#pg10> Specifically, the objective is Lrob(θ) = Es∼D w top1 s(τ)l top1 s(θ) + w pair s(τ)l pair s(θ), where weights are determined by the closed-form CVaR allocation based on regret ∆b s (30) This ensures that the planner's gradients are guided by the scene-adaptive vulnerabilities discovered by the AWM, focusing on rare but critical failures

Verification and Analysis

Empirical results demonstrate that AWM-Planner improves five of the six nuPlan settings, with the largest gains on Test14-hard NR and R (+2.60 and +1.28) Furthermore, AWM as a Transferable Adversarial Simulator reduces the sim-agent closed-loop score for every planner, indicating that the learned adversary captures planner-agnostic interaction stress The analysis of the decoupled solver shows that freezing Stage A results in an error bounded by εA s + δcov s(πref) + 2ΛH,γδϵ This confirms that the structured class induced by AWM is a local Stackelberg approximation whose error is controlled by interpretable quantities The final safety-constrained dual update parameterization ensures that the planner avoids exploiting the objective by violating traffic rules

Conclusion

The paper introduces AWM, a multi-agent self-play fine-tuning framework leveraging predictive world models as structured adversaries for robust motion planning By integrating scene-adaptive coalition learning with counterfactual credit assignment, AWM efficiently exposes sparse, synergistic interaction failures within the planner’s inherent autoregressive interface Subsequently, a regret-aware robust best response, with reference-anchored trust region constraints, fine-tunes the planner against the induced tail risk Theoretical analysis and empirical evaluations demonstrate that our framework significantly improves planner robustness in rare, complex interactions while maintaining nominal performance However, AWM remains limited by the fidelity of the tokenized world model and lacks exhaustive scenario coverage or formal closed-loop safety guarantees.

--- Page 1 ---

World Models as Adversaries: Multi-Agent Self-Play Fine-Tuning for Robust Motion Planning

Tong Nie,1,2,† Yuewen Mei,2,† Junlin He,1 Yihong Tang,3,4 Jian Sun,2,B Wei Ma1.

--- Page 2 ---

Abstract

Robust motion planning in dense traffic requires autonomous vehicles to interact in rare and safety-critical scenarios that are underrepresented in naturalistic driving data <ref:2607.10630#pg2> Although adversarial training offers a feasible solution, existing methods often rely on external scenario generators, heuristic perturbations, or simulator-heavy rollouts, which makes them difficult to integrate with modern autoregressive planners <ref:2607. p. 010630#pg3> Here, we cast adversarially robust planner learning as a constrained min-max game and propose Adversarial World Modeling (AWM), a theoretically grounded multi-agent self-play fine-tuning framework <ref:2607. p. 010630#pg4>.

--- Page 3 ---

  1. Preliminary and Problem Formulation

2.1 Autoregressive Motion Generation

Tokenized Autoregressive Rollout. Recent generative driving models formulate multi-agent traffic evolution as sequential next-token prediction over a learned discrete motion vocabulary [38, 33, 46, 52] <ref:2607. p. 110630#pg9> Consider a scene s with ego agent e, non-ego agents Vs = The paper formulates the Markovian generative process as p(z1:H cs) = Q H t=1 p(zt xt) <ref:2607. p. 110630#pg10>

--- Page 4 ---

2.2 Problem Formulation: Robust Planning as a Constrained Game

We evaluate a rollout by a scalarized utility that trades off progress and safety The standard planning objective is to maximize Us under the benign prior πpred Robust planning requires the planner to perform well when some non-ego agents behave adversarially, while still preserving normal driving ability We therefore replace the benign environment with a controllable Adversarial World Model (AWM) πawm and formulate the ideal robust objective as a constrained two-player zero-sum game: max πplan min πawm, As Es∼D [Us(πplan, πawm, As)] s.t. As ≤ Kmax, As ⊆ Vs, ∀s (Attack Budget) Ex∼D Dtrust πplan(· x)∥ πnom plan(· x) ≤ ϵ (Trust Region Constraint)

--- Page 5 ---

3 Self-Play Fine-tuning with Adversarial World Models

We instantiate the idealized game in Eq. 2 as a decoupled self-play fine-tuning framework over the shared autoregressive factorization in Eq. p. 2 1 The key design is to separate the two roles of the game without changing the rollout interface Stage A involves learning an adversarial world model for Eq. p. 2 3, where role conditioning defines how an agent attacks and coalition learning decides who should attack Stage B optimizes the ego factor as a constrained robust best response to the AWM’s calibrated risk distribution

--- Page 8 ---

4 Experiments

Closed-Loop Planning and Adversarial Testing

We first evaluate whether the robust best-response update improves ordinary closed-loop planning performance AWM-Planner improves Plan-R1 by +2.10 on full InterPlan, +4. p. 6 p. 6 p. 697 on InterPlan10, and +5. p. 6 p. 6 p. 681 on InterPlan-LongTail We next test whether the learned adversary captures planner-agnostic interaction stress The simulator-side diagnostics in Table 6 further show that AWM utilizes nearly the full two-agent budget while remaining constrained by feasibility gates, indicating that the score drop is not a result of an unconstrained simulator collapse

Pruning redundant reasoning tokens

Closed-Loop Planning and Adversarial Testing We first evaluate whether the robust best-response update improves ordinary closed-loop planning performance AWM-Planner improves Plan-R1 by +2.10 on full InterPlan, +4.97 on InterPlan10, and +5.81 on InterPlan-LongTail We next test whether the learned adversary captures planner-agnostic interaction stress The simulator-side diagnostics in Table 6 further show that AWM utilizes nearly the full two-agent budget while remaining constrained by feasibility gates, indicating that the score drop is not a result of an unconstrained simulator collapse

Stage A involves role-conditioned adversarial generation where non-ego agents receive a latent role assignment ρi ∈ bg, atk, allowing the AWM to learn sparse, scene-adaptive attack coalitions via counterfactual credit assignment The coalition learning proceeds through a sequence: proposal probing, primary selection based on fused features xi = hprobe i, I(i ∈ Ps), rank(i) Bprop, qi, q¯i, ψS i (21), conditional pair ranking using the relational feature ϕ(i⋆s, j) to predict marginal coalition gain gˆs(i⋆s, j) (23), and a scene-adaptive calibrator qη that decides admission probability ps based on the pair gain Gpair s = Us(i⋆s) − Us(i⋆s, ˆjs) (70)

The planner-side optimization in Stage B utilizes a regret-aware objective that prioritizes informative tail vulnerabilities, coupled with reference-anchored trust-region regularization to preserve nominal driving behavior Specifically, the objective is Lrob(θ) = Es∼D w top1 s(τ)l top1 s(θ) + w pair s(τ)l pair s(θ), where weights are determined by the closed-form CVaR allocation based on regret ∆b s (30) This ensures that the planner's gradients are guided by the scene-adaptive vulnerabilities discovered by the AWM, focusing on rare but critical failures

Empirical results demonstrate that AWM-Planner improves Plan-R1 by +2.10 on full InterPlan, +4.97 on InterPlan10, and +5.81 on InterPlan-LongTail The analysis of the decoupled solver shows that freezing Stage A results in an error bounded by εA s + δcov s(πref) + 2ΛH,γδϵ This confirms that the structured class induced by AWM is a local Stackelberg approximation whose error is controlled by four interpretable quantities

--- Page 9 ---

Figure 1: Decoupled self-play fine-tuning with AWM. The full pipeline is described in Algorithm 1

--- Page 14 ---

Appendix

Appendix Contents

A Related Work 15

--- Page 15 ---

B Extended Architectural and Algorithmic Details

B.1 Discrete Autoregressive Planning Architecture and Role Conditioning. p. 0....... 16

--- Page 20 ---

B.2 Algorithmic Details of Adversarial Coalition Learning

B.2.1 Probe-Time Threat Signals and Residual Proposal Construction

--- Page 24 ---

B.3 Details of Regret-Aware Constrained Planner Optimization

B.3.1 Reference-Anchored Nominal Retention

--- Page 25 ---

B.4 AWM as a Closed-Loop Sim-Agent

--- Page 28 ---

C Theoretical Analysis

C.1 Notation and Autoregressive Local Stability

--- Page 31 ---

C.5 Regret as a Reference-Anchored Control Variate

--- Page 29 ---

C.6 Closed-Form CVaR on the AWM-Induced Two-Point Risk Distribution

--- Page 30 ---

C.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Robust motion planning in dense traffic can be achieved by casting robust planner learning as a constrained min-max game using Adversarial World Modeling (AWM). This allows for fine-tuning planners against hard cases while preserving nominal driving behavior through reference-anchored trust region regularization.

  2. The AWM framework introduces a decoupled solver consisting of an inner minimization stage that uses role-conditioned counterfactual credit assignment to discover sparse, scene-adaptive adversarial coalitions. This mechanism efficiently resolves the multi-agent reward ambiguity by training the adversary within the same autoregressive structure as the ego planner.

  3. The outer maximization stage optimizes the ego planner against a calibrated risk distribution derived from AWM's output, utilizing a regret-aware objective that prioritizes informative tail vulnerabilities and reference-anchored trust regions to improve hard-case recovery while preserving nominal driving behavior.

  4. The system can generate transferable adversarial interactions by using the learned AWM as an adversary simulator for other planners, demonstrating that the learned adversary captures planner-agnostic interaction stress across different planning architectures.

  5. The framework is designed to handle complex multi-agent scenarios by dynamically admitting or rejecting potential attacks using a scene-adaptive calibrator, ensuring that pair attacks should be admitted only when their conditional expected marginal gain is positive.

Sources

Related papers