Learned Controllers for Agile Quadrotors in Pursuit-Evasion Games

arXiv:2506.02849 · cs.RO, cs.LG · Submitted 2025-06-03 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: I'm Rosa.

Dev: And I'm Dev, here with Taro.

Rosa: We're looking at the paper: "Learned Controllers for Agile Quadrotors in Pursuit-Evasion Games".

Title and authors: Rosa: So we're looking at this paper called "Learned Controllers for Agile Quadrotors in Pursuit-Evasion Games." It’s about training a pursuer and an evader using reinforcement learning by having them compete against each other.

Dev: That adversarial setting is tricky because the policies keep changing during training, which makes the environment non-stationary, and agents can forget old strategies or just collapse into something bad.

Taro: Exactly, those policy changes mean the agents might get stuck in a loop or just totally fail to respond when things get chaotic.

Rosa: To tackle that, they propose something called Asynchronous Multi-Stage Population-Based training with Hedge sampling, which uses Policy-Space Response Oracles and adapts it for quadrotor reinforcement learning.

Dev: It sounds like they're trying to solve the problem of agents overfitting to their immediate opponent by having them train against a mixture of previously trained policies.

Taro: So instead of just fighting one person, the agent is being tested against a whole group of past behaviors at once, which should expose it to different strategies.

Rosa: That's right; in this method, each generation trains one agent using Proximal Policy Optimization against frozen opponent policies, and then a Hedge sampler decides which opponents are currently the hardest to beat.

Dev: The Hedge sampler assigns higher probability to those opponents that the agent is struggling against, while still keeping some exploration going through an exploration term.

Taro: So it’s like a curriculum where the system learns by being pushed by its own history and then focusing its training power on what it's currently bad at.

Rosa: They show that this population-based training helps the agents find new strategies while still keeping them competent against older ones, which means they reach a point where more training against just the latest opponent doesn't give much better results.

Dev: That makes sense; if you keep fighting only the newest thing, you might miss important underlying behaviors from earlier in the training process.

Taro: And when they test it on Crazyflie brushless quadrotors, they show this zero-shot sim-to-real transfer of agile pursuit evasion behavior against both handcrafted and learned adversaries up to four point eight seven meters per second <ref:2506.02849#pg2>.

Rosa: That speed is really impressive, especially since recent baselines reported real flight speeds around one meter per second in those scenarios <ref:2506.02849#pg1>.

Dev: The paper also notes that the AMSPBH evader manages to transfer zero-shot as well, escaping manual and learned pursuers in all the tested trials.

Taro: So it’s not just about the pursuer improving; the evasion side is also showing robust performance against various threats.

Rosa: It seems like they are using this population training to create a diverse set of pursuit and evasion behaviors that actually give them strategic diversity when they need it most.

Dev: The core improvement here is adaptive opponent curriculum via Hedge sampling, which allows the system to maintain competence against older strategies while adapting to new ones by assigning higher probability to opponents that are currently difficult to beat.

Title and authors: Taro: That mechanism creates an auto-curriculum where the oracle is trained as a single policy against the mixture, exposing each new controller to old, recent, heuristic, and learned strategies without making the reinforcement learning target fully non-stationary.

Rosa: They also demonstrate that population-based training generalizes across diverse and unseen opponent strategies better than just training against only the latest opponent.

Dev: The results show that AMSPBH is stronger or tied in most out-of-training evaluations, which proves its robustness against both handcrafted and learned adversaries.

Taro: And they’ve validated this on real hardware, confirming that the trained policies exhibit zero-shot sim-to-real transfer of agile pursuit evasion behavior on physical hardware reaching up to four point eight seven meters per second <ref:2506.02849#pg2>.

Rosa: It sounds like they are making a big deal about how this method bridges the gap between simulations and actual flight, which is always a challenge in robotics research.

Dev: The paper also points out that the policies being used are feed-forward neural networks, which means they have limitations regarding memory and capacity when dealing with partially observable mixtures of opponents.

Taro: That’s a fair caveat; their current architecture isn't as robust as it could be in a fully uncertain world.

Rosa: Future work mentioned in the paper suggests that the frozen population isn't just a training memory but also a controlled curriculum over opponent behaviors, which could be useful for other competitive agile-flight domains.

Dev: Another limitation they flag is that recurrent architectures might improve both robustness and behavioral diversity for these agents.

Taro: And they also mentioned the assumption of perfect state information, suggesting that incorporating onboard perception could remove that limitation in the future.

Rosa: To wrap up, this paper on Learned Controllers for Agile Quadrotors in Pursuit-Evasion Games introduces AMSPBH as a method based on Policy-Space Response Oracles to handle the non-stationarity and forgetting issues in adversarial pursuit games.

Dev: The main implication is that population-based training offers a way to discover new strategies while retaining competence against older ones, leading to better generalization across diverse opponent types.

Taro: For me, it’s interesting because it shows how you can structure the training process itself—the curriculum—to manage complexity instead of just relying on a single learning loop.

Rosa: Exactly; they show that by mixing in prior knowledge and focusing the training signal using that Hedge sampler, you can get high-performance policies that are genuinely robust when deployed in the real world.

Dev: It’s about making sure these learned controllers don't just work against the specific opponent they just fought, but actually have a solid understanding of what different behaviors look like overall.

Taro: So, for anyone building pursuit evasion systems, this suggests looking beyond single-opponent training and considering how to manage a diverse policy population throughout the learning process.

Rosa: That’s the gist of it; AMSPBH is about using a smart population strategy to keep those quadrotors agile and effective in unpredictable combat scenarios.

The paper's summary: Rosa: So, this paper is about how they trained pursuit and evasion quadrotors using reinforcement learning by making them fight each other in a competition against a population of previous strategies.

Dev: Right, yeah, they call it Asynchronous Multi-Stage Population-Based training with Hedge sampling. It’s basically trying to solve that problem where the agents keep forgetting what they learned earlier or just getting stuck repeating the same moves.

Rosa: Exactly, and this new approach uses something called Policy-Space Response Oracles to maintain a whole population of past behaviors, and then it uses that Hedge sampler to focus on training against the opponents that are currently giving them trouble.

Dev: That’s the core mechanism, right? So instead of just training one agent against one frozen opponent at a time, they mix in previous policies and give more weight to the ones that are hard to beat right now.

Rosa: It’s like an auto-curriculum where the system is constantly exposed to a history of strategies while simultaneously getting focused training on what it’s currently bad at.

Dev: And they show that this helps the agents discover new strategies while still keeping competence against older ones, which means they don't completely lose their skills when things change.

Rosa: The results are pretty strong; the learned policies can handle real-world Crazyflie quadrotors with zero-shot transfer of agile pursuit evasion behavior up to four point eight seven meters per second.

Dev: And it’s not just the pursuer that's doing well; the evader also shows good transfer, escaping both manual and learned pursuers in all the tests.

Rosa: It suggests this method is really good at bridging that gap between simulation and real flight, which is always a big hurdle in robotics.

Dev: The authors admit they are using feed-forward neural networks, so they point out that these policies have limitations regarding memory and capacity when dealing with a lot of uncertainty in the opponent's strategy.

Rosa: They suggest that the frozen population isn't just a training memory but also a controlled curriculum over opponent behaviors, which could be useful for other competitive agile-flight domains.

Dev: That’s interesting because they mention that using recurrent architectures might improve both robustness and behavioral diversity for these agents.

Rosa: So it seems like the big idea here is using this population-based training to create a set of behaviors that give the quadrotors strategic diversity when they need it most in unpredictable combat scenarios.

Dev: It’s about building controllers that don't just work against the specific opponent they just fought, but actually have a solid understanding of what different behaviors look like overall.

Rosa: And they show this robustness across diverse and unseen opponent strategies, which is a big deal for real-world deployment where you never know what kind of threat you're going to face.

Dev: So it’s not just about getting the best score against the current opponent; it's about making sure those learned controllers are solid enough to handle whatever comes next.

Rosa: Next time, we’ll look at how they actually set up that training signal construction and see exactly how they build that population of policies.

The paper's improvements: Taro: So, we've seen how they trained these agents using population training, and now we need to talk about what this paper suggests are the actual improvements they made to the system.

Rosa: Well, one big improvement is this adaptive opponent curriculum using Hedge sampling. It lets the system maintain competence against older strategies while simultaneously adapting to new ones by prioritizing opponents that are currently difficult to beat.

Dev: That’s smart, because it creates an auto-curriculum where the oracle learns against a mixture of policies, exposing each new controller to old and recent behaviors without making the training target completely non-stationary.

Rosa: It means the system is constantly being challenged by its own history while still focusing on what it needs to learn right now.

Dev: And another point they make is how this population-based training leads to better generalization across diverse and unseen opponent strategies compared to just training against only the latest one.

Taro: That’s important because when you deploy these controllers in the real world, you don't know exactly what kind of threat you're going to face, so being robust against things it hasn't seen before is key.

Rosa: The results show that this method is stronger or tied in most out-of-training evaluations, proving it handles both handcrafted and learned adversaries well.

Dev: And they’re showing high speed agility transfer on physical hardware, reaching four point eight seven meters per second in those tests with zero-shot sim to real transfer.

Taro: That speed is what makes it tangible for us; it means the policies aren't just theoretical models, but they actually perform well when there's real-world dynamics involved.

Rosa: The implication here is that we can build pursuit evasion systems that are genuinely robust across a wider range of unpredictable scenarios than before.

Dev: They also flag some limitations on their end; the policies they use are feed-forward neural networks, so they admit this limits how much memory and capacity the system has for dealing with those complex mixtures of opponents.

Taro: So while the method is solid for exploration, it seems like recurrent architectures might actually improve both robustness and behavioral diversity in these systems.

Rosa: That’s a good direction to look at; moving towards recurrent models could help them handle the complexity of a real-time, dynamic fight better.

Dev: And they also mention that incorporating onboard perception could remove the assumption that you have perfect state information, which is another big hurdle in these kinds of games.

Taro: So it sounds like the next steps for this research are focusing on improving the architecture itself rather than just tweaking how they sample their opponents.

Conclusion: Rosa: So we're wrapping up on "Learned Controllers for Agile Quadrotors in Pursuit-Evasion Games" by AMSPBH, which basically shows how population training can make pursuit and evasion agents really good at handling non-stationary opponents.

Dev: Yeah, the main thing is that this method helps agents discover new strategies while retaining competence against older ones, giving them better generalization.

Rosa: It’s a solid approach for building controllers that are tough when you deploy them in the real world because they’ve seen a lot of different behaviors.

Dev: The results with the Crazyflie quadrotors at four point eight seven meters per second really show that this training bridges the gap between simulation and actual flight speed.

Taro: I just want to emphasize how well it handles those unseen strategies, which is crucial because in real scenarios, you never know what kind of adversary you're going to run into.

Rosa: Exactly, and they’ve shown that the evader side also transfers well, escaping both manual and learned pursuers in all their tests.

Dev: It’s about building safety cases for these things; the fact that they're performing so reliably under pressure is what matters most for real-time systems.

Taro: The limitation they mentioned is that using feed-forward neural networks means there are some limits on how much memory and capacity those policies can actually hold when the opponent's strategy gets really complicated.

Rosa: So while the method is effective, we still have to consider future work, like moving towards recurrent architectures to potentially improve that memory aspect.

Dev: And they also noted that incorporating onboard perception would remove the assumption of perfect state information, which is another big thing for making these systems truly reliable.

Taro: If you look at this paper on "Learned Controllers for Agile Quadrotors in Pursuit-Evasion Games," it shows how population strategies can create a diverse set of behaviors that are useful when you need strategic variety in combat.

Rosa: It’s definitely a way to structure the training process itself to manage complexity instead of just relying on one learning loop.

Dev: For now, this is a strong piece for understanding how population-based training can tackle the non-stationarity inherent in these competitive games.

Taro: Next time, we'll look at some papers that deal with how agents coordinate when they aren't just fighting each other but working together in complex ways.

Alejandro Sanchez Roncero, Yixi Cai, Olov Andersson, 'Petter Ogren ƈ

Royal Institute of Technology (KTH) School of Electrical Engineering and Computer Science

cs.RO, cs.LG

Submitted: 2025-06-03

Updated: 2026-11-14

Comments: Accepted for publication in IEEE Robotics and Automation Letters (RA-L), 2026; to be presented at ICRA 2027. 8 pages, 8 figures. Video: https://youtu.be/U1tWZzHgv60

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 89/100

The gist: In this letter we study 1v1 quadrotor pursuit–evasion, where a pursuer and an evader are trained via reinforcement learning (RL) by competing against each other Such adversarial settings face…

Key concepts

Policy-Space Response Oracles (PSRO)
This is the core idea where a system maintains a collection of previously trained policies for both the pursuer and evader. Instead of training against just one opponent, it trains new policies against a mixture of these past opponents, helping to retain competence against older strategies.
Hedge Sampling
This technique is used to create an adaptive curriculum during training. It assigns higher probability to opponents that the current agent finds difficult to beat. This focuses the learning process on challenging scenarios while ensuring coverage of various strategies.
Population-Based Training
Instead of training against a single, constantly changing opponent, this method trains agents against a diverse population of previously learned policies. This approach helps prevent catastrophic forgetting and ensures the resulting policy is robust across many different strategies.
Zero-Shot Sim-to-Real Transfer
This means the trained RL policies can perform agile pursuit and evasion tasks successfully on physical hardware (like Crazyflie quadrotors) without needing extensive retraining. This confirms the learned skills are general enough to work in the real world.

Terminology

Summary

In this letter we study 1v1 quadrotor pursuit–evasion, where a pursuer and an evader are trained via reinforcement learning (RL) by competing against each other Such adversarial settings face well-known challenges: each agent’s policy changes during training, creating a non-stationary environment; agents might overfit to the current opponent and forget earlier strategies (catastrophic forgetting); and the competitive dynamics can cause strategy cycling or policy collapse To address these issues, we propose Asynchronous Multi-Stage Population-Based training with Hedge sampling (AMSPBH), a method based on Policy-Space Response Oracles (PSRO) and adapted to quadrotor RL control

The Core Proposal

The proposed AMSPBH algorithm is a method based on Policy-Space Response Oracles (PSRO) and adapted to quadrotor RL control PSRO maintains a population of previously trained policies and trains new approximate best responses against mixtures of that population instead of against a single opponent In AMSPBH, each generation trains one agent with Proximal Policy Optimization (PPO) against frozen opponent policies, while a Hedge sampler assigns higher probability to opponents that are currently difficult to beat The authors introduce AMSPBH, adapting PSRO-style population training to robotics and instantiating it for quadrotor pursuit–evasion, producing policies that continually improve while retaining robustness against prior strategies

Training Signal Construction

The training signal is built by maintaining a growing policy population for each role At each subsequent generation k, both roles train independently against previous-generation opponent populations The Hedge sampler converts these values into probabilities (12) where pj = γh/N + (1−γh)cηh j / ∑Nl=1 cηhl, which mixes challenge scores with uniform exploration With ηh > 0, the Hedge sampler concentrates training on challenging opponents while maintaining coverage through the exploration term

Performance Gains

The gains come from population-based training retaining competence against older strategies while adapting to new ones The results show that AMSPBH discovers new strategies while retaining competence against older opponents, reaching a regime where additional best-response training gives limited improvement Compared to training against only the latest opponent, population-based training generalizes better across diverse and unseen strategies The learned population contains distinct pursuit and evasion behaviors, providing useful strategic diversity for finding weaknesses and improving controller robustness

Validation and Results

The trained policies are validated with hardware experiments on Crazyflie brushless quadrotors, showing zero-shot sim-to-real transfer of agile, reactive pursuit–evasion behavior against both handcrafted and learned adversaries at up to 4.87 m/s AMSPBH achieves the best balanced learned-opponent score and is strongest or tied in most out-of-training evaluations The AMSPBH evader also transfers zeroshot, escaping manual and FRPN pursuers in all tested trials and remaining competitive against learned pursuers

Future Directions

The paper suggests that the frozen population is not only a training memory, but also a controlled curriculum over opponent behaviors The same principle could be useful in other competitive agile-flight domains, for example drone racing against diverse opponent policies or racing styles Several limitations remain, including the policies being feed-forward MLPs which limits memory and capacity in the partially observable mixture setting Recurrent architectures may improve both robustness and behavioral diversity Future work will incorporate onboard perception to remove the assumption of perfect state information

References

[1] S. Park, H. T. Kim, S. Lee, H. Joo, and H. p. 6 Kim, “Survey on AntiDrone Systems: Components, Designs, and Challenges,” IEEE Access, vol. p. 6 9, pp. 42 635–42 659 (ref:10)

[2] Y. Song, A. Romero, M. Muller, V. Koltun, and D. p. 7 Scaramuzza, “Reaching the Limit in Autonomous Racing: Optimal Control versus Reinforcement Learning,” Science Robotics, vol. p. 7 8, no. 82, p.eadg1462 (ref:10)

[3] E. Kaufmann, L. p. 7 Bauersfeld, and D. p. 7 Scaramuzza, “A Benchmark Comparison of Learned Control Policies for Agile Quadrotor Flight,” in 2022 International Conference on Robotics and Automation (ICRA), IEEE, 2022 (ref:10)

[4] D. Hanover, A. p. 7 Loquercio, L. p. 7 Bauersfeld, A. p. 7 Romero, R. Penicka, Y. p. 7 Song, G. Cioffi, E. Kaufmann, and D. p. 7 Scaramuzza, “Autonomous Drone Racing: A Survey,” IEEE Transactions on Robotics, vol. p. 7 40 (ref:10)

[5] C. de Souza, R. p. 7 Newbury, A. p. 7 Cosgun, P. Castillo, B. p. 7 Vidolov, and D. p. 7 Kulic, “Decentralized Multi-Agent Pursuit Using Deep Reinforcement Learning,” IEEE Robotics and Automation Letters, vol. p. 7 6 (ref:10)

[6] R. Zhang, Q. Zong, X. Zhang, L. Dou, and B. p. 7 Tian, “Game of Drones: Multi-UAV Pursuit-Evasion Game With Online Motion Planning by Deep Reinforcement Learning,” IEEE Transactions on Neural Networks and Learning Systems, vol. p. 7 34 (ref:10)

[7] J. Xiao and M. p. 7 Feroskhan, “Learning Multipursuit Evasion for Safe Target Navigation of Drones,” IEEE Transactions on Artificial Intelligence, vol. p. 7 5 (ref:10)

[8] O. Vinyals, I. p. 7 Babuschkin, W. p. 7 M. Czarnecki, M. p. 7 Mathieu, A. p. 7 Dudzik, J. Chung, D. HChoi, R. Powell et al., “Grandmaster level in StarCraft II using multi-agent reinforcement learning,” nature, vol 575 (ref:10)

[9] C. S De Witt, T Gupta, D Makoviichuk et al. p. 7, “Is Independent Learning All You Need in the Starcraft Multi-Agent Challenge? p. 7” arXiv preprint arXiv:2011. p. 709533 (ref:10)

[10] M Jaderberg, V Dalibard, S Osindero et al., “Population Based Training of Neural Networks,” arXiv preprint arXiv:1711. p. 709846 (ref:25)

[11] M Lanctot, V Zambaldi, A Gruslys et al., “A Unified Game-Theoretic Approach to Multiagent Reinforcement Learning,” Nov 2017 (ref:25)

[12] P Muller, S Omidshafiei, M Rowland et al., “A Generalized Training Approach for Multiagent Learning,” Feb. p. 7 p. 7 2020 (ref:25)

[13] J Chen, C Yu, G Li et al., “Online Planning for Multi-UAV Pursuit-Evasion in Unknown Environments Using Deep Reinforcement Learning,” IEEE Robotics and Automation Letters, pp. p. 7 p. 7 1–8 (ref:10)

[14] R Isaacs, Differential Games: A Mathematical Theory with Applications to Warfare and Pursuit, Control and Optimization. Courier Corporation, 1999 (ref:25)

[15] J F Fisac and S S Sastry, “The Pursuit-Evasion-Defense Differential Game in Dynamic Constrained Environments,” in 2015 54th IEEE Conference on Decision and Control (CDC), IEEE, 2015 (ref:10)

[16] X Fang, C Wang, L Xie, and J Chen, “Cooperative Pursuit With Multi-Pursuer and One Faster Free-Moving Evader,” IEEE transactions on cybernetics, vol. 52 (ref:10)

[17] A Ghotavadekar, F Nekova´ˇr, M Saska, and J Faigl, “Variable Time-Step MPC for Agile Multi-Rotor UAV Interception of Dynamic Targets,” IEEE Robotics and Automation Letters, vol. 10 (ref:10)

[18] M Pliska, M Vrba, T Ba´ca, and M Saska, “Towards Safe Mid-Air Drone Interception: Strategies for Tracking & Capture,” IEEE Robotics and Automation Letters, vol. 9 (ref:10)

[19] W Du, T Guo, J Chen et al., “Cooperative Pursuit of Unauthorized UAVs in Urban Airspace via Multi-agent Reinforcement Learning,” Transportation Research Part C: Emerging Technologies, vol. p. 7 128 (ref:10)

[20] M Kouzeghar, Y Song, M Meghjani, and R Bouffanais, “MultiTarget Pursuit by a Decentralized Heterogeneous UAV Swarm using Deep Multi-Agent Reinforcement Learning,” in 2023 IEEE International Conference on Robotics and Automation (ICRA), May 2023 (ref:10)

[21] Y Lin, H Gao, and Y Xia, “Distributed Pursuit–Evasion Game Decision-Making Based on Multi-Agent Deep Reinforcement Learning,” Electronics, vol. 14 (ref:10)

[22] E C etin, C Barrado, and E Pastor, “Countering a Drone in a 3D Space: Analyzing Deep Reinforcement Learning Methods,” Sensors, vol. 22 (ref:10)

[23] B Baker, I Kanitscheider et al., “Emergent Tool Use from Multi-Agent Autocurricula,” in International Conference on Learning Representations, 2019 (ref:10)

[24] V Pasumarti, L Bianchi, and A Loquercio, “Agile flight emerges from multi-agent competitive racing,” arXiv preprint arXiv:2512.11781 (ref:25)

[25] J Forster, “System identification of the crazyflie 2.0 nano quadro-copter,” 2015 (ref:10)

[26] J Schulman, F Wolski, P Dhariwal et al., “Proximal Policy Optimization Algorithms,” Aug. p. 7 2017 (ref:10)

[27] M Mittal, P Roth, J Tigue et al., “Isaac lab: A gpuaccelerated simulation framework for multi-modal robot learning,” arXiv preprint arXiv:2511. p. 704831 (ref:25)

[28] A Ma’Arif, W Rahmaniar, M A M Vera et al., “Artificial Potential Field Algorithm for Obstacle Avoidance in UAV Quadrotor for Dynamic Environment,” in 2021 IEEE International Conference on Communication, Networks and Satellite (COMNETSAT), Jul. p. 7 2021 (ref:10)

[3] E Kaufmann, L Bauersfeld, and D Scaramuzza, “A Benchmark Comparison of Learned Control Policies for Agile Quadrotor Flight,” in 2022 International Conference on Robotics and Automation (ICRA), IEEE, 2022 (ref:10)

[11] M Lanctot, V Zambaldi, A Gruslys et al., “A Unified Game-Theoretic Approach to Multiagent Reinforcement Learning,” Nov. p. 7 2017 (ref:10)

[12] P Muller, S Omidshafiei, M Rowland et al., “A Generalized Training Approach for Multiagent Learning,” Feb. 2020 (ref:10)

[13] J Chen, C Yu, G Li et al., “Online Planning for Multi-UAV Pursuit-Evasion in Unknown Environments Using Deep Reinforcement Learning,” IEEE Robotics and Automation Letters, pp. 1–8 (ref:10)

[2506.02849] is the arXiv ID of the paper. p. 0

--- Page 1 ---

Learned Controllers for Agile Quadrotors in Pursuit-Evasion Games

Abstract— In this letter we study 1v1 quadrotor pursuit–evasion, where a pursuer and an evader are trained via reinforcement learning (RL) by competing against each other Such adversarial settings face well-known challenges: each agent’s policy changes during training, creating a non-stationary environment; agents might overfit to the current opponent and forget earlier strategies (catastrophic forgetting); and the competitive dynamics can cause strategy cycling or policy collapse To address these issues, we propose Asynchronous Multi-Stage Population-Based training with Hedge sampling (AMSPBH), a method based on Policy-Space Response Oracles (PSRO) and adapted to quadrotor RL control

I. INTRODUCTION

The increasing proliferation of small unmanned aerial vehicles (UAVs) such as quadrotors in civilian and military contexts has raised urgent safety and security concerns [1] If used with malicious intent, UAVs can cause infrastructure damage, privacy violations, or mid-air collisions with manned aircraft To mitigate these risks, we explore strategies where a friendly pursuer UAV intercepts an intruder (see Fig. 1) In general, intercepting a maneuvering quadrotor is challenging: its strategy and dynamics are often unknown a priori, and quadrotors are highly maneuverable [2] To succeed, the interceptor must develop robust counter-strategies and potentially match the intruder’s agility Pursuit–evasion games (PEG) naturally model such interactions Reinforcement learning (RL) offers a promising approach: one specifies the task through a reward function (capture for the pursuer, survival for the evader) and trains a neural network policy that jointly handles perception, planning, and control [3] This has proven effective in drone racing, where RL controllers have surpassed expert human pilots by exploiting the full dynamic envelope [2], [4] However, existing RL approaches for PEG often assume simplified 2D dynamics, train against a single frozen opponent per stage, or use simple evaders that do not challenge the pursuer [4] Adversarial RL introduces challenges such as non-stationarity, catastrophic forgetting, strategy cycling, and policy collapse

--- Page 2 ---

population-based training: AMSPBH discovers new strategies while retaining competence against older opponents, reaching a regime where additional best-response training gives limited improvement

--- Page 3 ---

A. Quadrotor Dynamics and Control

We model each quadrotor as a rigid body of mass m and diagonal inertia tensor J = diag(Jxx,Jyy,Jzz) The state is x = (W p, q, W v, ω, omega), comprising position W p ∈ R 3 in the world frame

--- Page 4 ---

b) Evaluate π0jπ1jπk-1i vs. πk-1j??? who wins?

--- Page 5 ---

a) AMSPBH (ours) shows a strong triangular pattern, with later generations from both sides consistently winning against earlier opponents; AMSDRL (b) shows more erratic performance (less strong triangular pattern) due to forgetting

--- Page 6 ---

(a) AMSPBH (ours) is higher only against trajectory evaders, while AMSPBH remains high and performs best against the unseen APF evader The nominal γh = 0.7, ηh = 2 gives strong improvement and the best tested baselin

--- Page 7 ---

Fig. 7 and Table II summarize the physical experiments The AMSPBH pursuer captures several evader types, including hover, circle, manual (see below), AMSDRL, and AMSPBH policies Across the 42 physical flights summarized in Table II, the vehicles reach up to 4. p. 687 m/s linear speed and 6. p. 608 rad/s angular speed, confirming that the learned policies remain agile after transfer

--- Page 8 ---

TABLE II

REAL-WORLD PURSUIT–EVASION MATCHUP SUMMARY.

--- Page 9 ---

Fig. 3. Pursuer win-rate heatmaps across 13 generations

--- Page 10 ---

Fig. 4. Evaluation of win-rates (higher is better) for final-generation Pursuers and Evaders (ref:2506. p. 502849)

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Adaptive Opponent Curriculum via Hedge Sampling

The system can maintain competence against older strategies while adapting to new ones by using a Hedge sampler to assign higher probability to opponents that are currently difficult to beat (Equation 12). This creates an auto-curriculum where the oracle is trained as a single policy against the mixture, allowing it to expose each new controller to old, recent, heuristic, and learned strategies without making the RL target fully non-stationary.

  1. Robust Generalization Across Diverse Strategies

The population-based training generalizes better across diverse and unseen opponent strategies compared to training against only the latest opponent. The results show that AMSPBH is stronger or tied in most out-of-training evaluations, demonstrating robustness against both handcrafted and learned adversaries (Fig 4).

  1. High Speed Agility Transfer

The trained policies exhibit zero-shot sim-to-real transfer of agile pursuit evasion behavior on physical hardware reaching up to 4.87 m/s linear speed (Table II). This confirms that the population-based training successfully bridges the sim-to-real gap for high agility.

Sources

Related papers