Learned Controllers for Agile Quadrotors in Pursuit-Evasion Games
summary
The gist
In this letter we study 1v1 quadrotor pursuit–evasion, where a pursuer and an evader are trained via reinforcement learning (RL) by competing against each other Such adversarial settings face…
In short
The paper addresses challenges in training reinforcement learning agents for 1v1 quadrotor pursuit-evasion games, such as policy changes and forgetting old strategies. It proposes Asynchronous Multi-Stage Population-Based training with Hedge sampling (AMSPBH), which maintains a population of past policies. This method allows agents to continually improve by competing against mixtures of older and newer opponents, leading to robust controllers that generalize well.
Key concepts
- Policy-Space Response Oracles (PSRO)
- This is the core idea where a system maintains a collection of previously trained policies for both the pursuer and evader. Instead of training against just one opponent, it trains new policies against a mixture of these past opponents, helping to retain competence against older strategies.
- Hedge Sampling
- This technique is used to create an adaptive curriculum during training. It assigns higher probability to opponents that the current agent finds difficult to beat. This focuses the learning process on challenging scenarios while ensuring coverage of various strategies.
- Population-Based Training
- Instead of training against a single, constantly changing opponent, this method trains agents against a diverse population of previously learned policies. This approach helps prevent catastrophic forgetting and ensures the resulting policy is robust across many different strategies.
- Zero-Shot Sim-to-Real Transfer
- This means the trained RL policies can perform agile pursuit and evasion tasks successfully on physical hardware (like Crazyflie quadrotors) without needing extensive retraining. This confirms the learned skills are general enough to work in the real world.
Terminology used across episodes
This episode discusses
- Learned Controllers for Agile Quadrotors in Pursuit-Evasion Games · Paper Radio
- Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge?
- Population Based Training of Neural Networks
- Agile Flight Emerges from Multi-Agent Competitive Racing
- Isaac Lab: A GPU-Accelerated Simulation Framework for Multi-Modal Robot Learning
The paper
Learned Controllers for Agile Quadrotors in Pursuit-Evasion Games · Read on arXiv
Alejandro Sanchez Roncero, Yixi Cai, Olov Andersson, 'Petter Ogren ƈ
Royal Institute of Technology (KTH) School of Electrical Engineering and Computer Science
Transcript
Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.
Rosa: I'm Rosa.
Dev: And I'm Dev, here with Taro.
Rosa: We're looking at the paper: "Learned Controllers for Agile Quadrotors in Pursuit-Evasion Games".
Title and authors: Rosa: So we're looking at this paper called "Learned Controllers for Agile Quadrotors in Pursuit-Evasion Games." It’s about training a pursuer and an evader using reinforcement learning by having them compete against each other.
Dev: That adversarial setting is tricky because the policies keep changing during training, which makes the environment non-stationary, and agents can forget old strategies or just collapse into something bad.
Taro: Exactly, those policy changes mean the agents might get stuck in a loop or just totally fail to respond when things get chaotic.
Rosa: To tackle that, they propose something called Asynchronous Multi-Stage Population-Based training with Hedge sampling, which uses Policy-Space Response Oracles and adapts it for quadrotor reinforcement learning.
Dev: It sounds like they're trying to solve the problem of agents overfitting to their immediate opponent by having them train against a mixture of previously trained policies.
Taro: So instead of just fighting one person, the agent is being tested against a whole group of past behaviors at once, which should expose it to different strategies.
Rosa: That's right; in this method, each generation trains one agent using Proximal Policy Optimization against frozen opponent policies, and then a Hedge sampler decides which opponents are currently the hardest to beat.
Dev: The Hedge sampler assigns higher probability to those opponents that the agent is struggling against, while still keeping some exploration going through an exploration term.
Taro: So it’s like a curriculum where the system learns by being pushed by its own history and then focusing its training power on what it's currently bad at.
Rosa: They show that this population-based training helps the agents find new strategies while still keeping them competent against older ones, which means they reach a point where more training against just the latest opponent doesn't give much better results.
Dev: That makes sense; if you keep fighting only the newest thing, you might miss important underlying behaviors from earlier in the training process.
Taro: And when they test it on Crazyflie brushless quadrotors, they show this zero-shot sim-to-real transfer of agile pursuit evasion behavior against both handcrafted and learned adversaries up to four point eight seven meters per second <ref:2506.02849#pg2>.
Rosa: That speed is really impressive, especially since recent baselines reported real flight speeds around one meter per second in those scenarios <ref:2506.02849#pg1>.
Dev: The paper also notes that the AMSPBH evader manages to transfer zero-shot as well, escaping manual and learned pursuers in all the tested trials.
Taro: So it’s not just about the pursuer improving; the evasion side is also showing robust performance against various threats.
Rosa: It seems like they are using this population training to create a diverse set of pursuit and evasion behaviors that actually give them strategic diversity when they need it most.
Dev: The core improvement here is adaptive opponent curriculum via Hedge sampling, which allows the system to maintain competence against older strategies while adapting to new ones by assigning higher probability to opponents that are currently difficult to beat.
Title and authors: Taro: That mechanism creates an auto-curriculum where the oracle is trained as a single policy against the mixture, exposing each new controller to old, recent, heuristic, and learned strategies without making the reinforcement learning target fully non-stationary.
Rosa: They also demonstrate that population-based training generalizes across diverse and unseen opponent strategies better than just training against only the latest opponent.
Dev: The results show that AMSPBH is stronger or tied in most out-of-training evaluations, which proves its robustness against both handcrafted and learned adversaries.
Taro: And they’ve validated this on real hardware, confirming that the trained policies exhibit zero-shot sim-to-real transfer of agile pursuit evasion behavior on physical hardware reaching up to four point eight seven meters per second <ref:2506.02849#pg2>.
Rosa: It sounds like they are making a big deal about how this method bridges the gap between simulations and actual flight, which is always a challenge in robotics research.
Dev: The paper also points out that the policies being used are feed-forward neural networks, which means they have limitations regarding memory and capacity when dealing with partially observable mixtures of opponents.
Taro: That’s a fair caveat; their current architecture isn't as robust as it could be in a fully uncertain world.
Rosa: Future work mentioned in the paper suggests that the frozen population isn't just a training memory but also a controlled curriculum over opponent behaviors, which could be useful for other competitive agile-flight domains.
Dev: Another limitation they flag is that recurrent architectures might improve both robustness and behavioral diversity for these agents.
Taro: And they also mentioned the assumption of perfect state information, suggesting that incorporating onboard perception could remove that limitation in the future.
Rosa: To wrap up, this paper on Learned Controllers for Agile Quadrotors in Pursuit-Evasion Games introduces AMSPBH as a method based on Policy-Space Response Oracles to handle the non-stationarity and forgetting issues in adversarial pursuit games.
Dev: The main implication is that population-based training offers a way to discover new strategies while retaining competence against older ones, leading to better generalization across diverse opponent types.
Taro: For me, it’s interesting because it shows how you can structure the training process itself—the curriculum—to manage complexity instead of just relying on a single learning loop.
Rosa: Exactly; they show that by mixing in prior knowledge and focusing the training signal using that Hedge sampler, you can get high-performance policies that are genuinely robust when deployed in the real world.
Dev: It’s about making sure these learned controllers don't just work against the specific opponent they just fought, but actually have a solid understanding of what different behaviors look like overall.
Taro: So, for anyone building pursuit evasion systems, this suggests looking beyond single-opponent training and considering how to manage a diverse policy population throughout the learning process.
Rosa: That’s the gist of it; AMSPBH is about using a smart population strategy to keep those quadrotors agile and effective in unpredictable combat scenarios.
The paper's summary: Rosa: So, this paper is about how they trained pursuit and evasion quadrotors using reinforcement learning by making them fight each other in a competition against a population of previous strategies.
Dev: Right, yeah, they call it Asynchronous Multi-Stage Population-Based training with Hedge sampling. It’s basically trying to solve that problem where the agents keep forgetting what they learned earlier or just getting stuck repeating the same moves.
Rosa: Exactly, and this new approach uses something called Policy-Space Response Oracles to maintain a whole population of past behaviors, and then it uses that Hedge sampler to focus on training against the opponents that are currently giving them trouble.
Dev: That’s the core mechanism, right? So instead of just training one agent against one frozen opponent at a time, they mix in previous policies and give more weight to the ones that are hard to beat right now.
Rosa: It’s like an auto-curriculum where the system is constantly exposed to a history of strategies while simultaneously getting focused training on what it’s currently bad at.
Dev: And they show that this helps the agents discover new strategies while still keeping competence against older ones, which means they don't completely lose their skills when things change.
Rosa: The results are pretty strong; the learned policies can handle real-world Crazyflie quadrotors with zero-shot transfer of agile pursuit evasion behavior up to four point eight seven meters per second.
Dev: And it’s not just the pursuer that's doing well; the evader also shows good transfer, escaping both manual and learned pursuers in all the tests.
Rosa: It suggests this method is really good at bridging that gap between simulation and real flight, which is always a big hurdle in robotics.
Dev: The authors admit they are using feed-forward neural networks, so they point out that these policies have limitations regarding memory and capacity when dealing with a lot of uncertainty in the opponent's strategy.
Rosa: They suggest that the frozen population isn't just a training memory but also a controlled curriculum over opponent behaviors, which could be useful for other competitive agile-flight domains.
Dev: That’s interesting because they mention that using recurrent architectures might improve both robustness and behavioral diversity for these agents.
Rosa: So it seems like the big idea here is using this population-based training to create a set of behaviors that give the quadrotors strategic diversity when they need it most in unpredictable combat scenarios.
Dev: It’s about building controllers that don't just work against the specific opponent they just fought, but actually have a solid understanding of what different behaviors look like overall.
Rosa: And they show this robustness across diverse and unseen opponent strategies, which is a big deal for real-world deployment where you never know what kind of threat you're going to face.
Dev: So it’s not just about getting the best score against the current opponent; it's about making sure those learned controllers are solid enough to handle whatever comes next.
Rosa: Next time, we’ll look at how they actually set up that training signal construction and see exactly how they build that population of policies.
The paper's improvements: Taro: So, we've seen how they trained these agents using population training, and now we need to talk about what this paper suggests are the actual improvements they made to the system.
Rosa: Well, one big improvement is this adaptive opponent curriculum using Hedge sampling. It lets the system maintain competence against older strategies while simultaneously adapting to new ones by prioritizing opponents that are currently difficult to beat.
Dev: That’s smart, because it creates an auto-curriculum where the oracle learns against a mixture of policies, exposing each new controller to old and recent behaviors without making the training target completely non-stationary.
Rosa: It means the system is constantly being challenged by its own history while still focusing on what it needs to learn right now.
Dev: And another point they make is how this population-based training leads to better generalization across diverse and unseen opponent strategies compared to just training against only the latest one.
Taro: That’s important because when you deploy these controllers in the real world, you don't know exactly what kind of threat you're going to face, so being robust against things it hasn't seen before is key.
Rosa: The results show that this method is stronger or tied in most out-of-training evaluations, proving it handles both handcrafted and learned adversaries well.
Dev: And they’re showing high speed agility transfer on physical hardware, reaching four point eight seven meters per second in those tests with zero-shot sim to real transfer.
Taro: That speed is what makes it tangible for us; it means the policies aren't just theoretical models, but they actually perform well when there's real-world dynamics involved.
Rosa: The implication here is that we can build pursuit evasion systems that are genuinely robust across a wider range of unpredictable scenarios than before.
Dev: They also flag some limitations on their end; the policies they use are feed-forward neural networks, so they admit this limits how much memory and capacity the system has for dealing with those complex mixtures of opponents.
Taro: So while the method is solid for exploration, it seems like recurrent architectures might actually improve both robustness and behavioral diversity in these systems.
Rosa: That’s a good direction to look at; moving towards recurrent models could help them handle the complexity of a real-time, dynamic fight better.
Dev: And they also mention that incorporating onboard perception could remove the assumption that you have perfect state information, which is another big hurdle in these kinds of games.
Taro: So it sounds like the next steps for this research are focusing on improving the architecture itself rather than just tweaking how they sample their opponents.
Conclusion: Rosa: So we're wrapping up on "Learned Controllers for Agile Quadrotors in Pursuit-Evasion Games" by AMSPBH, which basically shows how population training can make pursuit and evasion agents really good at handling non-stationary opponents.
Dev: Yeah, the main thing is that this method helps agents discover new strategies while retaining competence against older ones, giving them better generalization.
Rosa: It’s a solid approach for building controllers that are tough when you deploy them in the real world because they’ve seen a lot of different behaviors.
Dev: The results with the Crazyflie quadrotors at four point eight seven meters per second really show that this training bridges the gap between simulation and actual flight speed.
Taro: I just want to emphasize how well it handles those unseen strategies, which is crucial because in real scenarios, you never know what kind of adversary you're going to run into.
Rosa: Exactly, and they’ve shown that the evader side also transfers well, escaping both manual and learned pursuers in all their tests.
Dev: It’s about building safety cases for these things; the fact that they're performing so reliably under pressure is what matters most for real-time systems.
Taro: The limitation they mentioned is that using feed-forward neural networks means there are some limits on how much memory and capacity those policies can actually hold when the opponent's strategy gets really complicated.
Rosa: So while the method is effective, we still have to consider future work, like moving towards recurrent architectures to potentially improve that memory aspect.
Dev: And they also noted that incorporating onboard perception would remove the assumption of perfect state information, which is another big thing for making these systems truly reliable.
Taro: If you look at this paper on "Learned Controllers for Agile Quadrotors in Pursuit-Evasion Games," it shows how population strategies can create a diverse set of behaviors that are useful when you need strategic variety in combat.
Rosa: It’s definitely a way to structure the training process itself to manage complexity instead of just relying on one learning loop.
Dev: For now, this is a strong piece for understanding how population-based training can tackle the non-stationarity inherent in these competitive games.
Taro: Next time, we'll look at some papers that deal with how agents coordinate when they aren't just fighting each other but working together in complex ways.
More episodes
- 2608.13438-ContactGuard: Pre-Contact Execution Monitoring with Action-Conditioned Latent World Models
- 2610.34593-Future Prediction as Training-Only Video Supervision for Pretrained VLA Policies
- 2610.35385-EmbodiedLab: Beyond Manipulation, Benchmarking Embodied Agents That Act to Know
- 2610.36378-Data-Driven Hamiltonian for Neural Optimal Control
- 2610.37275-A Heuristic for Scheduling Autonomous Vehicles at Intersections
- 2610.37288-Dynamic Coordination of Autonomous Vehicles at Intersections
- 2610.37632-Real-Time Coordination of Autonomous Vehicles at Intersections
- 2611.13779-Softness is something you also need! Why compliance helps control: information metrics of a passive walker with compliant ankle
- 2611.13411-Residual-Corrected Physics-Informed Neural Network KKL Observers with Input-to-State Stability Guarantees
- 2611.14488-Is Your Robot's Memory Just a Clock? Temporal Shortcuts in Memory-Augmented VLA Policies