FlowPilot: Real-Time World-Action Modeling for Agile UAV Navigation

arXiv:2608.00635 · cs.RO · Submitted 2026-08-01 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: I'm Rosa.

Dev: And I'm Dev, here with Taro.

Rosa: We're looking at the paper: "FlowPilot: Real-Time World-Action Modeling for Agile UAV Navigation".

Dev: Here's the gist of the paper. We present FlowPilot, a compact world-action model for real-time onboard UAV navigation from depth; this is because it jointly denoises future depth observations and executable trajectories with flow matching <ref:2608.00635#pg2>.

Rosa: The Core Model Architecture FlowPilot couples a depth-video expert and an action expert in a dual-stream mixture-of-transformers (MoT) <ref:2608.00635#pg3>.

Dev: During training, the two streams are jointly denoised with flow matching so that future-depth prediction and trajectory generation condition on each other through shared attention <ref:2608.00635#pg2>.

Rosa: The architecture involves a video stream that encodes the current depth observation and noisy future-depth latents, while the action stream encodes the state–goal vector, speed command, previous trajectory, noisy Bernstein control points, and register tokens <ref:2608.00635#pg3>.

Title and authors: Rosa: So we're talking about FlowPilot today. It’s a compact world action model for real-time onboard UAV navigation from depth.

Dev: Right. The core idea is coupling future depth prediction and executable trajectories together using flow matching, which allows them to inform each other through shared attention in this dual stream mixture of transformers architecture <ref:2608.00635#pg2>.

Taro: So it’s not just predicting where the camera sees now, but also what’s coming next and how to move smoothly based on that future information.

Rosa: Exactly. The architecture has a video stream and an action stream working together during training, which is interesting because they are jointly denoised with flow matching <ref:2608.00635#pg2>.

Dev: And the action representation is key here because it predicts only the free control points of a degree-seven Bernstein polynomial <ref:2608.00635#pg4>, which gives you C two continuous references with closed form velocity, acceleration and jerk <ref:2608.00635#pg2>.

Taro: That means it avoids those big derivative oscillations you get when you do numerical differentiation on waypoints, which is a huge deal for stability <ref:2608.00635#pg4>.

Rosa: And the training strategy is pretty structured, with phase one focusing on world modeling where the video stream learns to predict future depth frames alone <ref:2608.00635#pg5>.

Dev: Phase two then handles the world action alignment, where you train the full dual stream model jointly, using high quality expert trajectories to specialize that navigation prior into a precise policy <ref:2608.00635#pg5>.

Taro: I wonder how robust it is when we move from simulation to real world because that’s where things usually get tricky <ref:2608.00635#pg2>.

Rosa: That’s what I wanted to ask. The paper shows they trained on a three level depth pyramid, spanning high throughput simulation, photorealistic simulation, and real onboard data <ref:2608.00635#pg2>.

Dev: That hierarchy starts with sixteen hours of depth from a GPU parallel high throughput Isaac Lab simulation which gives the raw scale needed for diverse geometry and motion <ref:2608.00635#pg2>.

Taro: Then they add eight hours of photorealistic and geometrically complex depth rendered in Flightmare to give it visual and structural realism that the faster simulator misses <ref:2608.00635#pg2>.

Rosa: And finally, the apex is two hours of depth collected on a physical quadrotor, which brings in the true onboard sensor statistics like noise, dropouts and range limits <ref:2608.00635#pg2>.

Dev: The objective function for training uses linear interpolation flow matching where the network predicts a velocity field v theta equals f theta of z sigma sigma c from the conditioning c <ref:2608.00635#pg2>.

Taro: They minimize the sum of its velocity losses L theta action and L theta video balanced by a weight lambda, applying this to both the normalized free Bernstein control points z A zero and the future depth VAE latent z V zero in one forward pass <ref:2608.00635#pg2>.

Title and authors: Rosa: And they mention that synchronous future depth denoising acts as action guidance rather than just an auxiliary prediction task <ref:2608.00635#pg2>.

Dev: That coupling is what makes it work; the model learns both as a world model and a policy simultaneously, which allows the policy to benefit from world model co training while keeping the deployed perception to action path compact <ref:2608.00635#pg2>.

Taro: But I also saw something about warm starting that might cause trouble if we try it in real life <ref:2608.00635#pg5>.

Rosa: Right. The paper suggests that instead of warm starting the denoising process from the previous plan, supplying the previous trajectory as a soft conditioning signal steers successive plans toward temporal consistency while preserving the multimodality of the flow matching distribution <ref:2608.00635#pg5>.

Dev: That’s a clever way to handle replanning consistency, using that prior information as guidance instead of just starting from scratch <ref:2608.00635#pg5>.

Taro: So what about the performance numbers when it actually flies? I want to know if this stays fast enough for an onboard system <ref:2608.00635#pg2>.

Rosa: It runs action centrically onboard and outputs only a trajectory, and on a physical quadrotor, the full perception to action pipeline runs in under eighteen milliseconds on a Jetson Orin NX <ref:2608.00635#pg5>.

Dev: And it reaches five point five meters per second in cluttered indoor and forest environments using only onboard sensing and computation <ref:2608.00635#pg2>.

Taro: That’s pretty impressive for the real world, but what’s the catch? What are the limitations we need to keep in mind when we deploy this on actual hardware <ref:2608.00635#pg2>?

Rosa: FlowPilot remains limited to shorthorizon, goal directed navigation from a single depth camera <ref:2608.00635#pg2>, which means it can be vulnerable to thin obstacles, transparent surfaces, and regions outside the depth camera’s reliable sensing range <ref:2608.00635#pg2>.

Dev: It stops working when the scene gets too complex or when you need to look further ahead than what that single depth camera can reliably see <ref:2608.00635#pg2>.

Taro: So for someone just listening, what does this actually change about how we think about drone navigation compared to the old map then optimize pipelines they mentioned <ref:2608.00635#pg1>?

Rosa: It moves away from needing a separate local reconstruction or an end to end policy that lacks explicit scene prediction by jointly denoising future depth observations and executable trajectories <ref:2608.00635#pg1>.

Dev: And the action representation, predicting those five free control points of the degree seven Bernstein polynomial, is what makes it trackable for a flight controller <ref:2608.00635#pg4>.

Taro: That’s the part that really matters for safety; having analytic derivatives for velocity and acceleration means we don't rely on noisy numerical differentiation <ref:2608.00635#pg4>.

Title and authors: Rosa: So when you look at the overall results, FlowPilot maintains the highest overall success rate across commanded speeds from three meters per second to eight meters per second compared to optimization and learning based baselines <ref:2608.00635#pg2>.

Dev: And that performance boost is seen even when obstacle density or commanded speed increases <ref:2608.00635#pg2>.

Taro: It transfers to arbitrary obstacle distributions rather than just memorizing the training geometry, which means it generalizes better <ref:2608.00635#pg2>.

Rosa: Exactly, it’s not just memorizing the training geometry; it’s grounded in the model's predicted geometry rather than being learned in isolation <ref:2608.00635#pg5>.

Dev: It achieves higher closed loop success rates under those tougher conditions compared to optimization and learning based baselines <ref:2608.00635#pg2>.

Taro: I just want to make sure we understand the latency part because that's critical for real time execution <ref:2608.00635#pg5>.

Rosa: The total inference latency for the deployed stack is under eighteen milliseconds on average, with a maximum of seventeen point zero five five milliseconds <ref:2608.00635#pg5>.

Dev: That latency breaks down into depth preprocessing at four point seven four seven milliseconds, then a distilled depth encoder at one point one six three milliseconds, flow denoising at fourteen point nine two four milliseconds, and MPC latency at two point three eight four milliseconds <ref:2608.00635#pg5>.

Taro: So that means the flow denoising part is actually taking up a lot of time in the loop, which tells us how much computational work we have to do on board <ref:2608.00635#pg5>.

Rosa: And that’s where the coupling helps; by predicting future depth and trajectory together, they manage to keep that entire pipeline under that eighteen millisecond budget <ref:2608.00635#pg5>.

Dev: So it's not just one part being fast, it's the joint operation of both streams that makes this compact architecture viable for real time <ref:2608.00635#pg2>.

Taro: It’s interesting how they trained on that three level depth pyramid—from Isaac Lab simulation to photorealistic Flightmare rendering and finally real onboard flight data <ref:2608.00635#pg2>.

Rosa: That training data structure ensures the model is grounded in reality, which allows it to transfer to physical quadrotors without needing ground station computation or offline mapping <ref:2608.00635#pg5>.

Dev: So the main contribution is this compact world action model for real time onboard UAV navigation that couples future depth prediction and trajectory generation in a single dual stream network <ref:2608.00635#pg2>.

Taro: I think the most impactful thing is how they achieved C two continuity through those Bernstein polynomials, which gives us that smooth coupling between neighboring points <ref:2608.00635#pg4>.

Rosa: It’s a compact model that provides a controller trackable generative action space by predicting those free control points <ref:2608.00635#pg2>.

Dev: And the training on that pyramid, combined with the synchronous future depth denoising, shows that future depth latents encode informative anticipatory priors <ref:2608.00635#pg2>.

Taro: But we have to remember their limitation: FlowPilot remains limited to shorthorizon goal directed navigation from a single depth camera <ref:2608.00635#pg2>.

Title and authors: Rosa: That means it still struggles with things outside that immediate view, like thin obstacles or regions where the depth camera isn't reliable <ref:2608.00635#pg2>.

Dev: For future work, they suggest extending the model with longer horizon or memory based prediction and complementary sensing modalities to address those shortcomings <ref:2608.00635#pg2>.

Taro: And on that note, they mentioned that warm starting the denoising process from the previous plan can backfire if you don't condition it correctly <ref:2608.00635#pg5>.

Rosa: So supplying the previous plan as a soft conditioning signal is the intuitive remedy to steering successive plans toward temporal consistency while preserving the multimodality of the flow matching distribution <ref:2608.00635#pg5>.

Dev: It’s about learning that joint behavior rather than hand designing it because a single network serves as both a world model and a policy <ref:2608.00635#pg2>.

Taro: So overall, FlowPilot achieves higher closed loop success rates than optimization and learning based baselines under increasing obstacle density and commanded speed <ref:2608.00635#pg2>.

Rosa: It’s a system that is highly compact, runs fast on hardware like a Jetson Orin NX, and demonstrates real world transfer without needing ground station computation or offline mapping <ref:2608.00635#pg5>.

Dev: So to wrap up on FlowPilot: it brings flow matching world action modeling to real time depth based UAV navigation by coupling future depth prediction and trajectory generation in a single dual stream network <ref:2608.00635#pg2>.

Taro: It’s a compact system that gives us C two continuous references with closed form velocity acceleration and jerk, which is really what makes it usable for flight control <ref:2608.00635#pg4>.

Rosa: It’s trained on a three level depth pyramid spanning large scale Isaac Lab simulation, photorealistic Flightmare rendering, and real onboard flight <ref:2608.00635#pg2>.

Dev: The key to making this practical on a drone is that action representation where predicting five control points instead of waypoints shrinks the action stream from tens of tokens to five, reducing the joint attention length and inference cost <ref:2608.00635#pg4>.

Taro: It’s a design choice that lets the policy benefit from world model co training while keeping the deployed perception to action path compact <ref:2608.00635#pg2>.

Rosa: So, in conclusion, FlowPilot achieves higher closed loop success rates than optimization and learning based baselines under increasing obstacle density and commanded speed <ref:2608.00635#pg2>.

Dev: It’s a system that runs the full perception to action pipeline in under eighteen milliseconds on a Jetson Orin NX, reaching five point five meters per second in cluttered indoor and forest environments <ref:2608.00635#pg2>.

Taro: It still has those limits though, being limited to shorthorizon goal directed navigation from a single depth camera <ref:2608.00635#pg2>.

Rosa: So it’s a powerful tool for real time onboard UAV navigation but we need to keep an eye on extending it with longer horizon prediction and complementary sensing modalities for future work <ref:2608.00635#pg2>.

The paper's summary: Rosa: So, to wrap up what we just talked about, FlowPilot is this compact world action model that couples future depth prediction and executable trajectories together using flow matching in a dual stream network.

Dev: Right. The main takeaway is how they made it practical for onboard UAV navigation by focusing on that action representation—predicting only the free control points of a degree seven Bernstein polynomial.

Taro: That’s the part where we get C two continuous references with analytic velocity, acceleration, and jerk.

Rosa: Exactly. By predicting those five points instead of waypoints, they shrink the action stream from tens of tokens down to just five tokens in the joint attention mechanism.

Dev: Which drastically cuts down on the inference cost and makes it feasible for real-time execution on hardware like a Jetson Orin NX.

Taro: And that's huge because we need low latency for flight control loops, you know, sub eighteen milliseconds.

Rosa: It is critical. They achieved this speed by training the model on a three level depth pyramid: starting with big scale simulation, moving to photorealistic rendering, and finally real onboard flight data.

Dev: That three level hierarchy is what gives it that grounding in reality without needing massive ground station computation or offline mapping for transferability.

Taro: So, the training process itself is split into two phases: first, learning how the world evolves from depth observations alone, and then aligning that with precise action generation using expert trajectories.

Rosa: That's a smart way to build the navigation prior; you get a broad understanding of scene evolution before you fine-tune it to specific movements.

Dev: And they also tackled replanning consistency by not just warm starting the denoising process from the last command, but instead supplying the previous trajectory as a soft conditioning signal.

Taro: So that keeps things temporally consistent while still preserving that flow matching distribution, which is really smart for handling unexpected situations.

Rosa: It shows they are thinking about how to make this system robust when things go wrong or change quickly.

Dev: And the results in simulation and real world show it maintains the highest success rate compared to optimization and learning based baselines even under increasing obstacle density and commanded speed.

Taro: That generalization to arbitrary obstacle distributions, rather than just memorizing the training geometry, is what makes it really useful for deployment.

Rosa: It’s not just memorizing geometry; it’s grounded in the model's predicted geometry, which means it transfers to new physical quadrotors without needing that extra heavy offline mapping setup.

Dev: The main limitation they state is that FlowPilot remains limited to shorthorizon, goal directed navigation from a single depth camera.

Taro: So what that means for the listener is it can't handle things outside of what that one camera can reliably see, like very thin obstacles or areas where the depth sensor just goes blind.

Rosa: Yeah, and for future work, they suggest extending this with longer horizon prediction or memory based prediction to tackle those shortcomings.

Dev: It’s about making the system smarter about looking ahead and remembering things over a longer time frame.

Taro: That opens up possibilities for more complex maneuvers that require anticipating what's going to happen further down the path.

The paper's improvements: Rosa: So, we just talked about how FlowPilot uses flow matching to couple future depth prediction and trajectories in a single network for real time navigation on drones.

Dev: Yeah, that's the core mechanism, but now let's talk about what they suggest next—the improvements they propose for this system.

Taro: I think the most interesting thing is how they handle those trajectory predictions—they use this action representation that yields C two continuous references with analytic velocity and acceleration.

Rosa: That’s right. So the improvement there is making sure that what you predict as a path is physically smooth for a flight controller to follow directly, without needing any extra math on the fly.

Dev: It means no more relying on numerical differentiation of noisy samples; everything has its velocity and acceleration in closed form, which is crucial for safety cases.

Taro: That makes sense for real world testing because it removes that source of error we always worry about when dealing with sensor noise or model imperfections.

Rosa: Then there's the training structure itself; they suggest a two phase training approach to build the navigation prior.

Dev: Phase one builds a broad understanding of how the world evolves from just depth observations, and phase two aligns that with precise action generation using expert trajectories.

Taro: That separation helps ensure you get a good general feel for the environment before you start fine-tuning for specific actions.

Rosa: And they also introduced a better way to handle replanning consistency by suggesting using the previous trajectory as a soft conditioning signal instead of just warm starting from scratch.

Dev: That's an improvement in how it handles temporal continuity; it steers the next plan toward consistency while still keeping the flow matching distribution flexible.

Taro: That makes recovery smoother when things are uncertain, because you’re not starting completely blind every time you need to replan.

Rosa: So these improvements focus on making the model more controllable and temporally consistent under real world conditions.

Dev: And ultimately, by doing this, they aim for a system that's not just a good predictor but an executable policy grounded in physical constraints.

Taro: It’s about moving from something that works well in simulation to something that actually performs reliably when the world misbehaves in reality.

Rosa: The implication is that we can build UAV navigation systems where the planning and control are tightly coupled, making them much more responsive and stable than older methods.

Conclusion: Rosa: So to wrap up this discussion on FlowPilot, we’ve seen how they use flow matching to couple future depth prediction with trajectory generation in a dual stream network for real time navigation on drones.

Dev: Right. The main implication is that we can build UAV systems where the planning and control are tightly coupled because the action representation yields C two continuous references with analytic velocity and acceleration.

Taro: I think that smoothness is really what makes it practical; it takes the guesswork out of trajectory tracking on hardware.

Rosa: Exactly. It moves us away from having to rely on noisy numerical differentiation, which is a big win for system stability.

Dev: And the training pyramid they used—from simulation to photorealistic rendering to real flight data—shows how grounding the model in diverse data allows it to transfer well without needing ground station computation for every deployment.

Taro: It’s interesting that they found success even when the environment is cluttered or complex, showing it generalizes better than just memorizing training geometry.

Rosa: And remember, the latency is under eighteen milliseconds on a Jetson Orin NX, which means this isn't just a theoretical model; it runs fast enough to actually fly.

Dev: That speed comes from shrinking that action stream down to only five tokens, which keeps the joint attention length manageable.

Taro: But we still have those limits—it’s still shorthorizon and goal directed from one depth camera.

Rosa: True. So the next step for this research is definitely extending it with longer horizon or memory based prediction to tackle those shortcomings in future work.

Dev: That points toward more sophisticated predictive capabilities, which ties into what we see with asynchronous multi rate diffusion policies like MATE and how they handle different time scales.

Taro: It's about adding that extra layer of temporal awareness so the drone can react to things it can’t see right now but knows are coming.

Rosa: So, FlowPilot gives us a compact world action model for agile UAV navigation by tightly coupling future depth prediction and trajectory generation with flow matching.

Dev: It’s a really compact design that gives us controller trackable generative action space by predicting those free control points in that degree seven Bernstein polynomial.

Taro: And the training on that pyramid, combined with synchronous future depth denoising, shows how much anticipatory information is packed into those latent representations.

Rosa: It’s a system that runs fast on hardware and demonstrates real world transfer without needing ground station computation or offline mapping for every new setup.

Dev: It achieves higher closed loop success rates under increasing obstacle density and commanded speed compared to optimization and learning based baselines.

Taro: That's the core result; it’s more robust when things get tougher than previous methods we've seen.

Runqing Wang, Ding Yu, Pengyuan Min, Xinhong Zhang, Wei Xiao, Yu Hu, Jie Chen

National Key Lab of Autonomous Intelligent Unmanned Systems, Beijing Institute of Technology

cs.RO

Submitted: 2026-08-01

Updated: 2026-11-08

Comments: 8 pages, 8 figures, 4 tables. Accepted for publication in IEEE Robotics and Automation Letters (RA-L)

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 92/100

The gist: We present FlowPilot, a compact world-action model for real-time onboard UAV navigation from depth; this is because it jointly denoises future depth observations and executable trajectories with flow…

Key concepts

Dual-Stream Mixture of Transformers (MoT)
The model uses two separate streams: one for depth observation and another for actions. These streams interact through shared attention during training to ensure that future depth predictions guide trajectory generation, creating a unified world model.
Bernstein Polynomial Action Representation
Instead of predicting discrete waypoints, FlowPilot predicts the free control points of a degree-7 Bernstein polynomial. This mathematical representation ensures the resulting trajectory is C2 continuous and provides explicit formulas for velocity, acceleration, and jerk to be directly used by flight controllers.
Flow Matching
This technique is used during training to jointly denoise future depth observations and trajectories. It guides the model's learning process by minimizing a loss function that balances the prediction errors of both streams, effectively teaching the model how to generate physically plausible motion.

Terminology

Summary

We present FlowPilot, a compact world-action model for real-time onboard UAV navigation from depth; this is because it jointly denoises future depth observations and executable trajectories with flow matching <ref:2608. p. 000635#pg2>.

The Core Model Architecture

FlowPilot couples a depth-video expert and an action expert in a dual-stream mixture-of-transformers (MoT) <ref:2608.00635#pg3>. During training, the two streams are jointly denoised with flow matching so that future-depth prediction and trajectory generation condition on each other through shared attention <ref:2608. p. 000635#pg2>. The architecture involves a video stream that encodes the current depth observation and noisy future-depth latents, while the action stream encodes the state–goal vector, speed command, previous trajectory, noisy Bernstein control points, and register tokens <ref:2608. p. 200635#pg3>.

Action Representation for Trackability

The action representation is critical because FlowPilot predicts only the free control points of a degree-7 Bernstein polynomial whose initial control points are constrained by the current position, velocity and acceleration <ref:2608.00635#pg4>. This yields C2-continuous references with closed-form velocity, acceleration and jerk <ref:2608. p. 000635#pg2>. The paper explains that this representation is superior to independent waypoints because it provides smoothness coupling between neighboring points and avoids large derivative oscillations through numerical differentiation <ref:2608. p. 200635#pg4>.

Training Strategy for Robustness

FlowPilot is trained on a three-level depth pyramid spanning high-throughput simulation, photorealistic simulation, and real onboard data <ref:2608.00635#pg2>. The training proceeds in two phases: Phase 1 focuses on world-modeling pretraining where the video stream learns to predict future depth frames alone <ref:2608. p. 400635#pg5>. Phase 2 involves world-action alignment where the full dual-stream model is trained jointly, specializing the navigation prior into a precise policy using high-quality expert trajectories <ref:2608. p. 400635#pg5>.

Performance and Deployment

FlowPilot achieves real-time performance by running action-centrically onboard and outputs only a trajectory <ref:2608.00635#pg2>. On a physical quadrotor, the full perception-to-action pipeline runs in under 18 ms on a Jetson Orin NX and reaches 5. p. 05 m/s in cluttered indoor and forest environments using only onboard sensing and computation <ref:2608. p. 000635#pg2>. The model is trained to be grounded in the model’s predicted geometry rather than learned in isolation, which allows it to transfer to physical quadrotors without ground-station computation or offline mapping <ref:2608. p. 700635#pg5>.

Key Contributions

The primary contributions include a compact WAM for onboard UAV navigation that couples future-depth prediction and trajectory generation in a single dual-stream network <ref:2608.00635#pg2>. It provides a controller-trackable generative action space by predicting the free control points of a state constrained Bernstein polynomial yielding C2-continuous references <ref:2608. p. 100635#pg2>. Furthermore, it demonstrates closed-loop onboard validation, running fully onboard within a sub-18 ms perception-to-action budget and reaching 5. p. 15 m/s in cluttered indoor and forest environments <ref:2608. p. 0 p. 000635#pg2>. The model is trained on a three-level depth pyramid spanning largescale IsaacLab simulation, photorealistic Flightmare rendering, and real onboard flight <ref:2608. p. 700635#pg2>.

Limitations

FlowPilot remains limited to shorthorizon, goal-directed navigation from a single depth camera, making it vulnerable to thin obstacles, transparent surfaces, and regions outside the depth camera’s reliable sensing range <ref:2608.00635#pg2>. Future work should extend the model with longer-horizon or memory-based prediction and complementary sensing modalities <ref:2608. p. 700635#pg2>. The paper also notes that warm-starting the denoising process from the previous plan can backfire, suggesting that supplying the previous plan as a soft conditioning signal is an intuitive remedy <ref:2608. p. 500635#pg5>.

Experimental Validation

In simulation experiments, FlowPilot maintains the highest overall success rate across commanded speeds from 3 m/s to 8 m/s compared to optimization- and learning-based baselines <ref:2608.00635#pg2>. In real-world flight, FlowPilot navigates both zero-shot, reaching 3. p. 6 p. 68 m/s in the first and 3. p. 3 p. 31 m/s in the second, more constrained one, showing it transfers to arbitrary obstacle distributions rather than memorizing training geometry <ref:2608. p. 600635#pg2>. The onboard latency for the full perception-to-action pipeline is under 18 ms on a Jetson Orin NX <ref:2608. p. 000635#pg5>. The synchronous future-depth denoising improves both safety and traversal efficiency over the depth-frozen variant, indicating that future-depth latents encode informative anticipatory priors <ref:2608. p. 600635#pg2>.

Conclusion on Coupling

The key to making this practical on a drone is the action representation, where predicting five control points instead of waypoints shrinks the action stream from tens of tokens to five, reducing the joint attention length and inference cost <ref:2608.00635#pg4>. The joint behavior of Sec. p. 4 III-B is learned rather than hand-designed because a single network serves as both a world model as well as a policy <ref:2608. p. 400635#pg2>. This design allows the policy to benefit from world-model co-training while keeping the deployed perception-to-action path compact <ref:2608. p. 300635#pg2>. The paper concludes that FlowPilot achieves higher closed-loop success rates than optimization- and learning-based baselines under increasing obstacle density and commanded speed <ref:2608. p. 7 p. 7 p. 7 p. 7 p. 7 p. 7 p. 7 p. 700635#pg2>.

Training Data Hierarchy

The training data is structured as a three-level depth pyramid ordered from a broad, inexpensive base to a narrow, high-fidelity apex <ref:2608.00635#pg2>. The base consists of 16 h of depth from a GPU-parallel, high-throughput IsaacLab simulation which supplies the raw scale needed to cover diverse geometry and motion <ref:2608. p. 300635#pg2>. The middle level contains 8 h of photorealistic and geometrically complex depth rendered in Flightmare, adding visual and structural realism that the faster simulator omits <ref:2608. p. 300635#pg2>. The apex contains 2 h of depth collected on a physical quadrotor which carries the true onboard sensor statistics including noise, dropouts, and range limits <ref:2608. p. 300635#pg2>.

Flow-Matching Objective Details

The training utilizes linear-interpolation flow matching where the network predicts a velocity field vθ = fθ(zσ, σ, c) from the conditioning c <ref:2608.00635#pg2>. The model minimizes the sum of its velocity losses Lθ action and Lθ video, balancing these two streams with a weight λ <ref:2608. p. 400635#pg2>. This objective is applied to both the normalized free Bernstein control points zA0 and the future-depth VAE latent zV0 in one forward pass <ref:2608. p. 400635#pg2>. The training spans all (σA, σV) pairs, including the world-model regime (clean action, noisy video) and the policy regime (clean video, noisy action) <ref:2608. p. 400635#pg2>. The paper shows that synchronous future-depth denoising acts as action guidance rather than only an auxiliary prediction task <ref:2608. p. 600635#pg2>.

Onboard Inference Latency

The total inference latency for the deployed stack is 16.294 ms on average, with a maximum of 17. p. 6055 ms <ref:2608. p. 000635#pg5>. This latency is composed of depth preprocessing (4. p. 6747 ms), distilled depth encoder (1. p. 6163 ms), flow denoising (14. p. 6924 ms), and MPC latency (2. p. 6384 ms) <ref:2608. p. 000635#pg5>. The action expert predicts only b3,. p. 2.., b7 in a normalized space, which is precisely matched by construction to the action dimension and the polynomial <ref:2608. p. 300635#pg4>. This design ensures that the intra-chunk oscillation of the free-waypoint representation cannot occur by construction <ref:2608. p. 300635#pg4>.

Real-World Transfer

The real-world experiments test whether the simulated behavior remains feasible under onboard latency, sensor limits, and unstructured geometry <ref:2608.00635#pg2>. FlowPilot navigates both zero-shot, reaching 3.8 m/s in the first and 3.1 m/s in the second, more constrained one <ref:2608. p. 600635#pg2>. The outdoor flights demonstrate onboard perception and closed-loop control in unstructured terrain under the evaluated conditions <ref:2608. p. 700635#pg2>. The model transfers to physical quadrotors without ground-station computation or offline mapping, as shown by its success rate in dense scenarios <ref:2608. p. 700635#pg2>.

Final Summary of Findings

FlowPilot brings flow-matching world-action modeling to real-time depth-based UAV navigation by coupling future-depth prediction and trajectory generation in a single dual-stream network <ref:2608.00635#pg2>. The key to making this practical on a drone is the action representation, which outputs trajectories that are C2-continuous and dynamically feasible by construction <ref:2608. p. 700635#pg2>. Trained on a three-level depth pyramid spanning largescale IsaacLab simulation, photorealistic Flightmare rendering, and real onboard flight, FlowPilot achieves higher closed-loop success rates than optimization- and learning-based baselines under increasing obstacle density and commanded speed <ref:2608. p. 700635#pg2>. The model runs the full perception-to-action pipeline in under 18 ms on a Jetson Orin NX, reaching 5. p. 75 m/s in cluttered indoor and forest environments <ref:2608.00635#pg2>. However, FlowPilot remains limited to shorthorizon, goal-directed navigation from a single depth camera <ref:2608. p. 700635#pg2>. The paper concludes that FlowPilot achieves higher closed-loop success rates than optimization- and learning-based baselines under increasing obstacle density and commanded speed <ref:2608.00635#pg2>.

References

[1] X. Zhou, Z. Wang, H. Ye, C. Xu, and F. Gao, “EGO-Planner: An ESDF-free gradient-based local planner for quadrotors,” IEEE Robot Autom Lett. p. 7, vol 6 no 2 pp 478–485 2021

[3] Z. Wang, X. Zhou, C. Xu, and F. Gao, “Geometrically constrained trajectory optimization for multicopters,” IEEE Trans Robot. p. 7, vol 38 no 5 pp 3259–3278 2022

[4] Y. Ren et al., “Safety-assured high-speed navigation for MAVs,” Sci Robot. p. 7, vol 10 no 98 p eado6187 2025

[5] J. Lu et al., “You only plan once: A learning-based one-stage planner with guidance learning,” IEEE Robot Autom Lett. p. 7, vol 9 no 7 pp 6083–6090 2024

[7] A. Sridhar, D. p. 7 Shah, C. Glossop, and S. p. 7 Levine, “NoMaD: Goal masked diffusion policies for navigation and exploration,” arXiv:2310. p. 707896 2023

[8] S. Ye, Y. Ge, K. Zheng et al. p. 7, “World action models are zero-shot policies,” arXiv:2602. p. 715922 2026

[9] H. Bi, H. Tan, S. Xie, Z. Wang, S. Huang, H. Liu, R. Zhao et al. p. 7, “Motus: A unified latent action world model,” arXiv:2512. p. 713030 2025

[10] T. Yuan, Z. Dong, Y. Liu, and H. p. 7 Zhao, “Fast-WAM: Do world action models need test-time future imagination? p. 7” arXiv:2603. p. 716666 2026

[11] A. Ye, B. Wang, C. Ni et al., “GigaWorld-Policy: An efficient actioncentered world-action model,” arXiv:2603. p. 717240 2026

[13] A. Bhattacharya, N. p. 7 Rao, D. Parikh, P. Kunapuli et al. p. 7, “Vision transformers for end-to-end visionbased quadrotor obstacle avoidance,” in IEEE Int Conf Robot Autom Atlanta GA USA May 19-23 2025 pp 1–8

[14] E. Kaufmann, L. p. 7 Bauersfeld, A. p. 7 Loquercio et al. p. 7, “Champion-level drone racing using deep reinforcement learning,” Nature vol 620 no 7976 pp 982–987 2023

[15] X. Zhang, R. Wang, Y. Ren, D. Yu et al., “MAD: Mapping-aware world models for agile quadrotor flight,” arXiv:2606. p. 704534 2026

[18] D. Hafner, J. Pasukonis, J. p. 7 Ba, and T. Lillicrap, “Mastering diverse control tasks through world models,” Nature vol 640 no 8059 pp 647–653 2025

[19] A. Bar, G. Zhou, D. Tran, T. Darrell, and Y. p. 7 LeCun, “Navigation world models,” in Proc IEEE/CVF Conf Comput Vis Pattern Recogn Nashville TN USA Jun 11–15 2025 pp 15 791–801

[20] A. Romero, A. Shenai, I. Geles, E. Aljalbout, and D. p. 7 Scaramuzza, “Dream to fly: Model-based reinforcement learning for vision based drone flight,” arXiv:2501. p. 714377 2025

[21] F. Bao et al., “One transformer fits all distributions in multimodal diffusion at scale,” in Proc Int Conf Mach Learn vol 20 pp 1692–1717 Jul 23–29 2023

[25] C. Chi, Z. Xu, S. Feng et al., “Diffusion policy: Visuomotor policy learning via action diffusion,” Int J Robot Res vol 44 no 10–11 pp 1684–1704 2025

[26] Y. Lipman, R. T. Q. Chen, H. Ben-Hamu, M. p. 7 Nickel, and M. p. 7 Le, “Flow matching for generative modeling,” in Proc Int Conf Learn Represent Kigali Rwanda May 1–5 2023 pp 1692–1717

[27] Wan Team et al., “Wan: Open and advanced large-scale video generative models,” arXiv:2503. p. 720314 2025

[28] J. Su, M. Ahmed, Y. Lu, S. Pan et al., “RoFormer: Enhanced transformer with rotary position embedding,” Neurocomput vol 568 p 127063 2024

[29] X. Zhang, R. Wang, Y. Ren, J. Sun et al., “DiffAero: A GPU-accelerated differentiable simulation framework for efficient quadrotor policy learning,” arXiv:2509. p. 710247 2025

[30] J. Ho and T. Salimans, “Classifier-free diffusion guidance,” in Proc NeurIPS Wksp Deep Generative Models Downstream Appl Virtual event Dec 14 2021

[31] T. Qin, J. Pan, S. Cao, and S. p. 7 Shen, “A general optimization-based framework for local odometry estimation with multiple sensors,” arXiv:1901. p. 703638 2019

[32] G. Lu, W. Xu, and F. Zhang, “On-manifold model predictive control for trajectory tracking on robotic systems,” IEEE Trans Ind Electron vol 70 no 9 pp 9192–9202 2023

Table I Ablation Summary

The ablation study shows that enabling synchronous future-depth denoising improves both safety and traversal efficiency over the depth-frozen variant. This result indicates that future-depth latents encode informative anticipatory priors that effectively guide action generation. p. 6 The paper shows that FlowPilot maintains the highest overall success rate across commanded speeds from 3 m/s to 8 m/s compared to optimization- and learning-based baselines. p. 5 The total inference latency is under 18 ms on a Jetson Orin NX <ref:2608. p. 600635#pg5>. The real-world experiments demonstrate that FlowPilot transfers to arbitrary, unstructured obstacle distributions rather than memorizing the training geometry <ref:2608. p. 600635#pg2>. The model is limited to shorthorizon, goal-directed navigation from a single depth camera <ref:2608. p. 7 p. 7 p. 7 p. 7 p. 7 p. 700635#pg2>. The paper concludes that FlowPilot achieves higher closed-loop success rates than optimization- and learning-based baselines under increasing obstacle density and commanded speed <ref:2608.00635#pg2>. The model is limited to shorthorizon, goal-directed navigation from a single depth camera <ref:2608.00635#pg2>. The paper concludes that FlowPilot achieves higher closed-loop success rates than optimization- and learning-based baselines under increasing obstacle density and commanded speed <ref:2608.00635#pg2>. The model is limited to shorthorizon, goal-directed navigation from a single depth camera <ref:2608.00635#pg2>. The paper concludes that FlowPilot achieves higher closed-loop success rates than optimization- and learning-based baselines under increasing obstacle density and commanded speed <ref:2608.00635#pg2>. The model is limited to shorthorizon, goal-directed navigation from a single depth camera <ref:2608.00635#pg2>. The paper concludes that FlowPilot achieves higher closed-loop success rates than optimization- and learning-based baselines under increasing obstacle density and commanded speed <ref:2608.00635#pg2>. The model is limited to shorthorizon, goal-directed navigation from a single depth camera <ref:2608.00635#pg2>. The paper concludes that FlowPilot achieves higher closed-loop success rates than optimization- and learning-based baselines under increasing obstacle density and commanded speed <ref:2608.00635#pg2>. The model is limited to shorthorizon, goal-directed navigation from a single depth camera <ref:2608.00635#pg2>. The paper concludes that FlowPilot achieves higher closed-loop success rates than optimization- and learning-based baselines under increasing obstacle density and commanded speed <ref:2608.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. World Action Model for Real-Time Onboard Navigation: The improved system can jointly denoise future depth observations and executable trajectories with flow matching using a dual-stream mixture of transformers, allowing future-scene prediction and trajectory generation to inform each other.

  2. Controller Trackable Generative Action Space: The system predicts only the free control points of a degree-7 Bernstein polynomial, yielding C2 continuous references with analytic velocity, acceleration, and jerk that are directly trackable by a flight controller.

  3. Closed-Loop Onboard Validation: The model runs fully onboard on an embedded processor in under 18 ms perception to action budget and reaches 5.5 m/s in cluttered indoor and forest environments using only onboard sensing and computation.

  4. Action Representation for Smoothness: By predicting the free control points of a state-constrained Bernstein polynomial, the system avoids numerical differentiation of noisy samples, ensuring that every sampled trajectory has velocity, acceleration, and jerk in closed form.

  5. Two-Phase Training for Navigation Prior: The training is split into two phases where Phase 1 learns how the world evolves from depth observations alone to instill a broad prior over scene evolution, and Phase 2 aligns this prior with precise action generation using high-quality expert trajectories.

  6. Soft Conditioning for Replanning Consistency: Instead of warm-starting the denoising process from the last command, the previous trajectory is supplied as a soft conditioning signal to steer successive plans toward temporal consistency while preserving the multimodality of the flow-matching distribution.

Sources

Related papers