RefineFly: Failure-Aware Post-Training for Aerial Vision-Language Navigation

RecoverFly: A Failure-Aware Reinforcement Learning Post-Training Framework for Aerial Vision-Language Navigation

Abstract

Unmanned aerial vehicle vision-language navigation (UAVVLN) requires agents to translate visual observations and language instructions into reliable flight actions in complex environments. Although recent end-to-end UAV vision-language-action (UAV-VLA) policies reduce reliance on separately designed perception, planning, and control modules, their behavior-cloning objectives provide limited corrective supervision for interactive closed-loop execution. Reinforcement learning (RL) offers a promising solution, while its effectiveness is constrained by inefficient use of samples, long-tailed scene distributions, and policy distribution shift during optimization. To this end, we propose RecoverFly, a failure-aware RL post-training framework for end-to-end UAV-VLA policies. Specifically, RecoverFly adapts token-level RL for stable optimization of grammar-constrained autoregressive UAV actions, revisits unresolved failure cases to strengthen corrective learning and sample utilization, and combines a two-stage long-tail scene curriculum with reference-policy regularization to improve scene adaptation while preserving acquired capabilities. Experiments on the TravelUAV benchmark demonstrate that RecoverFly achieves the best performance on the seen, unseen-map, and unseen-object splits. Moreover, compared to the AerialVLA initialization, RecoverFly improves success rate by 3.12 to 8.37 percentage points under a total rollout budget of about 30% of the training-set size, validating its effectiveness, robustness, and generalization capabilities.

Introduction

Vision-language navigation (VLN) requires embodied agents to associate natural-language instructions with visual observations and execute trajectories toward semantic goals. Unmanned aerial vehicle (UAV) VLN extends this problem to large-scale three-dimensional (3D) environments, where an aerial agent searches for a language-described object while continuously adjusting forward motion, altitude, yaw, and termination decisions. Recent vision-language-action (VLA) models provide a promising end-to-end interface from onboard observations and instructions to executable controls, reducing reliance on separately engineered perception, planning, and control modules. However, closed-loop UAV navigation remains difficult for two reasons. First, rapidly changing viewpoints, occlusion, and long trajectories cause local errors to compound into collisions, navigation drift, premature landing, timeouts, or target misses. Behavior-cloned policies are especially vulnerable because they are trained on expert demonstrations while they may need to act on their own error-induced states at test time. Second, aerial datasets are costly and highly imbalanced across scenes, causing standard sampling to emphasize frequent scenes while neglecting rare ones, thus weakening transfer to unseen maps and difficult routes. These problems are coupled since rare or difficult cases that most need correction are also least likely to be revisited during standard training.

Existing UAV-VLN systems mitigate the challenges of long-horizon planning through waypoint prediction, hierarchical planners, memory mechanisms, or external perception modules. Although end-to-end UAV VLA policies such as AerialVLA can generate flight actions directly, their optimization relies primarily on behavior cloning, which provides no specific corrective signal for failures encountered during closed-loop execution. While reinforcement learning (RL) can leverage interactive rewards to improve policy behavior in specific states, its direct application to autoregressive action policies introduces four issues. First, a failure event may result from previous actions rather than from the instantaneous actions. Second, unresolved failure cases with substantial learning value may rapidly disappear from subsequent sampling batches. Third, scene imbalance may concentrate policy optimization on frequently observed environments. Finally, unconstrained policy updates may degrade previously acquired navigation capabilities. As a result, a practical post-training method should address these issues jointly rather than treating RL as a generic second-stage optimizer.

To address the aforementioned challenges, we propose RecoverFly, a failure-aware RL post-training framework for end-to-end UAV-VLA policies. Specifically, RecoverFly adapts RL to grammar-constrained UAV action tokens, enabling stable closed-loop optimization of executable autoregressive actions. Furthermore, RecoverFly incorporates failure-aware replay to retain and revisit unresolved cases with substantial learning value, thereby improving sample utilization and strengthening corrective learning during closed-loop execution. Building on this, a two-stage curriculum progressively adjusts the training distribution toward long-tailed scenes, while reference-policy Kullback-Leibler (KL) regularization limits excessive policy deviation and preserves previously acquired capabilities. Consequently, RecoverFly integrates fine-grained action optimization, failure-oriented learning, long-tail scene adaptation, and stable policy updating into a unified post-training process, thereby improving navigation performance and generalization across diverse environments.

The main contributions of this paper are as follows:

• RL Framework for End-to-End UAV-VLA. We develop RecoverFly as an integrated closed-loop RL post-training framework. Specifically, RecoverFly preserves the pretrained textual action interface and adapts token-level RL to grammar-constrained UAV commands, enabling autoregressive action generation to be optimized through online navigation feedback.

• Failure-Aware Replay Learning. We propose a failure-aware replay mechanism that retains and revisits unresolved navigation cases with substantial learning value. This design improves the utilization of sparse failure feedback without applying policy updates to stale trajectories.

• Long-Tail Adaptation with Stable Policy Updates. We introduce a two-stage long-tail scene curriculum together with reference-policy KL regularization. Building on this design, RecoverFly strengthens learning from underrepresented scenes while limiting policy distribution shift and preserving previously acquired navigation capabilities.

Related Work

VLN for UAVs: Early VLN methods established cross-modal instruction grounding and action prediction. AerialVLN and AVDN extended this setting to outdoor aerial navigation and dialog-conditioned target search, while UAV-ON and OpenFly broadened it toward open-world object goals and larger-scale aerial datasets. To execute long-horizon flights, TravelUAV predicts continuous waypoints, whereas CityNavAgent, SkyVLN, TypeFly, and training-free VLM approaches combine language reasoning with memory, trajectory generation, model-based control, or program synthesis. Moreover, NavFoM and LongFly further improve generalization through large-scale pretraining and spatiotemporal modeling. These methods substantially improve semantic planning, while most retain explicit interfaces between high-level reasoning and low-level flight execution, through which prediction and control errors can accumulate.

VLA Policies for UAV-VLN: RT-2 and OpenVLA show that robotic actions can be represented in language-compatible output spaces, allowing pretrained vision-language representations to support embodied control. This paradigm has been extended to fine-grained imitation, racing, and cognitive UAV control. Furthermore, AerialVLA maps onboard observations and linguistic prompts directly to continuous 3-degree-of-freedom (DoF) controls and an intrinsic landing decision, reducing reliance on oracle waypoints, external detectors, and separate landing modules. However, these UAV-VLA policies are trained mainly by supervised behavior cloning. Thus, they learn strong expert-action priors while receiving limited corrective learning from collisions, moving-away behavior, stuck actions, early stops, and timeouts induced by their own closed-loop execution.

RL in UAV-VLN: RL has also been explored directly in language-conditioned UAV control. SuReAL combines supervised position prediction with RL-based continuous control and evaluates the resulting policy on a physical quadcopter. More recently, HTNav integrates staged imitation and RL with tiered decision making for urban aerial VLN. OpenVLN employs value-based waypoint rewards and KL-regularized policy updates, while FlightGPT combines supervised fine-tuning with group relative policy optimization (GRPO)-style optimization for goal accuracy, reasoning quality, and output compliance. While these studies establish the value of RL for aerial navigation, online post-training of autoregressive end-to-end UAV-VLA policies remains underexplored. Specifically, existing methods fail to comprehensively address sparse closed-loop feedback, adaptation to long-tail scenarios, and the mitigation of policy drift. RecoverFly establishes a unified framework based on the standard proximal policy optimization (PPO) algorithm to address these complex and coupled challenges.

Method

Overview: We propose RecoverFly, a failure-aware online RL post-training framework for end-to-end UAV-VLA policies in UAV-VLN. Building on the native token-level VLA optimization in RLinf, RecoverFly preserves the pretrained textual action interface and integrates dynamic failure replay, a two-stage long-tail scene curriculum, and stage-wise reference-policy KL regularization to strengthen corrective learning and rare-scene adaptation while constraining policy drift.

Textual UAV Policy and Online RL Objective: We consider end-to-end UAV-VLN control, where the agent receives a natural-language instruction x and a visual observation ot at time step t. We denote the resulting policy context by ht = (x, ot), and the policy directly generates an action-token sequence zt = (zt,1,..., zt,Kt), where K is the maximum sequence length and Kt ≤ K is the realized length. The autoregressive policy factorizes as πθ(zt ht) = ∏Kt k=1 πθ(zt,k ht, zt,<k). Following AerialVLA, the action grammar contains three numerical control tokens followed by an optional LAND token. The decoded 3-DoF control is at = ⟨∆xt, ∆zt, ∆ψt⟩, corresponding to forward progression, vertical movement, and yaw adjustment, respectively. Each control dimension is uniformly quantized into 99 bins over [0, 5], [−5, 5], and [−π, π], respectively. A deterministic decoder maps valid sequences to continuous controls. Moreover, we use mt,k ∈ 0, 1 to select valid action-token positions after padding to length K, and define Mt = max(1, ∑K k=1 mt,k). The same action grammar constrains rollout generation and policy optimization, and a sequence that still fails parsing is mapped to a safe no-op action and receives an invalid-action penalty.

Event-Aware Reward and Advantage Estimation: In UAV-VLN, success and failure events provide direct supervision, whereas they occur sparsely. Therefore, RecoverFly combines dense distance progress with event-specific reward, which is as follows: rt = clip(κp(Dt−1 − Dt), rprog min, rprog max) + ∑e∈E Re I[et = e], where Dt is the Euclidean distance to the target and E = success, collision, stuck, away, early-stop, timeout denotes the event set. In long-horizon UAV navigation, the success or failure of a trajectory may become observable only several steps after the key actions, making the attribution of delayed rewards to earlier decisions essential. Therefore, we apply PPO with generalized advantage estimation (GAE), which propagate delayed returns to preceding actions while constraining the magnitude of each policy update to support effective credit assignment and stable policy optimization. Moreover, we attach a value head Vϕ(ht) to the base VLA policy and optimize it using the standard clipped PPO value loss LV. Consequently, let dt ∈ 0, 1 indicate whether the transition terminates the episode. For a rollout ending at step T, we compute Ât = ∑T−t−1 l=0 (γλ)l δt+l, δt = rt + γ(1 − dt)Vϕ(ht+1) − Vϕ(ht), where γ is the discount factor and λ controls the bias–variance trade-off. For terminal transitions, dt = 1 removes the bootstrap value, and non-terminal rollout truncations bootstrap from the final value estimate.

Token-Level Policy Optimization Backbone: Standard continuous-action PPO is not directly applicable, as the VLA policy parameterizes token probabilities rather than an explicit density over decoded commands. A sequence-level alternative forms a joint ratio from autoregressive token probabilities, while this applies a single importance weight and clipping decision to the entire action sequence. Thus, RecoverFly applies the clipped surrogate separately to each valid action token and assigns all tokens encoding the same action the shared action-level advantage Ât. For the k-th action token, the importance ratio is ρt,k(θ) = πθ(zt,kht,zt,<k) / πold(zt,kht,zt,<k), where πold denotes the rollout policy used to collect the current batch. Defining ρ̄t,k(θ) = clip(ρt,k(θ), 1 − ϵ, 1 + ϵ), the token-level objective is Jtoken PPO(θ) = Et[(1/Mt) ∑K k=1 mt,k min(ρt,k(θ)Ât, ρ̄t,k(θ)Ât)], where ϵ is the PPO clipping threshold. This formulation enables token-wise updates in the native autoregressive action space while preserving the action-step learning signal and avoiding a shared sequence-level clipping decision.

Dynamic Failure Replay: Although token-level PPO enables optimization of autoregressive VLA actions, ordinary on-policy sampling can rapidly dilute difficult failures, leading to insufficient learning of high-value samples. Prior work improves sample efficiency by relabeling unsuccessful experience in hindsight or by replaying levels with high estimated learning potential. In contrast, RecoverFly introduces a dynamic failure replay mechanism, which stores unresolved task initializations rather than old trajectories and regenerates each rollout with the current policy, thereby retaining on-policy PPO while repeatedly learning informative failures. Specifically, RecoverFly maintains a dynamic failure pool M = ξi, where each entry is represented as ξi = (idi, ci, fi, σi, ni), where idi, ci, and fi denote the task identifier, scene, and failure type, respectively. The state σi ∈ active, solved, dropped records the active state, and ni counts unsuccessful replay attempts. At each online reset, RecoverFly samples fresh and failed task instances according to a replay ratio η. After each rollout, failed fresh instances are added to the pool or reactivated within it, successful replays are marked as solved, and entries that remain unsuccessful after Nmax replay attempts are marked as dropped. As a result, the dynamic replay pool evolves with policy updates and increasingly focuses the training effort on unresolved failure cases.

Two-Stage Long-Tail Scene Curriculum: Although failure replay improves the use of observed failures, it does not correct scene-level imbalance in sampling. To mitigate this issue, RecoverFly introduces a two-stage long-tail scene curriculum that adjusts scene-sampling weights across training stages to balance the experience obtained from different scenarios. Curriculum learning controls the examples or distributions presented over training to improve optimization and generalization. In RecoverFly, the curriculum acts on scene frequency rather than trajectory difficulty. Specifically, stage I samples scenes according to their empirical frequencies as follows: P1(c) = Nc / ∑c′∈C Nc′, where C denotes the set of training scenes and Nc denotes the number of training episodes associated with scene c. This stage expands the basic navigation capability under the original data distribution. Subsequently, Stage II continues from Stage I and replaces proportional scene sampling with an equal quota for each scene. Compared with the empirical distribution used in Stage I, this allocation increases rare-scene training frequency and prevents certain scenes from dominating online sampling. Moreover, the curriculum couples balanced scene exposure with failu…

Improvements to AI Systems:

1. Failure-Aware Replay Mechanism for Autonomous Agents

• Improvement: Implement a dynamic failure pool that stores unresolved task initializations (not old trajectories) and regenerates them with the current policy. This ensures on-policy learning while repeatedly exposing the agent to high-value failure cases.

• Capability: The AI system can continuously learn from its own mistakes in closed-loop environments, converting sparse failure feedback into reusable corrective experience without stale data. This improves sample efficiency and accelerates learning from rare, difficult scenarios.

2. Token-Level PPO for Autoregressive Action Policies

• Improvement: Apply clipped PPO separately to each valid action token (e.g., numerical control tokens, LAND token) with shared action-level advantage, rather than using sequence-level importance weighting. This preserves the native autoregressive structure of language-compatible action outputs.

• Capability: The system can stably optimize discrete, grammar-constrained action sequences (e.g., UAV commands) using RL, enabling fine-grained credit assignment to individual tokens and avoiding instability from joint sequence-level updates.

3. Two-Stage Long-Tail Scene Curriculum

• Improvement: Stage I samples scenes by empirical frequency to build baseline capabilities; Stage II switches to equal scene quotas to increase rare-scene exposure. This curriculum adjusts training distribution over time without requiring explicit difficulty labels.

• Capability: The AI system can generalize better to underrepresented environments and novel scenarios by balancing experience across long-tailed distributions, preventing overfitting to frequent cases while retaining previously learned skills.

4. Stage-Wise Reference-Policy KL Regularization

• Improvement: Anchor policy updates to a frozen reference policy (initial VLA policy in Stage I, final Stage I policy in Stage II) via token-level KL divergence. This constrains policy drift during RL optimization.

• Capability: The system can adapt to new tasks or distributions while preserving previously acquired capabilities, reducing catastrophic forgetting and maintaining stable performance across different stages of training or deployment.

5. Event-Aware Reward with GAE for Delayed Credit Assignment

• Improvement: Combine dense distance-based progress rewards with sparse event rewards (success, collision, stuck, away, early-stop, timeout) and use generalized advantage estimation (GAE) to propagate delayed returns to earlier actions.

• Capability: The AI system can effectively learn from long-horizon tasks where outcomes (success/failure) are only observable many steps after key decisions, improving credit assignment and enabling corrective learning for compound errors.

6. Grammar-Constrained Action Decoding with Invalid-Action Penalty

• Improvement: Enforce a strict action grammar during both rollout generation and optimization; map unparseable sequences to safe no-op actions and penalize them.

• Capability: The system can safely explore in real-world or simulated environments without producing invalid or dangerous actions, improving robustness and reliability in closed-loop control.

What the Improved AI System Can Do:

• Self-Correct in Real Time: Continuously learn from its own failures during deployment or training, revisiting unresolved cases to improve success rates in complex, long-horizon tasks (e.g., UAV navigation, robotic manipulation).

• Generalize to Rare Scenarios: Adapt to unseen environments or objects by balancing training across long-tailed distributions, without sacrificing performance on frequent cases.

• Stable Multi-Stage Learning: Undergo multiple training phases (e.g., pretraining, RL fine-tuning, domain adaptation) without catastrophic forgetting, preserving core competencies while acquiring new skills.

• Efficient Sample Utilization: Achieve significant performance gains (e.g., 3–8% success rate improvement) with only 30% of the training-set rollout budget, making RL post-training practical for costly embodied AI systems.

• Safe Exploration: Operate in constrained action spaces (e.g., discrete control commands) with guaranteed valid outputs, reducing risks of system failure or unsafe behavior during learning.

Unmanned aerial vehicle vision-language navigation (UAV-VLN) requires agents to translate visual observations and language instructions into reliable flight actions in complex environments. Although recent end-to-end UAV vision-language-action (UAV-VLA) policies reduce reliance on separately designed perception, planning, and control modules, their behavior-cloning objectives provide limited corrective supervision for interactive closed-loop execution. Reinforcement learning (RL) offers a promising solution, while its effectiveness is constrained by inefficient use of samples, long-tailed scene distributions, and policy distribution shift during optimization. To this end, we propose RecoverFly, a failure-aware RL post-training framework for end-to-end UAV-VLA policies. Specifically, RecoverFly adapts token-level RL for stable optimization of grammar-constrained autoregressive UAV actions, revisits unresolved failure cases to strengthen corrective learning and sample utilization, and combines a two-stage long-tail scene curriculum with reference-policy regularization to improve scene adaptation while preserving acquired capabilities. Experiments on the TravelUAV benchmark demonstrate that RecoverFly achieves the best performance on the seen, unseen-map, and unseen-object splits. Moreover, compared to the AerialVLA initialization, RecoverFly improves success rate by 3.12 to 8.37 percentage points under a total rollout budget of about 30% of the training-set size, validating its effectiveness, robustness, and generalization capabilities.