From Semantic Grounding to Decision Optimization: A Unified Framework for Long-Horizon UAV Vision-Language Navigation

This paper proposes a unified semantic-to-decision framework for long-horizon UAV vision-language navigation (UAV-VLN). The authors identify three coupled issues in current approaches: weak grounding of instruction-relevant landmarks in visual observations, insufficient exploration of long-horizon history, and unstable decisions under local traps or repeated exploration. To address these, they propose a framework with three main components:

1. Instruction-grounded semantic enhancement module: This module injects object-level semantics and relative spatial cues into the current observation state. It uses an object detector to extract landmark proposals, back-projects them into 3D using depth and camera calibration, and computes relative spatial cues including bearing angle in the body frame, clipped normalized range, vertical offset, and box scale. Landmark relevance is modeled via cross-attention between the instruction query and landmark features, producing an instruction-grounded current-view feature.

2. Relevance-aware dynamic temporal aggregation (DTA): This strategy reweights the full history buffer while converting a few high-relevance frames into structured landmark prompts for the decoder. DTA computes relevance weights for all historical frames using the instruction query, aggregates them into a filtered temporal state, and additionally selects the top-2 high-relevance frames to convert into structured landmark prompts via a frozen grounding model (Grounding DINO), serialized with relative positional cues.

3. Topology-aware decision method: This combines local-optimum cognition (LOC) with group-relative policy optimization (GRPO) under progress, goal, semantic, and path-compliance rewards. LOC detects stagnation when no new nodes are explored within a recent window, and conditionally injects a frontier cue selected by balancing instruction compatibility and relative traversal cost. GRPO is used for policy refinement with group-normalized advantages and a clipped surrogate objective, anchored to a behavior-cloned reference policy via KL divergence.

The composite reward includes: progress reward (episode-normalized distance reduction), goal-completion bonus (terminal bonus for stopping within success radius), semantic matching reward (CLIP-based image-text consistency), and path compliance penalty (penalizing backtracking via estimated pose, altitude, and visual similarity checks, plus persistent deviation).

Experiments are conducted on AerialVLN-S and OpenFly benchmarks. On AerialVLN-S validation unseen split, the method achieves NE of 50.69, SR of 61.38, and OSR of 71.01, outperforming prior state-of-the-art Fly0 (NE 51.23, SR 60.07, OSR 69.22). On OpenFly test set, it achieves NE of 26.14, SR of 67.31, and OSR of 77.32, again outperforming Fly0 (NE 29.47, SR 64.67, OSR 72.81). Cumulative ablations show monotonic improvements from baseline (SR 35.85) through semantic enhancement (40.96), DTA (46.74), sparse grounding (51.32), LOC (56.47), and full GRPO (61.38). Additional ablations show SR of 46.74 without DTA, 50.90 without LOC, and 55.43 with simple history aggregation. The method also demonstrates robustness under Gaussian image noise and localization drift.

Improvements to AI systems:

1. Unified semantic-to-decision pipeline: Integrate object-level grounding, temporal aggregation, and topology-aware policy optimization into a single end-to-end trainable framework, replacing modular pipelines that lose information between perception and action.

2. Instruction-grounded semantic enhancement: Add a cross-attention module that injects object landmarks and relative spatial cues (bearing, range, vertical offset, box scale) directly into the observation state, improving the system's ability to focus on instruction-relevant objects rather than raw pixels.

3. Relevance-aware dynamic temporal aggregation (DTA): Implement a reweighting mechanism for full history buffers that selects high-relevance frames and converts them into structured landmark prompts (via a frozen grounding model) with positional cues, enabling better long-horizon memory without quadratic attention costs.

4. Topology-aware decision with local-optimum cognition (LOC): Add stagnation detection (no new nodes explored within a window) and conditional frontier injection based on instruction compatibility and traversal cost, preventing the system from getting stuck in local traps or repeated loops.

5. Group-relative policy optimization (GRPO) with KL anchoring: Use group-normalized advantages and a clipped surrogate objective, anchored to a behavior-cloned reference policy via KL divergence, for more stable and sample-efficient policy refinement compared to vanilla PPO.

6. Composite reward shaping: Combine progress (normalized distance reduction), goal-completion bonus, semantic matching (CLIP-based image-text consistency), and path-compliance penalties (backtracking, altitude, visual similarity, persistent deviation) to guide the agent toward both goal achievement and instruction adherence.

What the improved AI system can do:

• Navigate long-horizon, instruction-following tasks in 3D environments (e.g., UAV flight) with significantly higher success rate and lower navigation error, as demonstrated by outperforming prior state-of-the-art on AerialVLN-S (SR 61.38 vs 60.07) and OpenFly (SR 67.31 vs 64.67).

• Maintain robust performance under sensor noise (Gaussian image noise, localization drift) by leveraging semantic grounding and history aggregation rather than relying solely on raw observations.

• Recover from local optima and avoid repetitive exploration by detecting stagnation and injecting frontier cues, reducing backtracking and path deviation.

• Efficiently use long-horizon history by selectively attending to high-relevance frames, enabling scalable memory in tasks with hundreds of steps.

• Generalize to unseen environments (validation unseen splits) with monotonic performance gains from each added component, suggesting modular improvements can be applied to existing navigation systems.

UAV vision-language navigation (UAV-VLN) focuses on enabling an aerial agent to follow natural-language instructions in open 3D environments from egocentric visual observations. Current approaches suffer from three coupled issues: weak grounding of instruction-relevant landmarks in visual observations, insufficient exploitation of long-horizon history, and unstable decisions under local traps or repeated exploration. To address these issues, we propose a unified semantic-to-decision framework. First, we present an instruction-grounded semantic enhancement module that injects object-level semantics and relative spatial cues into the current observation state. Subsequently, we develop a relevance-aware dynamic temporal aggregation strategy that reweights the full history buffer while converting a few high-relevance frames into structured landmark prompts for the decoder. Finally, we devise a topology-aware decision method that combines local-optimum cognition with group-relative policy optimization under progress, goal, semantic, and path-compliance rewards. Experiments on the widely used AerialVLN and OpenFly benchmarks clearly demonstrate that our method achieves state-of-the-art performance.