Robust Vision-Language Navigation with Awareness of Instruction-Execution Progress
cs.CV
Submitted: 2026-10-09
Updated: 2026-10-09
License: http://creativecommons.org/licenses/by/4.0/
The gist: In Vision-Language Navigation (VLN) tasks involving multi-step instructions as guidance, an agent must associate its observations with the relevant instruction content at each decision point.
Abstract
In Vision-Language Navigation (VLN) tasks involving multi-step instructions as guidance, an agent must associate its observations with the relevant instruction content at each decision point. Most graph-based methods commonly match candidate trajectories with the entire instruction, allowing already executed content to interfere with subsequent decisions. Although instruction-execution progress provides a direct cue for separating executed and pending content, existing progress estimation approaches often rely on explicit progress supervision, sophisticated progress-learning stages, or costly online reasoning with LLMs. In this paper, we propose a robust VLN framework with Awareness of Instruction-Execution Progress (VLN-AIEP), which couples progress estimation with navigation decision-making. VLN-AIEP associates the visited trajectory with the instruction to suppress embeddings corresponding to executed content, and then matches candidate locations with the pending content. This coupling makes correct navigation decisions depend on accurate progress estimation, allowing the progress estimator to learn directly from navigation decision labels without explicit progress annotations. To support progress estimation, we construct a landmark-anchored trajectory representation that encodes the semantic labels and viewpoint-relative geometry of instruction-referred landmarks. These landmarks serve as spatial references for associating the visited trajectory with executed instruction content. Extensive experiments on R2R, RxR, CVDN, and R2R-CE demonstrate that the proposed method achieves favorable performance across multiple metrics.
Sources
- NaVid: Video-based VLM Plans the Next Step for Vision-and-Language Navigation
- MapGPT: Map-Guided Prompting with Adaptive Path Planning for Vision-and-Language Navigation
- BEVBert: Multimodal Map Pre-training for Language-guided Navigation
- Self-Monitoring Navigation Agent via Auxiliary Progress Estimation
- Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Grounding
- Learning to Navigate Unseen Environments: Back Translation with Environmental Dropout
- InstructNav: Zero-shot System for Generic Instruction Navigation in Unexplored Environment
- Fast-SmartWay: Panoramic-Free End-to-End Zero-Shot Vision-and-Language Navigation
- Constraint-Aware Zero-Shot Vision-Language Navigation in Continuous Environments
- YOLOE: Real-Time Seeing Anything
- Unsupervised Cross-lingual Representation Learning at Scale
- DINOv2: Learning Robust Visual Features without Supervision
- Decoupled Weight Decay Regularization
Related papers
- Loss Knows Best: Detecting Annotation Errors in Videos via Loss Trajectories
- AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memories
- Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift
- MambaX-Net: Dual-Input Mamba-Enhanced Cross-Attention Network for Longitudinal MRI Segmentation
- TeleOCR: Navigating Document Parsing Across Digital and Camera-Captured Documents
- A Survey on Efficient Vision-Language-Action Models