Talk2Escape: Conversational Grounding for Vision-and-Language Navigation
cs.RO, cs.HC
Submitted: 2026-09-23
Updated: 2026-09-23
Comments: IROS 2026
Project page: https://zeruili22.github.io/talk2escape
License: http://creativecommons.org/licenses/by/4.0/
The gist: While Vision-and-Language Navigation (VLN) has demonstrated remarkable success, the prevailing single-turn paradigm exposes a fundamental vulnerability: agents operate in a strictly open-loop manner.
Terminology
Abstract
While Vision-and-Language Navigation (VLN) has demonstrated remarkable success, the prevailing single-turn paradigm exposes a fundamental vulnerability: agents operate in a strictly open-loop manner. In practice, factors such as perceptual aliasing, sensor noise, and odometry drift can cause minor deviations to accumulate over time, often leading to catastrophic mission failures with no built-in mechanism for error recovery. To address this, we introduce Talk2Escape, a proactive and model-agnostic dialogue intervention framework that reframes navigation as a closed-loop interactive process. At its core, a lightweight vision-language module continuously monitors agent kinematics. Upon detecting localized looping or severe trajectory divergence, it translates raw egocentric observations into concise, grounded queries to solicit targeted corrective feedback from either an algorithmic oracle or a human-in-the-loop. Extensive evaluations in high-fidelity simulators, including R2R-CE, RxR-CE, and VLNVerse, demonstrate that Talk2Escape exhibits consistent improvements across diverse base agents. Empirically, Talk2Escape achieves a 66.0% Success Rate on R2R-CE, outperforming the current supervised and zero-shot state-of-the-art methods. We further validate its sim-to-real transfer on a Unitree Go2 quadruped, proving that proactive dialogue drastically improves navigation robustness in physical environments.
Sources
- VLNVerse: A Benchmark for Vision-Language Navigation with Versatile, Embodied, Realistic Simulation and Evaluation
- HA-VLN 2.0: An Open Benchmark and Leaderboard for Human-Aware Navigation in Discrete and Continuous Environments with Dynamic Multi-Human Interactions
- Nav-R1: Reasoning and Navigation in Embodied Scenes
- Hierarchical Cross-Modal Agent for Robotics Vision-and-Language Navigation
- SASRA: Semantically-aware Spatio-temporal Reasoning Agent for Vision-and-Language Navigation in Continuous Environments
- Language-Aligned Waypoint (LAW) Supervision for Vision-and-Language Navigation in Continuous Environments
- Fast-SmartWay: Panoramic-Free End-to-End Zero-Shot Vision-and-Language Navigation
- Boosting Zero-Shot VLN via Abstract Obstacle Map-Based Waypoint Prediction with TopoGraph-and-VisitInfo-Aware Prompting
- One Agent to Guide Them All: Empowering MLLMs for Vision-and-Language Navigation via Explicit World Representation
- Efficient-VLN: A Simple yet Strong Baseline for Efficient Vision-Language Navigation
- STRIDER: Navigation via Instruction-Aligned Structural Decision Space Optimization
- VLN-Zero: Rapid Exploration and Cache-Enabled Neurosymbolic Vision-Language Planning for Zero-Shot Transfer in Robot Navigation
Related papers
- FMT x: An Efficient and Asymptotically Optimal Extension of the Fast Marching Tree for Dynamic Replanning
- MPCFormer: A physics-informed data-driven approach for explainable socially-aware autonomous driving
- RoboLab: A High-Fidelity Simulation Benchmark for Analysis of Task Generalist Policies
- ARC: A Reasoning Recipe for Robot Foundation Models
- HRDexDB: A 4D Dexterous Grasping Dataset Across Human and Multiple Robot Embodiments
- APT: Action Expert Pretraining Improves Instruction Generalization of Vision-Language-Action Policies