One Agent to Guide Them All: Empowering MLLMs for Vision-and-Language Navigation via Explicit World Representation

arXiv:2602.15400 · cs.RO · Submitted 2026-02-17 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: I'm Rosa.

Dev: And I'm Dev, here with Taro.

Rosa: We're looking at the paper: "One Agent to Guide Them All".

Dev: Here's the gist of the paper. A navigable agent needs to understand both highlevel semantic instructions and precise spatial perceptions<ref:2602.15400#pg2> Building navigation agents centered on Multimodal Large Language Models (MLLMs) demonstrates a promising solution due to their powerful generalization ability<ref:2602.15400#pg2> In this work, we propose a decoupled design that separates low-level spatial state estimation from highlevel semantic planning<ref:2602.15400#pg2> Our method establishes a new zero-shot state-of-the-art, achieving 48.8% Success Rate (SR) in R2R-CE and 42.2% in RxRCE benchmarks<ref:2602.15400#pg2> Framework Architecture The proposed GUIDE THEM ALL (GTA) framework is a decoupled zero-shot VLN framework that mitigates error propagation and enables task-specific reasoning, achieving state-of-the-art zero-shot performance<ref:2602.15400#pg2> GTA first reasons about the global spatial structure and then performs semantic reasoning over language in an isolated and sequential manner for navigation<ref:2602.15400#pg2> The architecture involves a Metric Mapping Module that constructs a hybrid world representation Wt = ⟨Mvol, Gtopo⟩ asynchronously<ref:2602.15400#pg2> This module continuously constructs a metric map Mvol from high-frequency RGB-D observations using a rotating camera to encode spatial information, while Gtopo is built on top of it to encode historical trajectories<ref:2602.15400#pg2> The system renders the composite spatial-logic state via an Interactive Reasoning Interface into a structured prompt for Counterfactual Reasoning Brain driven by the frozen MLLM<ref:2602.15400#pg2> World Representation Details The core innovation is the construction of an Explicit Metric World Representation, which provides MLLMs with metric global spatial context for decision making<ref:2602.15400#pg2> This representation consists of two components: Volumetric Mapping (Mvol): We maintain a dense 3D geometry of the environment using a Truncated Signed Distance Field (TSDF)<ref:2602.15400#pg2> The TSDF volume is continuously updated by integrating these asynchronous point clouds via a weighted running average<ref:2602.15400#pg2> This explicit metric world representation provides a collision-free manifold for low-level planning<ref:2602.15400#pg2> 3 Topological Graph Abstraction (Gtopo): To support long-horizon memory, we abstract navigable space into a graph Gtopo = (V, E)<ref:2602.15400#pg2> Each node vi = ⟨pi, ci⟩ stores its metric position pi ∈ R squared and a visit count ci<ref:2602.15400#pg2> Upon reaching a new pose pcurr, we perform a spatial clustering update<ref:2602.15400#pg2> This graph structure enables the system to detect loops and backtracking behavior explicitly<ref:2602.15400#pg2> 3 Reasoning Enhancement Strategies To better elicit the capabilities of MLLMs, we introduce a counterfactual reasoning-based strategy that directly operates on the metric world representation Wt<ref:2602.15400#pg2> This strategy involves: 1.

Rosa: Reasoning over Procedural Reasoning Blueprints (PB): The Blueprint is decomposed into a dynamic checklist (e.g., “

x: Navigate to door”, “[] Turn left”), requiring the MLLM to update this blueprint at every step<ref:2602.15400#pg2> Incorporating PB leads to improvements across all metrics, most notably a substantial surge in OSR, jumping from 48.3% to 56.7% (↑8.4%)<ref:2602.15400#pg2> 2.

Dev: Counterfactual Queries: The MLLM performs logical rollouts via counterfactual queries, ensuring each simulated trajectory aligns with the decomposed instruction semantics<ref:2602.15400#pg2> This allows the agent to evaluate multiple potential action trajectories and select the one that aligns with the task objectives, while the metric world representation ensures physical grounding<ref:2602.15400#pg2> 3 Performance and Generalization The results demonstrate that explicitly modeling the world metrically is key to unlocking the navigation potential of generalist MLLMs<ref:2602.15400#pg2> GTA establishes a new state-of-the-art on both R2R-CE and RxR-CE benchmark, surpassing the previous leading zero-shot agent BZS-VLN by a substantial margin<ref:2602.15400#pg2> Furthermore, GTA exhibits performance comparable to fully supervised baselines, challenging the assumption that indomain imitation learning is mandatory<ref:2602.15400#pg2> The versatility of GTA is further highlighted on the RxR-CE benchmark, where it sustains a high success rate of 42.2% (compared to 48.8% on R2R-CE)<ref:2602.15400#pg2> Real-world deployment verifies that this metric representation serves as a robust, domain-invariant interface for embodied Vision-and-Language navigation<ref:2602.15400#pg2> Sim-to-Real Transfer We demonstrate zero-shot sim-toreal transfer across diverse embodiments, including a wheeled TurtleBot 4 and a custom-built aerial drone<ref:2602.15400#pg2> This is attributed to our EWR: the MLLM never sees the raw, noisy real-world pixels directly; instead, it reasons over the synthesized top-down metric map and topological graph<ref:2602.15400#pg2> Since the structural representation of a map is domaininvariant (a wall is a wall in both Sim and Real), the MLLM planner remains effective despite the domain shift<ref:2602.15400#pg2> The performance on real-world robots shows 40% in Wheeled Robot and 42% in Drone transferability<ref:2602.15400#pg2> Ablation Insights The ablation studies confirm the necessity of the core design choices<ref:2602.15400#pg2> Impact of Explicit Metric World Representation.: Baselines enhanced with EWR exhibit consistent and substantial performance gains across both discrete and continuous environments<ref:2602.15400#pg2> The EWR empowers the agent to perform global topological reasoning by inferring potential room connections and unexplored frontiers based on the mapped structure<ref:2602.15400#pg2> Scalability with MLLM Reasoning Capabilities.: A strong positive correlation is observed between the general reasoning capability of the backbone and the agent’s navigation performance<ref:2602.15400#pg2> Switching to Gemini-2.5 Pro yields a notable improvement to 42.3% SR, demonstrating the benefit of stronger multimodal alignment<ref:2602.15400#pg2> Impact of Procedural Reasoning Blueprints.: Incorporating PB leads to improvements across all metrics<ref:2602.15400#pg2> The Blueprint acts as a critical long-horizon compass, ensuring that the generated path remains semantically consistent with the instruction’s logical flow<ref:2602.15400#pg2> In this work, we present GTA, a hierarchical framework that enables generalist MLLMs to achieve state-of-the-art zero-shot navigation in continuous environments<ref:2602.15400#pg2> Our key innovation is the construction of an Interactive Metric World Representation, allowing MLLMs to interact and reason with it, ensuring the physical validity of the produced navigation actions<ref:2602.15400#pg2> This enables successful zero-shot transfer to physical environments without additional training<ref:2602.15400#pg2> The bottleneck in Embodied AI often lies not in the reasoning model itself, but in how the physical world is represented<ref:2602.15400#pg2> As MLLMs continue to scale, frameworks like GTA will be essential in translating foundational intelligence into robust physical action<ref:2602.15400#pg2> The future work will focus on integrating openvocabulary 3D perception to further enrich this representation in dynamic scenarios<ref:2602.

Paper summary: Rosa: So we're looking at this paper called "One Agent to Guide Them All: Empowering MLLMs for Vision-and-Language Navigation via Explicit World Representation". Basically, the main idea is that for an agent to navigate well in the real world using a multimodal model, it needs to grasp both big instructions and precise spatial details.

Dev: Right. And they argue that just having a tightly coupled design limits how well these agents perform on navigation tasks. So their solution is a decoupled design separating the low-level spatial state estimation from the high-level semantic planning, which they call the GUIDE THEM ALL framework.

Taro: I'm interested in how they do that decoupling because when things go wrong in reality, I want to know what happens at each level of that separation.

Rosa: Well, this framework introduces something they call an interactive metric world representation. They claim this lets the MLLM actively interact with the world to figure out a consistent global spatial structure during navigation instead of just relying on pre-defined maps or passive memory buffers eight, nine, ten, eleven <ref:2602.15400#pg1>.

Dev: So, instead of a fixed map, the agent is building this representation itself by interacting with it through a real-time simulator. That means it can actively seek the information it needs based on what's happening right now during the task, which is different from just reading a historical summary of observations.

Taro: I wonder how that interaction works in practice when the world isn't perfectly predictable, like when you have unexpected obstacles or poor lighting.

Rosa: To handle that uncertainty and to make sure the agent doesn't get lost, they propose adding a counterfactual reasoning strategy on top of this metric world representation. This lets the MLLM reason about "what will happen if" scenarios by interacting with the world model at each step, which ensures the planned actions line up with what was actually asked for in the instruction.

Dev: So it's like they give the MLLM a detailed map and then let it play out different possibilities to pick the best path forward, grounding those possibilities in that metric world representation. That’s a lot of reasoning happening concurrently.

Taro: Does this actually make a difference when we test these agents outside of the lab setting? I'm curious about how robust this interactive part is when the environment misbehaves.

Paper summary: Rosa: They did conduct experiments on both simulated and real-world environments, and they showed that explicitly modeling the world metrically is crucial for unlocking navigation potential in generalist MLLMs. They report a zero-shot success rate of forty-eight point eight percent in R2R-CE and forty-two point two percent in RxRCE benchmarks with this framework, which they claim surpasses previous leading zero-shot agents like BZS-VLN significantly <ref:2602.15400#pg2>.

Dev: The results are interesting because they also show performance comparable to fully supervised baselines, which challenges the idea that you absolutely have to do indomain imitation learning for this kind of task. They've even shown zero-shot sim-to-real transfer across different embodiments, like a wheeled TurtleBot four and a custom drone <ref:2602.15400#pg1>.

Taro: I see. So the reason they think this works for sim to real is because the MLLM isn't seeing raw, noisy pixels directly; it's reasoning over that synthesized top-down metric map and topological graph structure <ref:2602.15400#pg2>.

Rosa: Exactly. Since the structural representation of a map is domain invariant—meaning a wall is still just a wall whether it's simulated or real—the MLLM planner stays effective even when there's a domain shift, which they measure at forty percent transferability for wheeled robots and forty-two percent for drones <ref:2602.15400#pg2>.

Dev: And they also showed that adding procedural reasoning blueprints to this setup helps improve all the metrics, specifically boosting the success rate from forty-eight point three percent up to fifty-six point seven percent, which is an eight point four percent increase <ref:2602.15400#pg2>.

Taro: That procedural blueprint thing sounds like a way to give the MLLM a better long-horizon compass, ensuring the path it generates actually follows the logical flow of the instruction, not just some random sequence of steps.

Rosa: It does that. The blueprint acts as this dynamic checklist where you have to update it at every step, which helps improve performance across all metrics <ref:2602.15400#pg2>. So what we're seeing here is a system that combines explicit spatial understanding with active planning and structured instructions.

Dev: And the limitation they mention is that this whole thing is framed as a Partially Observable Markov Decision Process, or POMDP, defined by the tuple of Environment, Task, State, Action, Observation, and Time <ref:2602.15400#pg3>. That means it's still dealing with a lot of uncertainty in the environment and the state space.

Taro: So even with this strong representation and reasoning strategy, the underlying challenge is still figuring out how to handle all that inherent uncertainty when you're actually driving around somewhere.

Paper summary: Rosa: True. They do flag that their approach relies on this interactive metric world representation to provide physical grounding for the actions it produces <ref:2602.15400#pg2>. So the core idea is building a system where the model can reason over a consistent global spatial structure while making sure every action is physically valid according to that map.

Dev: The implication for us in controls engineering is that if we can get an AI model to reason about this metric structure, it gives us a much richer interface than just raw sensor data or simple heuristic maps. It’s about translating high-level language directly into physical constraints the agent can respect.

Taro: For me, what's important is that when the world misbehaves—say, a door opens unexpectedly—this system doesn't just crash; it should be able to use that metric world and its counterfactual reasoning to figure out the most graceful recovery path.

Rosa: That’s where they hope this structure helps; by having this explicit map and the ability to reason about alternatives, the agent can infer potential room connections or unexplored areas based on the mapped structure <ref:2602.15400#pg2>. It's about inferring what *could* be there rather than just reacting to what *is* there right now.

Dev: So, to wrap up this paper, the work is about using an interactive metric world representation and counterfactual reasoning to make MLLMs better at vision-and-language navigation in continuous environments <ref:2602.15400#pg1>. It’s a framework that decouples planning from state estimation to achieve strong zero-shot performance with good sim-to-real transfer capabilities.

Taro: I think the real world impact is that we might stop needing massive amounts of specific training data just to get agents to navigate rooms or complex spaces reliably, provided you give them this kind of structured, metric understanding.

Rosa: That’s what they’re aiming for; making the underlying representation robust enough so that foundational intelligence in the MLLM can translate into reliable physical action without needing tons of specific examples for every single environment.

Dev: And the future work they mentioned focuses on integrating openvocabulary three-dimensional perception to further enrich this metric representation, especially in more dynamic scenarios where things are constantly changing.

Taro: So it's a continuous improvement cycle: better world representation leads to better reasoning, which leads to better navigation under harder conditions.

Rosa: That's the core of the paper, "One Agent to Guide Them All: Empowering MLLMs for Vision-and-Language Navigation via Explicit World Representation". It’s about moving past simple text maps toward an interactive world model that keeps things physically grounded.

Conclusion: Rosa: So we’re wrapping up this look at the paper "One Agent to Guide Them All: Empowering MLLMs for Vision-and-Language Navigation via Explicit World Representation". Essentially, this work focuses on giving those big multimodal models a better way to navigate physical spaces.

Dev: Right. The authors of this paper are looking at how we can make these agents handle complex navigation tasks without needing tons of specific training data.

Taro: They’re proposing a decoupled design where the low-level spatial guessing is separate from the high-level planning over language instructions.

Rosa: Exactly. They build this explicit metric world representation, which means they’re giving the model a structured map—both physical geometry and historical paths—to reason over during navigation.

Dev: It’s about creating this interactive interface where the MLLM can actually talk to the map as it moves, instead of just guessing based on what it saw in a single snapshot.

Taro: The results they show are quite strong, especially with zero-shot transfer from simulation to real robots like wheeled cars and drones.

Rosa: They’re claiming success rates around forty percent for those transfers, which is good because it means the foundational intelligence in the model isn't completely wasted when you move it to a physical world.

Dev: But they also found that adding procedural reasoning blueprints actually helped push those success rates up significantly, showing that structuring the instruction helps ground the action better.

Taro: So when things go wrong outside of a perfect simulation, this way of grounding the plan in a metric structure seems like it gives the agent better recovery options.

Rosa: That’s what they are showing us: by making the world representation explicit and interactive, we can get these generalist models to perform navigation tasks robustly without needing massive amounts of indomain training.

Dev: It shifts the problem away from just having a smarter reasoning model toward how we represent the physical reality itself for that model.

Taro: And since they mentioned future work, they’re looking at openvocabulary three-dimensional perception to make this world representation even richer in more dynamic situations.

Zerui Li, Hongpei Zheng, Fangguo Zhao, Aidan Chan, Jian Zhou, Sihao Lin, Shijie Li

Australian Institute for Machine Learning, Adelaide University · The University of Manchester · Zhejiang University · Agency for Science, Technology and Research (A*STAR)

cs.RO

Submitted: 2026-02-17

Updated: 2026-11-07

Comments: CoRL 2026

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 99/100

The gist: A navigable agent needs to understand both highlevel semantic instructions and precise spatial perceptions<ref:2602.15400#pg2> Building navigation agents centered on Multimodal Large Language Models…

Key concepts

Metric Mapping Module
This module asynchronously builds a hybrid world representation called Wt. It uses high-frequency RGB-D data from a rotating camera to create a dense metric map (Mvol) and builds a topological graph (Gtopo) on top of it, encoding historical trajectories for long-horizon memory.
Volumetric Mapping (Mvol)
This component maintains a dense 3D geometry of the environment using a Truncated Signed Distance Field (TSDF). This explicit metric world representation provides a collision-free manifold, which is essential for low-level planning and ensuring physical validity of navigation actions.
Topological Graph Abstraction (Gtopo)
This abstracts navigable space into a graph Gtopo = (V, E), where each node stores its metric position and visit count. This structure allows the system to detect loops and backtracking behavior explicitly, supporting long-horizon memory during navigation.

Terminology

Summary

A navigable agent needs to understand both highlevel semantic instructions and precise spatial perceptions<ref:2602. p. 0

Where the paper says it

One Agent to Guide Them All: Empowering MLLMs for Vision-and-Language Navigation via Explicit World Representation Zerui Li1† , Hongpei Zheng2† , Fangguo Zhao3 , Aidan Chan1 , Jian Zhou1 , Sihao Lin1 , Shijie Li4 , Qi Wu1* Abstract— A navigable agent needs to understand both highlevel semantic instructions and precise spatial perceptions. Building navigation agents centered on Multimodal Large Language Models (MLLMs) demonstrates a promising solution due to their powerful generalization ability.

15400#pg2> Building navigation agents centered on Multimodal Large Language Models (MLLMs) demonstrates a promising solution due to their powerful generalization ability<ref:2602. p. 0
Where the paper says it

One Agent to Guide Them All: Empowering MLLMs for Vision-and-Language Navigation via Explicit World Representation Zerui Li1† , Hongpei Zheng2† , Fangguo Zhao3 , Aidan Chan1 , Jian Zhou1 , Sihao Lin1 , Shijie Li4 , Qi Wu1* Abstract— A navigable agent needs to understand both highlevel semantic instructions and precise spatial perceptions. Building navigation agents centered on Multimodal Large Language Models (MLLMs) demonstrates a promising solution due to their powerful generalization ability.

15400#pg2> In this work, we propose a decoupled design that separates low-level spatial state estimation from highlevel semantic planning<ref:2602. p. 0
Where the paper says it

However, the current tightly coupled design dramatically limits system performance. In this work, we propose a decoupled design that separates low-level spatial state estimation from highlevel semantic planning.

15400#pg2> Our method establishes a new zero-shot state-of-the-art, achieving 48. p. 0
Where the paper says it

We conduct comprehensive experiments in both simulated and real-world environments. Our method establishes a new zero-shot state-of-the-art, achieving 48.

8% Success Rate (SR) in R2R-CE and 42. p. 0
Where the paper says it

Our method establishes a new zero-shot state-of-the-art, achieving 48. 8% Success Rate (SR) in R2R-CE and 42.

2% in RxRCE benchmarks<ref:2602. p. 0
Where the paper says it

8% Success Rate (SR) in R2R-CE and 42. 2% in RxRCE benchmarks.

15400#pg2>

Framework Architecture

The proposed GUIDE THEM ALL (GTA) framework is a decoupled zero-shot VLN framework that mitigates error propagation and enables task-specific reasoning, achieving state-of-the-art zero-shot performance<ref:2602.15400#pg2> GTA first reasons about the global spatial structure and then performs semantic reasoning over language in an isolated and sequential manner for navigation<ref:2602. p. 0

Where the paper says it

Specifically, as shown in Fig. 1, GTA first reasons about the global spatial structure and then performs semantic reasoning over language in an isolated and sequential manner for navigation.

15400#pg2> The architecture involves a Metric Mapping Module that constructs a hybrid world representation Wt = ⟨Mvol, Gtopo⟩ asynchronously<ref:2602. p. 3
Where the paper says it

An overview of navigation loop is shown in Algorithm 1. Interactive Metric World Representation To enable the agent to be aware of both spatial and historical information during navigation, we introduce the Metric Mapping Module, which constructs a hybrid world representation Wt = ⟨Mvol, Gtopo⟩ asynchronously.

15400#pg2> This module continuously constructs a metric map Mvol from high-frequency RGB-D observations using a rotating camera to encode spatial information, while Gtopo is built on top of it to encode historical trajectories<ref:2602. p. 3
Where the paper says it

Interactive Metric World Representation To enable the agent to be aware of both spatial and historical information during navigation, we introduce the Metric Mapping Module, which constructs a hybrid world representation Wt = ⟨Mvol, Gtopo⟩ asynchronously. It continuously constructs a metric map Mvol from high-frequency RGB-D observations using a rotating camera to encode spatial information, while Gtopo is built on top of it to encode historical trajectories.

15400#pg2> The system renders the composite spatial-logic state via an Interactive Reasoning Interface into a structured prompt for Counterfactual Reasoning Brain driven by the frozen MLLM<ref:2602. p. 3
Where the paper says it

We construct the Interactive Metric World Representation by unifying this geometric reconstruction with Procedur al Reasoning Blueprints, which comprise the logical “TODO List” and topological history. This composite spatial-logic state is rendered via our Interactive Reasoning Interface into a structured prompt for Counterfactual Reasoning Brain driven by the frozen MLLM (right).

15400#pg2>

World Representation Details

The core innovation is the construction of an Explicit Metric World Representation, which provides MLLMs with metric global spatial context for decision making<ref:2602.15400#pg2> This representation consists of two components:

** Volumetric Mapping (Mvol): We maintain a dense 3D geometry of the environment using a Truncated Signed Distance Field (TSDF)<ref:2602.15400#pg2> The TSDF volume is continuously updated by integrating these asynchronous point clouds via a weighted running average<ref:2602. p. 3

Where the paper says it

Given the camera intrinsic matrix K and the agent’s estimated pose Twb(s) ∈ SE(3) at capture time s, we project each pixel u = (u, v) with depth d into the world frame: pw = Twb(s) · Tbc(θs) · π −1 (u, d) (2) where Tbc(θs) represents the camera extrinsic calibration with rotation angle θs at capture time s. The TSDF volume is continuously updated by integrating these asynchronous point clouds via a weighted running average.

15400#pg2> This explicit metric world representation provides a collision-free manifold for low-level planning<ref:2602. p. 3
Where the paper says it

For a voxel v with centroid x, the signed distance St(v) is updated as: St(v) = Wt−1(v)St−1(v) + wt · sdft(x) Wt−1(v) + wt (3) where sdft(x) is the projective signed distance from the current observation. This explicit metric world representation provides a collision-free manifold for low-level planning.

15400#pg2> 3**

** Topological Graph Abstraction (Gtopo): To support long-horizon memory, we abstract navigable space into a graph Gtopo = (V, E)<ref:2602.15400#pg2> Each node vi = ⟨pi, ci⟩ stores its metric position pi ∈ R squared and a visit count ci<ref:2602. p. 3

Where the paper says it

: To support long-horizon memory, we abstract navigable space into a graph Gtopo = (V, E). Each node vi = ⟨pi , ci⟩ stores its metric position pi ∈ R 2 and a visit count ci .

15400#pg2> Upon reaching a new pose pcurr, we perform a spatial clustering update<ref:2602. p. 3
Where the paper says it

Each node vi = ⟨pi , ci⟩ stores its metric position pi ∈ R 2 and a visit count ci . Upon reaching a new pose pcurr, we perform a spatial clustering update: V ← ( Update(vnearest) if dmin < δmerge V ∪ {Node(pcurr)} otherwise (4)

15400#pg2> This graph structure enables the system to detect loops and backtracking behavior explicitly<ref:2602. p. 4
Where the paper says it

p∥2 represents the distance to nearest existing node vnearest, and we set δmerge = 0. This graph structure enables the system to detect loops and backtracking behavior explicitly.

15400#pg2> 3**

Reasoning Enhancement Strategies

To better elicit the capabilities of MLLMs, we introduce a counterfactual reasoning-based strategy that directly operates on the metric world representation Wt<ref:2602.15400#pg2> This strategy involves:

  1. Reasoning over Procedural Reasoning Blueprints (PB): The Blueprint is decomposed into a dynamic checklist (e. p. 4

    Where the paper says it

    a) Procedural Reasoning Blueprints. : Specifically, at each step, the interactive reasoning interface will provide MLLMs with: Tprompt = Itask ⊕ State(Gtopo) ⊕ History(Ht) ⊕ Iinstr (6) Detailed as follows: • Dynamic Task Plan (Itask): We decompose the navigation goal into a dynamic checklist (e.

    g., “[x] Navigate to door”, “[] Turn left”), requiring the MLLM to update this blueprint at every step<ref:2602. p. 4
    Where the paper says it

    , “[x] Navigate to door”, “[ ] Turn left”). The MLLM is required to update this blueprint at every step, ensuring the immediate action aligns with the high-level plan.

    15400#pg2> Incorporating PB leads to improvements across all metrics, most notably a substantial surge in OSR, jumping from 48. p. 7
    Where the paper says it

    III, incorporating PB leads to improvements across all metrics. Most notably, we observe a substantial surge in OSR, jumping from 48.

    3% to 56.7% (↑8.4%)<ref:2602. p. 0
    Where the paper says it

    As shown in Fig. 2, arXiv:2602.

    15400#pg2>

  2. Counterfactual Queries: The MLLM performs logical rollouts via counterfactual queries, ensuring each simulated trajectory aligns with the decomposed instruction semantics<ref:2602. p. 1

    Where the paper says it

    At each step, the MLLM evaluates candidate actions against a structured goal list. It performs logical rollouts via counterfactual queries, ensuring each simulated trajectory aligns with the decomposed instruction semantics.

    15400#pg2> This allows the agent to evaluate multiple potential action trajectories and select the one that aligns with the task objectives, while the metric world representation ensures physical grounding<ref:2602. p. 4
    Where the paper says it

    Counterfactual Reasoning Brain Finally, MLLMs output the navigation action by reasoning over the constructed metric world representation Wt through the interactive reasoning interface. With the proposed counterfactual reasoning strategy, the system is able to evaluate multiple potential action trajectories and select the one that aligns with the task objectives, while the metric world representation ensures physical grounding.

    15400#pg2> 3

Performance and Generalization

The results demonstrate that explicitly modeling the world metrically is key to unlocking the navigation potential of generalist MLLMs<ref:2602.15400#pg2> GTA establishes a new state-of-the-art on both R2R-CE and RxR-CE benchmark, surpassing the previous leading zero-shot agent BZS-VLN by a substantial margin<ref:2602. p. 5

Where the paper says it

: As evidenced in Tab. I, GTA establishes a new state-of-the-art on both R2R-CE and RxR-CE benchmark, surpassing the previous leading zero-shot agent BZS-VLN (#14) by a substantial margin.

15400#pg2> Furthermore, GTA exhibits performance comparable to fully supervised baselines, challenging the assumption that indomain imitation learning is mandatory<ref:2602. p. 5
Where the paper says it

Furthermore, it effectively narrows the gap with recent state-of-the-art supervised models like EfficientVLN [35]. This result challenges the assumption that indomain imitation learning is mandatory.

15400#pg2> The versatility of GTA is further highlighted on the RxR-CE benchmark, where it sustains a high success rate of 42. p. 5
Where the paper says it

In contrast, GTA demonstrates superior resilience. Despite the increased difficulty, it sustains a high success rate of 42.

2% (compared to 48. p. 5
Where the paper says it

Despite the increased difficulty, it sustains a high success rate of 42. 2% (compared to 48.

8% on R2R-CE)<ref:2602. p. 0
Where the paper says it

As shown in Fig. 2, arXiv:2602.

15400#pg2> Real-world deployment verifies that this metric representation serves as a robust, domain-invariant interface for embodied Vision-and-Language navigation<ref:2602. p. 0
Where the paper says it

Furthermore, to validate the versatility of our metric representation, we demonstrate zero-shot sim-toreal transfer across diverse embodiments, including a wheeled TurtleBot 4 and a custom-built aerial drone. These real-world deployments verify that our decoupled framework serves as a robust, domain-invariant interface for embodied Vision-andLanguage navigation.

15400#pg2>

Sim-to-Real Transfer

We demonstrate zero-shot sim-toreal transfer across diverse embodiments, including a wheeled TurtleBot 4 and a custom-built aerial drone<ref:2602.15400#pg2> This is attributed to our EWR: the MLLM never sees the raw, noisy real-world pixels directly; instead, it reasons over the synthesized top-down metric map and topological graph<ref:2602. p. 8

Where the paper says it

50 transferability with 40% in Wheeled Robot and 42% in Drone. This is attributed to our EWR: the MLLM never sees the raw, noisy real-world pixels directly; instead, it reasons over the synthesized top-down metric map and topological graph.

15400#pg2> Since the structural representation of a map is domaininvariant (a wall is a wall in both Sim and Real), the MLLM planner remains effective despite the domain shift<ref:2602. p. 8
Where the paper says it

This is attributed to our EWR: the MLLM never sees the raw, noisy real-world pixels directly; instead, it reasons over the synthesized top-down metric map and topological graph. Since the structural representation of a map is domaininvariant (a wall is a wall in both Sim and Real), the MLLM planner remains effective despite the domain shift.

15400#pg2> The performance on real-world robots shows 40% in Wheeled Robot and 42% in Drone transferability<ref:2602. p. 8
Where the paper says it

50 transferability with 40% in Wheeled Robot and 42% in Drone. This is attributed to our EWR: the MLLM never sees the raw, noisy real-world pixels directly; instead, it reasons over the synthesized top-down metric map and topological graph.

15400#pg2>

Ablation Insights

The ablation studies confirm the necessity of the core design choices<ref:2602.15400#pg2>

** Impact of Explicit Metric World Representation.: Baselines enhanced with EWR exhibit consistent and substantial performance gains across both discrete and continuous environments<ref:2602. p. 6

Where the paper says it

As presented in Tab. II, baselines enhanced with EWR exhibit consistent and substantial performance gains across both discrete and continuous environments.

15400#pg2> The EWR empowers the agent to perform global topological reasoning by inferring potential room connections and unexplored frontiers based on the mapped structure<ref:2602. p. 6
Where the paper says it

We attribute this robustness to the EWR. In the absence of detailed instructions, the EWR empowers the agent to perform global topological reasoning by inferring potential room connections and unexplored frontiers based on the mapped structure.

15400#pg2> **

** Scalability with MLLM Reasoning Capabilities.: A strong positive correlation is observed between the general reasoning capability of the backbone and the agent’s navigation performance<ref:2602. p. 6

Where the paper says it

As shown in Tab. IV, we observe a strong positive correlation between the general reasoning capability of the backbone and the agent’s navigation performance.

15400#pg2> Switching to Gemini-2. p. 7
Where the paper says it

8 establishes a solid baseline with 37. Switching to Gemini-2.

5 Pro yields a notable improvement to 42. p. 7
Where the paper says it

Switching to Gemini-2. 5 Pro yields a notable improvement to 42.

3% SR, demonstrating the benefit of stronger multimodal alignment<ref:2602. p. 7
Where the paper says it

5 Pro yields a notable improvement to 42. 3% SR, demonstrating the benefit of stronger multimodal alignment.

15400#pg2> **

** Impact of Procedural Reasoning Blueprints.: Incorporating PB leads to improvements across all metrics<ref:2602. p. 7

Where the paper says it

As shown in Tab. III, incorporating PB leads to improvements across all metrics.

15400#pg2> The Blueprint acts as a critical long-horizon compass, ensuring that the generated path remains semantically consistent with the instruction’s logical flow<ref:2602. p. 7
Where the paper says it

GTA ensures that the generated path remains semantically consistent with the instruction’s logical flow. This prevents early divergence and ensures the agent successfully navigates to the target area, validating that logical decomposition is essential for robust long-horizon execution.

15400#pg2> **

Conclusion

In this work, we present GTA, a hierarchical framework that enables generalist MLLMs to achieve state-of-the-art zero-shot navigation in continuous environments<ref:2602.15400#pg2> Our key innovation is the construction of an Interactive Metric World Representation, allowing MLLMs to interact and reason with it, ensuring the physical validity of the produced navigation actions<ref:2602. p. 8

Where the paper says it

CONCLUSION In this work, we present GTA, a hierarchical framework that enables generalist MLLMs to achieve state-of-the-art zero-shot navigation in continuous environments. Our key innovation is the construction of an Interactive Metric World Representation, allowing MLLMs to interact and reason with it, ensuring the physical validity of the produced navigation actions.

15400#pg2> This enables successful zero-shot transfer to physical environments without additional training<ref:2602. p. 8
Where the paper says it

Crucially, our real-world deployment demonstrates that this metric representation serves as a robust, domain-invariant interface, abstracting low-level visual discrepancies and bridging the gap between semantic reasoning and physical execution. This enables successful zero-shot transfer to physical environments without additional training.

15400#pg2> The bottleneck in Embodied AI often lies not in the reasoning model itself, but in how the physical world is represented<ref:2602. p. 8
Where the paper says it

This enables successful zero-shot transfer to physical environments without additional training. These findings suggest that the bottleneck in Embodied AI often lies not in the reasoning model itself, but in how the physical world is represented.

15400#pg2> As MLLMs continue to scale, frameworks like GTA will be essential in translating foundational intelligence into robust physical action<ref:2602. p. 8
Where the paper says it

These findings suggest that the bottleneck in Embodied AI often lies not in the reasoning model itself, but in how the physical world is represented. As MLLMs continue to scale, frameworks like GTA will be essential in translating foundational intelligence into robust physical action.

15400#pg2> The future work will focus on integrating openvocabulary 3D perception to further enrich this representation in dynamic scenarios<ref:2602. p. 8
Where the paper says it

As MLLMs continue to scale, frameworks like GTA will be essential in translating foundational intelligence into robust physical action. Our future work will focus on integrating openvocabulary 3D perception to further enrich this representation in dynamic scenarios.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Decoupled spatial reasoning from semantic planning allows for task-specific reasoning strategies for different modules, which is not feasible in previous tightly coupled systems. This separation effectively prevents error propagation from spatial reasoning to semantic decision making.

  2. The interactive metric world representation provides MLLMs with metric global spatial context for decision making, avoiding reliance on humandesigned priors or passive memory summaries. This allows the agent to actively seek the information required for decision making based on the current context, rather than relying on predefined summaries of historical observations.

  3. Counterfactual reasoning allows MLLMs to evaluate multiple potential action trajectories by reasoning about what will happen if . . . , through interaction with the metric world representation. This ensures that the selected actions are physically valid by grounding the decision in a physical simulation.

  4. The Metric Mapping Module constructs a hybrid world representation Wt = ⟨Mvol, Gtopo⟩ which continuously updates a dense 3D geometry via TSDF and builds a topological graph Gtopo to encode historical trajectories. This composite spatial-logic state is then rendered into a structured prompt for the MLLM, enabling it to directly infer the next metric waypoint (x, y, z) collision-free trajectory to reach the next state st+1.

  5. The Interactive Reasoning Interface selects orthogonal views based on minimizing both spatial and angular distances while maintaining temporal relevance. This asynchronous selection strategy ensures comprehensive spatial coverage while maintaining temporal relevance of the visual information for accurate reasoning.

  6. Counterfactual reasoning is enhanced by Procedural Reasoning Blueprints, which include a Dynamic Task Plan (Itask) and "Topological & Physical State (State(Gtopo))" that integrate safety mechanisms like vertical awareness and loop detection. This provides the MLLM with a structured prompt Tprompt = Itask ⊕ State(Gtopo) ⊕ History(Ht) ⊕ Iinstr, ensuring the generated path remains semantically consistent with the instruction’s logical flow.

  7. The framework achieves zero-shot sim-to-real transfer across diverse embodiments by having the MLLM reason over a synthesized top-down metric map and topological graph rather than raw, noisy real-world pixels. This is attributed to our EWR: Since the structural representation of a map is domaininvariant (a wall is a wall in both Sim and Real), the MLLM planner remains effective despite the domain shift.

Abstract

A navigation agent must understand high-level semantic instructions and reason about precise spatial structure. Multimodal Large Language Models (MLLMs), with their strong generalization, are a promising foundation for such agents. We propose GTA, a decoupled design that separates low-level spatial state estimation from high-level semantic planning. We introduce an explicit metric world representation that combines accumulated geometry with navigation history and exposes this information to the MLLM through a multimodal interface for decision-making. We further introduce counterfactual branch reasoning that checks anticipated outcomes against subsequent observations and revises navigation decisions when they disagree. The world representation grounds waypoint selection in reconstructed metric geometry. GTA achieves 48.8% Success Rate (SR) on the full R2R-CE val-unseen split and 46.2% on a 260-episode English RxR-CE val-unseen subset. Additional comparisons on shared R2R-CE episodes and mechanism ablations support the framework's effectiveness. We also demonstrate transfer to wheeled, quadrupedal, and aerial robots. Project page: https://zeruili22.github.io/GTA/

Sources

Related papers