VLD: Visual Learned Distance as an Interface for Reinforcement Learning Navigation

arXiv:2512.07976 · cs.RO, cs.CV · Submitted 2025-12-08 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: We're looking at the paper: "VLD: Visual Learned Distance as an Interface for Reinforcement Learning Navigation".

Dev: Here's the gist of the paper. Training end-to-end policies from image data to directly predict navigation actions for robotic systems has proven inherently difficult The VLD framework decouples perception learning from policy learning by training a self-supervised distance-to-goal predictor on internet-scale video data and using it as a compact signal for reinforcement learning navigation policies This design allows the RL policy to be trained entirely in simulation using privileged geometric distance signals, with injected noise to mimic the uncertainty of the trained distance predictor The framework enables a scalable path toward reliable, multimodal navigation policies by inheriting semantic goal information from large-scale visual training while retaining robust low-level navigation behaviors learned in simulation.

Rosa: Pruning redundant reasoning tokens The core idea of the Vision–Language Distance (VLD) learning is to decouple perception learning from policy learning by training a self-supervised distance-to-goal predictor on internet-scale video data and using it as a compact signal for reinforcement learning navigation policies.

Dev: Instead of relying on raw sensory inputs during policy training, the framework first trains this predictor to generalize across both image- and text-based goals, providing a distance signal that can be minimized by an RL policy.

Rosa: This allows the RL policy to consume VLD predictions to navigate in simulated and real-world environments, inheriting semantic goal information from large-scale visual training while retaining robust low-level navigation behaviors learned in simulation.

Paper summary: Rosa: So we're looking at this paper called "VLD: Visual Learned Distance as an Interface for Reinforcement Learning Navigation". The main idea here is to tackle how hard it is to train these end-to-end navigation policies directly from images, especially because of that gap when you move from simulation to the real world.

Dev: Right. It's about decoupling perception learning from policy learning by training a distance-to-goal predictor on massive amounts of video data and using that as a signal for reinforcement learning policies. This approach lets the RL policy train entirely in simulation using these geometric distance signals, even with some noise to mimic how uncertain the predictor actually is.

Taro: I'm curious about what this means for when we actually put these systems on a physical robot. If we can get robust low-level navigation learned in simulation, but then replace those simulator signals with VLD predictions at deployment, that sounds like a big win for sim-to-real transfer.

Rosa: Exactly. The authors are using internet scale video data to train this distance predictor so it understands both image and text goals. This gives the RL policy something much richer than just raw sensor inputs during training, allowing it to inherit semantic goal information from that large visual training set while keeping its basic navigation skills solid from simulation.

Dev: They build this temporal distance function, which we call Td of observation time and goal specification, by doing a self-supervised regression problem. They create positive examples by just taking observation pairs along the same path so the temporal distance is simply the difference in time between them.

Taro: And they also use negative mining to make sure the model recognizes when goals are mismatched across different paths, assigning a maximum temporal distance to those outliers. How do they actually train this predictor to give us reliable distances?

Rosa: They use an inlier-outlier Gaussian mixture Negative Log-Likelihood loss. That's what gives them calibrated predictions and uncertainty estimates, which is important because it tells the policy how much it can trust the distance signal.

Dev: Before we get into how they train the policy, Rosa mentioned something about validating these distance predictors on their own merits, not just by seeing if the downstream navigation policy succeeds. They introduce a direct evaluation method based on ordinal consistency using Kendall's tau, which checks if predicted distances maintain the correct ordering of progress toward the goal.

Paper summary: Taro: So they are treating the distance estimator as a product unto itself? That seems like it could be really useful for debugging perception models later on.

Rosa: It is. They show that VLD outperforms prior temporal distance approaches like ViNT and VIP by giving us a principled way to judge the predictor in isolation, not just looking at how well the navigation policy does its job downstream.

Dev: And for deployment, they inject noise into the geometric distance signals with a lightweight MLP trained to mimic the error distribution of their learned predictor. That prepares the RL policy for when it sees those noisy signals in reality.

Taro: What kind of structured errors are we talking about? Does this noise model capture things like when the goal is just out of view or behind the agent?

Rosa: Yes, they design this geometric overlap noise to mimic those uncertainty patterns by conditioning it on features like projection success ratio and relative camera rotation. This means the error structure isn't just random; it has specific shapes that match what their VLD model learns under that mixture negative log-likelihood objective.

Dev: That structured error pattern is key because it induces behaviors in the policy, like performing periodic three hundred sixty degree scans at important locations, which is something you want for real-world navigation.

Taro: So the noise isn't just random static; it’s mimicking the actual failure modes of their perception model. That makes sense for making a robot behave reasonably when things get messy outside the lab.

Rosa: It also helps with robustness even when the goal is visually out of view, because VLD can still maintain ordinal consistency by correctly identifying the closer viewpoint based on ground truth geodesic distance, even without direct visual overlap.

Dev: That's a strong claim for long-horizon perception gaps. The multimodal version of their work tracks the image and language goals very closely, suggesting that linguistic cues can reinforce the visual distance estimation while keeping that ordering structure intact.

Taro: If text alone can still provide semantically grounded distances, even with weaker supervision than a full image goal, that opens up some possibilities for systems where visual input is unreliable.

Rosa: It shows they are building a scalable bridge between large scale perception models and reinforcement learning control by training the predictor on diverse datasets like synthetic environments, internet walking videos, and embodied datasets.

Paper summary: Dev: The deployment stage involves training policies conditioned on sensory information and that distance-to-goal signal, where the simulator signal gets swapped out for VLD predictions. They report a success rate of zero point seven three when swapping the distance signal because policies trained on noisy ground truth distances transfer effectively to the VLD at deployment without retraining.

Taro: So they are showing zero-shot replacement capability based on geometric overlap noise inducing robustness to those characteristic error patterns of VLD, which is pretty powerful for field testing a robot.

Rosa: And in real-world experiments on a physical robot, the VLD-driven policy achieved a success rate of ninety-three point three percent, which outperforms image-based navigation policies they tested. That's where the practical validation comes in for me and Dev.

Dev: The paper also explores how this works across different goal specifications, including image-only and text-only goals, showing that the text-only VLD model still achieves ordinal consistency above random performance even after those baselines are fine tuned.

Taro: That suggests VLD can form meaningful navigation distances from text alone despite having a weaker supervision signal than when you have an image goal.

Rosa: It confirms their hypothesis that decoupling perception from control helps improve sim-to-real robustness, and it gives us this scalable path toward reliable, multimodal navigation policies by combining robust and transferable reinforcement learning control with scalable semantic understanding.

Dev: The authors conclude that the VLD framework successfully bridges large scale perception models and embodied action by training a distance-to-goal predictor on large scale visual data and using it as a compact signal for reinforcement learning navigation policies.

Taro: What about the limitations? Where does this framework stop working?

Rosa: They flag that while the multimodal model closely tracks the image goal variant, they still have to carefully manage how those different goal specifications interact during deployment. That is one thing they point out as an area needing more work.

Dev: The real limitation is that the performance still depends heavily on the quality and diversity of the internet scale video data used for training their predictor; if that data doesn't cover a specific scene type, the generalization might not hold up as well as expected.

Taro: So it’s not a perfect solution out of the box; you still need good data to keep those robust performance metrics high.

Rosa: Exactly. But overall, they show that even with this interface, you get effective navigation behavior and a scalable bridge between large scale perception models and reinforcement learning control. That's what we saw in these results for the VLD paper.

Conclusion: Rosa: So we're wrapping up on VLD, which is this framework that uses learned distance predictions from images to guide reinforcement learning policies for navigation in robots.

Dev: Yeah, so basically, they decoupled how the robot learns to move from what it sees by training a separate predictor on massive video data and feeding that signal into the control loop.

Rosa: The authors call it VLD—Visual Learned Distance as an Interface for Reinforcement Learning Navigation. It’s really about creating this bridge between huge perception models and actual robot action.

Dev: It seems like the core idea is that you don't need the policy to understand every single pixel perfectly; you just need a reliable distance estimate, and they showed how you can train that distance estimator on diverse data—both images and text—to be really robust.

Rosa: And what’s interesting is how they tested this predictor separately using ordinal consistency checks, which means they were judging the quality of the distance signal before even putting it into a robot loop.

Dev: That's crucial for safety, because if the distance estimate is wrong in a weird way, you need to know why before you let it drive anything.

Rosa: The results show that when you take that signal and deploy it on a real robot, the policy transfers pretty well from simulation to reality without needing retraining.

Dev: That’s the sim-to-real transfer part we were talking about; they found a success rate of ninety three point three percent in physical tests, which is solid for this kind of interface.

Rosa: It proves that even if you have a minimal interface like this distance signal, you can get effective navigation behavior that handles things like goal being out of view reasonably well.

Dev: And it also shows that the system explores by doing things like spinning around periodically when it's uncertain about the direction, which is a good way to handle those real-world perceptual gaps.

Rosa: So, VLD seems to be building a scalable path toward navigation policies that have both strong low-level control and high-level semantic understanding from big data.

Dev: It’s definitely a step in making these systems more reliable across different real-world conditions, but we still need to see how this holds up when the visual data is completely different from what it was trained on.

Lazar Milikic, Manthan Patel, Jonas Frey

ETH Zurich · EPFL · Stanford University · UC Berkeley

cs.RO, cs.CV

Submitted: 2025-12-08

Updated: 2026-11-06

Journal ref: CoRL 2026 Workshop on Spatial Intelligence and Structured Reasoning for Embodied Navigation (SISREN)

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 86/100

The gist: Training end-to-end policies from image data to directly predict navigation actions for robotic systems has proven inherently difficult The VLD framework decouples perception learning from policy…

Key concepts

VLD Framework
This framework separates perception learning from policy learning. It first trains a predictor on massive video data to estimate the distance between an image and a specified goal. This distance prediction then serves as a compact signal for the reinforcement learning policy to learn how to navigate, allowing policies to be trained in simulation using these learned signals.
Temporal Distance Function (Td)
This function estimates the expected time or horizon between where the robot is currently observing and a specific goal. It is trained by sampling observation pairs along the same path for positive examples and using negative mining to identify mismatched goals, producing calibrated predictions with uncertainty estimates.
Geometric Overlap Noise (GeoNoise)
This noise model mimics the uncertainty patterns of the VLD predictor by conditioning perturbations on features like projection success ratio and camera rotation. It generates structured errors, such as lower confidence when a goal is out of view, which helps train the RL policy to behave correctly in challenging visual situations.

Terminology

Summary

Training end-to-end policies from image data to directly predict navigation actions for robotic systems has proven inherently difficult The VLD framework decouples perception learning from policy learning by training a self-supervised distance-to-goal predictor on internet-scale video data and using it as a compact signal for reinforcement learning navigation policies This design allows the RL policy to be trained entirely in simulation using privileged geometric distance signals, with injected noise to mimic the uncertainty of the trained distance predictor The framework enables a scalable path toward reliable, multimodal navigation policies by inheriting semantic goal information from large-scale visual training while retaining robust low-level navigation behaviors learned in simulation. p. 0

Where the paper says it

The RL policy can be trained entirely in simulation using privileged geometric distance signals, with injected noise to mimic the uncertainty of the trained distance predictor. At deployment, the policy consumes VLD predictions, inheriting semantic goal information—“where to go”—from large-scale visual training while retaining the robust low-level navigation behaviors learned in simulation.

Pruning redundant reasoning tokens

The core idea of the Vision–Language Distance (VLD) learning is to decouple perception learning from policy learning by training a self-supervised distance-to-goal predictor on internet-scale video data and using it as a compact signal for reinforcement learning navigation policies. Instead of relying on raw sensory inputs during policy training, the framework first trains this predictor to generalize across both image- and text-based goals, providing a distance signal that can be minimized by an RL policy. p. 0

Where the paper says it

Instead of relying on raw sensory inputs during policy training, we first train a self-supervised distance-togoal predictor on internet-scale video data. This predictor generalizes across both image- and text-based goals, providing a distance signal that can be minimized by a reinforcement learning (RL) policy.

This allows the RL policy to consume VLD predictions to navigate in simulated and real-world environments, inheriting semantic goal information from large-scale visual training while retaining robust low-level navigation behaviors learned in simulation. p. 0
Where the paper says it

The RL policy can be trained entirely in simulation using privileged geometric distance signals, with injected noise to mimic the uncertainty of the trained distance predictor. At deployment, the policy consumes VLD predictions, inheriting semantic goal information—“where to go”—from large-scale visual training while retaining the robust low-level navigation behaviors learned in simulation.

How the training signal is built

The temporal distance function, denoted as Td(ot, g), estimates the expected temporal horizon between a current observation and a goal specification. This function is trained using a self-supervised regression problem where positive examples are formed by sampling observation pairs along the same trajectory such that td = j − i is their temporal distance. p. 3

Where the paper says it

Training Procedure We frame distance prediction as a (self-)supervised regression problem. For each trajectory we sample observation pairs oi and oj such that i ≤ j and the td = j − i is their temporal distance, where td ≤ tdmax for a fixed tdmax, forming the dataset of triplets {(oi , oj , j −i)}i,j .

To encourage decisive recognition of mismatched goals, negative mining is employed, where observations from different trajectories are paired and assigned a maximum distance tdmax. The training objective utilizes an inlier–outlier Gaussian mixture Negative Log-Likelihood (NLL) loss to produce calibrated predictions and uncertainty estimates. p. 3
Where the paper says it

Then the training objective is the negative loglikelihood (NLL) over a batch of N samples: L = − 1 N X N n=1 log" cˆn 1 √ 2π σR exp − (tdn−tˆn) 2 2σ 2 R + (1 − cˆn) 1 √ 2π σO exp − (tdn−tˆn) 2 2σ 2 O # . (1) This inlier–outlier Gaussian mixture NLL formulation has two benefits: (i) it encourages calibrated uncertainty estimates that can be propagated to downstream policies, and (ii) it reduces the penalty for inherently ambiguous predictions, preventing the model from overfitting to noisy supervision.

Where the gains come from

The primary gains are demonstrated through ordinal consistency, which is assessed using Kendall’s τ, a rank correlation coefficient, to test whether predicted distances preserve the correct ordering of progress toward the goal. Experiments show that VLD outperforms prior temporal distance approaches like ViNT and VIP by providing a principled way to evaluate distance predictors in isolation rather than only through downstream policy success. p. 1

Where the paper says it

Together, these components form the foundation of our framework for scalable, multimodal, and transferable navigation policies. Our main contributions are: (i) a scalable framework for training vision–language distance (VLD) predictors on internet-scale real-world data, outperforming prior designs for temporal distance functions such as ViNT [41] and VIP [27]; (ii) a novel evaluation methodology for temporal distance functions, introducing ordinal consistency as the first principled way to evaluate distance predictors in isolation rather than only through downstream policy success; and (iii) a navigation framework that decouples RL policy training in simulation from perception learning on internet-scale data, combining robust and transferable RL control with scalable semantic understanding.

Furthermore, when deployed in simulation, the RL policy is exposed to noisy distance signals generated by a lightweight MLP trained to mimic the error distribution of the learned predictor, which prepares it for deployment. p. 1
Where the paper says it

On the control side, to reflect the inherent uncertainty of distance-function predictions, we expose the RL policy during simulation training to noisy distance signals generated by a lightweight MLP trained to mimic the error distribution of our learned predictor. This prepares the policy for the noisy predictions it will encounter at deployment.

Application and Transfer

The application stage involves training policies conditioned on sensory information and the distance-to-goal signal, where the distance signal is provided directly by the simulator during training but replaced by VLD predictions at deployment. The key result is that policies trained on noisy ground-truth distances transfer effectively to VLD at deployment, yielding a success rate of 0. p. 8

Where the paper says it

The key result is that policies trained on noisy groundtruth distances transfer effectively to VLD at deployment. Simply swapping the distance signal yields a success rate of 0.

73 when swapping the distance signal. p. 8
Where the paper says it

The key result is that policies trained on noisy groundtruth distances transfer effectively to VLD at deployment. Simply swapping the distance signal yields a success rate of 0.

This demonstrates that geometric overlap noise induces robustness to the characteristic error patterns of VLD, enabling strong zero-shot replacement without retraining the policy. p. 8
Where the paper says it

73, corresponding to only a ∼17% relative drop from the privileged baseline. This demonstrates that geometricoverlap noise induces robustness to the characteristic error patterns of VLD, enabling strong zero-shot replacement without retraining the policy.

The VLD-driven policy achieves a success rate of 93. p. 8
Where the paper says it

We deploy both policies directly on a TurtleBot4-Light platform [32] using the same discrete action space as in the simulation. In real-world experiments, the VLD-driven policy achieves a success rate of 93.

3% in real-world experiments on a physical robot, outperforming image-based navigation policies. p. 9
Where the paper says it

Policies trained in simulation with structured noise can operate effectively with learned distance predictors in deployment, enabling strong zero-shot replacement of simulator signals. Importantly, this design transfers effectively to real-world robotics: a policy driven by VLD predictions outperforms a strong image-based navigation policy when deployed on a physical robot.

The framework's robustness is further enhanced by using geometric overlap noise, which is designed to mimic the uncertainty patterns of the VLD model by conditioning perturbations on overlap features like projection success ratio (PSR) and relative camera rotation. This learned noise model produces structured error patterns, such as larger errors and lower confidence when the goal is out of view or behind the agent, which aligns with the behavior expected from the VLD model trained under a mixture NLL objective. p. 4

Where the paper says it

This produces smooth, nondiscrete samples while respecting the predicted uncertainty. Ultimately, the model outputs a noisy pair ( ˆd, cˆ) conditioned on overlap and relative pose, thereby reproducing structured error patterns of the VLD predictor: larger errors and lower confidence when the goal is out of view or behind the agent, and tighter, higher-confidence estimates when visual alignment is strong.

The GeoNoise model induces policy behaviors crucial for downstream navigation driven by a vision-language-based distance function, such as performing periodic 360◦ scans at key locations. p. 1
Where the paper says it

(A) Training stage: we separately train a temporal Vision–Language Distance (VLD) function on diverse real-world and synthetic video datasets, and an RL navigation policy in simulation using geometric distance-to-goal signals with injected noise to mimic real predictor uncertainty. (B) Deployment stage: the trained RL policy consumes predictions from the learned VLD model—specified by either image or text goals—to navigate in simulated and real-world environments.

The superior performance of VLD is evident in its ability to maintain ordinal consistency even when the goal is visually out of view, where it correctly identifies the closer viewpoint relative to the goal according to ground-truth geodesic distance. The multimodal version tracks the image-goal variant closely, indicating that linguistic cues can reinforce visual distance estimation while preserving ordinal structure. p. 7

Where the paper says it

The multimodal version closely tracks the image-only-goal predictions, demonstrating that linguistic cues can reinforce—but do not override—visual distance estimation. Ordinal consistency (Kendall’s τ ) on text-specified goals (HM3D).

This robustness to long-horizon perception gaps suggests that VLD provides a meaningful navigation signal even when no visual overlap is present. p. 7
Where the paper says it

In scenarios where the goal lies completely outside the field of view, confidence scores drop markedly and the predicted distances become noisier, reflecting true perceptual uncertainty rather than model failure. This motivates analyzing whether VLD still provides a meaningful navigation signal even when no visual overlap is present.

The final results show that the VLD-driven policy achieves competitive navigation performance even when deployed on a physical robot, proving the effectiveness of decoupling perception from control. p. 9
Where the paper says it

Importantly, this design transfers effectively to real-world robotics: a policy driven by VLD predictions outperforms a strong image-based navigation policy when deployed on a physical robot. While a scalar signal inevitably limits the information available to the policy, our results show that even this minimal interface can induce effective navigation behavior and provide a scalable bridge between large-scale perception models and reinforcement learning control.

The framework's success in real-world experiments is attributed to the fact that the control policy is trained primarily from geometric signals rather than raw visual observations, making it less sensitive to the visual domain shift between simulation and reality. The VLD-driven policy exhibits exploration behavior by periodically performing 360◦ rotations to evaluate how predicted visual-language distance changes across directions before committing to forward motion This exploration strategy proves beneficial in the real-world setting where perceptual cues may be subtle or appear from viewpoints different from the goal image. The policy's ability to detect partial visual cues from the goal, marked by a sharp increase in confidence and a corresponding drop in predicted distance, gives it a decisive signal to commit to the correct direction

The effectiveness of VLD over other distance predictors is confirmed by its highest success rate and path length among all swapped variants, as it provides the most informative and stable signal. The results show that even when the goal is not directly visible, VLD can guide the agent toward the goal over extended navigation, reducing uncertainty as new observations are acquired. p. 7

Where the paper says it

52 and τ = 0. These results show that VLD continues to provide a meaningful navigation signal even when the goal is visually out of view, demonstrating robustness to longhorizon perception gaps.

This capability to provide directional guidance through visual-language distance is what allows the policy to progressively move toward the target. p. 9
Where the paper says it

Both policies observe the target object (printer) in one of the views. The VLD-driven policy immediately moves toward the goal as distance predictions drop sharply, while FGPrompt-EF initially moves in the opposite direction, illustrating VLD’s sensitivity to goal-relevant visual cues.

The overall design provides a scalable bridge between large-scale perception models and reinforcement learning control. p. 9
Where the paper says it

Importantly, this design transfers effectively to real-world robotics: a policy driven by VLD predictions outperforms a strong image-based navigation policy when deployed on a physical robot. While a scalar signal inevitably limits the information available to the policy, our results show that even this minimal interface can induce effective navigation behavior and provide a scalable bridge between large-scale perception models and reinforcement learning control.

p. 9
Where the paper says it

Importantly, this design transfers effectively to real-world robotics: a policy driven by VLD predictions outperforms a strong image-based navigation policy when deployed on a physical robot. While a scalar signal inevitably limits the information available to the policy, our results show that even this minimal interface can induce effective navigation behavior and provide a scalable bridge between large-scale perception models and reinforcement learning control.

The framework's ability to handle different goal specifications is shown by its performance across image-only, text-only, and multimodal goals, where the multimodal model closely tracks the image-goal variant. The text-only VLD model still achieves ordinal consistency well above random and remains comparable to the image-based baselines even after those baselines are fine-tuned. p. 6

Where the paper says it

When using text-only goals, performance drops noticeably, which is expected since natural language descriptions are inherently less precise than images in specifying spatial targets, and our text labels are automatically generated and therefore coarse. However, the text-only VLD model still achieves ordinal consistency well above random and remains comparable to the image-based baselines even after those baselines are fine-tuned.

This suggests that VLD is able to form semantically grounded navigation distances from text alone, despite the weaker supervision signal. p. 6
Where the paper says it

However, the text-only VLD model still achieves ordinal consistency well above random and remains comparable to the image-based baselines even after those baselines are fine-tuned. This suggests that VLD is able to form semantically grounded navigation distances from text alone, despite the weaker supervision signal.

The confidence scores naturally decrease when distance predictions are unreliable, which aligns with the behavior expected from the VLD model trained under a mixture NLL objective. p. 3
Where the paper says it

where tˆ≥ 0 is the predicted temporal distance and cˆ ∈ [0, 1] is a confidence score. The confidence aligns naturally with our Gaussian mixture NLL loss (Section 3.

The GeoNoise model replicates the structural nature of VLD errors more faithfully than Ornstein–Uhlenbeck noise, leading to significantly better transfer performance. p. 4
Where the paper says it

Thus, by conditioning perturbations on overlap features and learning to map geometric distances into temporally realistic errors, we produce noise that more faithfully reflects the structure of prediction uncertainty encountered in practice. To this end, we design a learned noise model based on geometric overlap features: for each observation and goal pair, we extract a 13-dimensional feature vector: 1.

The distance accuracy evaluation metric shows that VLD is able to identify the closer viewpoint well above random even when there is no direct visual overlap with the goal. This indicates that the model learns a spatially coherent representation that provides meaningful directional guidance even when the goal is not visible, complementing long-horizon ordinal consistency results. The final comparison shows that the VLD-driven policy achieves a success rate of 0. p. 8

Where the paper says it

We deploy both policies directly on a TurtleBot4-Light platform [32] using the same discrete action space as in the simulation. In real-world experiments, the VLD-driven policy achieves a success rate of 93.

87 in real-world experiments, confirming its effectiveness as a navigation signal This result supports the hypothesis that decoupling perception from control can improve sim-to-real robustness. p. 9
Where the paper says it

These results provide evidence supporting our hypothesis that decoupling perception from control can improve sim-to-real robustness. Because the control policy is trained primarily from geometric signals rather than raw visual observations, it appears less sensitive to the visual domain shift between simulation and the real world.

The work provides a scalable path toward reliable, multimodal navigation policies by combining robust and transferable RL control with scalable semantic understanding. p. 1
Where the paper says it

Together, these components form the foundation of our framework for scalable, multimodal, and transferable navigation policies. Our main contributions are: (i) a scalable framework for training vision–language distance (VLD) predictors on internet-scale real-world data, outperforming prior designs for temporal distance functions such as ViNT [41] and VIP [27]; (ii) a novel evaluation methodology for temporal distance functions, introducing ordinal consistency as the first principled way to evaluate distance predictors in isolation rather than only through downstream policy success; and (iii) a navigation framework that decouples RL policy training in simulation from perception learning on internet-scale data, combining robust and transferable RL control with scalable semantic understanding.

p. 1
Where the paper says it

Together, these components form the foundation of our framework for scalable, multimodal, and transferable navigation policies. Our main contributions are: (i) a scalable framework for training vision–language distance (VLD) predictors on internet-scale real-world data, outperforming prior designs for temporal distance functions such as ViNT [41] and VIP [27]; (ii) a novel evaluation methodology for temporal distance functions, introducing ordinal consistency as the first principled way to evaluate distance predictors in isolation rather than only through downstream policy success; and (iii) a navigation framework that decouples RL policy training in simulation from perception learning on internet-scale data, combining robust and transferable RL control with scalable semantic understanding.

p. 1
Where the paper says it

Together, these components form the foundation of our framework for scalable, multimodal, and transferable navigation policies. Our main contributions are: (i) a scalable framework for training vision–language distance (VLD) predictors on internet-scale real-world data, outperforming prior designs for temporal distance functions such as ViNT [41] and VIP [27]; (ii) a novel evaluation methodology for temporal distance functions, introducing ordinal consistency as the first principled way to evaluate distance predictors in isolation rather than only through downstream policy success; and (iii) a navigation framework that decouples RL policy training in simulation from perception learning on internet-scale data, combining robust and transferable RL control with scalable semantic understanding.

p. 1
Where the paper says it

Together, these components form the foundation of our framework for scalable, multimodal, and transferable navigation policies. Our main contributions are: (i) a scalable framework for training vision–language distance (VLD) predictors on internet-scale real-world data, outperforming prior designs for temporal distance functions such as ViNT [41] and VIP [27]; (ii) a novel evaluation methodology for temporal distance functions, introducing ordinal consistency as the first principled way to evaluate distance predictors in isolation rather than only through downstream policy success; and (iii) a navigation framework that decouples RL policy training in simulation from perception learning on internet-scale data, combining robust and transferable RL control with scalable semantic understanding.

The paper concludes that the VLD framework successfully bridges large-scale perception models and embodied action by training a distance-to-goal predictor on large-scale visual data and using it as a compact signal for reinforcement learning navigation policies. The design allows the navigation policy to focus on robust local control while highlevel goal understanding is learned from diverse real-world data. p. 9

Where the paper says it

We introduce Vision–Language Distance (VLD), a framework that decouples perception learning from control by training a distanceto-goal predictor on large-scale visual data and using it as a compact signal for reinforcement learning navigation policies. Although VLD compresses rich visual observations into a single scalar distance, this abstraction allows the navigation policy to focus on robust local control while highlevel goal understanding is learned from diverse real-world data.

Policies trained in simulation with structured noise can operate effectively with learned distance predictors in deployment, enabling strong zero-shot replacement of simulator signals. p. 9
Where the paper says it

Although VLD compresses rich visual observations into a single scalar distance, this abstraction allows the navigation policy to focus on robust local control while highlevel goal understanding is learned from diverse real-world data. Policies trained in simulation with structured noise can operate effectively with learned distance predictors in deployment, enabling strong zero-shot replacement of simulator signals.

The work provides a scalable path toward reliable, multimodal navigation policies by combining robust and transferable RL control with scalable semantic understanding.

The study demonstrates that even a minimal interface can induce effective navigation behavior and provide a scalable bridge between large-scale perception models and reinforcement learning control. The final results show that even this minimal interface can induce effective navigation behavior and provide a scalable bridge between large-scale perception models and reinforcement learning control. p. 9

Where the paper says it

Importantly, this design transfers effectively to real-world robotics: a policy driven by VLD predictions outperforms a strong image-based navigation policy when deployed on a physical robot. While a scalar signal inevitably limits the information available to the policy, our results show that even this minimal interface can induce effective navigation behavior and provide a scalable bridge between large-scale perception models and reinforcement learning control.

p. 9
Where the paper says it

Importantly, this design transfers effectively to real-world robotics: a policy driven by VLD predictions outperforms a strong image-based navigation policy when deployed on a physical robot. While a scalar signal inevitably limits the information available to the policy, our results show that even this minimal interface can induce effective navigation behavior and provide a scalable bridge between large-scale perception models and reinforcement learning control.

The work provides a scalable path toward reliable, multimodal navigation policies by combining robust and transferable RL control with scalable semantic understanding. The paper demonstrates that even this minimal interface can induce effective navigation behavior and provide a scalable bridge between large-scale perception models and reinforcement learning control. p. 9
Where the paper says it

Importantly, this design transfers effectively to real-world robotics: a policy driven by VLD predictions outperforms a strong image-based navigation policy when deployed on a physical robot. While a scalar signal inevitably limits the information available to the policy, our results show that even this minimal interface can induce effective navigation behavior and provide a scalable bridge between large-scale perception models and reinforcement learning control.

The framework's success in real-world experiments is attributed to the fact that the control policy is trained primarily from geometric signals rather than raw visual observations, making it less sensitive to the visual domain shift between simulation and reality. The VLD-driven policy exhibits exploration behavior by periodically performing 360◦ rotations to evaluate how predicted visual-language distance changes across directions before committing to forward motion This exploration strategy proves beneficial in the real-world setting where perceptual cues may be subtle or appear from viewpoints different from the goal image. The policy's ability to detect partial visual cues from the goal, marked by a sharp increase in confidence and a corresponding drop in predicted distance, gives it a decisive signal to commit to the correct direction. The overall design provides a scalable bridge between large-scale perception models and reinforcement learning control.

The framework's success in real-world experiments is attributed to the fact that the control policy is trained primarily from geometric signals rather than raw visual observations, making it less sensitive to the visual domain shift between simulation and reality. The VLD-driven policy exhibits exploration behavior by periodically performing 360◦ rotations to evaluate how predicted visual-language distance changes across directions before committing to forward motion This exploration strategy proves beneficial in the real-world setting where perceptual cues may be subtle or appear from viewpoints different from the goal image. The policy's ability to detect partial visual cues from the goal, marked by a sharp increase in confidence and a corresponding drop in predicted distance, gives it a decisive signal to commit to the correct direction The overall design provides a scalable bridge between large-scale perception models and reinforcement learning control.

The framework's success in real-world experiments is attributed to the fact that the control policy is trained primarily from geometric signals rather than raw visual observations, making it less sensitive to the visual domain shift between simulation and reality. The VLD-driven policy exhibits exploration behavior by periodically performing 360◦ rotations to evaluate how predicted visual-language distance changes across directions before committing to forward motion This exploration strategy proves beneficial in the real-world setting where perceptual cues may be subtle or appear from viewpoints different from the goal image The policy's ability to detect partial visual cues from the goal, marked by a sharp increase in confidence and a corresponding drop in predicted distance, gives it a decisive signal to commit to the correct direction The overall design provides a scalable bridge between large-scale perception models and reinforcement learning control.

The study demonstrates that even a minimal interface can induce effective navigation behavior and provide a scalable bridge between large-scale perception models and reinforcement learning control. The final results show that even this minimal interface can induce effective navigation behavior and provide a scalable bridge between large-scale perception models and reinforcement learning control. The work provides a scalable path toward reliable, multimodal navigation policies by combining robust and transferable RL control with scalable semantic understanding. The paper demonstrates that even this minimal interface can induce effective navigation behavior and provide a scalable bridge between large-scale perception models and reinforcement learning control <ref:2512. p. 9

Where the paper says it

Importantly, this design transfers effectively to real-world robotics: a policy driven by VLD predictions outperforms a strong image-based navigation policy when deployed on a physical robot. While a scalar signal inevitably limits the information available to the policy, our results show that even this minimal interface can induce effective navigation behavior and provide a scalable bridge between large-scale perception models and reinforcement learning control.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Decoupling perception from policy allows training RL policies in simulation using privileged geometric distance signals while allowing deployment to consume Vision-Language Distance predictions, inheriting semantic goal information—'where to go'—from large-scale visual training.

  2. The framework enables the system to be trained on internet-scale video data for distance prediction, which generalizes across both image- and text-based goals, providing a distance signal that can be minimized by the RL policy.

  3. The use of ordinal consistency as an evaluation methodology allows researchers to assess distance functions directly by testing whether predicted distances rank pairs in the same order as ground-truth distances, rather than only through downstream policy success.

  4. The system can perform robust navigation even when no visual overlap exists with the goal, as shown by results where VLD continues to provide a meaningful navigation signal even when the goal is visually out of view.

  5. The RL policy training can be made more robust by using geometric overlap noise, which is conditioned on overlap features and learning to map geometric distances into temporally realistic errors, thus mimicking the uncertainty patterns of the trained distance predictor.

  6. The policy can actively seek informative viewpoints by utilizing VLD feedback, as demonstrated when it periodically performs 360◦ rotations to evaluate how the predicted visual-language distance changes across directions before committing to forward motion.

Abstract

Training end-to-end policies from image data to directly predict navigation actions for robotic systems has proven inherently difficult. Existing approaches often suffer from either the sim-to-real gap during policy transfer or limited action-labeled training data. To address this problem, we introduce Visual Learned Distance (VLD), a scalable framework for image-goal navigation that decouples perception learning from policy learning. We first train a self-supervised distance-to-goal predictor on about 3,000 hours of internet-scale and simulated video, providing a distance signal that a reinforcement learning (RL) policy can minimize. The RL policy is trained separately in simulation using privileged geometric distances perturbed by GeoNoise, a learned noise model that reproduces the viewpoint- and overlap-dependent errors of the trained distance predictor. At deployment, the policy consumes VLD predictions zero-shot, without retraining, inheriting ``where to go'' from large-scale visual training while retaining the robust navigation behaviors learned in simulation. This error modeling is key to the transfer: with VLD inputs, GeoNoise-trained policies reach 58.7% success on HM3D and 76.2% on Gibson, whereas policies trained with generic temporally correlated noise collapse to 4.9% and 11.5%. On a physical robot, the VLD-driven policy succeeds in 88% of trials, compared with 60% for a strong end-to-end image-goal baseline that outperforms it in simulation. Our decoupled design thus provides an alternative and scalable path toward reliable navigation policies that transfer from simulation to the real world.

Sources

Related papers