VEGA: Learning Navigation VLAs from In-the-Wild Egocentric Video with Geometric Trajectory Supervision

summary

The gist

VEGA introduces an approach for training navigation VisionLanguage-Action (VLA) models from unlabeled egocentric navigation videos by reconstructing local scene geometry and generating obstacle-aware…

In short

VEGA trains navigation VisionLanguage-Action (VLA) models using unlabeled egocentric videos by reconstructing local scene geometry. It extracts goal-conditioned trajectories from this geometry and uses them to supervise a flow-matching policy. This method allows the model to learn obstacle-aware planning directly from video data without needing manual robot demonstrations.

Key concepts

Flow-Matching VLA Policy
This is the type of AI model being trained. It learns how to predict navigation actions by matching the distribution of future actions in a video sequence to a target distribution. VEGA uses this framework, supervised by geometric data, to learn safe navigation behaviors.
Euclidean Signed Distance Field (ESDF)
The ESDF is a map created from the video frames that tells the system which areas are free space and which are occupied or unobserved. It defines distances to obstacles, allowing the model to understand immediate geometric constraints for planning.
Goal-Conditioned Waypoint Prediction
Instead of just aiming for a single point, VEGA predicts a sequence of short waypoints toward specific goals. These goals can be text descriptions or locations on an image. This allows the VLA model to learn how to navigate toward diverse destinations using multimodal inputs.
Model Predictive Path Integral (MPPI) Planner
This is the planning tool used during training to generate safe trajectories. It searches for a path that minimizes costs like collision risk, clearance from obstacles, and control effort while moving toward a specific waypoint goal.

Terminology used across episodes

This episode discusses

The paper

VEGA: Learning Navigation VLAs from In-the-Wild Egocentric Video with Geometric Trajectory Supervision · Read on arXiv

Gershom Seneviratne, Yohan Abeysinghe, Jianyu An, Vaibhav Shende, Rahul Kumar Dinesh Manocha

University of Maryland

We introduce VEGA, an approach for training navigation VisionLanguage-Action (VLA) models from unlabeled egocentric navigation videos. Internet-scale egocentric videos provide a scalable source of navigation-relevant visual observations, capturing cluttered scenes, close-range obstacles, and natural human motion through real-world spaces. However, these videos are not directly usable for policy learning because they do not provide obstacle-aware trajectories conditioned on explicit navigation goals in the robot's coordinate frame. VEGA addresses this gap by reconstructing local scene geometry from monocular video, sampling navigation goals (represented as text, image, or spatial waypoints) and generating obstacle-aware trajectories using the constructed geometry. The resulting trajectory distribution is then used to train a flow-matching VLA navigation policy. By using geometry exclusively during training, VEGA distills obstacle-aware planning directly into a vision-based policy. Furthermore, we introduce VEGA-Bench, a benchmark containing 250k scenes and approximately 5 million navigation goals paired with scene geometry, designed to evaluate goal progress, collision avoidance, and obstacle clearance of VLAs. Our evaluation shows that VEGA achieves competitive goal progress while reducing collisions by 33.0% and improving obstacle clearance by 17.9% over the strongest baseline on VEGABench, while improving success by at least 150.0%, reducing collisions by at least 66.7%, and improving obstacle clearance by at least 60.0% in real-world trials. Ultimately, we demonstrate that video-derived geometric supervision provides a scalable and effective signal for training obstacle-aware navigation VLAs. The code and benchmark will be released at the time of publication.

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: We're looking at the paper: "VEGA: Learning Navigation VLAs from In-the-Wild Egocentric Video with Geometric Trajectory Supervision".

Dev: Here's the gist of the paper. VEGA introduces an approach for training navigation VisionLanguage-Action (VLA) models from unlabeled egocentric navigation videos by reconstructing local scene geometry and generating obstacle-aware trajectories to supervise a flow-matching VLA policy Internet-scale egocentric videos provide a scalable source of navigation-relevant visual observations, capturing cluttered scenes, close-range obstacles, and natural human motion through real-world spaces VEGA addresses this gap by reconstructing local scene geometry from monocular video, sampling navigation goals (represented as text, image, or spatial waypoints) and generating obstacle-aware trajectories using the constructed geometry The resulting trajectory distribution is then used to train a flow-matching VLA navigation policy By using geometry exclusively during training, VEGA distills obstacleaware planning directly into a vision-based policy Pruning redundant reasoning tokens Existing robot navigation datasets provide useful demonstrations for social navigation, visual goal reaching, and cross-embodiment policy learning but they provide only sparse goalconditioned supervision Training a broadly goal-conditioned navigation policy therefore requires far denser supervision than simply recording one path through each environment; for each scene, the policy must observe how feasible trajectories vary across many possible goals, obstacle configurations, and start locations while preserving traversability and clearance This creates a need for methods that combine the semantic reasoning capabilities of VLAs with the geometric safety of classical planning To address this gap while avoiding the high cost of robot data collection, we turn to large-scale egocentric videos, such as walking tours, household recordings, and first-person footage These videos provide diverse layouts, viewpoints, obstacles, traversability cues, and natural motion patterns but they lack robot actions, explicit goals, and trajectories grounded in a robot’s action space The challenge is to recover goal-conditioned obstacle-aware supervision from such action-free video How the training signal is built The training signal for VEGA consists of goal-conditioned trajectories derived from reconstructed scene geometry, which supervise a flow-matching VLA that predicts navigation actions from RGB observations and a goal at inference time The process involves several key steps: 1.

Rosa: Geometry Recovery and Visibility-Aware ESDF Construction: For each frame It, VEGA reconstructs a dense metric point map Pt using MoGe-2 This point map is then used to construct a visibility mask (Mvis) and compute a Euclidean Signed Distance Field ESDF(x, y) over the local frame The ESDF defines known free space with positive values and occupied or unobserved regions with negative values 2.

Dev: Multimodal Goal Anchoring: VEGA uses Florence-2 to extract object labels and bounding boxes from each RGB frame It For each detected object j, a waypoint goal gwpj is obtained by querying the corresponding region in the MoGe-2 point map and projecting it onto the robot-centric ground plane Auxiliary waypoint goals are also sampled from observed free-space subset Xfree to provide supervision for arbitrary traversable locations 3.

Rosa: MPPI-Based Trajectory Generation: For each waypoint goal gwpj, VEGA uses a Model Predictive Path Integral (MPPI) planner to generate a target waypoint trajectory yˆt:t+H,j Candidate trajectories are scored using ESDF-based goal-reaching, collision, clearance, control-effort, and smoothness costs The lowest-cost collision-free rollout is used as the reference trajectory for supervision 4.

Title and authors: Rosa: So we're looking at this paper called "VEGA: Learning Navigation VLAs from In-the-Wild Egocentric Video with Geometric Trajectory Supervision." It’s by Gershom Seneviratne Yohan Abeysinghe Jianyu An Vaibhav Shende Rahul Kumar Dinesh Manocha. Basically, they’re tackling the problem of training navigation VisionLanguageAction models using videos that don't actually have robot actions or clear goals.

Dev: Right, so the core idea here is to take those huge collections of egocentric videos—like walking tours or household footage—and turn them into something useful for learning how to navigate in the real world.

Rosa: Exactly. The paper points out that these videos give us a massive source of visual observations, showing everything from cluttered scenes to natural human motion, but they lack trajectories that are tied to explicit navigation goals in the robot's frame of reference.

Taro: So the naive question is, how do you get obstacle-aware trajectories from just raw video where nothing is explicitly labeled as a goal?

Dev: That’s what VEGA tries to solve by reconstructing the local scene geometry from those monocular videos. They use a tool called MoGe-two to build a dense metric point map <ref:2606.18426#pg1>.

Rosa: And that map lets them build something crucial: visibility-aware Euclidean Signed Distance Fields, or ESDFs. This ESDF defines where the robot can safely go—positive values mean free space, negative values mean an obstacle is there.

Taro: But how do they turn that static geometry into a dynamic goal-conditioned signal that a navigation policy can actually learn from?

Dev: They anchor goals using Florence-two which extracts labels and bounding boxes from the RGB frames <ref:2606.18426#pg1>. For each detected object, they query the MoGe-two map to find a waypoint goal near it <ref:2606.18426#pg1>.

Rosa: And they don't stop there; they also sample auxiliary waypoint goals right from the observed free space within that video subset to give supervision for just any traversable location.

Taro: So, once you have these goals, how do you generate the actual training signal that tells the AI what to do?

Dev: They use a Model Predictive Path Integral planner, or MPPI. For each waypoint goal they identified, they run an MPPI planner to generate a target waypoint trajectory over a horizon of H plus J steps.

Title and authors: Rosa: And these candidate trajectories get scored against several costs—things like collision risk, clearance around obstacles, control effort smoothness—and the lowest-cost, collision-free rollout becomes the reference trajectory for training.

Taro: So what’s the big payoff here? What makes VEGA different from just using a standard VLA model with some robot data?

Dev: The improvement is that they use geometry *exclusively* during training to distill obstacle-aware planning directly into a vision-based policy. It bypasses the need for dense, manual demonstrations or expensive robot data collection.

Rosa: That’s huge because it means we can use these massive internet-scale videos as a scalable source of supervision that captures real-world clutter and motion patterns without needing someone to manually film every possible path.

Taro: But the paper points out some limitations, right? What does VEGA not do when the world misbehaves?

Dev: They flagged two main things. First, because the policy is memoryless, it only predicts actions based on what it sees right now and the goal condition; if a goal or an obstacle moves outside the camera view, it can become brittle.

Rosa: And second, their local ESDF supervision doesn't model dynamic obstacle motion explicitly. It plans around where obstacles are in that moment but doesn't predict their future movement over the planning horizon.

Taro: So for someone just listening to the show, what does this mean for real-world robots? How robust is this geometric supervision when things aren't perfectly static?

Dev: They say they showed improvement on VEGA-Bench where it achieved comparable goal progress with lower collision rates and larger obstacle clearances compared to previous baselines. In real world trials, they reported improving goal reaching by at least one hundred fifty percent and reducing collisions by at least sixty-six point seven percent.

Rosa: That improvement in clearance, the paper claims it’s at least sixty percent better than the strongest baseline they tested against in actual robot trials. This suggests that video-derived geometric supervision is a scalable way to give obstacle-aware navigation VLAs a safety net when they are deployed.

Title and authors: Taro: The deployment architecture uses ROS two to decouple the policy inference from the low-level control, so the robot can keep tracking its current path while waiting for the next prediction chunk <ref:2606.18426#pg1>. That’s practical for real systems.

Dev: And for goal conditioning, they made it flexible by introducing a trainable waypoint encoder that maps text goals, image region goals from Florence-two and metric coordinate goals all into a shared prefix interface for the VLA model <ref:2606.18426#pg1>.

Rosa: So you can condition the policy using text instructions or just pointing at an area in the video to tell it where to go next. That makes it much more versatile than before.

Taro: It does make sense that they need that flexibility, but I'm still wondering about those limitations regarding dynamic motion. If a person suddenly walks into the path of the planned trajectory, how does this geometric approach handle that surprise?

Dev: That’s where the authors admit their current method doesn't explicitly model future obstacle motions. They rely on replanning when newly observed obstacles appear, but they don't predict those future motions during the initial planning step.

Rosa: So for now, it seems best suited for scenarios where the environment is relatively static or where obstacles move slowly enough that a short-term geometric plan is sufficient. It’s a solid signal for getting to a goal safely when you have good video input.

Taro: I think the key thing they are proving is that we can get robust, obstacle-aware planning supervision just from geometry extracted from common videos instead of needing every single robot to drive through every possible scenario manually.

Dev: Precisely. The full paper, "VEGA: Learning Navigation VLAs from In-the-Wild Egocentric Video with Geometric Trajectory Supervision," shows how you can use this geometry to train a flow-matching VLA policy, and it’s a significant step in making navigation models more grounded in physical safety.

Rosa: It certainly moves the field forward by showing how we can scale supervision from expensive robot runs to readily available video data, provided we handle those geometric modeling gaps carefully.

Taro: For next time, I'm looking forward to seeing how future work tackles that dynamic motion problem they identified.

The paper's summary: Rosa: So, VEGA is taking these massive collections of everyday videos—walking tours, household recordings—and using them to train navigation models without ever needing a robot to actually drive or record a perfect path.

Dev: Right, it’s distilling obstacle-aware navigation behavior straight out of in-the-wild video by reconstructing the local scene geometry and then generating trajectories to supervise a flow matching model.

Rosa: The key thing is how they build that training signal. They use tools to reconstruct a dense point map from the video, which lets them create what they call an ESDF, or Euclidean Signed Distance Field.

Dev: That field tells you exactly where the robot can go—positive numbers for clear space, negative numbers for obstacles—and that geometry is used to anchor goals. They use object detectors to find items in the video and map those locations onto that geometric field to get waypoint goals.

Rosa: And then they take those waypoints and use a model predictive planner, an MPPI planner, to generate a safe path between each goal, scoring the candidate paths based on things like how close they get to obstacles or how smooth the movement is.

Dev: The final output for training is these goal-conditioned trajectories derived from that geometry, which supervise the flow matching VLA model when it’s making decisions in real time.

Rosa: What this means for us on the ground is that we don't need millions of hours of expensive robot driving data to teach an AI how to navigate around clutter or handle close-range objects.

Dev: It shifts the burden from manual data collection onto using existing internet-scale videos as a scalable source for training, which is a huge deal for making navigation models practical outside of controlled labs.

Rosa: The results they showed are pretty strong, improving goal progress and reducing collisions in real-world trials by percentages like sixty percent clearance improvement.

Dev: But there are two big caveats the authors mention. First, the policy is memoryless; it only looks at what’s happening right now, so if a goal or an obstacle suddenly appears outside the camera view, it can get brittle fast.

Rosa: And second, their geometric supervision doesn't actually model how dynamic obstacles move over time; it plans based on where they are in that single snapshot.

Dev: So for deployment, you’ve got a system that uses these geometric cues to make safe predictions, but the authors flag that future work needs to add trajectory prediction or time indexed distance fields to handle those moving things better.

Rosa: That points toward the next challenge: how do we make this geometrically supervised approach robust when we aren't dealing with perfectly static environments?

The paper's improvements: Rosa: So, we’re looking at how VEGA suggests improving these navigation VLAs by focusing on their training structure and deployment flexibility.

Dev: The paper points out that they've adapted the pi zero point five VLA architecture by adding specific trainable components for goal-conditioned waypoint prediction.

Rosa: They handle text goals using the language backbone, image region goals are encoded with a frozen vision encoder called SigLIP, and metric coordinate goals are conditioned through a new trainable waypoint encoder.

Dev: That’s clever because it lets them condition the entire model through a shared prefix interface, so they can use text instructions, visual regions from an image, or specific coordinates all at once.

Rosa: This makes the system much more versatile for different kinds of navigation tasks than just relying on one type of goal.

Dev: And on the deployment side, they use a ROS two based execution architecture that separates policy inference from low level control, so the robot can track its current trajectory while a new prediction is being computed in parallel.

Rosa: That decoupling is pretty important for real-world systems because it lets you manage latency better when you’re trying to keep up with the loop rate.

Dev: It’s about ensuring that while the AI is calculating the next waypoint, the physical robot isn't stalled waiting for a response.

Rosa: But we have to talk about those limitations they identified, which are pretty practical for field testing.

Dev: They noted that because this policy is memoryless, it’s brittle if something outside the camera view—a goal or an obstacle—moves into the frame and it can cause issues with prediction.

Rosa: Plus, their local ESDF supervision doesn't explicitly model how dynamic obstacles move over time; they rely on replanning when new obstacles show up rather than predicting their future motions during the initial planning step.

Dev: So for anyone looking at putting this on a physical robot right now, you’re looking at a system that needs careful tuning to handle unexpected things that aren't perfectly static.

Rosa: The authors suggest that future work should definitely include trajectory prediction or time indexed distance fields to account for those moving obstacles more effectively over the planning horizon.

Dev: That would move the system from reacting to what's in front of it to actually anticipating where things are going, which is a major step for safety.

Rosa: It sounds like the next big hurdle is getting that temporal awareness into the geometric supervision pipeline.

Conclusion: Rosa: So to wrap up, VEGA is using geometry derived from action-free egocentric video to train navigation VLAs by generating obstacle aware trajectories that supervise a flow matching policy.

Dev: It’s essentially using the physical structure of the scene, like an ESDF, to create a dense set of training examples for the AI.

Rosa: The implication here is that we can move away from needing massive amounts of manual robot driving data just to teach an AI how to navigate safely in complex, real-world environments.

Dev: It scales supervision from expensive robot runs to readily available video data, which is a huge win for making navigation models more practical and accessible.

Rosa: But we have to remember the caveats the authors pointed out—the memoryless nature of the policy and not explicitly modeling dynamic obstacle motion over time.

Dev: Exactly. So while it gets us strong performance on benchmarks and in some real-world trials, we need future work to build in that temporal prediction capability so it can handle unexpected movements more gracefully.

Rosa: That’s right, the next big step is making that geometric supervision robust enough to handle the messy reality of moving objects.

Dev: Speaking of which, we've got a paper on exact and approximate convex reformulation of linear stochastic optimal control with chance constraints coming up next that might help us build those more robust planning loops.

Rosa: Definitely, let’s see how those mathematical formulations translate into practical safety guarantees for our robots.

Dev: We will be talking about that right after this break.

More episodes

← Home