LocalNav: Distilling Frontier VLMs and Embodied RL for On-Device Object Goal Navigation

summary

The gist

Vision Language Models (VLMs) have emerged in the robotic domain as a powerful tool that enables environmental perception with language context…

In short

LocalNav enables ObjectNav for resource-constrained robots by distilling complex reasoning from large cloud models into a lightweight 4B local VLM. This method uses Scene Graph (SG) data and minimal reasoning traces to train a student model, achieving strong navigation performance (34.5% Success Rate) locally, significantly reducing reliance on the cloud.

Key concepts

LocalNav
A framework designed to run Vision Language Models (VLMs) for Object Navigation directly on small, local robotic hardware instead of relying on distant cloud servers. It achieves this by distilling knowledge from powerful models into a smaller, efficient student model.
SFT Distillation
A training technique where a smaller 'student' model learns the complex decision-making skills of a much larger 'teacher' model. In this case, the large frontier models provide reasoning traces that are used to train the 4B local VLM, allowing it to acquire sophisticated spatial and semantic navigation abilities.
E-RLVR with Token Generation (TG) Regularization
A technique used during training to compress the output sequence length of a model. By regularizing how tokens are generated, this method reduces the length of the model's response, which directly lowers generation latency and improves efficiency for real-time deployment on physical robots.

Terminology used across episodes

This episode discusses

The paper

LocalNav: Distilling Frontier VLMs and Embodied RL for On-Device Object Goal Navigation · Read on arXiv

Nicolas Baumann, Liam Boyle, Pu Deng, Edoardo Ghignone, Boyang Sun, Marc Pollefeys, Luca Benini, Michele Magno

Center for Project-Based Learning, ETH Zurich

Vision Language Models (VLMs) have emerged in the robotic domain as a powerful tool that enables environmental perception with language context, serving as a catalyst for open- vocabulary tasks like Object Goal Navigation (ObjectNav). Yet, their computational footprint typically confines them to cloud execution, hindering low-latency inference with local deployment on resource-constrained robots. To address this challenge, we present a distillation strategy that aligns a lightweight, 4B- parameter local VLM to our spatial-semantic navigation interface via frontier model traces for edge execution on embedded GPU devices (e.g., Jetson Orin). While cloud models establish a baseline achieving 41.61% Success Rate (SR) on the HM3D OVON benchmark, zero-shot deployment of our proposed architecture with Qwen3.5-4B reaches only 25.97% SR. We demonstrate that distilling frontier reasoning traces and subsequent finetuning with Embodied Reinforcement Learning from Verifiable Rewards (E- RLVR) obtains a 37.36% SR, substantially narrowing the gap to large cloud models. However, while deployable on memory- constrained devices, the model cannot compete with round-trip cloud inference time: we identify Token Generation (TG) as the main driver of latency and reduce it with Token Generation (TG)- regularized E-RLVR. The combined method reduces latency by 69.29% and retains a SR of 35.53%. Combined with quantization, this joint strategy yields a cumulative 80.21% reduction in overall inference latency without significantly sacrificing performance, presenting a viable paradigm for local, low-latency VLM execu- tion on mobile robots.

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: We're looking at the paper: "LocalNav: Distilling Frontier VLMs and Embodied RL for On-Device Object Goal Navigation".

Dev: Here's the gist of the paper. Vision Language Models (VLMs) have emerged in the robotic domain as a powerful tool that enables environmental perception with language context, serving as a catalyst for open-vocabulary tasks like ObjectNav The agent appears to be in a simplistic kitchen and I can see a silver appliance resembling a microwave which could be Success: +26% points-71.8% Pruning redundant reasoning tokens The paper presents LocalNav, a framework enabling VLM execution for ObjectNav on resource-constrained robotic platforms, eliminating cloud dependency By developing a SG-based architecture driven by frontier models, we establish a SotA baseline achieving a 39.7% SR on the HM3D OVON benchmark We demonstrate that a minimal amount of roughly 500 reasoning traces is enough to distill these complex spatial-semantic capabilities into a lightweight 4B student model, preserving robust navigation with a SR of 34.5%, narrowing the gap to cloud-based performance How the training signal is built The distillation strategy involves first establishing a State of the Art (SotA), Scene Graph (SG)-based pipeline using Claude Sonnet 4.6, achieving a 39.7% SR on the HM3D OVON benchmark We then demonstrate that fine-tuning Qwen3.5-4B on just ∼500 frontier reasoning traces effectively enables navigation capabilities, yielding a SR of 34.5%, narrowing the gap to the performance of large cloud models To achieve this, we record raw reasoning traces from Gemini 3.1 Pro, GPT-5.4, and Claude Sonnet 4.6, comprising image pairs and SG states generated directly by the aforementioned frontier models All training traces are gathered exclusively from the HM3D OVON train set Where the gains come from The downstream optimization introduces E-RLVR with Token Generation (TG) regularization to compress output sequence lengths for physical deployment while grounding the agent in its task This technique reduces TG overhead by 72.1% and latency by 71.8% Combined with quantization, this joint strategy yields a cumulative 82.8% reduction in overall inference latency without significantly sacrificing performance The E-RLVR framework natively regularizes the policy to compress its token output lengths during environment rollouts By transitioning from explicit linguistic prompt engineering to native execution constraints, the model retains necessary reasoning components within a condensed token footprint Performance and Deployment The system is designed with a modular architecture where the VLM makes high-level decisions from an explicit scene representation, while low-level navigation is delegated to a PointGoal policy This modularity enables the system to rely on dedicated navigation policies to safely traverse environments and avoid obstacles The final model marginally increases SR to 49%, cuts redundant object visitations in half (0.31 repetitions per episode), and explicitly compresses the output sequence length via Rbrevity to just 149.36 tokens per query, a 72.1% reduction in output length, directly reducing the overall action generation latency by 71.8% The chosen IQ4-XS format provides the best trade-off between prefill throughput and TG throughput without significantly degrading the KLD Combined with the E-RLVR-based brevity optimization, quantization further reduces the average runtime per episode from 86.0 s to 52.5 s, achieving an overall latency reduction of 82.

Title and authors: Rosa: So we're looking at this paper called "LocalNav: Distilling Frontier VLMs and Embodied RL for On-Device Object Goal Navigation". It’s about taking those big language models that can see the world and giving them a way to actually navigate physical spaces on smaller, less powerful robots.

Dev: Right. The focus here seems to be making these complex spatial understanding capabilities run locally without needing constant connection to a cloud server, which is pretty important for real-world deployment.

Taro: I’m curious about how they handle the open-vocabulary part of this, since that’s where things get tricky when you move away from just finding pre-defined objects like cabinets or chairs.

Rosa: Exactly. The paper suggests that by distilling knowledge from really big frontier models—like Claude Sonnet four point six and Gemini three point one Pro—they can create a much smaller, lighter student model, specifically a four billion parameter model called Qwen3 point 5-4B, and still get pretty good navigation results.

Dev: And that distillation process is key because they’re not just training the small model from scratch; they’re using those reasoning traces generated by these massive models to guide the learning process, which is a smart way to inject that high-level spatial understanding.

Taro: So, if I were driving this robot out into a messy kitchen and I told it "find the drawers below the microwave," I’m expecting it to understand that instruction because of all those examples from bigger models.

Rosa: That’s the goal, yeah. They show that by using just about five hundred of these detailed reasoning traces, they can distill those complex spatial and semantic skills into this smaller model, achieving a Success Rate of thirty-four point five percent on the HMthree dee OVON benchmark when fine-tuned on Qwen3 point 5-4B >

Dev: Thirty-four point five percent is a solid number for moving from cloud performance down to what's runnable on a Jetson Orin AGX, which is the hardware they’re targeting > I wonder how robust that thirty-four point five percent SR holds up when the kitchen gets unexpectedly cluttered or messy during actual testing > <ref:2606.27871#pg3>

Taro: That’s where my concern lies. If the environment misbehaves, does this distilled model just freeze because it can't generalize beyond those five hundred traces, or does it have some way to recover gracefully >

Rosa: The paper addresses that by adding a layer of improvement called E-RLVR, which stands for Embodied Reinforcement Learning from Verifiable Rewards. This framework is designed to ground the agent in its task and regularize the token generation length during the environment rollouts >

Dev: That sounds like a way to make sure it doesn't just produce a long string of text when it should be producing a short action command, which directly ties into reducing latency > The paper claims this technique cuts generation latency by seventy-one point eight percent > <ref:2606.27871#pg1,generation latency by 71.8>

Title and authors: Taro: So, that means the system isn't just smarter in its reasoning; it’s also faster at outputting what it needs to do in real time when it’s actually navigating > That would make a huge difference for safety loops on a real robot >

Rosa: It does. And they pair that with some quantization, which means shrinking the model itself, leading to an overall reduction in inference latency by eighty-two point eight percent compared to the baseline > So we’re talking about a system that can run much faster on-board than what you'd get just from running a big cloud model >

Dev: It sounds like they’ve managed to bridge that gap between high-level understanding and low-level, fast execution, which is usually where these kinds of systems struggle > I wonder about the trade-off here—does getting that latency reduction mean we lose some of the nuanced reasoning?

Taro: The paper mentions it does preserve robust navigation with a Success Rate of thirty-four point five percent, so they’re saying the performance hit isn't too steep, which is good for practical use > I just want to know if there are any specific failure modes they flagged when testing this on actual hardware > <ref:2606.27871#pg3>

Rosa: They do mention that the modular architecture is what allows them to drop in different VLMs, which means if Qwen3 point 5-4B isn't perfect for a new task, you could swap it out without redesigning the whole navigation pipeline > That flexibility is a big plus for field testing > <ref:2606.27871#pg1>

Dev: I see. So the core idea here is modularity combined with aggressive distillation and output compression to get complex object goal navigation running locally on constrained hardware > It’s like taking a super smart brain, training it on specific examples, and then teaching it to speak in very short, efficient sentences for its physical actions >

Taro: I think the main implication is that we can finally start deploying these kinds of open-vocabulary agents in environments where they actually need to operate autonomously without constant internet access > That’s a big deal for robotics research on the ground >

Rosa: It definitely sounds like they’ve made a lot of headway by showing that you don't need the biggest models in the cloud to get good results if you can strategically distill their knowledge into something much smaller and more efficient > We’ll keep an eye on how this performs when tested outside of controlled labs >

Dev: Yeah, it’s a solid step toward making these systems practical for real-world use where latency is critical, which brings us to the conclusion of this look at "LocalNav: Distilling Frontier VLMs and Embodied RL for On-Device Object Goal Navigation" >

The paper's summary: Rosa: So, LocalNav is about taking these big language models that can see and understand the world—these frontier VLMs—and shrinking them down so they can actually run on small robots without needing a constant connection to a powerful computer somewhere else.

Dev: Right. The core idea is distillation. They take all that massive spatial understanding from the biggest models, like Claude Sonnet and Gemini, and squeeze that knowledge into a much smaller four billion parameter model called Qwen3 point five-4B > It’s about making intelligence fit the hardware > <ref:2606.27871#pg2>

Rosa: Exactly. And they show you don't need a huge training set to get that distillation to work. They only needed about five hundred of these detailed reasoning traces from those big models, and that’s enough to teach the smaller model how to navigate complex spaces >

Dev: That’s a low number for such complex spatial tasks. What they did next was introduce something called E-RLVR, which is their way of optimizing the robot's behavior while it’s actually running in the real world > It helps compress the output sequence lengths, which means the robot doesn't have to say as much stuff when it’s trying to figure out how to move >

Rosa: And that compression is key because it directly cuts down on generation latency, which is how long it takes for the robot to decide what action to take next > They report a reduction in generation latency of seventy-one point eight percent, which helps a lot with real-time control >

Dev: That’s where the engineer’s view comes in. Onboard deployment means you can't have a few seconds of delay because you're waiting for the cloud to talk back; if that latency drops from some high number down to something much lower, that opens up safer operational windows > But what happens when the environment throws a wrench in the works?

Rosa: That’s where Taro gets involved. The paper shows how this modular system works. The VLM makes the big, high-level plan based on a Scene Graph, and then it delegates the low-level physical movement to a simpler policy called PointGoal > This lets them swap out different VLMs easily if one isn't performing well for a specific task >

Dev: So you’re getting flexibility in choosing your brain while still having that structured high-level decision-making framework, which is good for building safety cases because you can isolate the VLM component > But what about when the robot runs into something totally unexpected, like a weird piece of furniture not in its training set?

Rosa: The paper suggests that because the VLM is grounded in an explicit scene representation—that Scene Graph—it has a better chance of making a smart decision even if it’s slightly outside its immediate learned examples > It allows for open-vocabulary tasks, meaning it can handle things it hasn't seen before >

Dev: So they’re saying the combination of distillation for smaller size and E-RLVR for fast output length gives you a system that’s both smart enough to understand new objects and fast enough to react in real time on a small chip > It moves the entire VLM capability from just being theoretical in the cloud to being practically useful on the ground >

Rosa: Exactly. The implication here is that we can start deploying agents into messy, unstructured real environments where they don't need constant internet access or massive compute power to function > But we still have to figure out how much of that thirty-four point five percent success rate holds up when things get really chaotic outside the lab.

The paper's improvements: Dev: So we're looking at how they actually make this run fast on hardware, not just how smart it is in theory. They introduce something called E-RLVR with Token Generation regularization to deal with the robot's output sequence lengths > It compresses the text the model spits out during navigation, reducing that overhead by seventy-two point one percent >

Rosa: That compression has a direct impact on speed. When you cut down on how much text is generated for every single move, it directly cuts down on generation latency by seventy-one point eight percent > It’s about making the robot act faster because it doesn't have to write a long instruction first >

Dev: And they pair that token compression with quantization. Quantization is just shrinking the model itself, making its math run on less hardware, and together they get an overall reduction in inference latency of eighty-two point eight percent > That’s a big win for deployment because it means the entire process—from seeing to acting—is much quicker than a standard cloud-based setup >

Rosa: That takes the system from being just "smart" to being practically runnable on things like a Jetson Orin AGX, which is what we usually use for field trials > But they have to be careful. They say this joint strategy doesn't significantly sacrifice performance, meaning you get the speed boost without losing much of that spatial understanding we talked about earlier >

Dev: That’s the balancing act. The trade-off between being super fast and still being accurate is always there in these kinds of papers > I just wonder if that extra speed comes with any kind of reasoning compromise when things get really messy on the ground > Taro, what's your take on that?

Taro: From my testing view, it seems like the system stays grounded because the E-RLVR framework is designed to regulate policy compression during those rollouts > It’s not just guessing; it’s following constraints meant to keep it in line with the task > So, when things go wrong, does this compression make it more brittle or more stable?

Rosa: The authors imply stability. They show that the system maintains a Success Rate of thirty-four point five percent even after these aggressive optimizations > It suggests that the distillation and regularization work together to preserve enough reasoning for robust navigation > It's about making sure the model doesn't just stop working when it hits something new >

Dev: So, to sum up these improvements, they’ve taken a massive cloud-based capability and turned it into a local system that’s distilled down small, compressed, and optimized for speed on actual robot hardware > It moves the goal from theoretical possibility to something you could actually put in the field today > But we still need more real-world data to prove that this level of performance holds up across all possible environments.

Conclusion: Rosa: So to wrap up, LocalNav shows that we can take those massive models from the cloud and distill them down into something small enough to run locally on a robot with limited power > The core idea is using distillation and then compressing the output sequence length with E-RLVR to keep it fast and efficient >

Dev: Right. The numbers show that this approach cuts generation latency by seventy-one point eight percent, which is a big deal for real-time control loops, especially when you’re on resource-constrained hardware like a Jetson Orin AGX > It really changes the safety case because it means faster decision making on board >

Taro: I just want to stress that while they get this speed and size reduction, we still have to watch how robust it is when the environment goes completely sideways, because the success rate they showed on the benchmark is thirty-four point five percent > That doesn't mean it handles every single weird obstacle perfectly in a chaotic real world >

Rosa: True. So LocalNav proves that you don't need an enormous cloud connection to get good object navigation for open-vocabulary tasks; you can get a solid baseline performance by focusing on distillation and output compression > It’s about making powerful AI practical for the field, not just theoretical experiments >

Dev: It’s a solid step toward deploying these kinds of sophisticated agents in environments where latency is critical, which brings us to the next thing we need to look at: how these models handle genuine uncertainty during real-world interaction.

More episodes

← Home