WildOS: Open-Vocabulary Object Search in the Wild
summary
The gist
Autonomous navigation in complex, unstructured outdoor environments requires robots to operate over long ranges without prior maps and limited depth sensing, and this work presents WildOS…
In short
WildOS is a system for robots to navigate long distances in complex outdoor areas without maps using both geometry and vision. It combines building a spatial graph with visual reasoning, allowing robots to intelligently search for specific objects or directions by scoring potential paths based on both safety and semantic meaning.
Key concepts
- Navigation Graph Construction
- This component builds a sparse map of the environment to help the robot remember where it has been and where it can go. It creates a structure that allows the robot to maintain spatial memory, which is essential for planning long-range paths in unknown environments.
- ExploRFM Module
- This module uses a foundation model to predict visual information from images. It generates maps showing where traversable areas are, where potential new areas (frontiers) exist, and what objects might be present in the scene based on the robot's goal query.
- Scoring Mechanism
- This process combines geometric data with visual predictions to assign a score to potential paths. The score considers how safe a path is geometrically, how likely it is to lead toward the goal semantically, and other factors derived from the visual module.
- Goal Triangulation Module
- This part estimates where a desired goal might be located beyond what the robot can currently see. It helps in finding candidate locations for exploration by projecting possibilities into a larger space, guiding the robot toward unseen targets.
Terminology used across episodes
This episode discusses
- WildOS: Open-Vocabulary Object Search in the Wild · Paper Radio
- STEP: Stochastic Traversability Evaluation and Planning for Risk-Aware Off-road Navigation
- NeBula: Quest for Robotic Autonomy in Challenging Environments; TEAM CoSTAR at the DARPA Subterranean Challenge
- RayFronts: Open-Set Semantic Ray Frontiers for Online Scene Understanding and Exploration
- On Evaluation of Embodied Navigation Agents
- Uni-NaVid: A Video-based Vision-Language-Action Model for Unifying Embodied Navigation Tasks
- NaVILA: Legged Robot Vision-Language-Action Model for Navigation
- GNM: A General Navigation Model to Drive Any Robot
- ViNT: A Foundation Model for Visual Navigation
- VLD: Visual Learned Distance as an Interface for Reinforcement Learning Navigation · Paper Radio
- TerrainNet: Visual Modeling of Complex Terrain for High-speed, Off-road Navigation
- RADIOv2.5: Improved Baselines for Agglomerative Vision Foundation Models
- DINOv2: Learning Robust Visual Features without Supervision
- Segment Anything
- SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features
The paper
WildOS: Open-Vocabulary Object Search in the Wild · Read on arXiv
Hardik Shah, Erica Tevere, Deegan Atha, Marcel Kaufmann, Shehryar Khattak, Manthan Patel, Marco Hutter, Jonas Frey, Patrick Spieler
Jet Propulsion Laboratory (JPL) · Swiss Federal Institute of Technology (ETH Zurich) · FieldAI Inc. · Stanford University · University of California Berkeley
Autonomous navigation in complex, unstructured outdoor environments requires robots to operate over long ranges without prior maps and limited depth sensing. In such settings, relying solely on geometric frontiers for exploration is often insufficient. In such settings, the ability to reason semantically about where to go and what is safe to traverse is crucial for robust, efficient exploration. This work presents WildOS, a unified system for long-range, open-vocabulary object search that combines safe geometric exploration with semantic visual reasoning. WildOS builds a sparse navigation graph to maintain spatial memory, while utilizing a foundation-model-based vision module, ExploRFM, to score frontier nodes of the graph. ExploRFM simultaneously predicts traversability, visual frontiers, and object similarity in image space, enabling real-time, onboard semantic navigation tasks. The resulting vision-scored graph enables the robot to explore semantically meaningful directions while ensuring geometric safety. Furthermore, we introduce a particle-filter-based method for coarse localization of the open-vocabulary target query, that estimates candidate goal positions beyond the robot's immediate depth horizon, enabling effective planning toward distant goals. Extensive closed-loop field experiments across diverse off-road and urban terrains demonstrate that WildOS enables robust navigation, significantly outperforming purely geometric and purely vision-based baselines in both efficiency and autonomy. Our results highlight the potential of vision foundation models to drive open-world robotic behaviors that are both semantically informed and geometrically grounded. Project Page: https://leggedrobotics.github.io/wildos/
Transcript
Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.
Rosa: We're looking at the paper: "WildOS: Open-Vocabulary Object Search in the Wild".
Dev: Here's the gist of the paper. Autonomous navigation in complex, unstructured outdoor environments requires robots to operate over long ranges without prior maps and limited depth sensing In such settings, the ability to reason semantically about where to go and what is safe to traverse is crucial for robust, efficient exploration This work presents WildOS, a unified system for long-range, open-vocabulary object search that combines safe geometric exploration with semantic visual reasoning The resulting vision-scored graph enables the robot to explore semantically meaningful directions while ensuring geometric safety System Architecture WildOS is a unified, real-time system that combines complementary strengths of visual and geometric perception for long-range navigation and object search Its core consists of five main components: 1) Navigation Graph Construction, which builds a sparse navigation graph to maintain spatial memory, 2) ExploRFM Module, a foundation-model-based vision module that predicts visual traversability, visual frontiers, and object similarity in image space, 3) Goal Triangulation Module, which estimates candidate goal positions beyond the robot’s immediate depth horizon, 4) Fusion Module, which projects geometric frontier nodes into the image plane and scores them using visual-semantic cues, and 5) Planning and Control, which selects a path based on the scored graph to guide exploration ExploRFM Module The ExploRFM module is designed to provide long-range semantic perception that complements LiDAR-based geometry Given the current RGB frame It and a goal text query Qgoal, ExploRFM produces three dense prediction maps: Tvis t, (visual traversability map), Fvis t, (visual frontier score map), and Svis t, (object similarity mask) The visual traversability map Tvis t estimates the semantic traversability of each pixel distinguishing between safe and unsafe regions based on scene semantics The visual frontier score map Fvis t highlights semantic regions likely to lead to novel observations or unexplored spaces Scoring Mechanism The fusion module combines the complementary strengths of the two modalities by projecting each geometric frontier node from world coordinates into the image plane of It and assigning a visual-semantic exploration score si This score is defined as si = fscoreTvis t, Fvis t, π(xvi), pˆgoal, xt, which is computed using terms for Goal Confidence (Gconf), Reachability Confidence (Rconf), and Frontier Confidence (Fconf) The scoring function is implemented to be goal-agnostic by evaluating fscore across a discretized set of possible goal headings around each frontier node vi Planning Strategy The planner selects a frontier node vi that minimizes the expected total cost to reach the goal, formulated as v∗ = argmin vi∈Fgeo t fj∈Fvis t h Clocal(xT, vi)+Ctrans(vi, fj)+Cgoal(fj, pˆgoal) This cost formulation links geometric understanding within Hd and visual reasoning up to Hv, motivating the design of the scored navigation graph The transition cost Ctrans is estimated using the minimum-cost image path (MCIP) between projected frontier vi and a visual frontier pixel fj Experimental Validation Extensive field experiments across diverse off-road and urban terrains demonstrate that WildOS significantly improves efficiency and robustness over both purely geometric and purely vision-based baselines The system outperforms LRN in handling dead-ends, as WildOS uses the navigation graph to reroute effectively, whereas LRN exhibits oscillatory behavior due to its memoryless nature Furthermore, WildOS shows strong generalization across diverse outdoor terrains without retraining, highlighting the effectiveness of foundation-model features In conclusion, WildOS presents a unified system for open-vocabulary object search and long-range navigation in unstructured outdoor environments Limitations include oscillatory behavior in long-horizon exploration and a lack of visual feature memory for query-driven retrieval Future work suggests incorporating visual memory for query-driven retrieval and integrating visual traversability into the geometric map to create a unified local representation REFERENCES
Title and authors: Rosa: To start off, let's talk about the title and who wrote this paper. "WildOS: Open-Vocabulary Object Search in the Wild." It’s pretty descriptive of what they’re trying to achieve.
Dev: The authors are a team from JPL, Caltech, ETH Zurich, and a few others. They clearly have expertise spanning robotics, vision systems, and control engineering.
Taro: I'm interested in how they frame the problem by using that title. "Open-Vocabulary Object Search" implies the robot isn't just looking for pre-defined objects; it can search for anything semantically meaningful out there.
Rosa: That’s right, and then you have "in the Wild," which immediately tells you this isn't some controlled lab setting. They are aiming for unstructured outdoor environments where things are unpredictable.
Dev: From my side, I see the implication that they aren't just trying to navigate around obstacles; they’re trying to reason about *where* to go based on what those obstacles look like, which is a big step toward true autonomy.
Taro: It means the robot needs a way to connect what it sees visually with where it physically can go, not just following lines in a map.
The paper's summary: Rosa: So, the paper summarizes WildOS as a unified, real-time system that merges visual and geometric perception for long-range navigation and object search. It’s about building this navigation graph that stores spatial memory while using vision to score those frontiers.
Dev: Essentially, they combine the spatial awareness of geometry with the semantic understanding of vision to plan paths through semantically promising directions while maintaining geometric safety.
Taro: They're using a particle-filter based goal localization module to estimate candidate goal locations beyond the depth horizon, which is key because it helps with that long-range planning.
Rosa: And they achieve this by projecting geometric frontier nodes into the image plane and scoring them with visual-semantic cues, which defines a score based on goal confidence, reachability confidence, and frontier confidence.
Dev: That scoring function is designed to be goal-agnostic by evaluating the score across several possible goal headings around each node. It’s about finding directions that look promising visually while still being physically reachable and safe.
The paper's improvements: Taro: I want to talk about what they suggest as improvements. One major point is adding a quality or uncertainty score for frontier labels so the system can tell the difference between reliable and risky exploration opportunities.
Rosa: That makes sense, because if a path looks visually interesting but is actually dangerous, the robot needs to know not to go there without hesitation. It allows for more informed decision-making in ambiguous areas.
Dev: I also see them projecting visual traversability into the geometric traversability map, which creates a unified local representation that uses both semantic and geometric cues for planning around things like water hazards.
Taro: That’s important because geometry alone doesn't tell you if something is actually drivable or safe to traverse in a real outdoor setting.
Rosa: Another improvement they mention is developing a category-adaptive thresholding mechanism for open-vocabulary grounding, which lets the system vary its similarity threshold depending on what kind of object it's looking for.
Dev: That should help improve detection precision by not using one fixed threshold across every single object category. It’s about making the visual reasoning smarter about what it’s seeing.
Conclusion: Rosa: So, to wrap up this WildOS paper, it presents a unified system for open-vocabulary object search and long-range navigation in unstructured outdoor environments by combining geometric exploration with vision-scored graph planning.
Dev: The main implication is that robots can explore semantically meaningful directions while ensuring they stay geometrically safe, which is a big step beyond purely geometric or purely vision-based methods.
Taro: What I see is that it handles dead ends better because the navigation graph lets the robot reroute effectively when it hits a wall, unlike older systems that might just oscillate.
Rosa: And they show strong generalization across diverse outdoor terrains without needing to be fully retrained, which suggests the foundation model features are really powerful here.
Dev: One limitation they point out is oscillatory behavior in long-horizon exploration, which means the system can get stuck in loops if the visual cues aren't clear enough over a very long distance.
Taro: They also mention a lack of visual feature memory for query-driven retrieval, meaning if you ask it to find something specific based on past observations, it doesn't have that history built in yet.
Rosa: It’s still a big step forward for open-vocabulary mapping, even with those limitations. WildOS is now the unified system for this kind of task.
More episodes
- 2608.13438-ContactGuard: Pre-Contact Execution Monitoring with Action-Conditioned Latent World Models
- 2610.34593-Future Prediction as Training-Only Video Supervision for Pretrained VLA Policies
- 2610.35385-EmbodiedLab: Beyond Manipulation, Benchmarking Embodied Agents That Act to Know
- 2610.36378-Data-Driven Hamiltonian for Neural Optimal Control
- 2610.37275-A Heuristic for Scheduling Autonomous Vehicles at Intersections
- 2610.37288-Dynamic Coordination of Autonomous Vehicles at Intersections
- 2610.37632-Real-Time Coordination of Autonomous Vehicles at Intersections
- 2611.13779-Softness is something you also need! Why compliance helps control: information metrics of a passive walker with compliant ankle
- 2611.13411-Residual-Corrected Physics-Informed Neural Network KKL Observers with Input-to-State Stability Guarantees
- 2611.14488-Is Your Robot's Memory Just a Clock? Temporal Shortcuts in Memory-Augmented VLA Policies