OmniGuide: Universal Spatial Guidance Fields for Enhancing Generalist Robot Policies
cs.RO, cs.LG
Submitted: 2026-03-09
Updated: 2026-11-02
Comments: Accepted to CoRL 2026. Project page: https://omniguide.github.io/
Project page: https://omniguide.github.io
License: http://creativecommons.org/licenses/by/4.0/
The gist: Vision-language-action (VLA) models have shown great promise as generalist policies for a large range of relatively simple tasks.
Terminology
Abstract
Vision-language-action (VLA) models have shown great promise as generalist policies for a large range of relatively simple tasks. However, they demonstrate limited performance on more complex tasks, such as those requiring complex spatial or semantic understanding, manipulation in clutter, or precise manipulation. We propose OmniGuide, a flexible framework that improves VLA performance on such tasks by leveraging arbitrary sources of guidance, such as 3D foundation models, semantic-reasoning VLMs, and human pose models. We show how many kinds of guidance can be naturally expressed as differentiable energy functions with task-specific attractors and repellers located in 3D space, that influence the sampling of VLA actions. In this way, OmniGuide enables guidance sources with complementary task-relevant strengths to improve a VLA model's performance on challenging tasks. Extensive experiments in both simulation and real-world environments, across diverse sources of guidance, demonstrate that OmniGuide enhances the performance of state-of-the-art generalist policies (e.g., π 0.5, GR00T N1.6) significantly across success and safety rates. Critically, our unified framework matches or surpasses the performance of prior methods designed to incorporate specific sources of guidance into VLA policies. Project page: https://omniguide.github.io/
Sources
- $\pi_0$: A Vision-Language-Action Flow Model for General Robot Control
- Real-Time Execution of Action Chunking Flow Policies
- RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control
- GR-3 Technical Report
- TGRPO :Fine-tuning Vision-Language-Action Model via Trajectory-wise Group Relative Policy Optimization
- Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robots
- Diffusion Posterior Sampling for General Noisy Inverse Problems
- Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities
- SafeFlow: Safe Robot Motion Planning with Flow Matching via Control Barrier Functions
- Neural MP: A Generalist Neural Motion Planner
- DynaGuide: Steering Diffusion Polices with Active Dynamic Guidance
- Spatial Forcing: Implicit Spatial Representation Alignment for Vision-language-action Model
- A Taxonomy for Evaluating Generalist Robot Manipulation Policies
- UMI-on-Air: Embodiment-Aware Guidance for Embodiment-Agnostic Visuomotor Policies
- VLSA: Vision-Language-Action Models with Plug-and-Play Safety Constraint Layer
- ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulation
- $\pi^{*}_{0.6}$: a VLA That Learns From Experience
- $\pi_{0.5}$: a Vision-Language-Action Model with Open-World Generalization
- ContextVLA: Vision-Language-Action Model with Amortized Multi-Frame Context
- FlowDPS: Flow-Driven Posterior Sampling for Inverse Problems
Related papers
- Rethinking Actions: Continuity-Aware Priors for World-Model Planning
- MOSAIC: Modular Obstacle and Spatial-Aware Framework for Manipulation In Cluttered Scenes
- SAGAS: Semantic-Aware Graph-Assisted Stitching for Offline Temporal Logic Planning
- DexSeed: Expanding Human Hand-Object Interactions into Robot Demonstrations for Dexterous Policy Learning
- LaST0.5: A Generalist World Action Model with Interaction-Centric Latent Reasoning
- CIALLO: Learning Shared Intent Representations for Cross-Embodiment Manipulation