EmboAlign: Aligning Video Generation with Compositional Constraints for Zero-Shot Manipulation
cs.RO
Submitted: 2026-03-05
Updated: 2026-09-16
License: http://creativecommons.org/licenses/by/4.0/
The gist: Video generative models (VGMs) pretrained on large-scale internet data can produce temporally coherent rollout videos that capture rich object dynamics, offering a compelling foundation for zero-shot
Terminology
Abstract
Video generative models (VGMs) pretrained on large-scale internet data can produce temporally coherent rollout videos that capture rich object dynamics, offering a compelling foundation for zero-shot robotic manipulation. However, VGMs often produce physically implausible rollouts, and converting their pixel-space motion into robot actions through geometric retargeting further introduces cumulative errors from imperfect depth estimation and keypoint tracking. To address these challenges, we present EmboAlign, a data-free framework that aligns VGM outputs with compositional constraints generated by vision-language models (VLMs) at inference time. The key insight is that VLMs offer a capability complementary to VGMs: structured spatial reasoning that can identify the physical constraints critical to the success and safety of manipulation execution. Given a language instruction, EmboAlign uses a VLM to automatically extract a set of compositional constraints capturing task-specific requirements, which are then applied at two stages: (1) constraint-guided rollout selection, which scores and filters a batch of VGM rollouts to retain the most physically plausible candidate, and (2) constraint-based trajectory optimization, which uses the selected rollout as initialization and refines the robot trajectory under the same constraint set to correct retargeting errors. We evaluate EmboAlign on six real-robot manipulation tasks requiring precise, constraint-sensitive execution, improving the overall success rate by 43.3 percentage points over the strongest baseline without any task-specific training data.
Sources
- Large Video Planner Enables Generalizable Robot Control
- Robotic Manipulation by Imitating Generated Videos Without Physical Demonstrations
- CreFlow: Corrective Reflow for Sparse-Reward Embodied Video Diffusion RL
- VideoAgent: Self-Improving Video Generation
- Dream2Flow: Bridging Video Generation and Open-World Manipulation with 3D Object Flow
- Robot Learning from a Physical World Model
- Video as the New Language for Real-World Decision Making
- Gen2Act: Human Video Generation in Novel Scenarios enables Generalizable Robot Manipulation
- Envision: Embodied Visual Planning via Goal-Imagery Video Diffusion
- DreamGen: Unlocking Generalization in Robot Learning through Video World Models
- NovaFlow: Zero-Shot Manipulation via Actionable Flow from Generated Videos
- Spatiotemporal Predictive Pre-training for Robotic Motor Control
- Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representations
- Unified Video Action Model
- RoboDreamer: Learning Compositional World Models for Robot Imagination
- Grounding Generated Videos in Feasible Plans via World Models
- Code-as-Monitor: Constraint-aware Visual Programming for Reactive and Proactive Robotic Failure Detection
- SafeBimanual: Diffusion-based Trajectory Optimization for Safe Bimanual Manipulation
- Empowering Autonomous Driving with Large Language Models: A Safety Perspective
- SENTINEL: A Multi-Level Formal Framework for Safety Evaluation of Foundation Model-based Embodied Agents
Related papers
- FMT x: An Efficient and Asymptotically Optimal Extension of the Fast Marching Tree for Dynamic Replanning
- MPCFormer: A physics-informed data-driven approach for explainable socially-aware autonomous driving
- RoboLab: A High-Fidelity Simulation Benchmark for Analysis of Task Generalist Policies
- ARC: A Reasoning Recipe for Robot Foundation Models
- HRDexDB: A 4D Dexterous Grasping Dataset Across Human and Multiple Robot Embodiments
- APT: Action Expert Pretraining Improves Instruction Generalization of Vision-Language-Action Policies