MAESTRO: Orchestrating Robotics Modules with Coding Agents for Zero-Shot Generalist Robots
cs.RO, cs.AI
Submitted: 2025-11-02
Updated: 2026-11-01
Comments: 34 pages. Accepted at CoRL 2026. Substantially revised with expanded experiments, ablations, and discussion. Project website: https://maestro-robot.github.io
Project page: https://robotics-transformerx.github.io
License: http://creativecommons.org/licenses/by/4.0/
The gist: Today's dominant route to generalist robots scales robotics data to train end-to-end policies.
Terminology
Abstract
Today's dominant route to generalist robots scales robotics data to train end-to-end policies. We instead use a vision-language model (VLM) as a coding agent that writes robot programs using a curated library of perception, planning, and control tools. MAESTRO's orchestration harness packages multi-step perception and action procedures into callable tools and guides their use through a plan-act-replan loop, using visual and execution feedback to verify outcomes and recover from failures. With this tool library and harness, MAESTRO matches or surpasses state-of-the-art vision-language-action (VLA) models zero-shot in our tabletop comparison and performs diverse mobile manipulation tasks, without collecting new teleoperation data or invoking a VLA tool. Ablations show that expanding the tool library improves average task progress with the full harness, while adding tools without it does not reliably improve progress. Case studies illustrate MAESTRO's potential for autonomous code improvement across trials, faster execution with VLA tools, and automatic robot data generation. We show that expanding robotics tool libraries together with their orchestration harnesses offers a complementary route to general-purpose robots alongside scaling end-to-end robot training data. Project site: https://maestro-robot.github.io. Code: https://github.com/penn-pal-lab/maestro.
Sources
- $\pi_0$: A Vision-Language-Action Flow Model for General Robot Control
- Gemini Robotics: Bringing AI into the Physical World
- GR00T N1: An Open Foundation Model for Generalist Humanoid Robots
- Gemini Robotics 1.5: Pushing the Frontier of Generalist Robots with Advanced Embodied Reasoning, Thinking, and Motion Transfer
- Robotic Control via Embodied Chain-of-Thought Reasoning
- Training Strategies for Efficient Embodied Reasoning
- CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Models
- $\pi_{0.5}$: a Vision-Language-Action Model with Open-World Generalization
- MolmoAct: Action Reasoning Models that can Reason in Space
- Code as Policies: Language Model Programs for Embodied Control
- Manipulate-Anything: Automating Real-World Robots using Vision-Language Models
- Closed-Loop Open-Vocabulary Mobile Manipulation with GPT-4V
- Sim-to-Real Reinforcement Learning for Vision-Based Dexterous Manipulation on Humanoids
- EMMA: Scaling Mobile Manipulation via Egocentric Human Data
- ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulation
- GraspGen: A Diffusion-based Framework for 6-DOF Grasping with On-Generator Training
- cuRobo: Parallelized Collision-Free Minimum-Jerk Robot Motion Generation
- Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V
- Articulate-Anything: Automatic Modeling of Articulated Objects via a Vision-Language Foundation Model
- VLMgineer: Vision Language Models as Robotic Toolsmiths
Related papers
- Rethinking Actions: Continuity-Aware Priors for World-Model Planning
- MOSAIC: Modular Obstacle and Spatial-Aware Framework for Manipulation In Cluttered Scenes
- SAGAS: Semantic-Aware Graph-Assisted Stitching for Offline Temporal Logic Planning
- DexSeed: Expanding Human Hand-Object Interactions into Robot Demonstrations for Dexterous Policy Learning
- LaST0.5: A Generalist World Action Model with Interaction-Centric Latent Reasoning
- CIALLO: Learning Shared Intent Representations for Cross-Embodiment Manipulation