Make Code as Policy Great Again: Frontier Agents Write, Call, and Evolve Robot Tools
cs.RO, cs.AI
Submitted: 2026-09-30
Updated: 2026-09-30
Code: https://github.com/lykycy123/RoboJEV
License: http://creativecommons.org/licenses/by/4.0/
The gist: Frontier models can control robots, but reasoning through every reach, grasp, and retreat makes manipulation slow and token-intensive.
Terminology
Abstract
Frontier models can control robots, but reasoning through every reach, grasp, and retreat makes manipulation slow and token-intensive. We revisit code as policy with a different division of labor: models build executable tools, code handles multi-phase motions, and models decide what to do next. We introduce URAI (Universal Robot-Agent Interface), which couples a programming agent that constructs robot tools with an execution agent that uses them in a feedback loop. The programming agent writes reusable and task-specific tools from task intent and refines them through execution feedback and human guidance. The execution agent selects and parameterizes these tools from current observations; each call runs a complete motion locally before returning control to the agent. Unlike delegating subsequent decisions to a generated program, this design retains model-level decision-making between tool executions. Validated tool revisions persist across episodes without updating foundation-model weights, and a shared GUI and API make the same tools available to humans and agents. Across five RoboDojo tasks and four frozen execution agents, URAI raises aggregate success from 18.0% to 53.0% relative to direct fingertip control, with the largest gain on Swap Blocks; with the same tools, a program written in advance reaches only 24% against 56% for two agents deciding after each call. Three of the four agents also finish episodes 1.3-1.5 times faster with 1.5-1.7 times fewer execution-agent output tokens; DeepSeek-V4-Flash's cost barely changes. We further evaluate URAI on seven real-world AgileX dual-arm tasks, spanning object manipulation, cloth folding, and human-interactive tic-tac-toe. URAI connects the coding and decision-making capabilities of frontier agents, organizing robot control around reusable tools that agents can both invoke and revise.
Sources
- OSWorld-Human: Benchmarking the Efficiency of Computer-Use Agents
- Agent S: An Open Agentic Framework that Uses Computers Like a Human
- Large Language Models as Tool Makers
- RoboDojo: A Unified Sim-and-Real Benchmark for Comprehensive Evaluation of Generalist Robot Manipulation Policies
- Show-Harness: Just a VLM Agent Can Play Robots
- In-Context Robot Learning with VLM Agents
- Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robots
- RHO: Your Coding Agent is Secretly a Roboticist
- CaP-X: A Framework for Benchmarking and Improving Coding Agents for Robot Manipulation
- VGGT-DP: Generalizable Robot Control via Vision Foundation Models
- RT-Trajectory: Robotic Task Generalization via Hindsight Trajectory Sketches
- VIA: Visual Interface Agent for Robot Control
- Inner Monologue: Embodied Reasoning through Planning with Language Models
- RoboStream: Weaving Spatio-Temporal Reasoning with Memory in Vision-Language Models for Robotics
- Sketch Interface for Teleoperation of Mobile Manipulator to Enable Intuitive and Intended Operation: A Proof of Concept
- Agent as Policy for Robotic Manipulation
- HAMSTER: Hierarchical Action Models For Open-World Robot Manipulation
- MOKA: Open-World Robotic Manipulation through Mark-Based Visual Prompting
- Spatial Policy: Guiding Visuomotor Robotic Manipulation with Spatial-Aware Modeling and Reasoning
- ASPIRE: Agentic /Skills Discovery for Robotics
Related papers
- FMT x: An Efficient and Asymptotically Optimal Extension of the Fast Marching Tree for Dynamic Replanning
- MPCFormer: A physics-informed data-driven approach for explainable socially-aware autonomous driving
- RoboLab: A High-Fidelity Simulation Benchmark for Analysis of Task Generalist Policies
- ARC: A Reasoning Recipe for Robot Foundation Models
- HRDexDB: A 4D Dexterous Grasping Dataset Across Human and Multiple Robot Embodiments
- APT: Action Expert Pretraining Improves Instruction Generalization of Vision-Language-Action Policies