CodeActionBench: Evaluating Agentic Code-as-Policy for Embodied Manipulation
cs.RO, cs.AI
Submitted: 2026-09-27
Updated: 2026-09-27
Comments: 28 pages including references and appendices, 10 figures. Project website: https://codeactionbench.org
Code: https://github.com/openai/codex
License: http://creativecommons.org/licenses/by/4.0/
The gist: How well can general-purpose multimodal models turn visual understanding and reasoning into embodied manipulation via executable code? We introduce CodeActionBench, a benchmark of 25 manipulation
Terminology
Abstract
How well can general-purpose multimodal models turn visual understanding and reasoning into embodied manipulation via executable code? We introduce CodeActionBench, a benchmark of 25 manipulation tasks that evaluates this capability through agentic Code-as-Policy. Without task-specific fine-tuning, demonstrations, external specialist perception or grasp modules, privileged scene state, or predefined task policies, agents should select visual evidence, form task-relevant 3D estimates, construct manipulation targets, and iteratively execute and revise their policies. A shared robot API provides RGB observations, calibrated geometric operations, robot feedback, and bounded motion, leaving task-dependent decisions to the evaluated agent. Fixed task instances, resource budgets, and a hidden physical-outcome verifier support controlled comparisons across models and harness configurations. Extensive evaluations across nine configurations and 675 attempts achieve success rates ranging from 2.7% to 73.3%. The strongest configuration, GPT-6 Astra with Codex CLI, solves 22 of 25 tasks at least once in three attempts, demonstrating the best performance while still leaving substantial room for improvement. Trajectory analyses reveal difficulties in spatial alignment, object retention, and completion judgment, including task failures despite successfully completed motions. CodeActionBench provides a controlled testbed for measuring how general-purpose models translate their capabilities into manipulation behavior and for examining typical failure scenarios in that process.
Sources
- SAM 3: Segment Anything with Concepts
- RoboScript: Code Generation for Free-Form Manipulation Tasks across Real and Simulation
- RoboTwin 2.0: A Scalable Data Generator and Benchmark with Strong Domain Randomization for Robust Bimanual Robotic Manipulation
- ETA: A New Agentic Paradigm for Embodied Tasks
- CaP-X: A Framework for Benchmarking and Improving Coding Agents for Robot Manipulation
- Instruct2Act: Mapping Multi-modality Instructions to Robotic Actions with Large Language Model
- Kimi K3: Open Frontier Intelligence
- DyPES-VLA: Learning Shared Dynamics Priors and Embodiment-Specific Control for Cross-Embodiment Manipulation
- From Fixed to Free Cameras: Calibration-Free View-Robust Vision-Language-Action Model
- Evaluating Real-World Robot Manipulation Policies in Simulation
- Being-H0.5: Scaling Human-Centric Robot Learning for Cross-Embodiment Generalization
- Robotic Long-Horizon Manipulation with Progressive In-Context Code Generation and Episodic Feedback
- G cubed VLA: Geometric Inductive Bias for Vision-Language-Action Models
- cuRobo: Parallelized Collision-Free Minimum-Jerk Robot Motion Generation
- In-Context VLA: Endowing Vision-Language-Action Models with Language via In-Context Post-Training and Agentic Tool Use
- GeoProp: Grounding Robot State in Vision for Generalist Manipulation
Related papers
- FMT x: An Efficient and Asymptotically Optimal Extension of the Fast Marching Tree for Dynamic Replanning
- MPCFormer: A physics-informed data-driven approach for explainable socially-aware autonomous driving
- RoboLab: A High-Fidelity Simulation Benchmark for Analysis of Task Generalist Policies
- ARC: A Reasoning Recipe for Robot Foundation Models
- HRDexDB: A 4D Dexterous Grasping Dataset Across Human and Multiple Robot Embodiments
- APT: Action Expert Pretraining Improves Instruction Generalization of Vision-Language-Action Policies