Learning the Task or Learning the Verifier? Rethinking Criteria Exposure in Embodied Self-Improvement
cs.AI, cs.RO
Submitted: 2026-11-05
Updated: 2026-11-05
Comments: 22 pages, 8 figures, 10 tables. Project page: https://dagroup-pku.github.io/PyRUA-Lean/RSIJudge
Project page: https://dagroup-pku.github.io/PyRUA-Lean/RSIJudge
License: http://creativecommons.org/licenses/by/4.0/
The gist: Self-improving embodied agents are increasingly being developed and compared on robot simulation benchmarks.
Terminology
Abstract
Self-improving embodied agents are increasingly being developed and compared on robot simulation benchmarks. Several recent works expose the benchmark's success criteria to the learning agent, a practice we call Criteria Exposure (CE). These criteria, originally written to verify policies trained by imitation learning, often demand more than the corresponding language instructions, so a score from such a verifier may not faithfully reflect whether the instructed task was completed, especially for robot-control programs written by an agent. As embodied coding agents report ever higher benchmark scores, a natural question arises: when a coding agent is exposed to the benchmark's criteria, does it generate code that better performs the instructed task, or code that simply overfits to the verification criteria? We conduct controlled experiments with a self-improving agent with and without CE, and find that with CE the agent quickly produces code with a higher success rate under the given criteria, but this advantage disappears when the criteria are changed slightly without departing from the original instructions. More importantly, the skills the agent develops under CE transfer poorly across tasks. We further propose instruction-faithful criteria and show that they better reflect the performance of agent-generated programs.
Sources
- Concrete Problems in AI Safety
- Compile Once, Run Cheap: Frozen, Portable Skills for Embodied Agents, and What Recursive Self-Improvement Adds
- Demonstrating specification gaming in reasoning models
- SAM 3: Segment Anything with Concepts
- RoboTwin 2.0: A Scalable Data Generator and Benchmark with Strong Domain Randomization for Robust Bimanual Robotic Manipulation
- RoboDojo: A Unified Sim-and-Real Benchmark for Comprehensive Evaluation of Generalist Robot Manipulation Policies
- ETA: A New Agentic Paradigm for Embodied Tasks
- Vision-Language Models as Success Detectors
- HexaShell: A Universal Harness for Robot Coding Agents in Simulation and the Real World
- Scaling Online Exploration for Robot Agents
- AI Agents That Matter
- EmbodiedEvo: Self-Evolving Coding Agent For Physical Intelligence
- ASPIRE: Agentic /Skills Discovery for Robotics
- AlphaEvolve: A coding agent for scientific and algorithmic discovery
- The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models
- $\pi_{0.5}$: a Vision-Language-Action Model with Open-World Generalization
- World Action Models: A Survey
- EmbodiedSWE: Coding Agents for Long Horizon Dexterous Robotics
- Reflexion: Language Agents with Verbal Reinforcement Learning
- Fewer Tokens, Better Action: GPT-6 Astra Robot Agents with 14% Higher Success Rate but 65% Fewer Tokens
Related papers
- Multi-Objective Structured Pruning of LLMs for Latency and Model Size Optimization
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Embeddings for Preferences, Not Semantics
- Can We Trust Item Response Theory for AI Evaluation?
- Diagnosing Task Insensitivity in Language Agents