EmbodiedLab: Beyond Manipulation, Benchmarking Embodied Agents That Act to Know
summary
The gist
EmbodiedLab introduces a benchmark where an embodied agent must act to know by changing a physical system and reading instruments through cameras…
In short
EmbodiedLab is a benchmark where an agent must act physically to discover information, rather than just following pre-set instructions. The agent interacts with a simulated laboratory, changes physical states, and reads instrument data via cameras to 'know' the answer. Results show that success heavily depends on reliably interacting with the apparatus and manipulating objects.
Key concepts
- Acting to Know
- This is an intelligent process where an agent uses physical actions—like moving parts or touching things—as a method to gain knowledge about its environment. The goal isn't just to reach a final state, but to explore the system until it learns what the required procedure is, using interaction as the primary way to uncover information.
- Embodied Agent
- An embodied agent is an AI system that exists in a physical or simulated world and interacts with it directly. Unlike software agents that only process data, these agents use their body and sensory inputs (like cameras) to physically manipulate objects and change the state of the environment to achieve goals.
- Interface Levels (T0-T2)
- These levels describe how much control an agent has over its actions. T0 is basic, providing raw camera images and joint control. T1 adds higher-level perception and planning capabilities. T2 incorporates learned manipulation skills from stronger agents, allowing the agent to perform more complex physical tasks.
- Capability Dimensions
- These define the specific skills an agent needs to succeed in EmbodiedLab, such as Fine Manipulation (precise object handling), Long-horizon work (planning for multi-stage tasks), and Fine Reading (accurately interpreting instrument data). These dimensions show what kind of physical interaction is required to solve a given experiment.
Terminology used across episodes
This episode discusses
- EmbodiedLab: Beyond Manipulation, Benchmarking Embodied Agents That Act to Know · Paper Radio
- CaP-X: A Framework for Benchmarking and Improving Coding Agents for Robot Manipulation
- Self-Evolving Coding Agents: From Digital Programs to Physical-World Intelligence · Paper Radio
- EmbodiedSWE: Coding Agents for Long Horizon Dexterous Robotics · Paper Radio
- WetRobo: A Reproducible Robot Kit for Coding Agents in Biological Laboratories · Paper Radio
- LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning
- RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robots
- RoboTwin 2.0: A Scalable Data Generator and Benchmark with Strong Domain Randomization for Robust Bimanual Robotic Manipulation
- RoboDojo: A Unified Sim-and-Real Benchmark for Comprehensive Evaluation of Generalist Robot Manipulation Policies
- HexaShell: A Universal Harness for Robot Coding Agents in Simulation and the Real World · Paper Radio
- Fewer Tokens, Better Action: GPT-6 Astra Robot Agents with 14% Higher Success Rate but 65% Fewer Tokens · Paper Radio
- RHO: Your Coding Agent is Secretly a Roboticist
- When Robots Do the Chores: A Benchmark and Agent for Long-Horizon Household Task Execution
- Compiling and Benchmarking Task-State Horizons for Embodied Agents
- LabUtopia: High-Fidelity Simulation and Hierarchical Benchmark for Scientific Embodied Agents
- AutoBio: A Simulation and Benchmark for Robotic Automation in Digital Biology Laboratory
- SciHorizon-eLab: An Agentic Protocol-to-Task Compiler for Scalable Benchmarking of Scientific Embodied Agents · Paper Radio
- Pipette: An Embodied Simulation Platform, Benchmark, and Data-Efficient Augmentation Framework for Wet-Lab Robotics
- ScienceWorld: Is your Agent Smarter than a 5th Grader?
- DISCOVERYWORLD: A Virtual Environment for Developing and Evaluating Automated Scientific Discovery Agents
The paper
EmbodiedLab: Beyond Manipulation, Benchmarking Embodied Agents That Act to Know · Read on arXiv
Hongcheng Gao, Zelin Zheng, Alex Zhou, Jay Zhu, Shijia Ge, Yazhe Wang, Xiping Li, Jianshu Zeng, Di Wu, Xuan Shangguan, © Yexing Wan
HexaFuture Inc. · “Tsinghua University” · “National University of Singapore” · “Peking University”
Embodied agents are expected to accomplish real-world goals by deciding for themselves how and when to interact with their surroundings. For many such goals, changing the physical state of the world is the means rather than the end: the agent changes the state of a system through interaction, observes the feedback, and explores until it either achieves the goal itself (e.g., discovering physical laws) or learns what the goal requires; it acts to know. Manipulation benchmarks stop at acting to change, with the target state given; scientific discovery benchmarks evaluate knowing but replace physical experimentation with symbolic actions or numerical queries. We introduce EmbodiedLab, a benchmark in which a robot must act to know. Quantitative science experiments make this setting controllable: the parameters of the system are resampled for every instance, so the result cannot be known in advance, yet it can be scored exactly. Across 30 experiments in five disciplines on 14 apparatus platforms, the agent receives an objective and the apparatus, chooses its own procedure, and operates a physics-based simulated laboratory, reading the instruments through its cameras. Evaluation checks what the agent came to know, grounded in readings recorded during the episode, as well as the physical state the task requires; every experiment is verified to be solvable. We evaluate six frontier models in their developers' coding agents with native observation and control, with added perception and motion planning, and with manipulation skills distilled from the strongest agents. With native control, even the strongest agent succeeds less than half the time; for every agent, most failures arise in interacting with the apparatus, and manipulation skills help weaker agents more than perception and planning. The benchmark and its evaluation code are available at https://github.com/HexaFuture/EmbodiedLab.
Transcript
Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.
Rosa: We're looking at the paper: "EmbodiedLab: Beyond Manipulation, Benchmarking Embodied Agents That Act to Know".
Paper summary: Rosa: So this paper is about something called EmbodiedLab. It's a benchmark designed for embodied agents to act to know. The big idea is that many real-world goals aren't just about reaching a target state, but figuring out what you need to do first and how to do it through interaction with the world.
Dev: Right, so instead of just changing the physical state of something for a set outcome, the agent has to change that state through interaction, observe what happens, and explore until it either achieves its own goal or learns what information it needs >
Taro: I mean, so we're moving past benchmarks where you just tell the robot where to go. This is about the agent deciding how and when to interact with its surroundings based on what it sees and feels >
Rosa: Exactly. EmbodiedLab sets up these thirty quantitative experiments across five disciplines like mechanics, electricity, optics, thermal physics, and chemistry using fourteen different apparatus platforms >
Dev: And what's interesting is that the setup is designed to be controllable because the parameters of the system are resampled for every single experiment so you can't know the result in advance but you can score it exactly >
Taro: The evaluation metric isn't just about getting to the final state; it checks what information the agent came to know, grounded in readings recorded during that episode and matching the physical state required by the task >
Rosa: That sounds like a real test of understanding, not just following a pre-programmed path. It’s testing whether the agent can act to know >
Conclusion: Dev: Looking at this paper, "EmbodiedLab: Beyond Manipulation, Benchmarking Embodied Agents That Act to Know," the authors are clearly pushing past simple manipulation tasks and focusing on that core concept of acting to know >
Rosa: I think the title itself tells you a lot. They're saying this benchmark goes beyond just manipulating objects into something where the agent has to uncover knowledge through physical action >
Dev: And it implies that for agents to be truly useful in complex, unstructured environments, they need that ability to figure out what they don't already know >
Taro: What this means practically is that when we build these systems, we can't just focus on making them follow instructions perfectly; we have to design them so they can handle the unknown and discover what’s required >
Rosa: It really boils down to the fact that acting to know is still a tough problem. The paper shows that even with these complex setups, most failures happen when the agent messes up in interacting with the apparatus or observing it >
Dev: And they quantify that too; they found that acting on the apparatus and restoring it together accounts for seventy-five to one hundred percent of every agent's failed runs >
More episodes
- 2608.13438-ContactGuard: Pre-Contact Execution Monitoring with Action-Conditioned Latent World Models
- 2610.34593-Future Prediction as Training-Only Video Supervision for Pretrained VLA Policies
- 2610.36378-Data-Driven Hamiltonian for Neural Optimal Control
- 2610.37275-A Heuristic for Scheduling Autonomous Vehicles at Intersections
- 2610.37288-Dynamic Coordination of Autonomous Vehicles at Intersections
- 2610.37632-Real-Time Coordination of Autonomous Vehicles at Intersections
- 2611.13779-Softness is something you also need! Why compliance helps control: information metrics of a passive walker with compliant ankle
- 2611.13411-Residual-Corrected Physics-Informed Neural Network KKL Observers with Input-to-State Stability Guarantees
- 2611.14488-Is Your Robot's Memory Just a Clock? Temporal Shortcuts in Memory-Augmented VLA Policies
- 2611.14470-TanCRAFT: Context-Reasoned Anticipatory Feedback for Assembly Tasks