MolmoB0T: Large-Scale Simulation Enables Zero-Shot Manipulation

summary

The gist

MolmoBot introduces MolmoBot-Engine to generate large-scale procedural data and demonstrates that zero-shot transfer to real-world mobile manipulation is effective without any real-world fine-tuning.

In short

MolmoBot introduces MolmoBot-Engine to create massive procedural data for robots in simulation. It demonstrates that policies trained only on this synthetic data can transfer effectively to real-world manipulation without any real-world fine-tuning. This challenges the idea that simulation alone is enough for robust physical control, providing an open pipeline and a large dataset for diverse tasks and robots.

Key concepts

MolmoBot
This is a vision-language model based on Molmo2, designed to handle multi-frame vision and language inputs. It uses a flow-matching action head to generate control signals, making it capable of learning complex manipulation skills directly from simulated visual data.
MolmoBot-Pi0
This policy class replicates the standard $\pi_0$ architecture used in reinforcement learning for comparison. It serves as a baseline to isolate whether performance gains come from the specific MolmoBot design or simply from the quality and scale of the training data.
MolmoBot-SPOC
This is a lightweight policy designed specifically for deployment on edge devices. It is suitable for mobile robots and can be fine-tuned using reinforcement learning, offering a balance between performance derived from simulation and computational efficiency in real-world settings.

Terminology used across episodes

This episode discusses

The paper

MolmoB0T: Large-Scale Simulation Enables Zero-Shot Manipulation · Read on arXiv

Abhay Deshpande, Maya Guru, Rose Hendrix, Snehal Jauhri, Ainaz Eftekhar, Rohun Tripathi MaxArgus, Jordi Salvador Haoquan Fang Matthew Wallingford Wilbert Pumacay Yejin Kim Quinn Pfeifer Ying-Chun Lee Piper Wolters Omar Rayyan Mingtong Zhang Jiafei Duan Karen Farley Winson Han Eli Vanderbilt Dieter Fox Ali Farhadi Georgia Chalvatzaki Dhruv Shah Ranjay Krishna

Allen Institute for AI · University of Washington · University of California, Los Angeles · Technische Universität Darmstadt · Princeton University

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: I'm Rosa.

Dev: And I'm Dev, here with Taro.

Rosa: We're looking at the paper: "MolmoB0T: Large-Scale Simulation Enables Zero-Shot Manipulation".

Dev: Here's the gist of the paper. MolmoBot introduces MolmoBot-Engine to generate large-scale procedural data and demonstrates that zero-shot transfer to real-world mobile manipulation is effective without any real-world fine-tuning.

Rosa: This capability is significant because it challenges the assumption that simulation alone is sufficient for robust manipulation policies.

Dev: The work provides an open-source pipeline and a dataset of 1.7 million expert trajectories to enable this transfer across diverse tasks and robots.

Rosa: Policy Architectures Three policy classes are trained on the MolmoBot-Data, including MolmoBot, which is a Molmo2-based multi-frame vision-language model with a flow-matching action head.

Title and authors: Rosa: So we're looking at this paper today, "MolmoB0T: Large-Scale Simulation Enables Zero-Shot Manipulation." It sounds like they're tackling that big problem where training robots in the real world is super expensive and slow.

Dev: Yeah, they’ve built a system called MolmoBot to generate a massive amount of procedural data in simulations, and the main claim is that you can use this data to get these policies into the real world without doing any extra, dedicated fine-tuning.

Rosa: It's about using simulation to create a really big foundation model for manipulation agents, which is something that’s become really important lately because building those models usually takes huge resources.

Taro: I'm thinking about the authors—they’ve got a lot of people listed on the title page, which shows how much effort went into making this data generation pipeline work across all those different robots and tasks.

Dev: Exactly, and it suggests they are trying to provide a more complete picture of what it takes to build these generalist agents, since we often don't see all the details behind the strongest models.

The paper's summary: Rosa: Okay, so what’s actually in this paper? They’re talking about using MolmoBot-Engine, which is this whole open-source pipeline for generating procedural data across different robots and environments in their simulation space called MolmoSpaces.

Dev: That engine uses a huge set of 232k environments with forty-eight thousand manipulable objects spread across eight different task types, and they do extensive domain randomization, which means they randomize the environment, the actions taken, and even how the camera sees things.

Rosa: It’s not just generating data; it’s about creating this dataset of one point seven million expert trajectories that allows these policies to generalize really well to things they haven't seen before in reality.

Taro: What I find interesting is that they aren't just training one thing; they trained three different policy classes: MolmoBot, which is a vision-language model with a flow-matching action head, MolmoBot-Pi0, which replicates the old pi zero architecture for comparison, and MolmoBot-SPOC.

Dev: That separation into classes is key because it lets them see if the performance gains come from the specific architecture of the policy or just from having this massive amount of diverse training data.

The paper's improvements: Rosa: They show some really solid results, particularly when they test these policies on two different robots: the Franka FR3 for tabletop tasks and the Rainbow Robotics RB-Y1 mobile manipulator.

Dev: On the tabletop pick-and-place tasks, their best MolmoBot policy gets a success rate of seventy nine point two percent in real world evaluations across four different settings, which is significantly better than pi zero point five at thirty nine point two percent.

Taro: That comparison between the best MolmoBot and pi zero point five is telling because it shows that their model, which uses this large-scale synthetic data, can perform better than a policy based on a much smaller dataset trained from scratch.

Rosa: And they also demonstrated zero-shot transfer. They showed that these policies can move directly from simulation to unseen real-world objects and environments without needing any extra real-world fine-tuning at all.

Dev: That zero-shot capability is what makes this work so significant, because it challenges the idea that you absolutely have to collect a lot of real world data or do task-specific fine tuning to get a robust manipulation policy.

Conclusion: Rosa: So, to wrap up on "MolmoB0T: Large-Scale Simulation Enables Zero-Shot Manipulation," the main point is that procedural environment generation combined with diverse assets allows policies to generalize broadly from simulation to the real world.

Dev: They conclude that zero-shot transfer works effectively for both static and mobile manipulation, and they open up the MolmoBot-Engine pipeline so other groups can extend this approach to new robots and tasks.

Taro: I just want to add that they did show some limitations; they didn't mention what happens when the real world misbehaves in a way their simulation didn't predict, which is always something we have to consider when moving these models out of the lab.

Rosa: That’s fair, and it makes sense because this is all about pushing the boundaries of what simulation alone can do for robotics.

Dev: Yeah, so we have this open-source engine and a dataset of one point seven million expert trajectories that shows that massive scale synthetic data can really help build generalist foundation models for manipulation agents.

Taro: Alright, yeah, that’s the picture for this paper. We'll take a quick break and then I think we’ll look at some papers on real-time execution next.

More episodes

← Home