AgiBot World Colosseo: A Large-scale Manipulation Platform for Scalable and Intelligent Embodied Systems

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: We're looking at the paper: "AgiBot World Colosseo".

Dev: Here's the gist of the paper. The gist: AgiBot World introduces a large-scale manipulation platform comprising over 1 million trajectories across 217 tasks in five deployment scenarios, achieving an order-of-magnitude increase in data scale compared to existing datasets.

Rosa: AgiBot World Platform and Data AgiBot World is described as a full-stack and open-source embodied intelligence ecosystem based on the AgiBot G1 hardware platform, providing high-quality data for challenging tasks spanning a wide spectrum of real-life scenariosWe construct AgiBot World—an open-source robot manipulation dataset collected by more than 100 homogeneous robots, providing high-quality data for challenging tasks spanning a wide spectrum of real-life scenariosThe latest version contains 1,001,552 trajectories, with a total duration of 2976.4 hours, covering 217 specific tasks, 87 skills, and 106 scenesThe dataset spans over five major domains—domestic, retail, industrial, restaurant, and office environmentKey statistics of our dataset are presented in Fig.

Dev: 3AgiBot World provides extensive coverage across five key domains: domestic, retail, industrial, restaurant, and office environmentsIt features over 3,000 distinct objects in a variety of scenesThe dataset covers a wide range of scenarios and scene setupsOur hardware setup includes mobile base humanoid robots with whole-body control, dexterous hands, and visuo-tactile sensorsEach episode is meticulously designed, featuring multiple camera views, depth information, camera calibration, and language annotations for both the overall task and each individual sub-stepsWe embrace a human-in-the-loop framework to ensure high qualityHuman feedback plays a critical role not only in post-collection review but also in actively guiding the data collection processFailure recovery data, referred to as failure recovery data, constitute approximately one percent of the datasetWe consider them invaluable for achieving policy alignment and failure reflectionThe dataset is available under the CC BY-NC-SA 4.0 license, along with the model checkpoints and code Data Collection Protocol and Quality Assurance The data collection session can be broadly divided into three phases(1) Before formally commencing data collection, we first conduct preliminary data acquisition to validate the feasibility of each task and establish corresponding collection standards(2) After feasibility validation and review of the collection standards, skilled teleoperators arrange the initial scene and formally begin data collection according to the established standards(3) During post-processing, the data annotators will verify whether each episode meets the collection standards established in phase 1 and provide language annotationsAll data undergoes an initial validity verification locally, such as verifying the absence of missing framesDuring data collection, teleoperators may occasionally commit errors, such as inadvertently dropping objects while manipulating the robotic armsRather than discarding such trajectories, we retain them and manually annotate each with corresponding failure reasons and timestampsConcurrent with feedback collection from data annotators, we adopt a human-in-the-loop approach to assess and refine data qualityThis process involves an iterative cycle of collecting a small set of demonstrations, training a policy, and deploying the resulting policy to evaluate data availabilityFor instance, during real-world deployment, the model exhibits prolonged pauses at the onset of actionsIn response, we revise the data collection protocols and introduce a post-processing step to eliminate idle framesThis feedback-driven methodology ensures continuous improvement in data quality Policy Learning Framework (ViLLA) To effectively utilize our high-quality AgiBot World dataset and enhance the policy’s generalizability, we propose a hierarchical Vision-Language-Latent-Action (ViLLA) framework with three training stagesCompared to Vision-Language-Action (VLA) model where action is vision-language conditioned, the ViLLA model predicts latent action tokens, conditioned on the generation of subsequent robot control actionsIn Stage 1, we project consecutive images into a latent action space by training an encoder-decoder latent action model (LAM) on internet-scale heterogeneous dataThis allows the latent action to serve as an intermediate representation, bridging the gap between general image-text inputs and robotic actionsIn Stage 2, these latent actions act as pseudolabels for the latent planner, facilitating embodiment-agnostic long-horizon planning and leveraging the generalizability of the pre-trained VLMFinally, in Stage 3, we introduce the action expert and jointly train it with the latent planner to support the learning of dexterous manipulationThe latent action model is constructed around an inverse dynamics model-based encoder I(ztIt,It+H) and a forward dynamics model-based decoder F(It+HIt,zt)The latent action tokens are quantized using a VQ-VAE objectiveThe latent planner consists of 24 transformer layers, which enable layer-by-layer conditioning from the VLM backbone with full bidirectional attentionSpecifically, given multiview input images I h t,I l t,I r t at timestep t, along with a language instruction l describing the ongoing task, the latent planner predicts latent action tokens: P ztI h t,I l t,I r t,lThe action expert decodes low-level action chunks, denoted by At =

at,at+1,…,at+H: with H = 30During inference, the VLM, latent planner, and action expert are synergistically combined within the generalist policy GO-1 Experimental Results and Performance Policies pre-trained on our dataset achieve an average success rate improvement of 30% compared to those trained on the prior large-scale robot dataset OXENotably, even when utilizing only a fraction of our dataset—equivalent to 1/10 of the data volume in hours compared to OXE—the generalizability of pretrained policies is elevated by 18%Models pre-trained on the AgiBot World dataset demonstrate a significant improvement in the “Table Bussing” task, nearly tripling performanceOn average, the completion score increases by 0.30 and 0.29 for in-distribution and out-of-distribution setups, respectivelyThe policy’s performance exhibits a predictable power-law scaling relationship with the number of trajectories, supported by a Pearson correlation coefficient of r = 0.97Being larger in quantity does not necessarily translate to improved performance, while a smaller set of human-verified data yields a 0.

Title and authors: Rosa: So we've got a platform with this massive amount of data now. The paper "AgiBot World Colosseo: A Large-scale Manipulation Platform for Scalable and Intelligent Embodied Systems" is detailing how they built this system.

Dev: They introduce the concept of AgiBot World as a full-stack, open-source ecosystem based on the AgiBot G1 hardware platform. It’s essentially a comprehensive toolkit for embodied intelligence.

Taro: The title suggests they're aiming for something scalable and intelligent, which implies they're not just collecting data but designing a whole system around it.

Rosa: They highlight that this platform comes with the dataset, the models, and even benchmarks. It’s designed to be open-source, which is important for community building in this space.

Dev: The authors mention deploying a suite of one hundred dual-arm humanoid robots to collect all this data and model their capabilities across those varied environments.

Taro: So the point here is that they're not just looking at one specific robot or one task; they’re building an ecosystem capable of handling a huge variety of real-world interactions.

Rosa: They focus on covering five major domains, from domestic settings to industrial ones, making sure the data isn't limited to just one type of interaction.

The paper's summary: Dev: Now let's look at what they actually summarized in the paper. They are focusing on how this dataset compares to previous efforts like Open X-Embodiment or DROID, and where it falls short.

Taro: They point out that older methods often end up with policies that are stuck on very simple, short-horizon tasks and struggle when they encounter situations outside the training data.

Rosa: They argue that existing work hasn't always done enough to focus on true generalizability across different objects and environments, especially for complex real-world scenarios.

Dev: AgiBot World Colosseo aims to move beyond that by providing data with high-quality annotations and human feedback integrated right into the collection process.

Taro: They also mention incorporating failure recovery data, which is about one percent of the dataset, because they see that learning from how things go wrong is crucial for real-world robustness.

Rosa: So essentially, they are showing how their approach addresses those limitations by focusing on scale and quality control simultaneously.

The paper's improvements: Dev: The authors suggest several improvements to their own methodology, which they call the ViLLA framework. It’s a hierarchical Vision-Language-Latent-Action approach instead of just a standard vision-language model leading directly to an action.

Taro: That latent action space idea is interesting because it acts as this intermediate representation, bridging the gap between what you see and what you actually do in the robot's control system.

Rosa: They break it down into three stages. Stage one trains a latent action model on internet-scale data to create these representations first.

Dev: Then stage two uses those latent actions as pseudolabels for a planner, which helps with long-horizon planning, leveraging the generalizability of the pre-trained vision language model.

Taro: And stage three involves an action expert that works with the planner to help figure out those precise, dexterous manipulation skills.

Rosa: This structure seems designed to make the AI system better at performing multi-step tasks that require continuous action, not just single steps.

Conclusion: Dev: So to wrap up, what's the main implication here for how we think about these embodied systems? It’s about moving away from policies that only solve short, controlled lab tasks toward something more robust and capable in complex real-world situations.

Taro: I think it means we need datasets that genuinely capture the messy reality of interaction, including failures, not just clean success scenarios. The emphasis on human feedback during collection is a big deal for building trust in that data.

Rosa: I agree with Taro on the quality control aspect. The way they are actively guiding data collection instead of just reviewing it later makes a huge difference in how good the final policy ends up being.

Dev: And from an engineering standpoint, the ViLLA framework seems to be a solid path for achieving better long-horizon execution because it gives us that temporal reasoning through those latent action predictions.

Taro: It sets a new standard for what constitutes high-quality data in this area. We’re seeing this focus on scale and quality control as essential ingredients for making AI truly embodied and useful outside of the lab.

Rosa: So, to summarize, the AgiBot World Colosseo paper presents a massive dataset built on real-world hardware that is structured to improve generalizability through a new hierarchical learning framework.

Dev: It’s a big step forward in providing data for training policies that can handle complex tasks with better success rates and more resilience when things go wrong.

Taro: We're looking at how this platform allows us to push the limits of what embodied AI can do when it has enough diverse, high-quality examples to learn from.