Data Pyramid for Embodied Manipulation: A Survey

arXiv:2607.24744 · cs.RO, cs.CV · Submitted 2026-07-27 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: We're looking at the paper: "Data Pyramid for Embodied Manipulation: A Survey".

Paper summary: Rosa: To wrap up, the paper "Data Pyramid for Embodied Manipulation: A Survey" lays out this five layer structure of data sources to handle the requirements of embodied foundation models.

Dev: The authors organize this pyramid around the tension between scalability and robot alignment, showing how different data types fit together depending on what you need from a model.

Rosa: What it means in simple terms is that we don't just need one perfect dataset anymore; we need a whole ecosystem of complementary data sources to build models capable of understanding the world and interacting with it physically.

Dev: They discuss how these data properties map onto model capabilities like perception reasoning planning and action generation, but they also point out the practical hurdles, especially building those large scale tactile datasets.

Rosa: The authors conclude that the optimal way to collect this data recipe is still undecided, suggesting future research should focus on recipes that are aware of the architecture of the models they are training.

Dev: Ultimately, this survey provides a roadmap for how different embodied foundation models can be supported by a rich and heterogeneous collection of data from real robot interaction all the way down to general vision language data.

Conclusion: Rosa: So we're wrapping up this look at the "Data Pyramid for Embodied Manipulation: A Survey" paper by these authors, and what it really means for how we build robot models and systems.

Dev: They lay out this five layer structure of data sources, showing how they fit together to support different types of embodied foundation models.

Rosa: It's about organizing all that data—from real robot movements to general vision language stuff—based on the trade-off between being scalable and being accurate for robot alignment.

Dev: Right, so it’s not just one perfect dataset anymore; they show we need a whole ecosystem of complementary data types to build models that can both perceive the world and physically interact with it.

Rosa: The paper connects these data properties—like semantic coverage or physical fidelity—to what the models can actually do, such as planning or generating actions.

Dev: And they highlight some big challenges, like how hard it is to collect really large scale tactile datasets and failure recovery data for those real robot interactions.

Rosa: So the authors point out that there isn't one perfect recipe for all this data; the way we collect it needs to change depending on what kind of model you’re trying to train.

Dev: Exactly, and they suggest future work should focus on creating recipes that are aware of the specific architecture of these foundation models.

Rosa: It really shows how moving from just real robot trajectories toward mixing all five layers is where we need to go for more robust systems.

Dev: It sets up a lot of interesting questions about how we can best combine these different types of data for the next generation of embodied intelligence.

PKU University · NTU University of Science and Technology University of Science and Technology Hong Kong University of Science and Technology Hong Kong University of Science and Technology Hong Kong National University of Singapore Chinese University of Hong Kong The Duke Institute for Advanced Study UCB Graduate School National Grand Battery Unilink Shanghai Jiao Tong University

cs.RO, cs.CV

Submitted: 2026-07-27

Updated: 2026-11-17

Comments: Awesome Embodied Data Pyramid; Project Page at https://jasper-aaa.github.io/embodied-data-pyramid/ GitHub Repo at https://github.com/worldbench/awesome-embodied-data-pyramid

Code: https://github.com/Genesis-Embodied-AI/Genesis

Project page: https://genesis-embodied-ai.github.io

License: http://creativecommons.org/licenses/by-sa/4.0/

Importance score: 95/100

The gist: The work organizes embodied data into a "pyramid" spanning five complementary sources: real-robot data, UMI-style data, egocentric and exocentric data, simulation data…

Key concepts

Real-Robot Data
This is the highest fidelity data, consisting of direct recordings from a physical robot interacting with its environment. While it provides the most accurate representation of real dynamics and actions, collecting this data is very expensive and time-consuming to obtain.
UMI Data
This data allows for scalable collection in everyday environments without needing a robot. It retains supervision from an end-effector, making it a cost-effective way to complement expensive real-robot datasets by providing robot-free interaction examples.
Egocentric / Exocentric Data
These are authentic observations of the real world captured from different perspectives (first-person or third-person). They offer rich, diverse data about human and object interactions but lack the direct control signals needed for robotic execution.
Simulation Data
This involves creating synthetic environments where robots can practice actions cheaply and safely. It is excellent for scaling up training because it allows for privileged labeling, though its physical fidelity might not perfectly match real-world interactions.

Terminology

Summary

The work organizes embodied data into a pyramid spanning five complementary sources: real-robot data, UMI-style data, egocentric and exocentric data, simulation data, and general vision-language (VLA) data. This organization is structured around the tension between scalability and robot alignment. p. 0

Where the paper says it

In this work, we organize the embodied data ecosystem as a “pyramid” spanning five complementary sources: 1 real-robot data, 2UMI-style data, 3 egocentric and exocentric data, 4 simulation data, and 5general vision-language data. We organize the pyramid around the tension between scalability and robot alignment, and further characterize each source in terms of data quality, diversity, reusability, and physical fidelity.

Each source is characterized by metrics such as quality (validity, consistency), diversity (coverage across tasks/objects/scenes), reusability (transfer across tasks/embodiments), and physical fidelity (faithfulness to real interaction dynamics). p. 2
Where the paper says it

Quality concerns the validity, consistency, informativeness, and task relevance of the collected experience; simply increasing data volume may provide limited benefit when trajectories are repetitive, annotations are inaccurate, signals are poorly synchronized, or collection conditions are systematically biased [211]. Diversity reflects coverage across tasks, objects, scenes, viewpoints, instructions, embodiments, sensing configurations, behaviors, and outcomes, and is important for robustness under distribution shift [64, 113].

The five layers are ordered from apex to base:

  1. Real-Robot Data: Directly executable; offers high fidelity but is expensive to collect.

  2. UMI Data: Allows robot-free collection in everyday environments while retaining end-effector supervision, serving as a scalable complement to real-robot data.

  3. Egocentric / Exocentric Data: Provides authentic, diverse real-world observations but lacks robot proprioception and direct execution signals.

  4. Simulation Data: Offers scalable and controllable experience with privileged labels at low cost, though it is limited by the sim-to-real gap in physical fidelity compared to real data.

  5. General Data: Web-scale corpora that ground perception and reasoning but lack direct robot-environment interaction, serving as a broad prior over semantics and reasoning.

The pyramid reflects a transition from strong physical grounding (real-robot data) toward greater accessibility and scalability (general data). p. 3

Where the paper says it

From the apex to the base, we order the five categories as real-robot data, UMI-style data (Universal Manipulation Interface), egocentric and exocentric data, simulation data, and general data. This ordering reflects an overall transition from stronger physical grounding and more directly actionable robot supervision toward greater accessibility and scalability.

This heterogeneous composition supports different embodied foundation models: embodied brain models use broad multimodal data for perception/reasoning; vision-language-action (VLA) models require robot-compatible action supervision; and world action models (WAMs) exploit action-free temporal data. p. 4
Where the paper says it

Embodied brain models primarily use broad multimodal and embodied data to support perception, spatial and temporal reasoning, affordance understanding, memory, and high-level planning [74, 158, 258]. Visionlanguage-action models require robot-compatible action supervision to map observations and instructions to executable behaviors [20, 23, 174].

The analysis connects dataset properties—including semantic coverage, temporal structure, action supervision, embodiment alignment, diversity, and physical fidelity—to model capabilities such as perception, reasoning, planning, action generation, and world prediction. p. 6

Where the paper says it

We examine how heterogeneous data sources support embodied vision-language models, vision-language-action models, and world action models. Our analysis connects dataset properties, including semantic coverage, temporal structure, action supervision, embodiment alignment, diversity, and physical fidelity, with model capabilities such as perception, reasoning, planning, action generation, and world prediction.

Key challenges identified include building large-scale tactile datasets and collecting failure/recovery data. p. 0
Where the paper says it

For embodied brain models, vision-language-action models, and world-action models alike, we relate data composition to capabilities in perception, reasoning, planning, action generation, and world prediction. We close by discussing six open challenges: building large-scale tactile datasets, collecting failure and recovery data, developing scalable data-collection pipelines, aligning actions across embodiments, leveraging egocentric data for dexterous manipulation, and designing principled data recipes for robot learning.

The optimal data recipe remains unresolved; future work should investigate architecture-aware recipes. p. 0
Where the paper says it

We close by discussing six open challenges: building large-scale tactile datasets, collecting failure and recovery data, developing scalable data-collection pipelines, aligning actions across embodiments, leveraging egocentric data for dexterous manipulation, and designing principled data recipes for robot learning. We hope this work paves the foundation for the design of next-generation embodied systems.

The evolution of data utilization shows a shift from early reliance on real-robot trajectories toward mixtures of all five layers. p. 5

Where the paper says it

Evolution of data utilization in representative embodied foundation models. Early systems primarily rely on real-robot trajectories, whereas recent systems increasingly explore co-training with real-robot, egocentric, simulation, general, and UMI-style data.

Action-labeled data are crucial for training VLA models as they explicitly specify executable behaviors, often by discretizing continuous actions into tokens. p. 4
Where the paper says it

Embodied brain models primarily use broad multimodal and embodied data to support perception, spatial and temporal reasoning, affordance understanding, memory, and high-level planning [74, 158, 258]. Visionlanguage-action models require robot-compatible action supervision to map observations and instructions to executable behaviors [20, 23, 174].

Conversely, action-free data provide scalable observational priors, which can be leveraged via action proxies (e. p. 4
Where the paper says it

Visionlanguage-action models require robot-compatible action supervision to map observations and instructions to executable behaviors [20, 23, 174]. World action models additionally exploit action-free temporal data, action-conditioned interaction, and synthetic experience to model environment evolution and action conse5

g., point trajectories) or used to train WAMs that model world dynamics. p. 2
Where the paper says it

Embodied agents must additionally understand physical states and dynamics, reason about how actions transform the environment, and execute appropriate behaviors in the physical world [20, 23, 174, 258]. These requirements fundamentally change the nature of supervision needed for pretraining and raise a central question: What data should be used to train embodied foundation models with such capabilities?

Egocentric data is increasingly important as a bridge between general web data and embodiment-specific trajectories due to its rich coverage of human-object interactions. p. 2
Where the paper says it

The community has explored several complementary sources of supervision for embodied foundation models. These include observation-action trajectories collected from physical robots and simulation, egocentric and exocentric recordings of human interaction, general image, video, language, and vision-language data, and, more recently, UMI-style demonstrations that capture object- and end-effector-centric manipulation without requiring a robot during collection [23, 24, 26, 62, 119, 359].

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Development of a data pyramid taxonomy guided by scalability and robot alignment will allow researchers to systematically categorize data sources into five complementary categories (real-robot, UMI-style, egocentric/exocentric, simulation, and general) based on how efficiently they can be expanded and how directly they support physical robot learning.

  2. Embodied foundation models can achieve superior perception and reasoning capabilities by utilizing the Data Composition and Action Representation framework to strategically select and mix data from different pyramid layers based on the model's specific needs, such as using general data for foundational vision-language skills and real-robot data for grounding actions.

  3. Vision-Language-Action models can be improved by aligning action representations across different embodiments, reconciling differences in action dimensionality, control interfaces, and action semantics through methods like MinMax normalization to enable policies learned from one platform to be deployed on another.

  4. World Action Models can enhance environmental modeling by leveraging action-free temporal data and action-conditioned interaction from simulation or egocentric sources to model environment evolution more effectively than relying solely on rigid physics simulators.

  5. Dexterous manipulation policies can be refined by integrating egocentric priors for hand policy learning, using data from egocentric and exocentric sources to capture natural human demonstrations at scale without requiring execution on a target robot.

  6. Scalable data collection pipelines will be improved by implementing Human-in-the-loop enhancement, such as ThriftyDAgger or CR-DAgger, which selectively augments datasets with failure and recovery trajectories, transforming errors encountered during deployment into targeted training data.

  7. Cross-embodiment state-action alignment can be enhanced by using Egocentric Priors for Dexterous Hand Policy Learning to map human interaction experience to robot manipulation by utilizing techniques like EgoMimic or H-RDT to normalize demonstrations for joint policy training.

  8. Simulation data will be more effective as a scalable complement by employing world models as simulators, which generate imagined interaction trajectories conditioned on observations and actions, allowing for policy optimization without repeated physical interaction.

  9. Sim-to-real transfer performance can be increased by addressing observation mismatch and dynamic gaps through the use of digital twin reconstruction and careful modeling of actuator latency, compliance, torque saturation, backlash in simulation backends.

Abstract

Multimodal foundation models learned to see and to speak by consuming the whole internet. Embodied agents admit no such shortcut, since they require data that couple observations with physical states and actions. These signals can be provided, to varying degrees, by multiple data sources. In this work, we organize the embodied data ecosystem as a "pyramid'' spanning five complementary sources: real-robot data, UMI-style data, egocentric and exocentric data, simulation data, and general vision-language data. We organize the pyramid around the tension between scalability and robot alignment, and further characterize each source in terms of data quality, diversity, reusability, and physical fidelity. We then analyze recent embodied foundation models through the lens of data recipes, examining how different sources are selected, aligned, and mixed during pretraining. For embodied brain models, vision-language-action models, and world-action models alike, we relate data composition to capabilities in perception, reasoning, planning, action generation, and world prediction. To facilitate community use, we additionally consolidate over 10,000 hours of trajectories from widely used robot datasets into a unified LeRobot format, with transparent frame-level quality annotations for filtering. We close by discussing six open challenges: building large-scale tactile datasets, collecting failure and recovery data, developing scalable data-collection pipelines, aligning actions across embodiments, leveraging egocentric data for dexterous manipulation, and designing principled data recipes for robot learning. We hope this work lays the foundation for the design of next-generation embodied systems.

Sources

Related papers