Data Pyramid for Embodied Manipulation: A Survey
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.
Rosa: We're looking at the paper: "Data Pyramid for Embodied Manipulation: A Survey".
Paper summary: Rosa: To wrap up, the paper "Data Pyramid for Embodied Manipulation: A Survey" lays out this five layer structure of data sources to handle the requirements of embodied foundation models.
Dev: The authors organize this pyramid around the tension between scalability and robot alignment, showing how different data types fit together depending on what you need from a model.
Rosa: What it means in simple terms is that we don't just need one perfect dataset anymore; we need a whole ecosystem of complementary data sources to build models capable of understanding the world and interacting with it physically.
Dev: They discuss how these data properties map onto model capabilities like perception reasoning planning and action generation, but they also point out the practical hurdles, especially building those large scale tactile datasets.
Rosa: The authors conclude that the optimal way to collect this data recipe is still undecided, suggesting future research should focus on recipes that are aware of the architecture of the models they are training.
Dev: Ultimately, this survey provides a roadmap for how different embodied foundation models can be supported by a rich and heterogeneous collection of data from real robot interaction all the way down to general vision language data.
Conclusion: Rosa: So we're wrapping up this look at the "Data Pyramid for Embodied Manipulation: A Survey" paper by these authors, and what it really means for how we build robot models and systems.
Dev: They lay out this five layer structure of data sources, showing how they fit together to support different types of embodied foundation models.
Rosa: It's about organizing all that data—from real robot movements to general vision language stuff—based on the trade-off between being scalable and being accurate for robot alignment.
Dev: Right, so it’s not just one perfect dataset anymore; they show we need a whole ecosystem of complementary data types to build models that can both perceive the world and physically interact with it.
Rosa: The paper connects these data properties—like semantic coverage or physical fidelity—to what the models can actually do, such as planning or generating actions.
Dev: And they highlight some big challenges, like how hard it is to collect really large scale tactile datasets and failure recovery data for those real robot interactions.
Rosa: So the authors point out that there isn't one perfect recipe for all this data; the way we collect it needs to change depending on what kind of model you’re trying to train.
Dev: Exactly, and they suggest future work should focus on creating recipes that are aware of the specific architecture of these foundation models.
Rosa: It really shows how moving from just real robot trajectories toward mixing all five layers is where we need to go for more robust systems.
Dev: It sets up a lot of interesting questions about how we can best combine these different types of data for the next generation of embodied intelligence.
PKU University · NTU University of Science and Technology University of Science and Technology Hong Kong University of Science and Technology Hong Kong University of Science and Technology Hong Kong National University of Singapore Chinese University of Hong Kong The Duke Institute for Advanced Study UCB Graduate School National Grand Battery Unilink Shanghai Jiao Tong University
cs.RO, cs.CV
Submitted: 2026-07-27
Updated: 2026-11-17
Comments: Awesome Embodied Data Pyramid; Project Page at https://jasper-aaa.github.io/embodied-data-pyramid/ GitHub Repo at https://github.com/worldbench/awesome-embodied-data-pyramid
Code: https://github.com/Genesis-Embodied-AI/Genesis
Project page: https://genesis-embodied-ai.github.io
License: http://creativecommons.org/licenses/by-sa/4.0/
Importance score: 95/100
The gist: The work organizes embodied data into a "pyramid" spanning five complementary sources: real-robot data, UMI-style data, egocentric and exocentric data, simulation data…
Key concepts
- Real-Robot Data
- This is the highest fidelity data, consisting of direct recordings from a physical robot interacting with its environment. While it provides the most accurate representation of real dynamics and actions, collecting this data is very expensive and time-consuming to obtain.
- UMI Data
- This data allows for scalable collection in everyday environments without needing a robot. It retains supervision from an end-effector, making it a cost-effective way to complement expensive real-robot datasets by providing robot-free interaction examples.
- Egocentric / Exocentric Data
- These are authentic observations of the real world captured from different perspectives (first-person or third-person). They offer rich, diverse data about human and object interactions but lack the direct control signals needed for robotic execution.
- Simulation Data
- This involves creating synthetic environments where robots can practice actions cheaply and safely. It is excellent for scaling up training because it allows for privileged labeling, though its physical fidelity might not perfectly match real-world interactions.
Terminology
Summary
The work organizes embodied data into a In this
work, we organize the embodied data ecosystem as a “pyramid” spanning five complementary sources:
1
real-robot data, 2UMI-style data, 3
egocentric and exocentric data, 4
simulation data, and 5general
vision-language data. We organize the pyramid around the tension between scalability and robot
alignment, and further characterize each source in terms of data quality, diversity, reusability, and
physical fidelity. Quality concerns the validity, consistency,
informativeness, and task relevance of the collected experience; simply increasing data volume may provide
limited benefit when trajectories are repetitive, annotations are inaccurate, signals are poorly synchronized,
or collection conditions are systematically biased [211]. Diversity reflects coverage across tasks, objects,
scenes, viewpoints, instructions, embodiments, sensing configurations, behaviors, and outcomes, and is
important for robustness under distribution shift [64, 113].pyramid
spanning five complementary sources: real-robot data, UMI-style data, egocentric and exocentric data, simulation data, and general vision-language (VLA) data. This organization is structured around the tension between scalability and robot alignment. p. 0Where the paper says it
Where the paper says it
The five layers are ordered from apex to base:
-
Real-Robot Data: Directly executable; offers high fidelity but is expensive to collect.
-
UMI Data: Allows robot-free collection in everyday environments while retaining end-effector supervision, serving as a scalable complement to real-robot data.
-
Egocentric / Exocentric Data: Provides authentic, diverse real-world observations but lacks robot proprioception and direct execution signals.
-
Simulation Data: Offers scalable and controllable experience with privileged labels at low cost, though it is limited by the sim-to-real gap in physical fidelity compared to real data.
-
General Data: Web-scale corpora that ground perception and reasoning but lack direct robot-environment interaction, serving as a broad prior over semantics and reasoning.
The pyramid reflects a transition from strong physical grounding (real-robot data) toward greater accessibility and scalability (general data). p. 3 From the apex to the base, we order the five categories as real-robot data, UMI-style data (Universal
Manipulation Interface), egocentric and exocentric data, simulation data, and general data. This ordering
reflects an overall transition from stronger physical grounding and more directly actionable robot supervision
toward greater accessibility and scalability. Embodied brain models primarily use broad multimodal and embodied data to support perception, spatial
and temporal reasoning, affordance understanding, memory, and high-level planning [74, 158, 258]. Visionlanguage-action models require robot-compatible action supervision to map observations and instructions
to executable behaviors [20, 23, 174].Where the paper says it
Where the paper says it
The analysis connects dataset properties—including semantic coverage, temporal structure, action supervision, embodiment alignment, diversity, and physical fidelity—to model capabilities such as perception, reasoning, planning, action generation, and world prediction. p. 6 We examine how heterogeneous data sources
support embodied vision-language models, vision-language-action models, and world action models. Our
analysis connects dataset properties, including semantic coverage, temporal structure, action supervision,
embodiment alignment, diversity, and physical fidelity, with model capabilities such as perception,
reasoning, planning, action generation, and world prediction. For
embodied brain models, vision-language-action models, and world-action models alike, we relate data
composition to capabilities in perception, reasoning, planning, action generation, and world prediction. We close by discussing six open challenges: building large-scale tactile datasets, collecting failure
and recovery data, developing scalable data-collection pipelines, aligning actions across embodiments,
leveraging egocentric data for dexterous manipulation, and designing principled data recipes for robot
learning. We close by discussing six open challenges: building large-scale tactile datasets, collecting failure
and recovery data, developing scalable data-collection pipelines, aligning actions across embodiments,
leveraging egocentric data for dexterous manipulation, and designing principled data recipes for robot
learning. We hope this work paves the foundation for the design of next-generation embodied systems.Where the paper says it
Where the paper says it
Where the paper says it
The evolution of data utilization shows a shift from early reliance on real-robot trajectories toward mixtures of all five layers. p. 5 Evolution of data utilization in representative embodied foundation models. Early
systems primarily rely on real-robot trajectories, whereas recent systems increasingly explore co-training with real-robot,
egocentric, simulation, general, and UMI-style data. Embodied brain models primarily use broad multimodal and embodied data to support perception, spatial
and temporal reasoning, affordance understanding, memory, and high-level planning [74, 158, 258]. Visionlanguage-action models require robot-compatible action supervision to map observations and instructions
to executable behaviors [20, 23, 174]. Visionlanguage-action models require robot-compatible action supervision to map observations and instructions
to executable behaviors [20, 23, 174]. World action models additionally exploit action-free temporal data,
action-conditioned interaction, and synthetic experience to model environment evolution and action conse5 Embodied agents must
additionally understand physical states and dynamics, reason about how actions transform the environment,
and execute appropriate behaviors in the physical world [20, 23, 174, 258]. These requirements fundamentally
change the nature of supervision needed for pretraining and raise a central question:
What data should be used to train embodied foundation models with such capabilities? The community has explored several complementary sources of supervision for embodied foundation models. These include observation-action trajectories collected from physical robots and simulation, egocentric and
exocentric recordings of human interaction, general image, video, language, and vision-language data, and,
more recently, UMI-style demonstrations that capture object- and end-effector-centric manipulation without
requiring a robot during collection [23, 24, 26, 62, 119, 359].Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Development of a data pyramid taxonomy guided by scalability and robot alignment will allow researchers to systematically categorize data sources into five complementary categories (real-robot, UMI-style, egocentric/exocentric, simulation, and general) based on how efficiently they can be expanded and how directly they support physical robot learning.
-
Embodied foundation models can achieve superior perception and reasoning capabilities by utilizing the
Data Composition and Action Representation
framework to strategically select and mix data from different pyramid layers based on the model's specific needs, such as using general data for foundational vision-language skills and real-robot data for grounding actions. -
Vision-Language-Action models can be improved by aligning action representations across different embodiments, reconciling differences in
action dimensionality, control interfaces, and action semantics
through methods like MinMax normalization to enable policies learned from one platform to be deployed on another. -
World Action Models can enhance environmental modeling by leveraging
action-free temporal data
andaction-conditioned interaction
from simulation or egocentric sources to model environment evolution more effectively than relying solely on rigid physics simulators. -
Dexterous manipulation policies can be refined by integrating egocentric priors for hand policy learning, using data from egocentric and exocentric sources to capture
natural human demonstrations at scale without requiring execution on a target robot.
-
Scalable data collection pipelines will be improved by implementing
Human-in-the-loop enhancement,
such as ThriftyDAgger or CR-DAgger, which selectively augments datasets with failure and recovery trajectories, transforming errors encountered during deployment into targeted training data. -
Cross-embodiment state-action alignment can be enhanced by using
Egocentric Priors for Dexterous Hand Policy Learning
to map human interaction experience to robot manipulation by utilizing techniques like EgoMimic or H-RDT to normalize demonstrations for joint policy training. -
Simulation data will be more effective as a scalable complement by employing world models as simulators, which generate
imagined interaction trajectories
conditioned on observations and actions, allowing for policy optimization without repeated physical interaction. -
Sim-to-real transfer performance can be increased by addressing observation mismatch and dynamic gaps through the use of
digital twin reconstruction
and careful modeling ofactuator latency, compliance, torque saturation, backlash
in simulation backends.
Abstract
Multimodal foundation models learned to see and to speak by consuming the whole internet. Embodied agents admit no such shortcut, since they require data that couple observations with physical states and actions. These signals can be provided, to varying degrees, by multiple data sources. In this work, we organize the embodied data ecosystem as a "pyramid'' spanning five complementary sources: real-robot data, UMI-style data, egocentric and exocentric data, simulation data, and general vision-language data. We organize the pyramid around the tension between scalability and robot alignment, and further characterize each source in terms of data quality, diversity, reusability, and physical fidelity. We then analyze recent embodied foundation models through the lens of data recipes, examining how different sources are selected, aligned, and mixed during pretraining. For embodied brain models, vision-language-action models, and world-action models alike, we relate data composition to capabilities in perception, reasoning, planning, action generation, and world prediction. To facilitate community use, we additionally consolidate over 10,000 hours of trajectories from widely used robot datasets into a unified LeRobot format, with transparent frame-level quality annotations for filtering. We close by discussing six open challenges: building large-scale tactile datasets, collecting failure and recovery data, developing scalable data-collection pipelines, aligning actions across embodiments, leveraging egocentric data for dexterous manipulation, and designing principled data recipes for robot learning. We hope this work lays the foundation for the design of next-generation embodied systems.
Sources
- TacSL: A Library for Visuotactile Sensor Simulation and Learning
- World Simulation with Video Foundation Models for Physical AI
- HapTile: A Haptic-Informed Vision-Tactile-Language-Action Dataset for Contact-Rich Imitation Learning
- Scalable Behavior Cloning with Open Data, Training, and Evaluation
- VQA: Visual Question Answering
- Green-VLA: Staged Vision-Language-Action Model for Generalist Robots
- V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning
- Introducing HOT3D: An Egocentric Dataset for 3D Hand and Object Tracking
- DexArt: Benchmarking Generalizable Dexterous Manipulation with Articulated Objects
- ARKitScenes: A Diverse Real-World Dataset For 3D Indoor Scene Understanding Using Mobile RGB-D Data
- DeepMind Lab
- Track2Act: Predicting Point Tracks from Internet Videos enables Generalizable Robot Manipulation
- RoboAgent: Generalization and Efficiency in Robot Manipulation via Semantic Augmentations and Action Chunking
- MapleGrasp: Mask-guided Feature Pooling for Language-driven Efficient Robotic Grasping
- Motus: A Unified Latent Action World Model
- GR00T N1: An Open Foundation Model for Generalist Humanoid Robots
- $\pi_0$: A Vision-Language-Action Flow Model for General Robot Control
- WEAR: An Outdoor Sports Dataset for Wearable and Egocentric Activity Recognition
- RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control
- RT-1: Robotics Transformer for Real-World Control at Scale
Related papers
- Rethinking Actions: Continuity-Aware Priors for World-Model Planning
- MOSAIC: Modular Obstacle and Spatial-Aware Framework for Manipulation In Cluttered Scenes
- SAGAS: Semantic-Aware Graph-Assisted Stitching for Offline Temporal Logic Planning
- DexSeed: Expanding Human Hand-Object Interactions into Robot Demonstrations for Dexterous Policy Learning
- LaST0.5: A Generalist World Action Model with Interaction-Centric Latent Reasoning
- CIALLO: Learning Shared Intent Representations for Cross-Embodiment Manipulation