EgoHTR: Egocentric 4D Demonstrations of Human Terrain Traversal

arXiv:2607.13472 · cs.RO, cs.CV · Submitted 2026-07-15 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: We're looking at the paper: "EgoHTR: Egocentric 4D Demonstrations of Human Terrain Traversal".

Dev: Here's the gist of the paper. Deploying humanoid robots in unstructured terrain remains an open problem While classic reinforcement learning struggles with the sheer complexity of real-world interactions, more promising methods leveraging human priors remain limited to models lacking contextual awareness The resulting dataset comprises over 150k frames, which we evaluate against motion-capture ground truth, demonstrating state-of-the-art accuracy and establishing a rigorous benchmark for human motion analysis and synthesis Further, we leverage this data to train perceptive locomotion policies, demonstrating hardware deployment on a Unitree G1 for reconstructed reference motions Dataset Introduction The Egocentric HumanTerrain Reconstruction (EgoHTR) dataset was introduced to bridge the gap between humanoid learning and scene reconstruction by providing 55 scene-aligned 4D human motion sequences in diverse, complex environments using a multi-sensor setup EgoHTR features 7 unstructured scenes, 8 subjects, and 55 diverse multi-modal sequences focused on rough terrain traversal, totaling 150 k frames with an average sequence length of 90 s The dataset includes modalities such as light blue ego- and exocentric Aria video and SLAM streams, a 3D mesh and point cloud, a parametrized model, and raw IMU Data Acquisition Pipeline The data acquisition pipeline is based on a multi-modal sensor suite consisting of (1) a head-mounted multi-sensor Project Aria glasses Gen.

Rosa: 1, (2) an IMU-based Rokoko Pro motion capture suit, and (3) a high-resolution Leica BLK2GO 3D scene scanner This setup is portable and designed to capture data in any unconstrained, real-world environment The reconstruction follows three stages: (I) body model parametrization using the SMPL-X model, (II) temporal synchronization of all sensor streams using a hand clap, and (III) spatial alignment in a common world frame Spatial alignment involves anchoring the body model kinematics to the closed-loop trajectory provided by the Aria glasses and refining it via Iterative Closest Point (ICP) against the dense environmental scan Dataset Evaluation Quantitative evaluation shows that EgoHTR reports an MPJPE of 73.2 mm and a PA-MPJPE of 54.3 mm for local pose estimation Compared to the strongest baseline, SLOPER4D

30: , EgoHTR achieves a 6.2% lower MPJPE and a 2.0% lower PA-MPJE Crucially, this performance improvement is achieved under substantially harder conditions where EgoHTR captures highly dynamic motions (e.g., backflips, parkour) across severe rough terrain The dataset serves as a foundational resource for training context-aware locomotion policies where current methods fail Applications and Utility The dataset enables the training of whole-body mimic policies on sparse, complex terrain by providing terrain-aligned reference motions together with per-frame foot-contact annotations on non-primitive terrain In simulation, a Unitree G1 humanoid robot is trained to track these human motion references using a policy that observes proprioception and receives a temporal foot-contact reward The dataset also facilitates comprehensive benchmarking of 4D human-scene reconstruction across exocentric, egocentric, and inertial paradigms Limitations Current limitations include the restriction of the data pipeline to static environments devoid of articulated objects Furthermore, the pipeline performance is subject to the physical constraints of hardware and potential failures in featureless environments or during high-acceleration maneuvers The dataset also lacks scale and diversity for comprehensive large-scale training The current data pipeline is restricted to static environments devoid of articulated objects EgoHTR is the first dataset providing 4D human-scene motion on rough terrain under in-the-wild conditions By open-sourcing our capture and reconstruction pipeline, we aim to build up a dataset capable of training context-aware foundation models for robot learning The dataset establishes a challenging benchmark for state-of-the-art human mesh recovery References

1: Y.

Dev: Zhang, Y.

Rosa: Seo, J.

Paper summary: Rosa: So, to wrap up on "EgoHTR: Egocentric 4D Demonstrations of Human Terrain Traversal", the paper is essentially presenting the first dataset providing four-dimensional human-scene motion on rough terrain under in the wild conditions <ref:2607.13472#pg1,EgoHTR: Egocentric 4D Demonstrations of Human Terrain Traversal>.

Dev: It’s about open-sourcing their capture and reconstruction pipeline so that we can build up a dataset capable of training context aware foundation models for robot learning.

Taro: This work establishes a challenging benchmark for state of the art human mesh recovery, which is important because it pushes what we think is achievable in motion tracking.

Rosa: The implication here is that by providing this data, we’re moving past indoor or controlled settings to something much more realistic for how robots need to learn to move around complex landscapes.

Dev: It sets a high bar for the accuracy of reconstructing human motion when you combine multiple sensor inputs like video, depth, and IMU.

Taro: If this reconstruction quality holds up when we feed it into locomotion policies, it could mean that humanoid robots can start learning to traverse really difficult outdoor terrain gracefully.

Rosa: That’s the big picture: moving toward systems that understand context rather than just following preprogrammed paths in simple environments.

Conclusion: Rosa: So, EgoHTR is about giving us those four-dimensional motion demonstrations of human terrain traversal using this new dataset.

Dev: It’s important to know who put this together—Yuan, Zhang, Chen and others—they built a pipeline that combines video with depth and IMU data.

Taro: They captured one hundred fifty k frames across seven scenes and eight subjects in rough environments like backflips or parkour.

Rosa: The evaluation showed state-of-the-art accuracy when we compared it to motion capture ground truth, hitting an MPJPE of seventy-three point two millimeters for local pose estimation.

Dev: That’s a six point two percent lower MPJPE than the strongest baseline they mentioned, SLOPER4D.

Taro: The thing that really stands out is that this accuracy holds up even when the motions are highly dynamic and the terrain is severe.

Rosa: It seems like this dataset is meant to be a foundation for training those context-aware locomotion policies we’ve been struggling with.

Dev: Exactly, it gives us terrain-aligned reference motions along with per-frame foot-contact annotations on non-primitive ground.

Taro: That means in simulation, we can train a robot like the Unitree G1 to track these human motions by observing proprioception and getting a reward for every foot contact.

Rosa: It opens up a way to test how much of this data actually helps when the world misbehaves outside of controlled settings.

Dev: The limitation they point out is that the current pipeline is restricted to static environments without articulated objects, which means we need more diversity there.

Taro: They also flag issues with hardware constraints and performance in featureless areas or during high-acceleration maneuvers.

Rosa: Despite those limitations, the authors are open-sourcing their capture and reconstruction pipeline for building up a larger dataset.

Dev: So they’re aiming to build a resource that helps train foundation models for robot learning in real-world conditions.

Taro: This work establishes a challenging benchmark for how accurately we can reconstruct human motion in complex 4D scenes <ref:2607.13472#pg1>.

ETH Zurich · ×Stanford University (implied by context) · “UC Berkeley” University (implied by context) · □TU Munich

cs.RO, cs.CV

Submitted: 2026-07-15

Updated: 2026-10-29

Comments: Project webpage: https://egohtr.github.io

Code: https://github.com/YanjieZe/GMR

Project page: https://egohtr.github.io

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 94/100

The gist: Deploying humanoid robots in unstructured terrain remains an open problem While classic reinforcement learning struggles with the sheer complexity of real-world interactions, more promising methods

Key concepts

EgoHTR Dataset
This dataset consists of 55 scene-aligned 4D human motion sequences across 7 unstructured scenes, totaling 150k frames. It combines various data types—video, SLAM streams, and point clouds—to capture how humans move while traversing rough terrain in real-world settings.
Multi-modal Sensor Suite
The data acquisition pipeline uses a combination of sensors: Project Aria glasses for ego/exocentric video and SLAM, an IMU suit for motion capture, and a Leica BLK2GO 3D scanner. This setup captures rich environmental context alongside the human's movement.
Spatial Alignment
This process anchors the human body model (using SMPL-X) to the environment. It involves aligning the body's kinematics with trajectories from the Aria glasses and refining this alignment using Iterative Closest Point (ICP) against a dense 3D environmental scan.

Terminology

Summary

Deploying humanoid robots in unstructured terrain remains an open problem While classic reinforcement learning struggles with the sheer complexity of real-world interactions, more promising methods leveraging human priors remain limited to models lacking contextual awareness The resulting dataset comprises over 150k frames, which we evaluate against motion-capture ground truth, demonstrating state-of-the-art accuracy and establishing a rigorous benchmark for human motion analysis and synthesis Further, we leverage this data to train perceptive locomotion policies, demonstrating hardware deployment on a Unitree G1 for reconstructed reference motions

Dataset Introduction

The Egocentric HumanTerrain Reconstruction (EgoHTR) dataset was introduced to bridge the gap between humanoid learning and scene reconstruction by providing 55 scene-aligned 4D human motion sequences in diverse, complex environments using a multi-sensor setup EgoHTR features 7 unstructured scenes, 8 subjects, and 55 diverse multi-modal sequences focused on rough terrain traversal, totaling 150 k frames with an average sequence length of 90 s The dataset includes modalities such as light blue ego- and exocentric Aria video and SLAM streams, a 3D mesh and point cloud, a parametrized model, and raw IMU

Data Acquisition Pipeline

The data acquisition pipeline is based on a multi-modal sensor suite consisting of (1) a head-mounted multi-sensor Project Aria glasses Gen. 1, (2) an IMU-based Rokoko Pro motion capture suit, and (3) a high-resolution Leica BLK2GO 3D scene scanner This setup is portable and designed to capture data in any unconstrained, real-world environment The reconstruction follows three stages: (I) body model parametrization using the SMPL-X model, (II) temporal synchronization of all sensor streams using a hand clap, and (III) spatial alignment in a common world frame Spatial alignment involves anchoring the body model kinematics to the closed-loop trajectory provided by the Aria glasses and refining it via Iterative Closest Point (ICP) against the dense environmental scan

Dataset Evaluation

Quantitative evaluation shows that EgoHTR reports an MPJPE of 73.2 mm and a PA-MPJPE of 54.3 mm for local pose estimation Compared to the strongest baseline, SLOPER4D [30], EgoHTR achieves a 6.2% lower MPJPE and a 2.0% lower PA-MPJE Crucially, this performance improvement is achieved under substantially harder conditions where EgoHTR captures highly dynamic motions (e.g., backflips, parkour) across severe rough terrain The dataset serves as a foundational resource for training context-aware locomotion policies where current methods fail

Applications and Utility

The dataset enables the training of whole-body mimic policies on sparse, complex terrain by providing terrain-aligned reference motions together with per-frame foot-contact annotations on non-primitive terrain In simulation, a Unitree G1 humanoid robot is trained to track these human motion references using a policy that observes proprioception and receives a temporal foot-contact reward The dataset also facilitates comprehensive benchmarking of 4D human-scene reconstruction across exocentric, egocentric, and inertial paradigms

Limitations

Current limitations include the restriction of the data pipeline to static environments devoid of articulated objects Furthermore, the pipeline performance is subject to the physical constraints of hardware and potential failures in featureless environments or during high-acceleration maneuvers The dataset also lacks scale and diversity for comprehensive large-scale training The current data pipeline is restricted to static environments devoid of articulated objects

Conclusion

EgoHTR is the first dataset providing 4D human-scene motion on rough terrain under in-the-wild conditions By open-sourcing our capture and reconstruction pipeline, we aim to build up a dataset capable of training context-aware foundation models for robot learning The dataset establishes a challenging benchmark for state-of-the-art human mesh recovery

References

[1] Y. Zhang, Y. Seo, J. Chen, Y. Yuan, K. Sreenath, P. Abbeel, C. Sferrazza, K. Liu, R. Duan, and G. Shi., Terrain Traversal., 4D Reconstruction,, Human Motion Dataset,, Mimic

[2] N. Rudin, J. He, J. Aurand, and M. Hutter., Parkour in the wild: Learning a general and extensible agile locomotion policy using multi-expert distillation and rl fine-tuning., arXiv preprint arXiv:2505.11164,, 2

[3] J. Sun, G. Han, P. Sun, W. Zhao, J. Cao, J. Wang, Y. Guo, and Q. Zhang., Dpl: Depthonly perceptive humanoid locomotion via realistic depth synthesis and cross-attention terrain reconstruction., arXiv preprint arXiv:2510.07152,, 3

[4] J. Long, J. Ren, M. Shi, Z. Wang, T. Huang, P. Luo, and J. Pang., Learning humanoid locomotion with perceptive internal model., 2025 IEEE International Conference on Robotics and Automation (ICRA), pages 9997–10003,, 4

[5] Z. Fu, Q. Zhao, Q. Wu, G. Wetzstein, and C. Finn., Humanplus: Humanoid shadowing and imitation from humans., In Conference on Robot Learning (CoRL), 2024,, 5

[6] K. Yin, W. Zeng, K. Fan, M. Dai, Z. Wang, Q. Zhang, Z. Tian, J. Wang, J. Pang., and W. Zhang., Unitracker: Learning universal whole-body motion tracker for humanoid robots., 2025 IEEE International Conference on Robotics and Automation (ICRA), pages 9997–10003,, 6

[7] Q. Liao, T. E. Truong, X. Huang, Y. Gao, G. Tevet, K. Sreenath., and C. K. Liu., Beyondmimic: From motion tracking to versatile humanoid control via guided diffusion., 2025 IEEE International Conference on Robotics and Automation (ICRA), pages 9997–10003,, 7

[8] Z. Chen, M. Ji, X. Cheng, X. Peng, X. B. Peng, and X. Wang., Gmt: General motion tracking for humanoid whole-body control., arXiv:2506.14770,, 8

[9] S. Zhao, Y. Ze, Y. Wang, C.-K Liu, P.-Abbeel, G. Shi., and R. Duan., Resmimic: From general motion tracking to humanoid whole-body loco-manipulation via residual learning., 2025 IEEE International Conference on Robotics and Automation (ICRA), pages 9997–10003,, 9

[10] Z. Zhang, J. Guo, C. Chen, J. Wang, C. Lin, Y. Lian, H. Xue, Z.-Wang., M.-Liu., J.-Lyu., H.-Liu., H.-Wang., and L.-Yi., Track any motions under any disturbances,, 2025 IEEE International Conference on Robotics and Automation (ICRA), pages 9997–10003,, 10

[11] Z. Luo, Y. Yuan, T. Wang, C. Li, S. Chen, F. Castaneda, Z.-A Cao., J. Li., D.-Minor., Q.-Ben., ˜X Da,, Sonic: Supersizing motion tracking for natural humanoid whole-body control,, 2025 IEEE International Conference on Robotics and Automation (ICRA), pages 9997–10003,, 11

[12] A. Allshire, H. Choi, J. Zhang, D. McAllister, A. Zhang, C.-M Kim, T. Darrell., P.-Abbeel, J. Malik., and A Kanazawa., Visual imitation enables contextual humanoid control., In Proceedings of the Conference on Robot Learning (CoRL), 2025,, 12

[13] Q. Zhang, J. Ma, P. Liu, S. Shi, Z. Su, Z.-Wang, J.-Sun., W Cui., J Yu., G Han., W Zhao., P Sun., K Yin., J Wang., J Cao., L Zhang., H Cheng,, X Hao,, Y Ji,, and J Liang.,Meshmimic: Geometry-aware humanoid motion learning through 3d scene reconstruction, 2026 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5152–5161,, 13

[14] W. Wang, L. Pan, H. Pi, Y Lou, X Ren, Y Wu, Z Liao., L Yang., R Dabral.

Improvements for AI systems

  1. Develop perceptive locomotion policies for humanoid robots that can reliably traverse uneven terrain by conditioning them on high-fidelity, scene-aligned reference motions from EgoHTR. This enables robots to move beyond flat ground and discrete obstacles by using terrain-aligned reference motions together with per-frame foot-contact annotations on non-primitive terrain.

  2. Enhance human mesh recovery systems to produce physically plausible interactions in the wild by leveraging multi-modal data fusion across egocentric, exocentric, and inertial paradigms. This allows for the reconstruction of physically plausible real-world interactions which existing unified vision models struggle with.

  3. Create a benchmark for evaluating human pose and shape estimation (HPS) and 4D human-scene reconstruction on rough terrain by utilizing EgoHTR's metrics, such as MPJPE, PA-MPJPE, W-MPJPE, and RTE. This establishes a standard to determine if foothold-critical tasks strictly require human-terrain coupling.

  4. Improve motion generation systems to create contextually aware locomotion policies by conditioning them on high-fidelity human motion data that specifies both foot placement and load bearing. This overcomes the limitation where existing generators fail entirely on unstructured environment, where continuous, high-resolution environmental geometry must dictate precise foot placement.

  5. Establish a scalable 4D human-scene reconstruction pipeline that balances deployment usability with reconstruction accuracy using a portable multi-sensor setup (Aria glasses, MoCap suit, 3D scanner). This pipeline enables centimeter-precision 4D human-scene reconstruction of dynamic trajectories in unconstrained environments.

Abstract

Deploying humanoid robots in unstructured terrain remains an open problem. While classic reinforcement learning struggles with the sheer complexity of real-world interactions, more promising methods leveraging human priors remain limited to models lacking contextual awareness. The restricted motion synthesis is a direct consequence of existing dataset pipelines failing to capture human-scene sequences in challenging environments. To bridge this gap between humanoid learning and scene reconstruction, we introduce the Egocentric Human-Terrain Reconstruction (EgoHTR) dataset. We develop and open-source a reconstruction pipeline capturing 55 scene-aligned 4D human motion sequences in diverse, complex environments using a multi-sensor setup of egocentric wearables and a portable 3D scanner. The resulting dataset comprises over 150k frames, which we evaluate against motion-capture ground truth, demonstrating state-of-the-art accuracy and establishing a rigorous benchmark for human motion analysis and synthesis. Further, we leverage this data to train perceptive locomotion policies, demonstrating hardware deployment on a Unitree G1 for reconstructed reference motions. Our pipeline enables community-driven dataset extensions and factors the problem to help researchers build foundational, context-aware robots that reliably traverse uneven terrain.

Sources

Related papers