World In Your Hands: A Large-Scale and Open-Source Ecosystem for Learning Human-Centric Manipulation in the Wild

arXiv:2512.24310 · cs.RO · Submitted 2025-12-30 · Read on arXiv

cs.RO

Submitted: 2025-12-30

Updated: 2026-09-21

Comments: This dataset represents the first large-scale collection of real-world, human-centric multimodal data integrating vision, language, tactile sensing, and action (VLTA) Github: https://github.com/tars-robotics/World-In-Your-Hands

Code: https://github.com/tars-robotics/World-In-Your-Hands

Project page: https://wiyh.tars-ai.com

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Terminology

• World In Your Hands — 27× in this paper
• Benchmark — 15× in this paper · explained in 2 episodes
• Accuracy — 14× in this paper · explained in 1 episode
• Consistency — 13× in this paper · explained in 3 episodes
• Spatial Referring — 10× in this paper
• Generalization — 7× in this paper · explained in 13 episodes
• Verification — 7× in this paper · explained in 2 episodes
• Calibration — 5× in this paper · explained in 7 episodes
• Fine-Tuning — 5× in this paper · explained in 5 episodes
• Transformer — 5× in this paper · explained in 2 episodes
• Gaussian Splatting — 4× in this paper · explained in 4 episodes
• Temporal consistency — 4× in this paper · explained in 2 episodes
• Alignment — 3× in this paper · explained in 1 episode
• Chain-of-Thought (CoT) — 3× in this paper · explained in 4 episodes
• Core — 3× in this paper · explained in 2 episodes
• Optical Flow — 3× in this paper · explained in 2 episodes
• Action Space — 2× in this paper · explained in 1 episode
• Causal Reasoning — 2× in this paper · explained in 2 episodes
• Fidelity — 2× in this paper · explained in 2 episodes
• Foundation Model — 2× in this paper · explained in 3 episodes
• Gaussian Splatting (GS) — 2× in this paper · explained in 3 episodes
• Task Decomposition — 2× in this paper · explained in 2 episodes
• Task Planning — 2× in this paper · explained in 2 episodes
• Action Flow — 1× in this paper · explained in 1 episode
• Adam — 1× in this paper · explained in 2 episodes
• Benchmarking — 1× in this paper · explained in 3 episodes
• Chain-of-Thought (CoT) Reasoning — 1× in this paper · explained in 2 episodes
• Embodied AI — 1× in this paper · explained in 2 episodes
• Few-Shot Learning — 1× in this paper · explained in 4 episodes
• Foundation Models — 1× in this paper · explained in 7 episodes
• Generative Model — 1× in this paper · explained in 2 episodes
• Generative Models — 1× in this paper · explained in 2 episodes
• Grounding — 1× in this paper · explained in 3 episodes
• Hallucination — 1× in this paper · explained in 3 episodes
• Learned Perceptual Image Patch Similarity — 1× in this paper
• Modular Architecture — 1× in this paper · explained in 4 episodes
• Peak Signal-to-Noise Ratio — 1× in this paper
• Robustness — 1× in this paper · explained in 5 episodes
• Structural Similarity Index Measure — 1× in this paper
• Unified Framework — 1× in this paper · explained in 5 episodes
• Vision-Language-Action (VLA) Models — 1× in this paper · explained in 2 episodes
• World Model — 1× in this paper · explained in 7 episodes

Sources

• GPT-4 Technical Report
• Qwen2.5-VL Technical Report
• Seed-Music: A Unified Framework for High Quality and Controlled Music Generation
• $\pi_0$: A Vision-Language-Action Flow Model for General Robot Control
• RT-1: Robotics Transformer for Real-World Control at Scale
• EgoDex: Learning Dexterous Manipulation from Large-Scale Egocentric Video
• DexMan: Learning Bimanual Dexterous Manipulation from Human and Generated Videos
• GPT-4o System Card
• OpenVLA: An Open-Source Vision-Language-Action Model
• Segment Anything
• LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models
• ViPRA: Video Prediction for Robot Actions
• LLaMA: Open and Efficient Foundation Language Models
• HO-Cap: A Capture System and Dataset for 3D Reconstruction and Pose Tracking of Hand-Object Interaction
• ActionCLIP: A New Paradigm for Video Action Recognition
• EgoVid-5M: A Large-Scale Video-Action Dataset for Egocentric Video Generation
• Diffusion-VLA: Generalizable and Interpretable Robot Foundation Model via Self-Generated Reasoning
• VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding
• DexUMI: Using Human Hand as the Universal Manipulation Interface for Dexterous Manipulation
• Qwen3 Technical Report

Related papers

• FMT x: An Efficient and Asymptotically Optimal Extension of the Fast Marching Tree for Dynamic Replanning
• MPCFormer: A physics-informed data-driven approach for explainable socially-aware autonomous driving
• RoboLab: A High-Fidelity Simulation Benchmark for Analysis of Task Generalist Policies
• ARC: A Reasoning Recipe for Robot Foundation Models
• HRDexDB: A 4D Dexterous Grasping Dataset Across Human and Multiple Robot Embodiments
• APT: Action Expert Pretraining Improves Instruction Generalization of Vision-Language-Action Policies