Kinematic Interface for the Wild: Modular Bimanual Loco-Manipulation Capture from 360 Cameras Alone
cs.RO
Submitted: 2026-09-19
Updated: 2026-09-19
Comments: 10 pages, 9 figures
License: http://creativecommons.org/licenses/by/4.0/
The gist: A wrist-mounted camera for UMI-style data collection must do two jobs: record the manipulation and localize in the scene.
Terminology
Abstract
A wrist-mounted camera for UMI-style data collection must do two jobs: record the manipulation and localize in the scene. Most handheld devices localize online from workspace-facing views crowded by hands and objects, or add dedicated tracking hardware. Room-scale bimanual capture therefore still tends to instrument the operator or the scene for accurate localization. We present KIWI (Kinematic Interface for the Wild), a capture kit whose only electronics are off-the-shelf cameras. Our core system splits the two jobs across the two lenses of a 360-degree camera. The rear lens faces the room and builds a shared metric map that registers both hands, and an optional head camera, in one frame without workspace co-visibility; the front lens records the manipulation, and offline IMU fusion bridges front-lens tracking loss. Through our quick-release plate, the camera module attaches to chopstick grippers, parallel-jaw grippers, hand-wrist mounts, or robot flanges. Across six bimanual recordings, combining the rear and front lenses failed to localize only 0.1% of query frames, whereas front-only bimanual feature alignment failed on 24.8% of frames and lost one recording entirely; against evaluation fiducials, localization error stayed within 4.5 mm. KIWI's recovered poses were sufficiently consistent for the four wrist streams alone to reconstruct the scene as a 3D Gaussian splat.
Sources
- UMI-3D: Extending Universal Manipulation Interface from Vision-Limited to 3D Spatial Perception
- What Enables In-Context Behavior Prompting for Manipulation?
- RDT2: Exploring the Scaling Limit of UMI Data Towards Zero-Shot Cross-Embodiment Generalization
- YUBI: Yielding Universal Bidigital Interface for Bimanual Dexterous Manipulation at Scale
- EgoMI: Learning Active Vision and Whole-Body Manipulation from Egocentric Human Demonstrations
- HiFi-UMI: Learning Deployable Manipulation Policies from High-Fidelity UMI Data Alone
- XRZero-G0: Pushing the Frontier of Dexterous Robotic Manipulation with Interfaces, Quality and Ratios
- On Bringing Robots Home
Related papers
- FMT x: An Efficient and Asymptotically Optimal Extension of the Fast Marching Tree for Dynamic Replanning
- MPCFormer: A physics-informed data-driven approach for explainable socially-aware autonomous driving
- RoboLab: A High-Fidelity Simulation Benchmark for Analysis of Task Generalist Policies
- ARC: A Reasoning Recipe for Robot Foundation Models
- HRDexDB: A 4D Dexterous Grasping Dataset Across Human and Multiple Robot Embodiments
- APT: Action Expert Pretraining Improves Instruction Generalization of Vision-Language-Action Policies