Learning Geometrically-Grounded Amodal 3D Representations for View-Generalizable Robotic Manipulation
cs.RO
Submitted: 2026-01-30
Updated: 2026-09-27
License: http://creativecommons.org/licenses/by/4.0/
The gist: Real-world robotic manipulation requires visuomotor policies capable of robust 3D scene reasoning under varying camera viewpoints.
Terminology
Abstract
Real-world robotic manipulation requires visuomotor policies capable of robust 3D scene reasoning under varying camera viewpoints. While recent 3D-aware manipulation policies have shown promise, they still face several limitations: (i) reliance on multi-view observations during inference, which is impractical in camera-constrained deployments; (ii) insufficient geometric fidelity in learned representations, limiting precise and viewpoint-robust control; and (iii) the lack of effective mechanisms for transferring pretrained 3D representations to downstream visuomotor policies. To address these challenges, we present GEM3D (Geometrically-Grounded 3D Manipulation), a unified 3D representation and policy learning framework for view-generalizable robotic manipulation. GEM3D learns amodal 3D representations from single-view RGB-D observations by jointly enforcing point-cloud reconstruction and Gaussian-splatting-based novel-view consistency during pretraining. A multi-step distillation strategy then transfers the learned geometric understanding into a deployable single-view visuomotor policy for downstream manipulation control. Experiments on 12 RLBench tasks and three real-world tasks demonstrate strong manipulation performance. Further zero-shot evaluations on eight RLBench tasks and the real-world tasks show robust generalization to unseen camera poses and substantial gains over existing methods.
Sources
- PolarNet: 3D Point Clouds for Language-Guided Robotic Manipulation
- On Pre-Training for Visuo-Motor Control: Revisiting a Learning-from-Scratch Baseline
- Perceiver IO: A General Architecture for Structured Inputs & Outputs
- DINOv2: Learning Robust Visual Features without Supervision
- DreamFusion: Text-to-3D using 2D Diffusion
- RRL: Resnet as representation for Reinforcement Learning
- Zero123++: a Single Image to Consistent Multi-view Diffusion Base Model
- Masked Visual Pre-training for Motor Control
- DNAct: Diffusion Guided Multi-Task 3D Policy Learning
- Deformable DETR: Deformable Transformers for End-to-End Object Detection
Related papers
- FMT x: An Efficient and Asymptotically Optimal Extension of the Fast Marching Tree for Dynamic Replanning
- MPCFormer: A physics-informed data-driven approach for explainable socially-aware autonomous driving
- RoboLab: A High-Fidelity Simulation Benchmark for Analysis of Task Generalist Policies
- ARC: A Reasoning Recipe for Robot Foundation Models
- HRDexDB: A 4D Dexterous Grasping Dataset Across Human and Multiple Robot Embodiments
- APT: Action Expert Pretraining Improves Instruction Generalization of Vision-Language-Action Policies