KeyGen: Unsupervised Keypoint based Object-Centric Representations for Category-Level Policy Generalization
cs.RO, cs.AI
Submitted: 2026-09-23
Updated: 2026-09-23
License: http://creativecommons.org/licenses/by/4.0/
The gist: Generalization in robotic manipulation requires policies to perform tasks across diverse unseen object instances that vary in shape, size, and pose.
Terminology
Abstract
Generalization in robotic manipulation requires policies to perform tasks across diverse unseen object instances that vary in shape, size, and pose. However, conventional behavior cloning (BC) methods often overfit to instance-specific geometry and appearance, limiting transfer to novel objects. We introduce KeyGen, a framework that learns canonicalized semantic 3D keypoints from point clouds and uses them as structured object-centric representations for policy learning. A visuomotor diffusion policy conditions on these keypoints together with object-centric geometry to predict full manipulation trajectories, enabling consistent geometric correspondence across object instances. To evaluate category-level generalization, we construct a photorealistic simulation benchmark with three manipulation tasks and a planning-driven data generation pipeline that produces expert trajectories across diverse object instances. Experiments show that KeyGen significantly outperforms prior methods on both seen and unseen objects under pose variation, scales effectively with additional demonstrations per object, maintains robustness to object rescaling, and achieves strong performance in both simulation and real-world manipulation.
Sources
- RT-1: Robotics Transformer for Real-World Control at Scale
- RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control
- Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulation
- RVT: Robotic View Transformer for 3D Object Manipulation
- BAKU: An Efficient Transformer for Multi-Task Policy Learning
- DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset
- LERF: Language Embedded Radiance Fields
- Distilled Feature Fields Enable Few-Shot Language-Guided Manipulation
- D$^3$Fields: Dynamic 3D Descriptor Fields for Zero-Shot Generalizable Rearrangement
- GenDP: 3D Semantic Fields for Category-Level Generalizable Diffusion Policy
- One-Shot Imitation Learning
- Neural Descriptor Fields: SE(3)-Equivariant Object Representations for Manipulation
- Skeleton Merger: an Unsupervised Aligned Keypoint Detector
- Key-Grid: Unsupervised 3D Keypoints Detection using Grid Heatmap Features
- USIP: Unsupervised Stable Interest Point Detection from 3D Point Clouds
- SC3K: Self-supervised and Coherent 3D Keypoints Estimation from Rotated, Noisy, and Decimated Point Cloud Data
- Canonical Capsules: Self-Supervised Capsules in Canonical Pose
- Symmetry-Robust 3D Orientation Estimation
- ShapeNet: An Information-Rich 3D Model Repository
- Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection
Related papers
- FMT x: An Efficient and Asymptotically Optimal Extension of the Fast Marching Tree for Dynamic Replanning
- MPCFormer: A physics-informed data-driven approach for explainable socially-aware autonomous driving
- RoboLab: A High-Fidelity Simulation Benchmark for Analysis of Task Generalist Policies
- ARC: A Reasoning Recipe for Robot Foundation Models
- HRDexDB: A 4D Dexterous Grasping Dataset Across Human and Multiple Robot Embodiments
- APT: Action Expert Pretraining Improves Instruction Generalization of Vision-Language-Action Policies