Pixel2Catch: Multi-Agent Sim-to-Real Transfer for Agile Manipulation with a Single RGB Camera
Pixel2Catch proposes a robotic catching system using only a single RGB camera to infer object motion from pixel-level features instead of estimating 3D coordinates. It uses image segmentation to capture changes in an object's position and scale over time. By employing a multi-agent reinforcement learning framework, the system successfully transfers policies trained in simulation directly to the real robot without fine-tuning.