Semantics-to-Contact: A Stagewise Framework for Robust Contact-Rich Manipulation
cs.RO
Submitted: 2026-11-05
Updated: 2026-11-05
Comments: NeurIPS 2026
Code: https://github.com/HumanSignal/labelstudio
Project page: https://edem-ai.github.io/S2C
License: http://creativecommons.org/licenses/by/4.0/
The gist: Contact-rich manipulation requires precise control under complex contact dynamics while remaining robust to diverse visual conditions.
Abstract
Contact-rich manipulation requires precise control under complex contact dynamics while remaining robust to diverse visual conditions. However, real-world reinforcement learning often overfits to the visual appearance of training scenes. We propose Semantics-to-Contact (S2C), a stagewise framework for visually robust contact-rich manipulation. The insight is to decompose the problem into two stages: semantic focusing and contact refinement. In the first stage, a vision-language model localizes task-relevant regions across visually diverse scenes, providing coarse spatial guidance and reducing the burden of exploration. In the second stage, a real-world human-in-the-loop residual reinforcement learning policy learns fine-grained contact behaviors within the localized region using force feedback. To further improve visual robustness, we introduce an object-centric visual augmentation strategy that randomizes background appearance while preserving the robot and task-relevant objects. Experiments on six contact-rich real-world manipulation tasks exhibit that S2C consistently improves success rates over strong baselines and maintains reliable performance under significant visual and positional variations. These results demonstrate that stagewise semantic focusing and contact refinement provide a practical path toward visually robust real-world contact-rich manipulation. Our demos can be seen in https://edem-ai.github.io/S2C/.
Sources
- Residual Off-Policy RL for Finetuning Behavior Cloning Policies
- Qwen3-VL Technical Report
- ConRFT: A Reinforced Fine-tuning Method for VLA Models via Consistency Policy
- MolmoPoint: Better Pointing for VLMs with Grounding Tokens
- Uncertainty Comes for Free: Human-in-the-Loop Policies with Diffusion Models
- RaC: Robot Learning for Long-Horizon Tasks by Scaling Recovery and Correction
- TRANSIC: Sim-to-Real Policy Transfer by Learning from Online Correction
- Preference-Calibrated Human-in-the-Loop Reinforcement Learning for Robotic Manipulation
- Human-in-the-loop Online Rejection Sampling for Robotic Manipulation
- WorldCoder-Bench: Benchmarking Physically Grounded 3D World Synthesis
- SignBot: Learning Human-to-Humanoid Sign Language Interaction
- OpenAI GPT-5 System Card
- Gemini Robotics 1.5: Pushing the Frontier of Generalist Robots with Advanced Embodied Reasoning, Thinking, and Motion Transfer
- Octo: An Open-Source Generalist Robot Policy
- Learning While Deploying: Fleet-Scale Reinforcement Learning for Generalist Robot Policies
- RL-ScanIQA: Reinforcement-Learned Scanpaths for Blind 360{\deg}Image Quality Assessment
- RLDG: Robotic Generalist Policy Distillation via Reinforcement Learning
- TwinRL: Digital Twin-Driven Reinforcement Learning for Real-World Robotic Manipulation
- From Reaction to Anticipation: Proactive Failure Recovery through Agentic Task Graph for Robotic Manipulation
- LIRA: Local Cross-Layer Information Routing for Vision-Language-Action Decoding
Related papers
- Rethinking Actions: Continuity-Aware Priors for World-Model Planning
- MOSAIC: Modular Obstacle and Spatial-Aware Framework for Manipulation In Cluttered Scenes
- SAGAS: Semantic-Aware Graph-Assisted Stitching for Offline Temporal Logic Planning
- DexSeed: Expanding Human Hand-Object Interactions into Robot Demonstrations for Dexterous Policy Learning
- LaST0.5: A Generalist World Action Model with Interaction-Centric Latent Reasoning
- CIALLO: Learning Shared Intent Representations for Cross-Embodiment Manipulation