HapticWAM: Distilling Imagined Touch into a World-Action Model without Inference-Time Tactile Sensing
cs.RO
Submitted: 2026-09-20
Updated: 2026-09-20
Comments: 10 pages, 5 figures, 4 tables, submitted to IEEE ICRA 2027
Project page: https://advanced-robotic-manipulation.github.io/websites/hapticwam/Abstract
License: http://creativecommons.org/licenses/by-nc-nd/4.0/
The gist: Contact-rich manipulation requires estimating forces, slip and contact geometry that can remain ambiguous in scene images.
Terminology
Abstract
Contact-rich manipulation requires estimating forces, slip and contact geometry that can remain ambiguous in scene images. Optical tactile sensors provide both visual observations of the contact surface and mechanical measurements, yet learning from these signals raises two challenges: representing contact beyond appearance and transferring its benefits to a policy that does not require fingertip observations at deployment. We introduce HapticWAM, a world-action model that combines heterogeneous tactile encoding, structured contact prediction and teacher-student distillation. Its teacher encodes gel images together with deformation, shear, distributed forces, resultant wrench and derived contact state into a frozen video backbone. Rather than predicting tactile pixels alone, the model jointly generates actions and a contact package describing future events and mechanics. Anticipatory Contact Coupling uses the previously imagined package to condition attention, preserving a contact-related input when direct tactile observations are unavailable. Haptic-Imagination Distillation transfers both contact futures and action predictions to a student that retains the generative contact head but removes its fingertip input branches. On a real-world setup, across three contact-rich pick-and-place tasks, HapticWAM Student achieves a 77% per-task mean success rate (41 of 50 starts, 82% pooled), reaching 95% on one of the tasks, outperforming the evaluated teacher and baseline configurations.
Sources
- RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control
- $\pi_0$: A Vision-Language-Action Flow Model for General Robot Control
- $\pi_{0.5}$: a Vision-Language-Action Model with Open-World Generalization
- Tactile-VLA: Unlocking Vision-Language-Action Model's Physical Knowledge for Tactile Generalization
- $N_0$-VTLA: Scaling Vision-Tactile-Language-Action Model with Latent Tactile Tokens
- HapticVLA: Contact-Rich Manipulation via Vision-Language-Action Model without Inference-Time Tactile Sensing
- Cosmos World Foundation Model Platform for Physical AI
- World Action Models are Zero-shot Policies
- World-Value-Action Model: Implicit Planning for Vision-Language-Action Systems
- Towards Predictive, Aligned, and Scalable Robot Learning
- Fast-WAM: Do World Action Models Need Test-time Future Imagination?
- Dream-Tac: A Unified Tactile World Action Model for Contact-Rich Robot Manipulation
- Tactile-WAM: Touch-Aware World Action Model with Tactile Asymmetric Attention
- $N_0$-TWAM: Scaling Tactile-Native World-Action Model for Contact-Rich Manipulation
- Being-H0.7: A Latent World-Action Model from Egocentric Videos
- VT-WAM: Visual-Tactile World Action Model for Contact-Rich Manipulation
- TacForeSight: Force-Guided Tactile World Model for Contact-Rich Manipulation
- FlowTouch: View-Invariant Visuo-Tactile Prediction
Related papers
- FMT x: An Efficient and Asymptotically Optimal Extension of the Fast Marching Tree for Dynamic Replanning
- MPCFormer: A physics-informed data-driven approach for explainable socially-aware autonomous driving
- RoboLab: A High-Fidelity Simulation Benchmark for Analysis of Task Generalist Policies
- ARC: A Reasoning Recipe for Robot Foundation Models
- HRDexDB: A 4D Dexterous Grasping Dataset Across Human and Multiple Robot Embodiments
- APT: Action Expert Pretraining Improves Instruction Generalization of Vision-Language-Action Policies