VTAM: Efficient Visuo-Tactile World-Action Models for Contact-Rich Manipulation without Tactile Pretraining
summary
The gist
VTAM introduces a generalist Video–Tactile Action Model that integrates tactile sensing into a predictive video world model to enable stable control in diverse contact-rich scenarios By grounding…
In short
VTAM introduces a generalist Video-Tactile Action Model that integrates tactile sensing into a predictive video world model to enable stable control in contact-rich scenarios. It augments video transformers with tactile streams using lightweight transfer finetuning, allowing the model to learn joint visuo-tactile dynamics for robust manipulation without needing prior tactile pretraining.
Key concepts
- Video-Action Models (VAMs)
- These models learn implicit world dynamics by processing raw video streams to predict temporally consistent actions. While good at long-horizon tasks, they struggle when critical interaction states are only partially visible from vision alone, especially in contact-rich situations.
- Shared Continuous Latent Space
- The model projects both visual observations and high-resolution tactile streams into a single latent space using a pre-trained Variational Autoencoder. This allows the system to model the temporal dynamics of the visual scene alongside fine physical deformations captured by touch, creating a unified representation of vision and touch.
- Deformation-Aware Virtual Force Regularization
- This mechanism is added to stabilize multimodal fusion at the action head. It provides targeted supervision specifically for tactile signals during policy optimization, ensuring that the control policy remains sensitive to critical contact transitions even when visual cues are insufficient.
Terminology used across episodes
This episode discusses
- VTAM: Efficient Visuo-Tactile World-Action Models for Contact-Rich Manipulation without Tactile Pretraining · Paper Radio
- RT-H: Action Hierarchies Using Language
- VLA-Touch: Enhancing Vision-Language-Action Models with Dual-Level Tactile Feedback
- pi 0: A Vision-Language-Action Flow Model for General Robot Control
- RT-1: Robotics Transformer for Real-World Control at Scale
- Genie: Generative Interactive Environments
- ImplicitRDP: An End-to-End Visual-Force Diffusion Policy with Structural Slow-Fast Learning
- PaLM-E: An Embodied Multimodal Language Model
- LTX-Video: Realtime Video Latent Diffusion
- TLA: Tactile-Language-Action Model for Contact-Rich Manipulation
- Sparsh: Self-supervised touch representations for vision-based tactile sensing
- 3D-ViTac: Learning Fine-Grained Manipulation with Visuo-Tactile Sensing
- VT-Refine: Learning Bimanual Assembly with Visuo-Tactile Feedback via Simulation Fine-Tuning
- Tactile-VLA: Unlocking Vision-Language-Action Model's Physical Knowledge for Tactile Generalization
- OpenVLA: An Open-Source Vision-Language-Action Model
- Auto-Encoding Variational Bayes
- Genie Envisioner: A Unified World Foundation Platform for Robotic Manipulation
- FACTR: Force-Attending Curriculum Training for Contact-Rich Policy Learning
- Decoupled Weight Decay Regularization
- Hierarchical Vision Language Action Model Using Success and Failure Demonstrations
- pi 0.5: a Vision-Language-Action Model with Open-World Generalization
The paper
VTAM: Efficient Visuo-Tactile World-Action Models for Contact-Rich Manipulation without Tactile Pretraining · Read on arXiv
University of Illinois Urbana-Champaign · Stanford University · Shanghai Jiao Tong University
Transcript
Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.
Rosa: We're looking at the paper: "VTAM: Efficient Visuo-Tactile World-Action Models for Contact-Rich Manipulation without Tactile Pretraining".
Dev: Here's the gist of the paper. VTAM introduces a generalist Video–Tactile Action Model that integrates tactile sensing into a predictive video world model to enable stable control in diverse contact-rich scenarios By grounding control in visuo–tactile dynamics and force-aware reasoning, VTAM enables stable control in diverse contact-rich scenarios Pruning redundant reasoning tokens Video-Action Models (VAMs) have emerged as a promising framework for embodied intelligence, learning implicit world dynamics from raw video streams to produce temporally consistent action predictions Although such models demonstrate strong performance on long-horizon tasks through visual reasoning, they remain limited in contact-rich scenarios where critical interaction states are only partially observable from vision alone To bridge this gap, we introduce the Video-Tactile Action Model (VTAM), a multimodal world modeling framework that incorporates tactile perception as a complementary grounding signal VTAM augments a pretrained video transformer with tactile streams via a lightweight modality transfer finetuning, enabling efficient cross-modal representation learning without tactile–language paired data or independent tactile pretraining How the training signal is built The model operates by projecting both multi-view visual observations and high-resolution tactile streams into a shared continuous latent space via a pre-trained Variational Autoencoder (VAE) Within this space, a multi-view diffusion process employing alternating intra-view and cross-view attention jointly models the temporal dynamics of the visual scene and the fine-grained physical deformations captured by the tactile sensor To address modality collapse, we introduce a deformation-aware virtual force regularization at the action head to stabilize multimodal fusion This mechanism provides targeted supervision of the tactile pathway, thereby stabilizing multimodal fusion and ensuring that the policy remains sensitive to critical contact transitions during downstream tasks Where the gains come from VTAM demonstrates superior performance in contact-rich manipulation, maintaining a robust success rate of 90% on average In challenging scenarios such as potato chip pick-and-place requiring high-fidelity force awareness, VTAM outperforms the π0.5 baseline by 80% The results highlight that integrating tactile feedback is essential for correcting visual estimation errors in world action models, providing a scalable approach to physically grounded embodied foundation models Pruning redundant reasoning tokens The ablation study on the Chip Pick-and-Place task confirms the necessity of the components of VTAM The Vision-only baseline yields a 0% success rate, as visual depth estimation suffers from severe occlusion during the final approach and cannot perceive subtle contact transitions Introducing tactile signals only at the action head also results in 0% success (Q2), demonstrating that raw force injection is insufficient without our hierarchical visuo-tactile world modeling Finally, removing the virtual-force regularization (No Reg.
Title and authors: Rosa: So we're looking at this paper today called VTAM: Efficient Visuo-Tactile World-Action Models for Contact-Rich Manipulation without Tactile Pretraining. It sounds like they are trying to figure out how robots can handle messy, physical interactions really well using video and touch data together.
Dev: Yeah, the title is pretty clear. They're focusing on contact rich manipulation, which is where things get complicated—like picking up a potato chip or peeling a cucumber—and they want to do that without needing those expensive tactile training datasets.
Rosa: Right. The authors are a big group from places like the University of Illinois Urbana-Champaign and Stanford, so they bring some solid robotics and vision expertise to this. They’re essentially trying to bridge the gap between what a robot sees and what it actually feels when it touches something.
Dev: What's interesting is that they aren't just slapping a sensor on top of an existing vision model. They are integrating the tactile sensing directly into the core world model, making it a unified system for predicting future actions based on both sight and feel.
Rosa: Exactly. It moves beyond just using vision for high-level understanding and language to actually modeling how forces evolve during a physical contact event itself, which is what vision alone struggles with when things get slippery or deformable.
Dev: So, the big question here is whether this integration allows the model to learn those contact dynamics automatically without needing specific labels for every single interaction they encounter.
The paper's summary: Rosa: Okay, so what VTAM actually does is propose a joint visuo-tactile prediction framework that forecasts both future visual and tactile streams in one shared latent space. This lets the model learn how contact dynamics evolve over time without needing those explicit semantic annotations for every single physical interaction.
Dev: That predictive aspect is key. They introduce something called a virtual force prediction objective to help stabilize the training, which they say successfully mitigates modality collapse during the learning process when you have both vision and touch inputs.
Rosa: Modality collapse is a big issue when you mix different data types; it means the model stops listening to one input because another input becomes too strong or confusing, so this force prediction objective is their way of keeping both signals relevant during training.
Dev: And they validate this whole thing on some pretty tough contact rich robotic tasks, including potato chip pick-and-place, cucumber peeling, and whiteboard wiping. They show large improvements in success rate compared to just using vision or just using naive tactile integration methods.
Rosa: The numbers are what really stand out; they demonstrate significant improvements in the success rate over those simpler baselines. It suggests that this joint modeling approach provides a more robust foundation for physically grounded embodied models, even without needing separate tactile pretraining data.
The paper's improvements: Dev: Now, let's talk about how they improved their original idea. They introduced a couple of key mechanisms to make this work better. First, they use a multi-view diffusion process employing alternating intra-view and cross-view attention to jointly model the temporal dynamics of both the visual scene and those fine-grained physical deformations from the tactile sensor.
Rosa: That part about modeling both visual scene dynamics and tactile deformations simultaneously in that shared space is clever because it directly targets learning temporally consistent contact features without needing explicit labels for those contact events.
Dev: Then, to fight that modality collapse we mentioned earlier, they add a deformation-aware virtual force regularization at the action head. This mechanism provides targeted supervision of the tactile pathway, which helps stabilize multimodal fusion and keeps the policy sensitive to critical contact transitions during downstream tasks.
Rosa: So, it’s not just about getting both modalities together; it’s about how they structure that learning process—first modeling the dynamics in a shared space, then using this force regularization to keep the action output tied correctly to those tactile signals.
Dev: And they use a two-stage training strategy where Stage I focuses on modeling those joint visuo-tactile latent dynamics, and Stage II handles the conditional joint action state force denoising. That way, they let the backbone build a coherent multimodal representation before optimizing the actual control policies.
Conclusion: Rosa: So to wrap up, VTAM shows that integrating tactile feedback is essential for correcting those visual estimation errors in world action models. It provides a scalable approach to building physically grounded embodied foundation models because you don't need massive amounts of specific tactile data upfront.
Dev: The results on tasks like chip pick-and-place showing an eighty percent improvement over the vision only baseline really hammers home that tactile feedback is necessary for correcting those errors when the visual information isn't enough to see what's happening at the contact point.
Taro: From my perspective, it’s important that they show how the system handles failure modes, and this paper seems to address that by stabilizing fusion during training so it doesn't just collapse into a single modality when things get tricky.
Rosa: Exactly. And looking at what they did for chip pick-and-place, cucumber peeling, and whiteboard wiping with success rates hitting eighty five percent to ninety five percent, it shows precise force regulation is achievable through this method.
Dev: The paper highlights that the system can generalize to visually ambiguous contact scenarios by using tactile perception as that complementary grounding signal they needed. But they do mention a limitation: the ablation study showed that removing the virtual-force regularization or relying only on raw force injection at the action head resulted in zero success rate for those tasks, meaning raw force isn't enough without their hierarchical modeling.
Taro: It seems like the main thing to watch next is how they would push this system into environments where those contact transitions are even more unpredictable and require faster recovery than what they tested here.
Rosa: Yeah, VTAM offers a solid way forward for building robots that can actually interact with the physical world reliably without needing perfectly labeled tactile data for every single interaction.
More episodes
- 2608.13438-ContactGuard: Pre-Contact Execution Monitoring with Action-Conditioned Latent World Models
- 2610.34593-Future Prediction as Training-Only Video Supervision for Pretrained VLA Policies
- 2610.35385-EmbodiedLab: Beyond Manipulation, Benchmarking Embodied Agents That Act to Know
- 2610.36378-Data-Driven Hamiltonian for Neural Optimal Control
- 2610.37275-A Heuristic for Scheduling Autonomous Vehicles at Intersections
- 2610.37288-Dynamic Coordination of Autonomous Vehicles at Intersections
- 2610.37632-Real-Time Coordination of Autonomous Vehicles at Intersections
- 2611.13779-Softness is something you also need! Why compliance helps control: information metrics of a passive walker with compliant ankle
- 2611.13411-Residual-Corrected Physics-Informed Neural Network KKL Observers with Input-to-State Stability Guarantees
- 2611.14488-Is Your Robot's Memory Just a Clock? Temporal Shortcuts in Memory-Augmented VLA Policies