NoContactNoWorries: Estimating Contact through Vision and Proprioception for In-Hand Dexterous Manipulation

summary

The gist

Perceiving physical contact is fundamental to dexterous manipulation This work proposes pseudo-tactile sensing for dexterous in-hand manipulation, i.e…

In short

This work proposes NoContactNoWorries, a method to estimate binary fingertip contact using only RGB-D vision and robot proprioception, eliminating the need for physical tactile sensors. The transformer-based framework fuses visual features with joint configurations over time to predict contact states. This pseudo-tactile signal allows reinforcement learning policies trained in simulation to be directly transferred to robotic hardware.

Key concepts

NoContactNoWorries
A transformer-based multimodal approach that fuses ego-centric RGB-D vision data with the robot's proprioception (joint states). It is designed to predict binary contact vectors at each fingertip site, effectively creating a pseudo-tactile signal without any dedicated tactile sensors.
Pose-Conditioned Spatial Fusion
This module uses cross-attention mechanisms to combine visual features extracted from RGB-D images with the robot's current and commanded joint configurations. This fusion step creates rich spatial representations that are specifically tailored to the robot's configuration, helping the model understand how vision relates to physical movement.
Causal Transformer Encoder
A temporal modeling component that uses a causal transformer architecture to process sequences of visual representations over time. It models how per-frame features evolve, allowing the system to accumulate evidence across multiple frames to infer persistent contact states from dynamic cues like trajectory changes.

Terminology used across episodes

This episode discusses

The paper

NoContactNoWorries: Estimating Contact through Vision and Proprioception for In-Hand Dexterous Manipulation · Read on arXiv

Soham Patil, Avirup Das, Sourabh Bhosale, Spandan Roy

Robotics Research Center (RRC) · International Institute of Information Technology (IIIT) Hyderabad · The University of Manchester

Perceiving physical contact is fundamental to dexterous manipulation. While robots often rely on dedicated hardware tactile sensors, humans exhibit a remarkable ability to infer contact by integrating visual information with an innate sense of their body's pose and movement. Inspired by this embodied perceptual skill, we investigate whether a robot can learn to infer contact from vision, an approach that also offers a scalable alternative to tactile hardware specifically for binary contact estimation, which faces practical challenges in cost, fragility, and integration. We present NoContactNoWorries, a transformer-based multimodal framework that fuses RGB-D vision with the robot's proprioception to infer binary contact states as a pseudo-tactile signal for hand-object interactions. We validate by training a single contact prediction model on multiple objects and show that the inferred contact signal supports downstream reinforcement learning agents for in-hand object reorientation, generalizing to novel objects. Experiments in both simulation and on a real-world robot validate our approach, highlighting the feasibility of inferring contact from vision and proprioception. Project Page: https://soham2560.github.io/no-contact-no-worries/

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: I'm Rosa.

Dev: And I'm Dev, here with Taro.

Rosa: We're looking at the paper: "NoContactNoWorries: Estimating Contact through Vision and Proprioception for In-Hand Dexterous Manipulation".

Dev: Here's the gist of the paper. Perceiving physical contact is fundamental to dexterous manipulation This work proposes pseudo-tactile sensing for dexterous in-hand manipulation, i.e., predicting binary fingertip contact from ego-centric RGB-D and proprioception without tactile measurements The Core Proposal The framework, named NoContactNoWorries, is a transformer-based multimodal approach that fuses RGB-D vision with the robot’s proprioception to infer binary contact states as a pseudo-tactile signal for hand-object interactions The goal is to predict binary contact vectors at each fingertip site using ego-centric RGB-D and proprioception without dedicated tactile sensors This approach offers a scalable alternative to tactile hardware specifically for binary contact estimation, addressing the practical challenges of cost, fragility, and integration.

Rosa: How the Training Signal is Built The training involves collecting contact-labeled demonstrations in simulation using the LEAP Hand platform built on NVIDIA Isaac Gym The model is trained using supervised learning over this dataset, optimizing a binary cross-entropy loss over all timesteps T to predict the binary contact vector.

Dev: Key components include: 1.

Rosa: RGB-D Feature Extraction, where an encoder maps RGB-D inputs into spatial features ft = Φ(It,Dt).

Title and authors: Rosa: So we're looking at this paper, NoContactNoWorries: Estimating Contact through Vision and Proprioception for In-Hand Dexterous Manipulation. Basically, it tackles how robots can figure out if their fingers are actually touching something without needing those expensive tactile sensors.

Dev: Right. It's about using what the robot already knows—its camera feed and its own joint positions—to create a pseudo-tactile signal for in-hand manipulation. It’s an approach that aims to be a scalable alternative to having dedicated hardware for just binary contact detection, which is really hard with cost and integration issues.

Taro: I'm curious how much of the uncertainty we're actually dealing with when we remove the tactile sensors entirely; does this work well when things get messy?

Rosa: Well, the core idea is to fuse RGB-D vision data with the robot’s proprioception using a transformer. This framework predicts binary contact states at each fingertip site directly from those inputs, without any actual touch measurements. It's essentially trying to mimic what humans do by integrating what they see with their sense of where their body is in space.

Dev: The training setup involves collecting labeled data in simulation using the LEAP Hand platform on NVIDIA Isaac Gym and then training a transformer model over that dataset using binary cross-entropy loss across all timesteps. So, they're teaching the model to output those contact vectors based on what it sees and knows about its own movement.

Taro: That sounds like a lot of data collection upfront. Does the way they structure the input—the spatial features from vision and those pose embeddings from proprioception—help them separate the relevant visual context from just random noise?

Rosa: They use a pose-conditioned spatial fusion module that uses cross-attention conditioned on both the current and commanded joint configurations to create those visual representations. This lets the model look at how things are positioned both now and where it's going next, which should help resolve the ambiguity in vision-based contact estimation.

Dev: And to handle the time aspect, they use a causal transformer encoder that models how those frame-by-frame representations evolve over a short window of time. This is key because contact isn't just about a single moment; it's about the dynamic interaction during movement, which they are capturing temporally.

Taro: So what’s the actual payoff here? If this model predicts binary contact states, how does that translate into something useful for a robot actually moving an object in its hand?

Rosa: The main payoff is that these predicted contact signals can be fed directly to downstream controllers as a surrogate tactile signal. This means you can take policies trained purely in simulation and transfer them straight to hardware without needing to retrain the entire policy from scratch on physical contact data.

Title and authors: Dev: That’s a significant win for deployment because it bypasses the need for real-world tactile sensor feedback during training, which is usually the biggest hurdle in robotics research. It lets you use simulation results more directly.

Taro: I wonder about generalization—if this model was trained on a few objects, does it actually work when we introduce something completely new to the hand?

Rosa: They validated its ability to generalize by showing that the inferred contact signal supports reinforcement learning agents for in-hand object reorientation, and they showed it generalizing to novel objects, like predicting contact for the letter 'R'. It seems pretty robust across different geometries.

Dev: But you have to remember the caveat: performance on actual hardware can degrade due to sensing noise and calibration mismatch. The authors note that even then, the full model stays robust by using those proprioceptive cues when visual evidence is incomplete.

Taro: So it’s not a perfect plug-and-play solution for every messy real-world scenario, but it handles missing information better than just relying on vision alone, which makes sense in physical contact situations.

Rosa: Exactly. The research also points out some things they didn't focus on yet. They are only estimating binary contact states at a fixed set of sensing sites, not full contact maps or the palm regions themselves.

Dev: And another limitation is that this current prediction doesn't estimate richer tactile quantities like force magnitudes or shear forces, which are absolutely crucial for dealing with deformable objects where pressure matters more than just knowing if something is touching.

Taro: So it’s a good start for detecting *if* something is touching, but it doesn't tell us *how hard* it's being touched or the friction involved. That leaves a gap for grasping things that squish or deform easily.

Rosa: Precisely. But they are making a practical case here: this method offers a viable substitute for hardware tactile sensors specifically for binary contact estimation, which solves the cost and fragility problems we always face with them in dexterous manipulation tasks.

Dev: So, to wrap up on NoContactNoWorries: Estimating Contact through Vision and Proprioception for In-Hand Dexterous Manipulation, this transformer-based multimodal framework successfully fuses vision and proprioception to infer binary contact states as a pseudo-tactile signal for hand interactions.

Taro: It seems like it provides a solid foundation for using vision and motion data to replace some of the most frustrating parts of tactile sensing hardware.

Rosa: It really does, especially when you think about how much easier it makes transferring simulation policies directly to real robots.

Dev: We'll keep an eye on those limitations regarding force estimation and coverage later, but for now, this paper shows a clear path toward using embodied perception to augment traditional sensing methods.

Taro: That’s all we have time for today on this paper.

The paper's summary: Rosa: So, this paper, NoContactNoWorries, is all about using just what the robot sees and what its own joints are doing to figure out if a finger is actually touching something in its hand without needing those expensive tactile sensors.

Dev: Right. They're proposing this transformer model that fuses RGB-D vision data with the robot's proprioception—that's your joint angles and positions—to predict binary contact states, which they call a pseudo-tactile signal.

Rosa: They trained this thing by looking at labeled demonstrations from simulation using the LEAP Hand platform on NVIDIA Isaac Gym, optimizing a binary cross entropy loss over all timesteps to get those contact vectors.

Dev: The training pipeline involves mapping the RGB-D inputs into spatial features, then using a pose conditioned spatial fusion module that uses cross attention based on both current and commanded joint configurations to make those visual representations.

Rosa: And then they use a causal transformer encoder that models the evolution of those frame by frame representations over time, which lets it look at dynamic cues like how a trajectory is deviating or when something gets briefly hidden.

Dev: The big win here is that these predicted contact signals can be used directly by downstream controllers as a surrogate tactile signal, meaning you can take policies trained in simulation and transfer them straight to hardware without needing to retrain the policy from scratch on real touch data.

Rosa: It’s pretty interesting because they show it handles things like novel objects, successfully predicting contact for something completely new like the letter 'R', and it stays robust even when contact regions are occluded by other parts of the hand or object.

Dev: That robustness is important, but there are still some limitations. The authors admit that the method only estimates binary contact states at a fixed set of sensing sites, not a full contact map or the palm region itself.

Rosa: And they also point out that it only predicts whether something is touching or not; it doesn't tell you how hard the force is or what kind of shear stress is happening, which would be important for grasping squishy things.

Dev: That’s a key distinction. They are providing a practical substitute for hardware tactile sensors specifically for the binary contact estimation part, which solves some of those major issues with cost and fragility when you need to do dexterous manipulation.

Rosa: So it’s not a perfect replacement yet, but it shows that we can get pretty good at inferring basic touch information using just vision and motion data.

Dev: And that capability to transfer simulation results directly to hardware is where this research really impacts the development cycle for complex robotic manipulation tasks.

The paper's improvements: Rosa: So, moving past just getting binary contact states at fixed points, they suggest making this system more comprehensive by looking at bigger areas of contact instead of just isolated fingertips or sensing spots on the palm.

Dev: That makes sense for a real-world scenario because if you're grasping something big, you need to know about the whole surface interaction, not just one tiny spot.

Rosa: Exactly. They’re looking at how to extend this approach so it can infer denser contact maps or even the entire palm region instead of just those specific fingertip sites they started with in the first place.

Dev: From a control perspective, that would give you a much richer signal for your controller, not just a simple on or off switch for each finger.

Rosa: And they also mention that this model isn't meant to be one-off; it suggests building a foundational model pre-trained on many different hand object interactions so it can actually tackle totally new tasks like tool use or assembly.

Dev: That’s a big shift; they want to move away from just training for "pick up this cube" and toward something more general, which means the AI needs to learn the physics of interaction itself.

Rosa: They also noted that right now, it only predicts contact states; it doesn't actually give you numbers like force magnitudes or shear forces, which are really important when dealing with deformable objects that you might be squeezing.

Dev: That’s a major limitation for safety and control because knowing the force tells you if something is about to slip or break, not just that it's touching.

Rosa: So the paper points toward future work focusing on adding those richer tactile quantities so we can move beyond simple presence detection to actual physical interaction understanding.

Dev: And that leads into a question about latency and loop rate; if you add more complex feature extraction for force estimation, how does that affect the real time performance of the whole system?

Rosa: That’s where you see the trade-off—the more detail you get from vision to estimate force, the slower your decision loop might become, so balancing accuracy with real time is going to be a big challenge.

Conclusion: Rosa: So, to wrap up, this paper NoContactNoWorries shows how you can use vision and robot movement data to create a pseudo-tactile signal for binary contact estimation in hand manipulation without needing physical sensors.

Dev: It's really about creating a way to bridge the gap between simulation training and real hardware deployment by using that model as a virtual sensor.

Rosa: Right, and the implication is that we can get policies trained in simulation to transfer directly to robots much faster because they don't have to retrain everything on physical touch data first.

Dev: I think the main thing is making contact detection practical for robotics where tactile sensors are too fragile or too expensive to put everywhere.

Taro: I just want to push on the failure modes; if the visual data gets noisy, does this system just fail completely, or can it still recover gracefully when things get weird?

Rosa: It can handle some mess, especially because it uses those proprioceptive cues—the robot's own joint positions—to keep going even when the visual evidence is incomplete.

Dev: That's where the safety case comes in; we need to make sure that if the prediction is wrong, our control loop handles that error without causing a crash or dangerous movement.

Taro: I think it’s promising because it shows a path toward making hand interaction perception more robust even when the environment doesn't behave exactly as expected during trials.

Rosa: So, overall, NoContactNoWorries gives us a solid method for inferring basic touch information using just vision and motion data to augment traditional sensing methods.

Dev: It’s a strong step toward using embodied perception to reduce the reliance on hardware for basic contact sensing in complex manipulation tasks.

Rosa: We'll keep watching how researchers expand this work, especially when they start adding those force and shear measurements we talked about earlier.

More episodes

← Home