FlowTouch: View-Invariant Visuo-Tactile Prediction

summary

The gist

Tactile sensation is essential for contact-rich manipulation tasks <ref:2603.08255#pg2> It provides direct feedback on object geometry, surface properties, and interaction forces…

In short

FlowTouch is a model that predicts tactile patterns from a 3D object mesh rather than just 2D images. It uses scene reconstruction to create 3D representations and Flow Matching to generate realistic tactile images, removing scene-specific details. This allows the model to bridge the sim-to-real gap and predict grasp stability for new sensor setups.

Key concepts

View-Invariant Visuo-Tactile Prediction
This is the main goal: predicting how a robot hand will feel (tactile patterns) based on what it sees (vision), regardless of the camera angle. It focuses on capturing the essential physical interaction information from a 3D object's shape rather than relying solely on 2D pictures.
Local 3D Mesh Encoding
Instead of using flat images, FlowTouch uses the object's local 3D mesh to store detailed geometric information. This mesh acts as a rich source of data that helps the model predict complex tactile sensations while ignoring irrelevant background details in the scene.
Flow Matching Model
This is a type of generative model used to create new images, specifically tactile ones. It learns how to transform an initial state (like a 2D input) into the desired output (the tactile image) by following a learned flow path, making the generated predictions highly realistic.
Domain Conditioning
This technique helps the model generalize to different real-world sensors. By adding a 'domain flag' to the input data, it forces the model to focus on high-level similarities between different sensor types, improving performance when tested on new hardware.

Terminology used across episodes

This episode discusses

The paper

FlowTouch: View-Invariant Visuo-Tactile Prediction · Read on arXiv

Seongjin Bien, Carlo Kneissl, §Tobias Julg ™, Frank Fundel, Thomas Ressler-Antal ™, Florian Walter §Bjorn Ommer §Gitta Kutyniok ℳ, §Wolfram Burgard

Department of Computer Science & Artificial Intelligence, University of Technology Nuremberg, Germany · Ludwig Maximilian University of Munich · TUM School of Computation, Information and Technology, Technical University of Munich, Germany

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: I'm Rosa.

Dev: And I'm Dev, here with Taro.

Rosa: We're looking at the paper: "FlowTouch: View-Invariant Visuo-Tactile Prediction".

Dev: Here's the gist of the paper. FlowTouch introduces FlowTouch, a novel model for view-invariant visuo-tactile prediction that uses an object’s local 3D mesh to encode rich information for predicting tactile patterns while abstracting away from scenedependent details Our results show that FlowTouch is able to bridge the sim-to-real gap and generalize to new sensor instances We further show that the resulting tactile images can be used for downstream grasp stability prediction The Core Idea of FlowTouch The key idea is to use an object’s local 3D mesh to encode rich information for predicting tactile patterns while abstracting away from scenedependent details This is achieved by lifting target objects from 2D observations into 3D representations using scene mesh generation foundation models FlowTouch integrates scene reconstruction and Flow Matching-based models for image generation The Prediction Framework The method consists of two components: the image-toPCN sampling pipeline, and the Flow Matching model that generates the tactile image conditioned on the input PCN The sampling pipeline uses SceneComplete to achieve object segmentation and image inpainting before feeding results into a 2D-to-3D model This process involves initializing a MuJoCo scene with the object mesh and contact pad meshes to sample the PCN from the intersection of the mesh surface with the visualization areas of the two pads Training and Domain Adaptation The model is trained using a Conditional Flow Matching (CFM) framework where a U-Net style Vision Transformer regresses the conditional velocity to satisfy the CFM objective To bridge the domain gap, a two-stage learning approach is employed: first pretraining only on synthetic data for 30k steps, and then a finetuning phase mixing in real data with only 20% weight Domain Conditioning is implemented by adding a domain flag to the PCN information used in cross attention to help the model focus on highlevel similarities between domains Sparsh Perceptual Loss is included as an objective to enforce that predictions are consistent in the Sparsh embedding space, which retains information relevant to downstream tasks Evaluation and Generalization The framework is evaluated on the ObjectFolderReal and YCB-Slide datasets for GelSight and DIGIT respectively Ablation studies show that Domain Conditioning significantly boosts performance across the board, being our best performing technique The Combined model scores slightly worse than the Domain Conditioning variant but is hypothesized to retain more tactile-relevant features in the image In zero-shot generalization tests, the model is able to generalize to some extent despite seeing a sensor instance during training Finally, the framework demonstrates capability in grasp stability estimation tasks Limitations and Future Work A limitation is that performance depends heavily on mesh quality and is highly sensitive to alignment in the robot frame The model struggles with unseen geometries, which could be alleviated by scaling up the diversity and size of the simulated dataset The framework does not explicitly encode force but relies implicitly on the depth of the sampled PCN to render meshes at different forces A promising direction for future work is to incorporate textures into the model’s conditioning to enable the prediction of high-resolution tactile features In this work, we introduced FlowTouch, a vision-totouch framework featuring a geometry-conditioned generative prediction model By leveraging point clouds and their corresponding normals sampled from meshes, our framework is able to effectively abstract away from scene-specific information inherent in image-only methods Furthermore, we show that FlowTouch’s output can be used to support downstream manipulation tasks even in zero-shot settings Finally, our framework uses scene reconstruction methods to make real-world deployment feasible References

Paper summary: Rosa: So, to recap where we are is that FlowTouch is this new model for view-invariant visuo-tactile prediction. The authors argue that tactile sensation is vital for manipulation, but getting data from sensors during planning isn't practical because you have to touch things first.

Dev: Right. They introduce FlowTouch as a generative framework conditioned on object geometry that predicts tactile signals without needing exploratory robot motion beyond the initial setup, which is a key claim they make in their summary.

Taro: It’s about using the local three dimensional mesh of an object to encode rich information for predicting tactile patterns while abstracting away from scene-dependent details, which is what makes it view-invariant.

Rosa: They achieve this by lifting objects into three dimensional representations using scene mesh generation foundation models and then integrating that with Flow Matching-based models for image generation.

Dev: The summary also highlights their contribution of proposing a simulation to real training method based on the mesh data of geometric primitives obtained via simulation, which they say improves output quality and reduces the need for costly real-world data collection.

Taro: They also show empirical verification of strong generalization across scene variations and sensor instances, along with its ability to generate relevant information for downstream manipulation tasks through grasp stability experiments.

Rosa: So, the paper is essentially arguing that by using geometry as a condition, you can predict tactile signals without needing to touch the object during every planning step.

Dev: And it’s important that they show this works across different scenes and different sensors, which speaks to its robustness beyond just one specific setup.

Taro: It moves the prediction from being tightly coupled with interaction images to being conditioned primarily on the geometric structure itself.

Rosa: That really shifts the focus toward using scene reconstruction methods to make real-world deployment feasible, as they point out in their summary.

Conclusion: Rosa: So looking at the title "FlowTouch: View-Invariant Visuo-Tactile Prediction," it captures the essence of what this work does—it’s about predicting touch from a view that doesn't depend on where you are or how you see it.

Dev: The authors, Bien, Kneissl, Julg, Fundel, Ressler-Antal, Walter, Ommer, Kutyniok and Burgard are the team behind this model that introduces this novel approach to visuo-tactile prediction.

Taro: What this means practically is that if we can use FlowTouch effectively in a real robot system, it could allow for much more intuitive manipulation where the robot can anticipate tactile feedback based on what it sees about the object's shape.

Rosa: Exactly. It suggests a path toward systems where perception and interaction planning happen much closer together, making grasping tasks more reliable even when things aren't perfectly aligned or when the sensor isn't in its ideal position.

Dev: And it paves the way for using these predictions for things like grasp stability estimation, which is a downstream task that relies on accurate tactile input.

Taro: So while there are limitations about mesh quality and alignment sensitivity, if we can scale up the simulated data diversity, this model could become a powerful tool for planning in complex scenarios.

More episodes

← Home