FlowTouch: View-Invariant Visuo-Tactile Prediction

arXiv:2603.08255 · cs.RO, cs.LG · Submitted 2026-03-09 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: I'm Rosa.

Dev: And I'm Dev, here with Taro.

Rosa: We're looking at the paper: "FlowTouch: View-Invariant Visuo-Tactile Prediction".

Dev: Here's the gist of the paper. FlowTouch introduces FlowTouch, a novel model for view-invariant visuo-tactile prediction that uses an object’s local 3D mesh to encode rich information for predicting tactile patterns while abstracting away from scenedependent details Our results show that FlowTouch is able to bridge the sim-to-real gap and generalize to new sensor instances We further show that the resulting tactile images can be used for downstream grasp stability prediction The Core Idea of FlowTouch The key idea is to use an object’s local 3D mesh to encode rich information for predicting tactile patterns while abstracting away from scenedependent details This is achieved by lifting target objects from 2D observations into 3D representations using scene mesh generation foundation models FlowTouch integrates scene reconstruction and Flow Matching-based models for image generation The Prediction Framework The method consists of two components: the image-toPCN sampling pipeline, and the Flow Matching model that generates the tactile image conditioned on the input PCN The sampling pipeline uses SceneComplete to achieve object segmentation and image inpainting before feeding results into a 2D-to-3D model This process involves initializing a MuJoCo scene with the object mesh and contact pad meshes to sample the PCN from the intersection of the mesh surface with the visualization areas of the two pads Training and Domain Adaptation The model is trained using a Conditional Flow Matching (CFM) framework where a U-Net style Vision Transformer regresses the conditional velocity to satisfy the CFM objective To bridge the domain gap, a two-stage learning approach is employed: first pretraining only on synthetic data for 30k steps, and then a finetuning phase mixing in real data with only 20% weight Domain Conditioning is implemented by adding a domain flag to the PCN information used in cross attention to help the model focus on highlevel similarities between domains Sparsh Perceptual Loss is included as an objective to enforce that predictions are consistent in the Sparsh embedding space, which retains information relevant to downstream tasks Evaluation and Generalization The framework is evaluated on the ObjectFolderReal and YCB-Slide datasets for GelSight and DIGIT respectively Ablation studies show that Domain Conditioning significantly boosts performance across the board, being our best performing technique The Combined model scores slightly worse than the Domain Conditioning variant but is hypothesized to retain more tactile-relevant features in the image In zero-shot generalization tests, the model is able to generalize to some extent despite seeing a sensor instance during training Finally, the framework demonstrates capability in grasp stability estimation tasks Limitations and Future Work A limitation is that performance depends heavily on mesh quality and is highly sensitive to alignment in the robot frame The model struggles with unseen geometries, which could be alleviated by scaling up the diversity and size of the simulated dataset The framework does not explicitly encode force but relies implicitly on the depth of the sampled PCN to render meshes at different forces A promising direction for future work is to incorporate textures into the model’s conditioning to enable the prediction of high-resolution tactile features In this work, we introduced FlowTouch, a vision-totouch framework featuring a geometry-conditioned generative prediction model By leveraging point clouds and their corresponding normals sampled from meshes, our framework is able to effectively abstract away from scene-specific information inherent in image-only methods Furthermore, we show that FlowTouch’s output can be used to support downstream manipulation tasks even in zero-shot settings Finally, our framework uses scene reconstruction methods to make real-world deployment feasible References

Paper summary: Rosa: So, to recap where we are is that FlowTouch is this new model for view-invariant visuo-tactile prediction. The authors argue that tactile sensation is vital for manipulation, but getting data from sensors during planning isn't practical because you have to touch things first.

Dev: Right. They introduce FlowTouch as a generative framework conditioned on object geometry that predicts tactile signals without needing exploratory robot motion beyond the initial setup, which is a key claim they make in their summary.

Taro: It’s about using the local three dimensional mesh of an object to encode rich information for predicting tactile patterns while abstracting away from scene-dependent details, which is what makes it view-invariant.

Rosa: They achieve this by lifting objects into three dimensional representations using scene mesh generation foundation models and then integrating that with Flow Matching-based models for image generation.

Dev: The summary also highlights their contribution of proposing a simulation to real training method based on the mesh data of geometric primitives obtained via simulation, which they say improves output quality and reduces the need for costly real-world data collection.

Taro: They also show empirical verification of strong generalization across scene variations and sensor instances, along with its ability to generate relevant information for downstream manipulation tasks through grasp stability experiments.

Rosa: So, the paper is essentially arguing that by using geometry as a condition, you can predict tactile signals without needing to touch the object during every planning step.

Dev: And it’s important that they show this works across different scenes and different sensors, which speaks to its robustness beyond just one specific setup.

Taro: It moves the prediction from being tightly coupled with interaction images to being conditioned primarily on the geometric structure itself.

Rosa: That really shifts the focus toward using scene reconstruction methods to make real-world deployment feasible, as they point out in their summary.

Conclusion: Rosa: So looking at the title "FlowTouch: View-Invariant Visuo-Tactile Prediction," it captures the essence of what this work does—it’s about predicting touch from a view that doesn't depend on where you are or how you see it.

Dev: The authors, Bien, Kneissl, Julg, Fundel, Ressler-Antal, Walter, Ommer, Kutyniok and Burgard are the team behind this model that introduces this novel approach to visuo-tactile prediction.

Taro: What this means practically is that if we can use FlowTouch effectively in a real robot system, it could allow for much more intuitive manipulation where the robot can anticipate tactile feedback based on what it sees about the object's shape.

Rosa: Exactly. It suggests a path toward systems where perception and interaction planning happen much closer together, making grasping tasks more reliable even when things aren't perfectly aligned or when the sensor isn't in its ideal position.

Dev: And it paves the way for using these predictions for things like grasp stability estimation, which is a downstream task that relies on accurate tactile input.

Taro: So while there are limitations about mesh quality and alignment sensitivity, if we can scale up the simulated data diversity, this model could become a powerful tool for planning in complex scenarios.

Seongjin Bien, Carlo Kneissl, §Tobias Julg ™, Frank Fundel, Thomas Ressler-Antal ™, Florian Walter §Bjorn Ommer §Gitta Kutyniok ℳ, §Wolfram Burgard

Department of Computer Science & Artificial Intelligence, University of Technology Nuremberg, Germany · Ludwig Maximilian University of Munich · TUM School of Computation, Information and Technology, Technical University of Munich, Germany

cs.RO, cs.LG

Submitted: 2026-03-09

Updated: 2026-10-30

Code: https://github.com/madebyollin/taesd

Project page: https://flowtouch.github.io

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 91/100

The gist: Tactile sensation is essential for contact-rich manipulation tasks <ref:2603.08255#pg2> It provides direct feedback on object geometry, surface properties, and interaction forces…

Key concepts

View-Invariant Visuo-Tactile Prediction
This is the main goal: predicting how a robot hand will feel (tactile patterns) based on what it sees (vision), regardless of the camera angle. It focuses on capturing the essential physical interaction information from a 3D object's shape rather than relying solely on 2D pictures.
Local 3D Mesh Encoding
Instead of using flat images, FlowTouch uses the object's local 3D mesh to store detailed geometric information. This mesh acts as a rich source of data that helps the model predict complex tactile sensations while ignoring irrelevant background details in the scene.
Flow Matching Model
This is a type of generative model used to create new images, specifically tactile ones. It learns how to transform an initial state (like a 2D input) into the desired output (the tactile image) by following a learned flow path, making the generated predictions highly realistic.
Domain Conditioning
This technique helps the model generalize to different real-world sensors. By adding a 'domain flag' to the input data, it forces the model to focus on high-level similarities between different sensor types, improving performance when tested on new hardware.

Terminology

Summary

Tactile sensation is essential for contact-rich manipulation tasks <ref:2603. p. 008255#pg2> It provides direct feedback on object geometry, surface properties, and interaction forces, enhancing perception and enabling fine-grained control <ref:2603. p. 008255#pg2>. Predicting tactile information from visual information can bridge this gap <ref:2603. p. 008255#pg2>.

FlowTouch Framework

This work introduces FlowTouch, a novel model for view-invariant visuo-tactile prediction <ref:2603.08255#pg2>. The key idea is to use an object’s local 3D mesh to encode rich information for predicting tactile patterns while abstracting away from scenedependent details <ref:2603. p. 008255#pg2>. FlowTouch integrates scene reconstruction and Flow Matching-based models for image generation <ref:2603. p. 008255#pg2>.

Methodology Overview

The method consists of two components: the image-toPCN sampling pipeline, and the Flow Matching model that generates the tactile image conditioned on the input PCN <ref:2603.08255#pg2>. The sampling process involves creating a mesh aligned with the robot coordinate frame using SceneComplete to properly position contacts from desired gripper poses <ref:2603. p. 108255#pg2>. This results in a Point Cloud of 3D positions and their respective normals (PCN) randomly sampled around the contact point on the mesh <ref:2603. p. 108255#pg2>.

Generative Model Details

The generative model utilizes a latent Flow Matching objective to regress the conditional velocity from noise to data <ref:2603.08255#pg2>. The background image encodes the sensor’s baseline appearance in the absence of contact and serves as a spatial prior <ref:2603. p. 208255#pg2>. The model uses Cross-attention Mesh Conditioning where the sampled PCN is linearly projected to form the cross-attention context <ref:2603. p. 208255#pg2>.

Training and Adaptation

The authors propose a simulation-to-real training method based on the mesh data of geometric primitives obtained via simulation and show that it improves output quality substantially reducing the need for costly real-world data collection <ref:2603.08255#pg2>. They employ a two-stage learning approach where they pretrain only on synthetic data for 30k steps with batch size 128 and then start a finetuning phase mixing in real data with only 20% weighting <ref:2603. p. 208255#pg2>. The Sparsh Perceptual Loss objective is included to enforce that predictions are consistent in the Sparsh embedding space <ref:2603. p. 308255#pg2>.

Evaluation and Results

The model is evaluated on ObjectFolderReal and YCB-Slide datasets for GelSight and DIGIT respectively <ref:2603.08255#pg2>. The results show that FlowTouch is able to bridge the sim-to-real gap and generalize to new sensor instances <ref:2603. p. 008255#pg2>. Grasp stability experiments demonstrate that the resulting tactile images can be used for downstream grasp stability prediction <ref:2603. p. 008255#pg2>. The Combined model scores slightly worse than the Domain Conditioning variant but retains more tactile-relevant features in the image <ref:2603. p. 508255#pg2>.

Limitations and Future Work

A limitation is that its performance depends heavily on mesh quality and it is highly sensitive to alignment in the robot frame <ref:2603.08255#pg2>. The model struggles with unseen geometries which could be alleviated by scaling up the diversity and size of the simulated dataset <ref:2603. p. 608255#pg2>. A promising direction for future work is to incorporate textures into the model’s conditioning <ref:2603. p. 608255#pg2>.

Conclusion

In this work, we introduced FlowTouch, a vision-totouch framework featuring a geometry-conditioned generative prediction model <ref:2603.08255#pg2>. By leveraging point clouds and their corresponding normals sampled from meshes, our framework is able to effectively abstract away from scene-specific information inherent in image-only methods <ref:2603. p. 608255#pg2>. Furthermore, we show that FlowTouch’s output can be used to support downstream manipulation tasks even in zero-shot settings <ref:2603. p. 608255#pg2>. Finally, our framework uses scene reconstruction methods to make real-world deployment feasible <ref:2603. p. 608255#pg2>.

Acknowledgment

This work has been partially supported by the project GeniusRobot funded by the German Federal Ministry of Education and Research (BMBF grant no. 01IS24083) <ref:2603. p. 008255#pg2>. It also has been partially supported by the German Federal Ministry of Research, Technology and Space (BMFTR) under the Robotics Institute Germany (RIG) <ref:2603. p. 708255#pg2>. The authors acknowledge the HPC resources provided by the Erlangen National HPC Center (NHR@FAU) under the BayernKI project no. p. 7 v106be <ref:2603. p. 008255#pg2>.

References

[1] W. Yuan, S. Dong, and E. p. 7 H. Adelson, “GelSight: High-resolution robot tactile sensors for estimating geometry and force,” Sensors, vol. p. 7 17, no. 12, 2017.

[2] M. Lambeta, P. p. 7 p. 7-W. Chou, S. Tian, B. Yang, B. Maloon et al. p. 7, “DIGIT: A novel design for a low-cost compact high-resolution tactile sensor with application to in-hand manipulation,” IEEE Robotics and Automation Letters, vol. p. 7 5, no. 3, 2020.

[3] Y. Chen, M. Van der Merwe, A. p. 7 Sipos, and N. p. 7 Fazeli, “Visuotactile transformers for manipulation,” in Proc. p. 7 p. 7 of the Conf. p. 7 p. 7 p. 7 p. 7 p. 7 on Robot Learning (CoRL), 2023 <ref:2603. p. 708255#pg2>.

[4] J. Hansen, F. Hogan, D. Rivkin, D. Meger, M. Jenkin, and G. p. 7 Dudek, “Visuotactile-RL: Learning Multimodal Manipulation Policies with Deep Reinforcement Learning,” in Proc. p. 7 of the IEEE Int. p. 7 Conf. on Robotics & Automation (ICRA), 2022 <ref:2603. p. 708255#pg2>.

[5] C. Higuera, J. p. 7 Ortiz, H. Qi, L. Pineda, B. Boots, and M. p. 7 Mukadam, “Perceiving Extrinsic Contacts from Touch Improves Learning Insertion Policies,” http://arxiv. p. 7org/abs/2309. p. 716652 <ref:2603. p. 008255#pg2>.

[6] R. Calandra, A. p. 7 Owens, M. Upadhyaya, W. p. 7 Yuan, J. Lin et al., “The feeling of success: Does touch sensing help predict grasp outcomes? p. 7” in Proc. of the Conf. on Robot Learning (CoRL), 2017 <ref:2603. p. 708255#pg2>.

[7] C. Keysers, J. p. 7 H. Kaas, and V. p. 7 Gazzola, “Somatosensation in social perception,” Nature Reviews Neuroscience, vol. p. 7 11, no. 6, 2010 <ref:2603. p. 008255#pg2>.

[8] N. Hedger, T. Naselaris, K. p. 7 Kay, and T. p. 7 Knapen, “Vicarious body maps bridge vision and touch in the human brain,” Nature, vol. p. 7 650, 2025 <ref:2603. p. 008255#pg2>.

[9] Z. Wu, Y. Lin, Y. Zhao, X. Zhang, Z. Chen et al. p. 7, “ViTacGen: Robotic pushing with vision-to-touch generation,” IEEE Robotics and Automation Letters, vol. p. 7 10, no. 11, 2025 <ref:2603. p. 008255#pg2>.

[10] R. Gao, Y.-Y. Chang, S. Mall, L. Fei-Fei, and J. p. 7 Wu, “ObjectFolder: A dataset of objects with implicit visual, auditory, and tactile representations,” in Proc. p. 7 of the Conf. on Robot Learning (CoRL), 2021 <ref:2603. p. 708255#pg2>.

[11] Y. Li, J.-Y. Zhu, R. Tedrake, and A. p. 7 Torralba, “Connecting touch and vision via cross-modal prediction,” in Proc. p. 7 of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), 2019 <ref:2603. p. 708255#pg2>.

[12] R. Gao, Y. Dou, H. Li, T. Agarwal, J. p. 7 Bohg et al. p. 7 p. 7, “The Object Folder Benchmark: Multisensory learning with neural and real objects,” in Proc. p. 7 of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), 2023 <ref:2603. p. 708255#pg2>.

[13] S. Zhong, A. Albini, O. P. Jones, P. Maiolino, and I. p. 7 Posner, “Touching a NeRF: Leveraging neural radiance fields for tactile sensory data generation,” in Proc. p. 7 of the Conf. on Robot Learning (CoRL), 2022 <ref:2603. p. 708255#pg2>.

[14] S. Wang, M. Lambeta, P.-W. Chou, and R. p. 7 Calandra, “TACTO: A fast, flexible, and open-source simulator for high-resolution vision-based tactile sensors,” IEEE Robotics and Automation Letters, 2022 <ref:2603. p. 708255#pg2>.

[15] C. Zhang, S. Cui, J. Hu, T. Jiang, T. Zhang et al. p. 7, “TacFlex: Multimode tactile imprints simulation for visuotactile sensors with coating patterns,” IEEE Transactions on Robotics, 2025 <ref:2603. p. 708255#pg2>.

[16] Z. Chen, S. Zhang, S. Luo, F. Sun, and B. p. 7 Fang, “Tacchi: A pluggable and low computational cost elastomer deformation simulator for optical tactile sensors,” IEEE Robotics and Automation Letters, 2023 <ref:2603. p. 708255#pg2>.

[17] Z. Si, G. Zhang, Q. Ben, B. Romero, Z. Xian et al. p. 7, “DIFFTACTILE: a physics-based differentiable tactile simulator for contact-rich robotic manipulation,” in Proc. p. 7 of the Int. p. 7 p. 7 p. 7 p. 7 Conf. on Learning Representations (ICLR), 2024 <ref:2603. p. 708255#pg2>.

[18] Z. Si and W. Yuan, “Taxim: An example-based simulation model for gelsight tactile sensors,” IEEE Robotics and Automation Letters, vol. p. 7 7, no. 2, 2022 <ref:2603. p. 008255#pg2>.

[19] W. Chen, Y. Xu, Z. Chen, P. Zeng, R. Dang et al. p. 7, “Bidirectional sim-to-real transfer for gelsight tactile sensors with cyclegan,” IEEE Robotics and Automation Letters, vol. p. 7 7, no. 3, 2022 <ref:2603. p. 008255#pg2>.

[20] G. M. Caddeo, A. Maracani, P. p. 7 D. Alfano, N. A. Piga, L. Rosasco, and L Natale, “Sim2real bilevel adaptation for object surface classification using vision-based tactile sensors,” in 2024 IEEE International Conference on Robotics and Automation (ICRA), 2024 <ref:2603. p. 708255#pg2>.

[21] R. Gao, Z. Si, Y.-Y. Chang, S. Clarke, J. Bohg et al., “ObjectFolder 2. p. 10: A multisensory object dataset for sim2real transfer,” in Proc. p. 7 of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), 2022 <ref:2603. p. 7 p. 708255#pg2>.

[23] F. Yang, C. Ma, J. Zhang, J. Zhu, W. Yuan, and A. p. 7 Owens, “Touch and Go: Learning from human-collected vision and touch,” in Advances in Neural Information Processing Systems, 2022 <ref:2603. p. 708255#pg2>.

[24] L. Fu, G. Datta, H. Huang, W. C.-H. Panitch, J. p. 7 Drake et al. p. 7, “A touch, vision, and language dataset for multimodal alignment,” in Proc. p. 7 of the Int. Conf. on Machine Learning (ICML), 2024 <ref:2603. p. 708255#pg2>.

[25] Y. Lipman, R. T. Q. Chen, H. Ben-Hamu, M. p. 7 Nickel, and M. p. 7 Le, “Flow matching for generative modeling,” in Proc. p. 7 of the Int. Conf. on Learning Representations (ICLR), 2022 <ref:2603. p. 708255#pg2>.

[26] A. Agarwal, G. p. 7 Singh, B. Sen, T. Lozano-Perez, and L. p. 7 P. Kaelbling, ´ “SceneComplete: Open-world 3d scene completion in cluttered real world environments for robot manipulation,” IEEE Robotics and Automation Letters, vol. p. 7 11, no. 1, 2026 <ref:2603. p. 008255#pg2>.

[27] E. Todorov, T. p. 7 Erez, and Y. p. 7 Tassa, “MuJoCo: A physics engine for model-based control,” in Proc. p. 7 of the IEEE/RSJ Int. p. 7 Conf. on Intelligent Robots and Systems (IROS), 2012 <ref:2603. p. 708255#pg2>.

[28] X. Wei, M. Liu, Z. Ling, and H. p. 7 Su, “Approximate convex decomposition for 3d meshes with collision-aware concavity and tree search,” ACM Transactions on Graphics, vol. p. 7 41, no. 4, 2022 <ref:2603. p. 008255#pg2>.

[29] R. Rombach, A. p. 7 Blattmann, D. p. 7 Lorenz, P. Esser, and B. p. 7 Ommer, “High-resolution image synthesis with latent diffusion models,” in Proc. p. 7 of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), 2022 <ref:2603.08255#pg2>.

[30] O. B. Bohan, “TAESD: Tiny autoencoder for stable diffusion,” https: //github. p. 7com/madebyollin/taesd, 2023 <ref:2603. p. 708255#pg2>.

[31] A. Dosovitskiy, L. p. 7 Beyer, A. Kolesnikov, D. p. 7 Weissenborn, X. p. 7 Zhai et al. p. 7, “An image is worth 16x16 words: Transformers for image recognition at scale,” in Proc. p. 7 of the Int. Conf. on Learning Representations (ICLR), 2021 <ref:2603. p. 708255#pg2>.

[32] H. Daume III, “Frustratingly easy domain adaptation,” in ´ Annual Meeting of the Association of Computational Linguistics, 2007 <ref:2603. p. 708255#pg2>.

[33] M. Johnson, M. p. 7 Schuster, Q. p. 7 Le, M. Krikun, Y. Wu et al., “Google’s multilingual neural machine translation system: Enabling zero-shot translation,” Transactions of the Association for Computational Linguistics, 2017 <ref:2603. p. 708255#pg2>.

[34] C. Higuera, A. p. 7 Sharma, C. K. Bodduluri, T. p. 7 Fan, P. Lancaster et al. p. 7, “Sparsh: Self-supervised touch representations for vision-based tactile sensing,” in Proc. p. 7 of the Conf. on Robot Learning (CoRL), 2024 <ref:2603. p. 708255#pg2>.

[35] M. Oquab, T. Darcet, T. Moutakanni, H. p. 7 V. Vo, M. Szafraniec et al. p. 7, “DINOv2: Learning robust visual features without supervision,” Transactions on Machine Learning Research, 2024 <ref:2603. p. 708255#pg2>.

[36] R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. p. 7 Wang, “The unreasonable effectiveness of deep features as a perceptual metric,” in Proc. p. 7 of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), 2018 <ref:2603. p. 708255#pg2>.

[37] Y. Li, Z.-X. Zou, Z. Liu, D. Wang, Y. Liang et al. p. 7, “Triposg: Highfidelity 3d shape synthesis using large-scale rectified flow models,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025 <ref:2603. p. 708255#pg2>.

[38] T. Julg, P. Krack, S. Bien, Y. Blei, K. Gamal ¨ et al. p. 7, “Robot Control Stack: A lean ecosystem for robot learning at scale,” https://arxiv. p. 7org/abs/2509. p. 714932 <ref:2603. p. 008255#pg2>.

Table I

LIST OF DATASETS USED FOR EVALUATION. INDICATES WITHIN-OBJECT SPLIT <ref:2603. p. 408255#pg2>.

Name Sensor Training/Validation/Self-D GelSight 887 (36) 135 (6) SIM-G 200K (135) 113K (60) YCB-D 2578 (7) 1084 (3) SIM-D — SELF-D — 547 (13) <ref:2603. p. 408255#pg2>.

Table II

ABLATION STUDY ON NETWORK DESIGN CHOICES EVALUATED ON THE GELSIGHT OFR DATASET. ALL METRICS ARE REPORTED ON THE HELD-OUT VALIDATION SET <ref:2603. p. 408255#pg2>.

Table III

QUANTITATIVE COMPARISON OF DIFFERENT SIM-TO-REAL TRAINING TECHNIQUES ACROSS THE PRETRAINING SIM-G, FINETUNING OFR-G, AND UNSEEN SELF-D DATASETS. VALUES FOR EXPERIMENTAL METHODS ARE SHOWN AS ABSOLUTE DIFFERENCES RELATIVE TO THE NAIVE FINETUNING BASELINE AND ARE CALCULATED ON THE VALIDATION DATASET <ref:2603. p. 608255#pg2>.

Table IV

TRAINING DATASET ABLATIONS FOR GRASP STABILITY TASK. VALUES ARE AVERAGED OVER 100 TRAININGS <ref:2603. p. 608255#pg2>.

Fig. 7

Tactile predictions from the grasp stability task on the dataset ablations listed in Tab. IV <ref:2603. p. 008255#pg2>.

**E.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Improved system can predict tactile sensations before physical contact by using an object’s local 3D mesh to encode rich information for predicting tactile patterns while abstracting away from scene-dependent details.

  2. The improved system can bridge the sim-to-real gap and generalize to new sensor instances by using a simulation-to-real training method based on the mesh data of geometric primitives obtained via simulation, which improves output quality, substantially reducing the need for costly real-world data collection.

  3. The improved system can be trained more robustly to domain shifts by incorporating Sparsh, a self-supervised encoder for tactile predictions that retains most of the relevant information of the original tactile image across a wide variety of sensor types.

  4. The improved system can achieve better generalization and focus on highlevel similarities between domains by utilizing Domain Conditioning, which is described as our best performing technique.

  5. The improved system can support downstream manipulation tasks in zero-shot settings by demonstrating that the resulting tactile images can be used for downstream grasp stability prediction even when tested on unseen sensor instances.

Sources

Related papers