Tac-DINO: Learning Tactile Features with Patch Alignment
summary
The gist
Touch is the primary medium through which humans interact with the environment.<ref:2606.12069#pg2> The authors propose a novel vision-tactile patch alignment (VTPA) method to solve vision-tactile…
In short
The authors propose Vision-Tactile Patch Alignment (VTPA) to solve vision-tactile feature learning by enforcing a localized contrastive loss, regardless of whether inputs are single or multi-modal. They developed Touch3D, a large tactile dataset, and demonstrated that VTPA outperforms whole-object alignment methods. Using vision input with auxiliary tactile patch alignment yielded the best results, showing how touch guides vision to achieve fine-grained physical understanding.
Key concepts
- Vision-Tactile Patch Alignment (VTPA)
- This method explicitly enforces a localized contrastive loss between visual and tactile data at the patch level. It ensures consistency between local visual and tactile inputs, whether they come from one modality or fused vision-tactile data, which is key to learning robust physical features.
- Touch3D Dataset
- This is a large 3D-Vision-Tactile dataset comprising 505 real-world objects and over 20,000 tactile samples. It was created using GelSight Mini sensors, providing the largest available dataset for training models to understand physical properties through touch and vision.
- Vis + CL Configuration
- This experimental setup uses vision as the primary input but adds an auxiliary tactile patch alignment loss. This configuration achieved the best performance because the tactile information guides the visual features toward a deeper, more detailed understanding of physical interactions.
Terminology used across episodes
This episode discusses
- Tac-DINO: Learning Tactile Features with Patch Alignment · Paper Radio
- See, Hear, and Read: Deep Aligned Representations
- Revisiting Feature Prediction for Learning Visual Representations from Video
- AnySkin: Plug-and-play Skin Sensing for Robotic Touch
- The Feeling of Success: Does Touch Sensing Help Predict Grasp Outcomes?
- UniForce: A Unified Latent Force Model for Robot Manipulation with Diverse Tactile Sensors
- Touch100k: A Large-Scale Touch-Language-Vision Dataset for Touch-Centric Multimodal Representation
- TacUMI: A Multi-Modal Universal Manipulation Interface for Contact-Rich Tasks
- X-Capture: An Open-Source Portable Device for Multi-Sensory Learning
- A Survey of Sim-to-Real Methods in RL: Progress, Prospects and Challenges with Foundation Models
- AnyTouch: Learning Unified Static-Dynamic Representation across Multiple Visuo-tactile Sensors
- AnyTouch 2: General Optical Tactile Representation Learning For Dynamic Tactile Perception
- ObjectFolder: A Dataset of Objects with Implicit Visual, Auditory, and Tactile Representations
- Generative Adversarial Networks
- Dexterity from Touch: Self-Supervised Pre-Training of Tactile Representations with Robotic Play
- Tactile-Conditioned Diffusion Policy for Force-Aware Robotic Manipulation
- ViTacFormer: Learning Cross-Modal Representation for Visuo-Tactile Dexterous Manipulation
- Visuo-Tactile World Models
- 3D-ViTac: Learning Fine-Grained Manipulation with Visuo-Tactile Sensing
- TacEx: GelSight Tactile Simulation in Isaac Sim -- Combining Soft-Body and Visuotactile Simulators
- DINOv2: Learning Robust Visual Features without Supervision
The paper
Tac-DINO: Learning Tactile Features with Patch Alignment · Read on arXiv
Hong Li, Yankang Dong, Yue Xu, Yihan Tang, Mingzhu Li, Jiamin Qiu, Qihang Yao, Xing Zhu, Yujun Shen
Shanghai Jiao Tong University · Ant Group · Shanghai Innovation Institute
Multisensory perception is essential for robotic manipulation, with tactile sensing providing critical information for fine-grained, contact-rich interactions. Although prior work integrates tactile and spatial signals to improve perception, vision-tactile learning commonly relies on object-level alignment. Addressing this limitation requires systematically aligned 3D vision-tactile data. We develop a data collection system and introduce Touch3D, a dataset comprising 505 real-world objects and 20,025 tactile contacts. Building on this dataset, we establish the Contact-Centric Multimodal Tactile Representation Benchmark (CoMT-Bench), which expands evaluation from object-level retrieval and tactile properties to include patch-level contact evaluation. We further propose Geometry-Relation Distilled Tactile-Centric Cross-Modal Learning (GeoTAC), which explores tactile learning across all multisensory data. In experiments on the Touch3D test set, patch-level alignment improves the average score by 5.32 over object-level alignment, with GeoTAC's geometric supervision improving it by a further 4.48. In robotic manipulation, GeoTAC improves TP by 19.14 and 11.34 over DINOv2 (Vision-only) and DINOv2 (Vision-Tactile), respectively. Project page: https://lihongcs.github.io/tacdino.
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "Tac-DINO: Learning Tactile Features with Patch Alignment".
Jane: Here's the gist of the paper. Touch is the primary medium through which humans interact with the environment.<ref:2606.12069#pg2> The authors propose a novel vision-tactile patch alignment (VTPA) method to solve vision-tactile feature learning, demonstrating that it exceeds the performance of methods without alignment and aligns with whole-object images<ref:2606.12069#pg9>.
Tom: Data Acquisition and Scale The authors first construct a data collection system to acquire a large-scale tactile dataset, with over 20 K tactile contacts from 505 real-world objects<ref:2606.12069#pg5>.
Jane: This system is proposed to gather the largest 3D-Vision-Tactile dataset to date, named Touch3D, which comprises 505 objects and 20,025 tactile samples<ref:2606.12069#pg5>.
Tom: The data collection platform utilizes a GelSight Mini as the visuo-tactile sensor to construct a vision-tactile data collection platform<ref:2606.12069#pg4>.
Title and authors: Tom: So we're looking at the paper "Tac-DINO: Learning Tactile Features with Patch Alignment," and it’s about figuring out how to get AI models to understand touch by aligning vision and touch data. It tackles the problem of vision-tactile feature learning, which is really hard because tactile signals are just local contact points, but we need global understanding Tom.
Jane: Right, so the paper says that while vision excels at object understanding from images, tactile sensing gives us info about local contact—things like texture or shape right where your finger touches something. The main gap they’re trying to bridge is connecting those local interactions to a bigger picture of the whole object Jane.
Lu: Essentially, they propose this Vision-Tactile Patch Alignment method, which forces the model to look at localized vision and tactile data together through a contrastive loss, even if one input is just vision or just touch. This should help them learn representations better than methods that don't do any alignment Lu.
Meng: But how much does this actually improve things when you move from training on local contacts to deploying a model for something real? We need to know if this works reliably in production, not just in a benchmark setting Meng.
Tom: Well, the experiments show that these vision-tactile fused approaches beat the vision-only and tactile-only baselines, which is consistent with what we already expect from combining modalities Tom. They even show that their patch alignment strategy is better than aligning with the whole object image Tom.
Jane: That's interesting because it moves beyond just looking at the whole picture. They introduce a benchmark called the Vis-Tac Holographic Matching Benchmark, which checks if this local-to-global alignment actually helps in understanding physical properties from local contact Jane.
Lu: The paper constructs a big dataset called Touchthree dee, which has over twenty K tactile contacts from five hundred five real objects, and then they design that benchmark to see how well the models align those labeled contact locations on a three-dimensional point cloud with the visible points in each view Lu <ref:2606.12069#pg1,over 20 K tactile contacts from 505 real>.
Meng: twenty thousand samples is a lot of data for collecting, so what does the practical cost look like for someone trying to set this up? We're talking about seven hundred hours just for data collection, which sounds significant for setting up a system Meng.
Tom: That's the scale of the effort involved in getting that dataset, including scanning three dee shapes and collecting all that vision-tactile data Tom <ref:2606.12069#pg1>. But they say it’s necessary to build this kind of large-scale dataset to truly test these alignment methods Tom.
Jane: So, how does their proposed patch alignment mechanism actually work in practice? They talk about enforcing a localized vision-tactile contrastive loss regardless of whether the local data is single-modality or multi-modality Jane.
Lu: For single-modality setups, they define this loss as Zm equals the transformation of Em(xm), and La equals the transformation of Ev(xv) and Et(xt), which ensures consistency between visual and tactile modalities locally Lu.
Title and authors: Tom: And for their vision-tactile fusion setup, they fuse both modalities first to get a unified representation before computing that same patch alignment loss between the individual local modalities Tom. It’s a layered approach to handling the different data types Tom.
Meng: If you're fusing everything upfront, does that just mean you're using more computational power during training without necessarily getting a better physical understanding? We need to know if the extra fusion is worth the cost at scale Meng.
Jane: The results suggest that vision-tactile fused approaches outperform vision-only and tactile-only baselines, which is what we’d expect, but they point out that using vision as input with auxiliary tactile patch alignment gets the best results Jane.
Lu: They attribute those best results to the idea that having touch guide the vision features toward a more fine-grained physical understanding capability Lu. It seems like touch provides a stronger signal for refining what the vision model sees Lu.
Tom: Exactly, it’s about using touch to sharpen the visual features toward a better physical understanding, which is how they show superior performance compared to just aligning with whole object images Tom. The Patch Tac configuration actually beats the baseline Tac by six point three zero in top-one k-NN results on Touchthree dee Tom.
Jane: It’s clear that this method helps classify objects and materials more accurately when using those specific metrics, like the k-NN accuracy for object classification and mean Average Precision for material classification Jane. That tells us it’s doing a better job at grasping what an object is made of compared to just looking at the image alone Jane.
Lu: The paper also flags some limitations. They note that performance can drop or stay the same when using vision-tactile fused local data as input because of insufficient patch-aligned data and the lack of a unified tactile backbone Lu.
Meng: That makes sense, if the underlying tactile representations aren't unified, fusing them might just be adding noise instead of clarity when you try to deploy it on real traffic Meng.
Tom: And they also say that geometric matching requires more advanced semantic strategies, suggesting that using semantic affordance prompts to constrain the manipulation space could lead to more meaningful results Tom. That sounds like a necessary next step for making this tool useful in physical tasks Tom.
Jane: So, while the alignment is strong locally, they think we need better ways to guide that geometry matching by adding semantic information about what things *can* be used for, rather than just trying to match points Jane.
Lu: Future work should focus on developing a unified tactile backbone and combining patch alignment with more elaborate losses to solve those challenges Lu. They also plan to explore integrating that affordance-constrained geometric matching in the next steps Lu.
Tom: So, overall, Tac-DINO shows that explicit local vision-tactile alignment is a solid way to improve feature learning for touch by using datasets like Touchthree dee and achieving better results than simpler methods Tom. It’s a strong step forward for getting AI to grasp physical properties through touch Tom.
Title and authors: Jane: It’s a big step because it shows that connecting vision and touch at the local patch level actually helps create a more robust global understanding of the object, even when you're dealing with complex real-world objects Jane.
Lu: The implication is that for robotics or virtual reality, this kind of method could allow AI to grasp physical properties from simple contact interactions much more effectively than just looking at pictures Lu. It’s about bridging the gap between seeing and feeling things in a way that matters for real-world interaction Lu.
Meng: From a deployment standpoint, if you can achieve better classification accuracy on objects and materials using this method, it means the model is less likely to fail when encountering novel objects or textures in a production setting Meng. That improved reliability is what we need to see Meng.
Tom: So, to wrap up on Tac-DINO: it’s about using Vision-Tactile Patch Alignment to learn features by forcing consistency locally, and the results show that this outperforms just using vision or just using touch alone on a large dataset Tom.
Jane: It really shows how important it is to connect those local sensory signals into a global understanding of the object, and they point toward needing semantic guidance for geometric matching in future work Jane.
Lu: The path forward seems to involve building that unified tactile backbone and adding those semantic prompts so the model knows what to focus on during manipulation Lu. It’s an ambitious direction for representation learning Lu.
Meng: For us at startup, the challenge will be whether we can make that unified backbone run efficiently enough without needing all of that massive data collection effort to get started Meng. The practicality of that is what matters most for scaling up Meng.
Tom: So, Tac-DINO gives us a solid mechanical method for alignment and shows the performance gains, but the next big hurdle is building that unified backbone efficiently enough to actually deploy it widely Tom.
Jane: It’s a really interesting piece of work because it proves that combining vision and touch via patch alignment is a viable way to move toward more physically grounded AI systems Jane. We'll see what they cook up next, but this is definitely worth keeping an eye on for how we can get AI to understand the physical world better Jane.
Lu: Agreed. The focus shifting toward semantic affordance prompts for geometric matching sounds like the right way to make it truly functional in manipulation tasks Lu.
Meng: I'll keep watching that, because if it can handle real-world traffic data without needing seven hundred hours of collection just to train the backbone, then that’s what we’ll be talking about next Meng.
Tom: Alright, so that wraps up our look at Tac-DINO. It's a solid paper showing how local alignment helps in vision-tactile learning, and the direction they point toward is really interesting for making AI interact with the physical world more intelligently Tom.
The paper's summary: Tom: So, we've talked about how these authors created this Vision-Tactile Patch Alignment method to fix the problem of learning features from touch data by forcing consistency locally Tom. Now, let's look at what they actually achieved in terms of their main results.
Jane: The core idea they are proving is that this local alignment helps bridge the gap between knowing what an object looks like and knowing how it feels when you touch it Jane. They show that their vision-tactile fused approaches beat the vision-only and tactile-only baselines, which is expected, but they go further.
Lu: What's really interesting is that they found the patch alignment strategy actually performs better than aligning with the whole object image Lu. The Patch Tac configuration beats the baseline Tac by six point three zero in those top one k nearest neighbor results on their Touchthree dee dataset Lu.
Meng: six point three zero on a specific benchmark, that’s a solid number, but how does that translate to something actually useful when you try to run this model on something messy and real? I mean, we're talking about five hundred five objects and twenty thousand tactile samples in Touchthree dee Meng.
Tom: That dataset size is huge, which is part of the whole point for them to prove their alignment method works at that scale Tom. They also show that using vision as input with auxiliary tactile patch alignment gets the best results Tom.
Jane: And they attribute that best result to something specific: having touch guide the vision features toward a more fine-grained physical understanding capability Jane. It means the touch signal is acting like a fine-tuning knob for what the computer sees Jane.
Lu: Exactly. The paper argues that this tactile guidance sharpens the visual features into something that understands physics better than just looking at pictures or just reading raw touch points Lu. They use a benchmark called Vis-Tac Holographic Matching to check this local-to-global alignment for understanding physical properties from local contact Lu.
Tom: So, the big picture here is that they aren't just making a classifier that gets slightly better; they are designing a mechanism that fundamentally changes how vision and touch data interact during training Tom. It's about making the model inherently more grounded in physics Tom.
Jane: It shows that this isn't just some clever trick for one type of data, but a way to structure the learning process itself so that local contact information actually contributes to global understanding Jane. This is what we need for any system interacting with the physical world.
Lu: The caveat they do mention is that using vision-tactile fused data locally can sometimes drop performance if you don't have a unified tactile backbone or enough patch-aligned data Lu. They also suggest that geometric matching needs more advanced semantic strategies, like using affordance prompts to constrain the manipulation space Lu.
Meng: So, they’re admitting that just having the alignment isn't enough; you need better semantic guidance for how the model should move or interact with things physically Meng. That makes sense if we're trying to build something that can actually grab and manipulate objects reliably on a real factory floor Meng.
Tom: The implication is that future work needs to focus on building that unified backbone and combining this patch alignment with those semantic prompts they mentioned Tom. It’s moving from just getting better classification numbers to building systems that truly understand how to use what they sense.
Jane: That's the direction they are pushing, moving beyond just matching points and into guiding physical action based on what the object *affords* you to do Jane. It really shows where the frontier is in this kind of multimodal AI research.
The paper's improvements: Tom: So, we’ve seen how the original paper proposed Patch Alignment to solve vision-tactile feature learning by enforcing that local contrastive loss regardless of whether you have single or multiple data types Tom. Now, let's talk about what they suggest we do next with this method Tom.
Jane: They are pointing out a few ways they think this system can be improved Jane. First, they say the geometric matching part needs more advanced semantic strategies to constrain the manipulation space Lu. That means instead of just trying to match points, the model should use information about what things are *for* Jane.
Lu: Exactly. They suggest introducing semantic affordance prompts so that geometric matching only focuses on functional areas, which should lead to more meaningful results when comparing tactile point clouds against scanned three dimensional shapes Lu. It’s about adding context to the geometry Lu.
Meng: That sounds like a necessary step for any real application, but what does it cost? If we have to design and label those semantic affordance prompts, does that just increase the training time or make deployment harder Meng?
Tom: The authors are planning future work to focus on developing a unified tactile backbone and combining this patch alignment with more elaborate losses Tom. That’s the heavy lifting they think is required for the system to be truly robust Tom.
Jane: And they also plan to explore integrating that affordance-constrained geometric matching in their next steps Jane. They want the system to not just see a shape but understand how you can actually use that shape physically Jane. It’s about making the AI more capable of interaction, not just recognition.
Lu: The implication is that the next generation of this research will focus on building that unified tactile backbone and using those elaborate losses to solve the challenges they identified Lu. They are aiming for a system where touch and vision work together seamlessly across all scales Lu.
Tom: So, they’re moving from just getting better classification accuracy on objects to building systems that actually understand how to use what they sense Tom. It’s about moving from passive recognition to active understanding Tom.
Jane: That makes sense because the current limitation they noted is that vision-tactile fused local data isn't always enough without that unified backbone Jane. They are trying to fix the foundation so these localized improvements stick globally Jane. This moves us toward truly physically grounded AI systems, which is a big deal for how we build robots or virtual reality experiences Jane.
Conclusion: Tom: So, to wrap up on Tac-DINO, it’s about using Vision-Tactile Patch Alignment to learn features by forcing consistency locally Tom. The big takeaway is that this method outperforms just using vision or just using touch alone on a large dataset Tom.
Jane: It really shows how important it is to connect those local sensory signals into a global understanding of the object, and they point toward needing semantic guidance for geometric matching in future work Jane. It’s about bridging the gap between seeing and feeling things in a way that matters for real-world interaction Jane.
Lu: They are aiming for a system where touch and vision work together seamlessly across all scales Lu. This is ambitious, but they've laid out a path forward by focusing on building that unified tactile backbone and adding those semantic prompts Lu. It’s about making the AI more capable of interaction, not just recognition Lu.
Meng: I still have to ask what it costs to run that system once we get those semantic prompts in there Meng. If the model gets way more complex because of all those losses, does it just become too slow or too expensive for us to deploy?
Tom: The authors admit that performance can drop or stay the same when using vision-tactile fused local data as input due to insufficient patch-aligned data and the absence of a unified tactile backbone Tom. So they’re being honest about where things are still shaky Tom.
Jane: That is fair, and it tells us we aren't done yet because they know the limitations of their current approach Jane. The implication is that solving those issues with a unified backbone is the next big hurdle for this field Jane.
Lu: I think the focus shifting toward semantic affordance prompts for geometric matching sounds like the right way to make it truly functional in manipulation tasks Lu. That’s where they need to go to get meaningful results beyond just classification numbers Lu.
Meng: If you can get that unified backbone running efficiently without needing all of that massive data collection effort, then that’s what we’ll be talking about next Meng. The practicality of that is what matters most for scaling up the whole concept Meng.
Tom: Alright, so Tac-DINO shows a solid mechanical method for alignment and shows the performance gains compared to simpler methods Tom. It’s a strong step forward for getting AI to grasp physical properties through touch Tom.
Jane: It really shows how important it is to connect those local sensory signals into a global understanding of the object Jane. They're moving toward systems that can actually interact with the physical world in a way that makes sense Jane.
Lu: The path forward seems to involve building that unified tactile backbone and adding those semantic prompts so the model knows what to focus on during manipulation Lu. It’s an ambitious direction for representation learning Lu.
Meng: I'll keep watching that, because if it can handle real-world traffic data without needing seven hundred hours of collection just to train the backbone, then that’s what we’ll be talking about next Meng. The real test is deployment readiness Meng.
Tom: Alright, so that wraps up our look at Tac-DINO. It's a solid paper showing how local alignment helps in vision-tactile learning, and the direction they point toward is really interesting for making AI interact with the physical world more intelligently Tom.
More episodes
- 2603.29765-Training-Free Dynamic Upcycling of Expert Language Models
- 2605.00778-Observable Performance Does Not Fully Reflect Adaptive System Organization: A Multi-Level Analysis of Gait Dynamics Under Occlusal Constraint
- 2605.06906-TraXion: Rethinking Pre-training Frameworks for Mobility and Beyond
- 2605.09165-Sparse Layers are Critical to Scaling Looped Language Models
- 2605.09439-Inverse Design for Conditional Distribution Matching
- 2605.10313-Signature Approach for Contextual Bandits with Nonlinear and Path-dependent Rewards: Regret Bound and Statistical Inference
- 2605.10878-Minimum Neural Weight Norm = Kolmogorov Complexity
- 2605.11617-MIST: Reliable Streaming Decision Trees for Online Class-Incremental Learning via McDiarmid Bound
- 2605.15085-From Data to Action: Accelerating Refinery Optimization with AI
- 2605.23500-GRTO: Bootstrapped Group Relative Tool Optimization for Referring Segmentation