AetheRock: An Arm-Worn Robot Teaching System for Force-Guided Vision-Tactile Learning

arXiv:2606.09777 · cs.RO · Submitted 2026-06-08 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: I'm Rosa.

Dev: And I'm Dev, here with Taro.

Rosa: We're looking at the paper: "AetheRock: An Arm-Worn Robot Teaching System for Force-Guided Vision-Tactile Learning".

Dev: Here's the gist of the paper. Force and tactile sensing are indispensable in contact-rich manipulation However, force-aware robot learning faces critical challenges due to the incompatible assembly of tactile and force sensors in handheld or wearable devices.

Rosa: To address these limitations, the work introduces AetheRock for gripper-force, vision, and tactile data collection.

Dev: Building on this, the proposed ForceVT is a representation learning framework that uses force and vision to guide fidelity-agnostic tactile learning.

Rosa: System Design and Hardware Innovations AetheRock is an arm-worn device designed for collecting gripper force, vision, and tactile data.

Title and authors: Dev: Let’s talk about how they actually built this system, AetheRock. They focused on three main hardware innovations to make sure the data collection is solid across the board.

Rosa: First, they have that resistive pressure sensor adapted for the central controller specifically for robust gripper force collection, which is a big step since force sensing isn't always straightforward on small devices.

Taro: They also introduced this new visuo-tactile GelSlim-MiniFab, which builds on the optics and circuit board design of the GelSlim four point zero but focuses on low cost manufacturing and making it easy to repair if something gets damaged <ref:2606.09777#pg1>.

Dev: And they used modular PCBs combined with a wearable kit to ensure stable connectivity and comfortable collection, which is important because you can't have a system that's too cumbersome for real use.

Rosa: It sounds like they’ve made a lot of effort into making the physical setup practical, moving away from high-cost, rigid setups that are hard to adapt to different robot grippers.

Taro: And they also mentioned integrating six-axis Force/Torque sensors into this system, which is significant because existing work often struggles with the size and cost of those sensors when trying to place them near a gripper <ref:2606.09777#pg2>.

Dev: That’s the point, they are directly addressing the challenge that applying a six-axis F/T sensor in current setups faces regarding size and high costs that hinder mass production <ref:2606.09777#pg2,applying a 6-axis F/T sensor in>.

Rosa: So, by combining the GelSlim-MiniFab with that pressure sensor and modular PCBs, they are creating a platform designed to collect qualified data quality for robot learning.

Taro: Qualified data quality means they're aiming for one hundred percent usability of the collected information across different modalities during training and deployment.

Dev: This hardware setup provides a continuous window of data collection that is four times longer than what UMI has done, which really increases the volume and consistency of training material available for policy learning.

Rosa: That volume and consistency are crucial because they feed directly into the representation learning framework they are developing next, ForceVT.

Taro: So we've covered the hardware that collects the data, now we need to look at how ForceVT actually processes that force, vision, and tactile information to make it useful for learning.

The paper's summary: Dev: Moving into the algorithm, ForceVT is a force-guided vision-tactile learning algorithm designed specifically to reduce the influence of inconsistency between the vision and tactile sensors during manufacturing or deployment.

Rosa: It’s about using that force data and vision to guide the tactile learning in a way that keeps it robust even when those two sensing modalities aren't perfectly aligned.

Taro: They have several components in this framework, starting with Tactile Random Masking, which perturbs the tactile observations using random spatial masking to increase the diversity of tactile situations and improve robustness against any specific tactile situation.

Dev: That masking step is important because it’s a direct way to make sure the model doesn't rely too heavily on one single pattern in the input data.

Rosa: Then they use modality-specific encoders that take the aligned wrist vision, force data, and those masked tactile observations and turn them into latent representations.

Taro: Those latent representations are then fed into a lightweight fusion module that combines the vision and force features to create a fused vision-force representation.

Dev: And finally, they use a soft alignment loss, Lsoft, which transfers relational similarity across different tactile fidelities from that fused vision-force space over to the actual tactile space.

Rosa: That’s interesting because it directly tries to bridge the gap between the idealized force-vision representation and what the actual imperfect tactile sensor is providing.

Taro: They also have a contrastive loss, Lcon, which computes cosine similarity between those vision-force representations and the tactile representations to achieve cross-modality alignment.

Dev: So, in short, they have masking for diversity, encoders for representation, fusion for combined features, and then two losses to align everything across the different sensory inputs.

Rosa: It sounds like a lot of moving parts working together to ensure that no matter how messy the input data is from manufacturing or use-case deployment, the learned policy can still perform well.

The paper's improvements: Taro: So they took this entire system and tested it across six different robotic tasks to see if all these algorithmic and hardware improvements actually translate into real performance gains.

Dev: The results show that AetheRock provides qualified data quality, but when you incorporate the force-tactile information using ForceVT, the policy performance improves across all tasks compared to baseline methods.

Rosa: For instance, on tasks like pick-and-place for rigid objects, they saw over eighty percent task progress with less than twenty minutes of demonstrations. That’s a lot of efficiency for data collection.

Taro: But there's a caveat here when you look at the scaling law analysis; when they extend the data collection and inference to a global generalization setting, the policy performance actually declines significantly compared to what you see in their earlier tables.

Dev: That decline suggests that while ForceVT helps locally, it might not fully solve the generalization issue when deploying it far outside of the training data distribution.

Rosa: But even with that caveat, ForceVT consistently improves performance across all six tasks when compared to baseline methods because it incorporates force information and achieves better alignment across all modalities.

Taro: And one specific example is on board erasing, where ForceVT actually got a fifty-eight point zero percent success rate compared to forty-seven point zero percent for the simple vision-tactile linear fusion baseline.

Dev: That comparison really shows how much incorporating the force information and achieving better alignment across all modalities leads to better success rates in contact-rich tasks.

Rosa: So, while they have these strong wins in controlled settings, we still need to pay attention to that scaling law because that tells us what happens when the system gets really far from its training environment.

Conclusion: Dev: Now let's look at what they admitted is tricky with this work. A primary limitation they point out is an inconsistency in the tactile contact signals between the time you are training and when you are actually making inferences.

Rosa: That means the system isn't perfect in translating its learned skills to real-world operation because that fidelity gap still exists between training and inference.

Taro: And to address this generalization issue, a focus for future work will be developing algorithms that involve inference feedback during the data collection process itself.

Dev: That sounds like they are looking at how the system can learn to correct its own mistakes or adapt in real-time based on immediate feedback during deployment.

Rosa: That’s smart thinking because trying to solve that data inconsistency issue by changing how you collect the data is a huge engineering challenge.

Taro: Another future direction they mentioned is studying pre-training with a large amount of data, which suggests that more extensive initial training could help build a more stable foundation for the system.

Dev: And they also suggest that assembling a robot system with a low-cost six-axis Force/Torque sensor, CoinFT, might actually align better with tactile signals than what they used in this study <ref:2606.09777#pg2>.

Rosa: It seems like their ultimate goal is to create a truly robust system where the data collection process itself helps solve the generalization problem rather than just relying on post-training adjustments.

Taro: So to wrap up, AetheRock and ForceVT show that combining force, vision, and tactile data can lead to better contact manipulation performance, even with noisy hardware.

Dev: Yes, they do improve success rates compared to baselines in tasks like board erasing by incorporating force information and achieving better alignment across all modalities.

Rosa: The paper on AetheRock: An Arm-Worn Robot Teaching System for Force-Guided Vision-Tactile Learning is a strong piece of work that shows how targeted hardware design and representation learning can make a difference in contact manipulation.

Hong Li, Yue Xu, Yihan Tang, Yankang Dong, Chenyuan Liu, Chenyang Yu, *Xuyang Li*, *Siyuan Huang*, Yujun Shen, Nan Xue2

Shanghai Jiao Tong University · Ant Group

cs.RO

Submitted: 2026-06-08

Updated: 2026-11-08

Project page: https://rhos.ai/research/AetheRock

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 87/100

The gist: Force and tactile sensing are indispensable in contact-rich manipulation However, force-aware robot learning faces critical challenges due to the incompatible assembly of tactile and force sensors in…

Key concepts

AetheRock
A wearable device worn on an arm designed to collect crucial sensory data for robotics. It integrates a resistive pressure sensor for gripper force, a visuo-tactile GelSlim-MiniFab for vision and touch, and modular PCBs to ensure stable connectivity while collecting high-quality data.
ForceVT
A representation learning framework that uses force and vision to guide tactile learning. It incorporates techniques like tactile random masking and cross-modality alignment losses to create robust representations that are less sensitive to inconsistencies between vision and tactile sensors during training.
Tactile Random Masking
A technique used in the algorithm where the raw tactile observations are intentionally perturbed using stochastic spatial masking. This process introduces diversity into the tactile data, making the resulting learned representations more robust and capable of handling various, unpredictable contact situations.
Cross-Modality Alignment Loss (Lcon)
A specific loss function used in ForceVT that computes the cosine similarity between the vision-force representation and the tactile representation. This loss encourages the learned features from different sensor modalities to be more similar, achieving better alignment across vision, force, and touch data.

Terminology

Summary

Force and tactile sensing are indispensable in contact-rich manipulation However, force-aware robot learning faces critical challenges due to the incompatible assembly of tactile and force sensors in handheld or wearable devices. p. 0

Where the paper says it

cn Abstract: Force and tactile sensing are indispensable in contact-rich manipulation. However, force-aware robot learning faces critical challenges due to the incompatible assembly of tactile and force sensors in handheld or wearable devices.

To address these limitations, the work introduces AetheRock for gripper-force, vision, and tactile data collection. p. 0
Where the paper says it

However, force-aware robot learning faces critical challenges due to the incompatible assembly of tactile and force sensors in handheld or wearable devices. To address these limitations, we first introduce AetheRock for gripper-force, vision, and tactile data collection, which is an arm-worn device featuring a modular and easily manufactured visuo-tactile sensor, GelSlim-MiniFab, at the fingertip, a resistive pressure sensor at the human finger contact region, a customized PCB module, and a wearable kit for comfortable and robust collection.

Building on this, the proposed ForceVT is a representation learning framework that uses force and vision to guide fidelity-agnostic tactile learning. p. 0
Where the paper says it

To address these limitations, we first introduce AetheRock for gripper-force, vision, and tactile data collection, which is an arm-worn device featuring a modular and easily manufactured visuo-tactile sensor, GelSlim-MiniFab, at the fingertip, a resistive pressure sensor at the human finger contact region, a customized PCB module, and a wearable kit for comfortable and robust collection. Building on this, we propose ForceVT, a representation learning framework that uses force and vision to guide fidelity-agnostic tactile learning, enabling robust inference in any tactile situation.

System Design and Hardware Innovations

AetheRock is an arm-worn device designed for collecting gripper force, vision, and tactile data. Key innovations in the hardware include:

  1. A resistive pressure sensor adapted to the central controller for robust gripper force collection. p. 1

    Where the paper says it

    Utilizing finger operations similar to FreeTacMan [16], the system collects UMI end-effector data enhanced by multi-sensor perception. Our system introduces three key innovations: (1) A resistive pressure sensor adapted to the central controller for robust gripper force collection; (2) A new visuo-tactile GelSlim-MiniFab builds on the optics and PCB design of GelSlim 4.

  2. A new visuo-tactile GelSlim-MiniFab which builds on the optics and PCB design of GelSlim 4. p. 1

    Where the paper says it

    Utilizing finger operations similar to FreeTacMan [16], the system collects UMI end-effector data enhanced by multi-sensor perception. Our system introduces three key innovations: (1) A resistive pressure sensor adapted to the central controller for robust gripper force collection; (2) A new visuo-tactile GelSlim-MiniFab builds on the optics and PCB design of GelSlim 4.

    0 with low-cost manufacturing and easy-to-repair features. p. 1
    Where the paper says it

    Our system introduces three key innovations: (1) A resistive pressure sensor adapted to the central controller for robust gripper force collection; (2) A new visuo-tactile GelSlim-MiniFab builds on the optics and PCB design of GelSlim 4. 0 [17], with low-cost manufacturing and easy-to-repair features; (3) Modular PCBs combined with an arm-worn kit for stable connectivity and comfortable collection.

  3. Modular PCBs combined with an arm-worn kit for stable connectivity and comfortable collection. p. 1

    Where the paper says it

    Our system introduces three key innovations: (1) A resistive pressure sensor adapted to the central controller for robust gripper force collection; (2) A new visuo-tactile GelSlim-MiniFab builds on the optics and PCB design of GelSlim 4. 0 [17], with low-cost manufacturing and easy-to-repair features; (3) Modular PCBs combined with an arm-worn kit for stable connectivity and comfortable collection.

Algorithm Development for Robust Learning

ForceVT is a force-guided vision-tactile learning algorithm designed to reduce the influence of vision-tactile inconsistency during manufacturing and deployment. This framework incorporates several components:

  1. Tactile Random Masking, which perturbs tactile observations through stochastic spatial masking to increase tactile diversity and improve robustness to any tactile situation. p. 4

    Where the paper says it

    Figure 4 presents ForceVT, a post-training framework featuring Robust Tactile Representation Learning for robustness to tactile variations and Cross-Modality Soft Alignment for preserving consistency between the relational structure of tactile features and that of fused vision–force features. 1 Robust Tactile Representation Learning To increase tactile diversity and improve robustness to any tactile situation, we introduce Tactile Random Masking, which perturbs tactile observations through stochastic spatial masking.

  2. Modality-specific encoders that encode the aligned wrist vision, force data, and masked tactile observations into latent representations. p. 4

    Where the paper says it

    For tactile observations from the k-th fidelity sensor, the process is formulated as: T˜(k) = Mρ,η(T (k) ), (1) where Mρ,η denotes the masking operator, and T (k) and T˜(k) represent the raw and masked tactile observations from the k-th fidelity sensor, respectively. The masked tactile input T˜(k) , together with aligned wrist vision Iv and force data F, is encoded by modality-specific encoders to obtain latent representations: zv = Ev(Iv), zf = Ef (F), z˜ (k) t = Et(T˜(k) ), (2) where Ev, Ef , and Et denote the vision, force, and tactile encoders, respectively.

  3. A lightweight fusion module that fuses vision and force features to obtain a fused vision-force representation. p. 5

    Where the paper says it

    2 ForceVT: Force Guided Vision Tactile Learning We first fuse vision and force features with a lightweight fusion module: zvf = F(zv, zf ), (3) where zvf denotes the fused vision-force representation. To learn fidelity-agnostic tactile representations, we introduce a soft alignment that transfers relational similarity across different tactile fidelities from the fused vision force space to tactile space: Lsoft = 1 N2 X i,j sim(z i vf , z j vf ) − sim(˜z i,(ki) t , z˜ j,(kj ) t ) 2 2 , (4) where ki and kj denote the tactile fidelity levels of samples i and j, respectively.

  4. A soft alignment loss (Lsoft) that transfers relational similarity across different tactile fidelities from the fused vision force space to tactile space. p. 5

    Where the paper says it

    2 ForceVT: Force Guided Vision Tactile Learning We first fuse vision and force features with a lightweight fusion module: zvf = F(zv, zf ), (3) where zvf denotes the fused vision-force representation. To learn fidelity-agnostic tactile representations, we introduce a soft alignment that transfers relational similarity across different tactile fidelities from the fused vision force space to tactile space: Lsoft = 1 N2 X i,j sim(z i vf , z j vf ) − sim(˜z i,(ki) t , z˜ j,(kj ) t ) 2 2 , (4) where ki and kj denote the tactile fidelity levels of samples i and j, respectively.

  5. A contrastive loss (Lcon) that computes cosine similarity between vision force and tactile representations to achieve cross-modality alignment. p. 5

    Where the paper says it

    Thus, the tactile similarity is computed between tactile representations that may come from different fidelity sensors. To achieve cross-modality alignment, we adopt a contrastive loss between vision force and tactile representations: Lcon = − 1 N X N i=1 log exp(sim(z i vf , z˜ i,(ki) t )/τ ) PN j=1 exp(sim(z i vf , z˜ j,(kj ) t )/τ ) , (5) where sim(·, ·) denotes cosine similarity, τ is a temperature parameter, and N is the batch size.

Experimental Validation and Results

Experiments evaluate the effectiveness of the hardware and algorithm in real-world robotics across six robotic tasks. The results demonstrate that AetheRock provides qualified data quality, and incorporating force-tactile information with ForceVT improves policy performance and robustness. p. 1

Where the paper says it

Experiments demonstrate that our hardware provides qualified data quality and that incorporating force-tactile information with the proposed algorithm improves policy performance and robustness. In summary, our primary contributions include: (1) An arm-worn robot teaching system, AetheRock, to robustly and conformably collect gripper force, vision, and tactile robot data.

Specifically:

  1. Data collection efficiency shows that pick-and-place tasks can attain over an 80% task progress for rigid objects with less than 20 minutes of demonstrations. p. 6

    Where the paper says it

    Figure 5 and Table 1 show the results of data collection efficiency across six tasks. Overall, without involving global generalization, pick-and-place tasks can attain over an 80% task progress for rigid objects and a 60% task progress for soft objects with less than 20 minutes of demonstrations.

  2. Post-training scaling law analysis reveals that when extending data collection and inference to a global generalization setting, the policy performance declines significantly compared to Table 5. p. 6

    Where the paper says it

    Overall, with data scaling, the policy performance improves to a certain extent. When extending data collection and inference to a global generalization setting, the policy performance declines significantly compared to Table 5.

  3. ForceVT consistently improves performance across all tasks compared to baseline methods because it incorporates force information and achieves better alignment across all modalities. p. 7

    Where the paper says it

    TacFiLM and TactileConcat generally perform worse than simple linear fusion, indicating that fusing vision and tactile modalities too early is less effective than later latent fusion and exhibits weaker generalization across tactile sensors, as our sensor is not trained on the T3 model. From Table 3, baseline methods exhibit substantial performance degradation when tactile fidelity decreases, with a maximum decrease close to 20%, whereas our method remains robust with changes below 5%.

Limitations and Future Work

A primary limitation of this work is an inconsistency in tactile contact signals between training and inference. Furthermore, developing algorithms involving inference feedback in data collection to solve this generalization issue will be a focus of future work. p. 8

Where the paper says it

AetheRock enables flexible manipulation and data collection, but real-robot inference, with its limited perception and execution, produces edge contacts and pad contacts. Developing algorithms and involving inference feedback in data collection to solve this generalization issue will be a focus of future work.

Future research will also focus on studying pre-training with a large amount of data. p. 8
Where the paper says it

Finally, tactile sensors show improvement in our work with contact status guidance signals, but this is weak due to our current study concentrating on post-training. Studying pre-training with a large amount of data will be future work.

The authors suggest that assembling a robot system with a low-cost 6-axis F/T sensor, Coin-FT [47], can better align with tactile signals. p. 8
Where the paper says it

Additionally, tactile sensing provides contact texture signals and also includes shear force perception. Assembling a robot system with a low-cost 6-axis F/T sensor, Coin-FT [47], can better align with tactile signals.

Task Definitions and Progress Metrics

The paper defines task progress in detail rather than relying solely on the success rate. Six tasks are included: Clamp Seal, Towel Hanging, Pick Bread, Erase Board, Pick Block, and Insert Flower. p. 5

Where the paper says it

0% Towel Hanging 206 27. 5% Pick Bread 199 17.

For four tasks (Erase Board, Screwdriving, Close Bottle, and Stowing), about 490 episodes are collected to verify the post-training scaling law. p. 7
Where the paper says it

Specifically, two-thirds of the failures in the stowing task fall into these two categories, while the remaining failures involve collisions with the block. Methods Erase Board Screwdriving Close Bottle Stowing Vision-only 18.

Task progress scores range from 0 to 1. p. 5
Where the paper says it

Only fisheye camera input to the model. Task Number of demos Data Collection Duration (min) Task progress Clamp Seal 229 28.

0 and describe specific outcomes for each task. p. 4
Where the paper says it

Figure 4: Schematic framework of the proposed ForceVT. Multi-fidelity tactile data are augmented via random masking, followed by modality-specific encoding.

The definition of task progress is detailed in Table 4. p. 6
Where the paper says it

For the global generalist setting, we moved the objects to five random positions and performed inference four times at each position. The detailed training and inference settings, as well as the task progress definition, are provided in the supplementary material.

GelSlim-MiniFab Manufacturing Details

The GelSlim-MiniFab sensor manufacturing involves several steps designed for low cost and easy repairability. The key steps are:

  1. Mold Installation, where a shell is designed to keep a consistent silicone thickness and protect useful components. p. 1

    Where the paper says it

    Currently, since the tactile sensor cost is relatively high, transferring to a new sensor due to slight scratches or lifted silicone would be a waste of resources. To address uponn challenges, we firstly present AetheRock, as shown in the left of Figure 1, an arm-worn robot teaching system designed for force, vision and tactile data collection.

  2. Low-Cost 2D scrylic cut and fix, refining the optical router to ensure it can adapt to only vertical cutting. p. 2

    Where the paper says it

    0 [17] optics and PCB design, we propose GelSlim-MiniFab, as shown in Figure 2, which is a visuo-tactile sensor with low-cost manufacturing and easy-to-repair features. GelSlim-MiniFab has three key innovations: (1) Low optical requirements, which permit users to use 2D-cut acrylic rather than high-cost 3D acrylic.

  3. Make transparent deformation layers using PDMS with a curing condition set at 65 Celsius for 60 minutes. p. 6

    Where the paper says it

    Figure 5 and Table 1 show the results of data collection efficiency across six tasks. Overall, without involving global generalization, pick-and-place tasks can attain over an 80% task progress for rigid objects and a 60% task progress for soft objects with less than 20 minutes of demonstrations.

  4. Make Reflective silicone, where adding one gram of ultrafine silver powder to 10 ml of IPA yields the best results.

  5. Make a Reflective layer by combining the transparent deformation layer with reflective silicone. p. 1

    Where the paper says it

    These inconsistencies cannot be thoroughly eliminated, which also influences the model’s performance. Currently, since the tactile sensor cost is relatively high, transferring to a new sensor due to slight scratches or lifted silicone would be a waste of resources.

  6. Dismantle the mold by cutting away the support material around the silicone.

Sensor Calibration Protocols

The paper details calibration protocols for both the resistive pressure sensor and the tactile measurements. For the resistive pressure sensor, a piecewise linear calibration protocol via a 16-bit I2C ADC interface is implemented, mapping raw analog signals to the target physical force span of 50–1000 g in Newtons (N). p. 4

Where the paper says it

To process the resistive pressure sensor readings, we implement a piecewise linear calibration protocol via a 16-bit I2C ADC interface. The raw analog signals are first mapped to the system voltage domain.

For tactile calibration, a geometric calibration pipeline is developed to reconstruct the complete sensor geometry by extrapolating valid contour edges and mapping distorted physical vertices onto a dynamically calculated, mathematically ideal regular polygon. p. 3
Where the paper says it

Next, we reconstruct the complete sensor geometry by extrapolating the valid contour edges to recover the hidden vertices, effectively capturing the inherent perspective tilt and spatial asymmetry. Finally, a homography transformation maps these distorted physical vertices onto a dynamically calculated, mathematically ideal regular polygon centered within the image frame.

The system time calibration uses a localized high-precision Network Time Protocol (NTP) synchronization mechanism to achieve stable temporal alignment with an average synchronization latency of 10 ms. p. 3
Where the paper says it

3 Data Process System Time Calibration. To guarantee precise temporal alignment across the PC and OrangePi central control during data collection, we developed a localized high-precision Network Time Protocol (NTP) synchronization mechanism.

Training and Inference Settings

The training setting utilized π0.5 [4] as the pretrained model for post-training on the collected data. p. 6

Where the paper says it

The insert flower task requires reasoning about the gripper pose, erroneous positions obstruct the observation, which is difficult for the current model. Post-training Scaling Law.

For ForceVT, training data was split into seven groups: one from 100% fidelity tactile data and five from 100% fidelity data, and one group at 75% and 50% fidelity. p. 5
Where the paper says it

Subsequently, we selected four tasks and employed three levels of tactile fidelity to simulate inconsistency. We divided the data into groups of 70 episodes each, collecting five groups at 100% fidelity, and one group at 75% and 50% fidelity.

Inference is performed on a Flexiv Rizon 4 robot arm with a Flexiv Grav adaptive gripper, using a unified init pose across all policies for fair comparison. p. 6
Where the paper says it

5 [4], revised to learn the relative pose as in UMI [10]. All trained models performed inference on a Flexiv Rizon 4 robot arm with a Flexiv Grav adaptive gripper.

The robot control frequency was set to 10 Hz. p. 9
Where the paper says it

Ichter, et al. pi 0: A vision-language-action flow model for general robot control.

Ablation Study of ForceVT Components

The ablation study shows the importance of different components in the proposed ForceVT method. The results indicate that both the force-vision-tactile alignment and cross-modality alignment yield relatively larger improvements compared to tactile random masking. p. 4

Where the paper says it

Figure 4 presents ForceVT, a post-training framework featuring Robust Tactile Representation Learning for robustness to tactile variations and Cross-Modality Soft Alignment for preserving consistency between the relational structure of tactile features and that of fused vision–force features. 1 Robust Tactile Representation Learning To increase tactile diversity and improve robustness to any tactile situation, we introduce Tactile Random Masking, which perturbs tactile observations through stochastic spatial masking.

Early fusion strategies underperform compared to late concatenation fusion when training data is limited during post-training. p. 7
Where the paper says it

This issue is caused by inconsistent grasp positions, where grasping occurs on a crossbar during training but on a ribbed shaft during inference. TacFiLM and TactileConcat generally perform worse than simple linear fusion, indicating that fusing vision and tactile modalities too early is less effective than later latent fusion and exhibits weaker generalization across tactile sensors, as our sensor is not trained on the T3 model.

The results indicate that the cross-sensor tactile representation exhibits poor generalization because TactileConcat utilizes T3 [46] as its pretrained model, which was trained on datasets from GelSight Finray [48], GelSight Svelte [49], GelSight Wedge [50], DenseTact 2. p. 7
Where the paper says it

This issue is caused by inconsistent grasp positions, where grasping occurs on a crossbar during training but on a ribbed shaft during inference. TacFiLM and TactileConcat generally perform worse than simple linear fusion, indicating that fusing vision and tactile modalities too early is less effective than later latent fusion and exhibits weaker generalization across tactile sensors, as our sensor is not trained on the T3 model.

0 [51], and GelSight 360 [52]. The full ForceVT framework comprises several core components including simple vision-tactile linear fusion, Tactile Random Masking, soft force-vision-tactile alignment, and cross-modality alignment. p. 4
Where the paper says it

Figure 4 presents ForceVT, a post-training framework featuring Robust Tactile Representation Learning for robustness to tactile variations and Cross-Modality Soft Alignment for preserving consistency between the relational structure of tactile features and that of fused vision–force features. 1 Robust Tactile Representation Learning To increase tactile diversity and improve robustness to any tactile situation, we introduce Tactile Random Masking, which perturbs tactile observations through stochastic spatial masking.

Policy Performance Comparison

Table 5 shows the ablation results for the proposed ForceVT method. The results indicate that our method consistently improves performance across all tasks compared to baseline methods because it incorporates force information and achieves better alignment across all modalities. p. 7

Where the paper says it

TacFiLM and TactileConcat generally perform worse than simple linear fusion, indicating that fusing vision and tactile modalities too early is less effective than later latent fusion and exhibits weaker generalization across tactile sensors, as our sensor is not trained on the T3 model. From Table 3, baseline methods exhibit substantial performance degradation when tactile fidelity decreases, with a maximum decrease close to 20%, whereas our method remains robust with changes below 5%.

Specifically, ForceVT consistently improves performance across all tasks. p. 7
Where the paper says it

Tables 2 and 3 report vision tactile policy performance and robustness, respectively. Overall, tactile sensing improves policy performance in both VisTacLinear and ForceVT.

The results indicate that by incorporating force information and achieving better alignment across all modalities, our ForceVT consistently improves performance across all tasks. p. 1
Where the paper says it

Finally, as shown on the right of Figure 1, we evaluate data collection efficiency on six robotic tasks and evaluate post-training scaling laws and vision-tactile learning on four robotic tasks. Experiments demonstrate that our hardware provides qualified data quality and that incorporating force-tactile information with the proposed algorithm improves policy performance and robustness.

Tactile Learning Algorithm Comparison

Figure 13 shows the results of different vision-tactile learning algorithms. Compared to the vision-only baseline, the simple visuo-tactile linear fusion method yields performance improvements in board erasing, bottle closing, and stowing. p. 7

Where the paper says it

TacFiLM and TactileConcat generally perform worse than simple linear fusion, indicating that fusing vision and tactile modalities too early is less effective than later latent fusion and exhibits weaker generalization across tactile sensors, as our sensor is not trained on the T3 model. From Table 3, baseline methods exhibit substantial performance degradation when tactile fidelity decreases, with a maximum decrease close to 20%, whereas our method remains robust with changes below 5%.

However, TacFilm and TactileConcat underperform relative to this simple linear fusion baseline. p. 7
Where the paper says it

TacFiLM and TactileConcat generally perform worse than simple linear fusion, indicating that fusing vision and tactile modalities too early is less effective than later latent fusion and exhibits weaker generalization across tactile sensors, as our sensor is not trained on the T3 model. From Table 3, baseline methods exhibit substantial performance degradation when tactile fidelity decreases, with a maximum decrease close to 20%, whereas our method remains robust with changes below 5%.

The results indicate that the simple visuo-tactile linear fusion method yields performance improvements in board erasing, bottle closing, and stowing. p. 7
Where the paper says it

TacFiLM and TactileConcat generally perform worse than simple linear fusion, indicating that fusing vision and tactile modalities too early is less effective than later latent fusion and exhibits weaker generalization across tactile sensors, as our sensor is not trained on the T3 model. From Table 3, baseline methods exhibit substantial performance degradation when tactile fidelity decreases, with a maximum decrease close to 20%, whereas our method remains robust with changes below 5%.

The results indicate that TacFilm and TactileConcat underperform relative to this simple linear fusion baseline. p. 7
Where the paper says it

TacFiLM and TactileConcat generally perform worse than simple linear fusion, indicating that fusing vision and tactile modalities too early is less effective than later latent fusion and exhibits weaker generalization across tactile sensors, as our sensor is not trained on the T3 model. From Table 3, baseline methods exhibit substantial performance degradation when tactile fidelity decreases, with a maximum decrease close to 20%, whereas our method remains robust with changes below 5%.

The results indicate that our method remains robust with changes below 5% when tactile fidelity decreases, demonstrating the robustness of our method across different tactile situations. p. 7
Where the paper says it

From Table 3, baseline methods exhibit substantial performance degradation when tactile fidelity decreases, with a maximum decrease close to 20%, whereas our method remains robust with changes below 5%. This demonstrates the robustness of our method across different tactile situations.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. ForceVT algorithm can learn fidelity-agnostic tactile representations by using force and vision to guide fidelity-agnostic tactile learning, enabling robust inference in any tactile situation. This allows the system to maintain performance even when visuo-tactile sensors exhibit manufacturing and utilization inconsistencies.

  2. The AetheRock hardware system provides qualified data quality for robot learning, enabling users to collect force and tactile data over a continuous window four times longer than UMI’s, with 100% data usability. This increases the volume and consistency of training data available for policy learning.

  3. The system can achieve improved policy performance across contact-rich tasks by incorporating force-tactile information, as shown by ForceVT consistently improving performance compared to baselines in Table 2 and Table 3. This leads to better success rates in tasks like board erasing (58.0% for ForceVT vs. 47.0% for VisTacLinear).

  4. The system can handle diverse tactile situations more robustly because tactile random masking significantly enhances inference robustness across various challenging tactile conditions. This means the AI model is less likely to fail when encountering novel or inconsistent contact signals during real-world operation.

Abstract

Force and tactile sensing are essential for contact-rich manipulation, but their integration into handheld or wearable devices is constrained by cost, size, and mechanical compatibility. Tactile signals are also sensitive to manufacturing variations and wear during use, posing further challenges to policy learning. To address these hardware and learning challenges, we introduce AetheRock, an arm-worn device for collecting gripper-force, visual, and tactile data. AetheRock incorporates GelSlim-Mini, our improved version of GelSlim 4.0 designed for low-cost, easy manufacturing, at the gripper fingertip, together with a resistive pressure sensor at the human finger contact area. To learn robust representations from the collected data, we propose ForceVT, a framework that aligns multimodal representations and uses force and vision to guide tactile learning, mitigating the effects of manufacturing variations and sensor degradation. Real-world experiments demonstrate efficient data collection with AetheRock. ForceVT improves average performance by 13% over the vision-tactile baseline and maintains an average task progress of 55.0% under reduced tactile fidelity, compared with 58.0% at full fidelity. Project page: https://lihongcs.github.io/aetherock.

Sources

Related papers