A Minimal Optical-Flow Representation for Vision-Based Tactile Rotation Classification in Robotic Manipulation Across Gravity Domains

arXiv:2610.12073 · cs.RO, cs.CV · Submitted 2026-10-08 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: I'm Rosa, and with me are Dev and Taro, guest researcher.

Dev: Today's paper: "A Minimal Optical-Flow Representation for Vision-Based Tactile Rotation Classification in Robotic Manipulation Across Gravity Domains".

Rosa: The gist The representation can be substantially reduced with little loss in accuracy,

Dev: First, who's behind it and why it matters.

Title and authors: Rosa: So we’ve seen that this paper is about making a smaller footprint for tactile sensing data processing to handle different gravity settings. The title itself points to using optical flow from vision-based tactile sensors across those four domains.

Dev: Yeah, and what they’re really digging into is whether you can build a classification system that doesn't need the gravity information as an extra input at all; it just needs the compact representation to handle the shift.

Rosa: Think of it like this: instead of feeding the gravity level into your classifier, you're trying to design a feature set so good that it naturally handles those different conditions when trained on a mix of them.

Dev: That’s what they’re testing, and their approach involves aggregating dense optical flow from simulated sensors over a fixed grid to get one hundred twenty-six features <ref:2610.12073#pg1>.

Rosa: So, the paper is essentially proposing this pipeline: take the tactile frames, calculate the dense optical flow using something called the Farnebäck polynomial-expansion method—that’s a specific math technique for that—aggregate it into these one hundred twenty-six features, and then use a lightweight classifier to guess if an object is rotating clockwise or counterclockwise <ref:2610.12073#pg1>.

Dev: That fixed grid aggregation is key because it reduces the high resolution of the flow field down to this manageable set of numbers, which they call x t = dx one <ref:2610.12073#pg2>..., dx sixty-three dy one..., dy sixty-three in their notation.

Rosa: It’s about finding that sweet spot where you reduce the feature set without losing too much accuracy, which is what they found when they reduced the grid from seven by nine down to four by five.

The paper's summary: Dev: The summary highlights a few major findings about how gravity actually impacts these features. They ran a PERMANOVA and found that gravity only accounts for about one point six percent of the variance in the representation itself, which is small but statistically significant <ref:2610.12073#pg1>.

Rosa: That’s interesting because if it’s just one point six percent of the variance, why does it still matter when you train a model only on Earth data <ref:2610.12073#pg1>?

Dev: Because even that small shift affects models trained only under Earth gravity; their accuracy drops from ninety-four point four percent on Earth to seventy-five point nine percent in orbit, which is a big gap <ref:2610.12073#pg1,from 94.4% on Earth to 75.9% in orbit>.

Rosa: So they’re showing that relying solely on training data from one condition—Earth—causes a noticeable performance dip when you move to Mars or the Moon, even if the physical change in the representation itself seems minor.

Dev: But the real strength they present is when you train a single model using data from all four gravity domains. That one model achieves ninety-six point three percent overall accuracy on unseen test sets, and it maintains performance between ninety-five point one percent and ninety-seven percent for the individual gravity conditions without needing gravity as an input at all during classification <ref:2610.12073#pg1,achieves 96.3% overall accuracy>.

Rosa: It seems the main point is that you don't need to explicitly tell your classifier what gravity it's in; if the representation is good enough, one model trained on everything works.

The paper's improvements: Dev: The authors propose two major improvements, first focusing on reducing that one hundred twenty-six-feature representation <ref:2610.12073#pg1>. They show that shrinking the spatial grid from seven by nine to four by five, which cuts the features down to forty only causes accuracy to drop from ninety-six point three percent down to ninety-five point seven percent.

Rosa: So they’re demonstrating that you can get a smaller representation while keeping performance very high, which is important for those resource-constrained platforms we talked about earlier.

Dev: Second improvement is the cross-gravity classification approach itself. Instead of building four separate models, they show that one single model trained across all four gravity domains performs consistently well across all of them without needing gravity as an input during the prediction phase.

Rosa: That’s powerful because it means we can deploy a single system in a robot that might move between environments, and it won't suddenly fail when the gravity changes.

Conclusion: Dev: To wrap up this paper on "A Minimal Optical-Flow Representation for Vision-Based Tactile Rotation Classification in Robotic Manipulation Across Gravity Domains," the authors confirm that a single model trained across all four gravity domains achieves ninety-six point three percent overall accuracy and maintains performance between ninety-five point one percent and ninety-seven percent across the individual domains without providing gravity as an input to the classifier <ref:2610.12073#pg1,A Minimal Optical-Flow Representation for Vision-Based Tactile Rotation Classification in>.

Rosa: It seems like a solid demonstration that you can create a compact tactile optical-flow representation that works robustly across gravity variations, even if you only train one model on all of them.

Dev: The main implication is that for field robotics, this means we can rely on a single classification system rather than needing complex logic to switch models based on the local gravity conditions.

Taro: From an autonomy standpoint, it suggests that as long as the representation captures enough of the motion cues—the flow from consecutive frames—the system can generalize its rotation understanding across different planetary bodies and orbital environments.

Rosa: We’ll keep an eye on this finding for how far this representation holds up when we move it out of a simulation and into a real-world robot.

Dev: Right, and next time we talk about how latency affects these types of feature extraction pipelines in motion tracking systems.

Oscar Martinez-Bernal, Mario Cavero-Vidal, Francesco Grella, Carol Martinez

SpaceR Research Group (SpaceR) · Interdisciplinary Centre for Security, Reliability and Trust (SnT), University of Luxembourg · Eurecat, Centre Tecnològic de Catalunya

cs.RO, cs.CV

Submitted: 2026-10-08

Updated: 2026-10-08

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 88/100

The gist: The gist The representation can be substantially reduced with little loss in accuracy, as a 4x5 grid reduces the feature set from 126 to 40 while decreasing accuracy only from 0.963 to 0.957This work

Key concepts

Optical Flow Representation
This representation captures the motion between two consecutive tactile sensor images by computing dense optical flow using the Farnebäck polynomial-expansion method. This dense field is then spatially reduced into a fixed grid, resulting in 126 features (dx and dy components) that describe how the tactile pattern moved from one moment to the next.
Gravity-Induced Domain Shift
The research quantified how gravity affects this optical flow representation. While gravity significantly influences individual features, it only accounts for a small portion of the total variance in the entire 126-feature set. Training on data from all four gravity domains improves generalization, showing that a model trained across all conditions can perform well on any single domain.
Ablation Study
This part of the work tests how much information is truly necessary for classification by reducing the feature set. By shrinking the spatial grid from 7x9 (126 features) to 4x5 (40 features), researchers observed that accuracy only dropped slightly, indicating that a relatively compact representation is sufficient for rotation classification.
XGBoost Classifier
This is the lightweight machine learning model used to make the final prediction about the object's rotation direction. It takes the extracted optical flow features as input and learns how to map those features to a specific output class. The study shows this classifier can generalize effectively across different gravity settings when trained comprehensively.

Terminology

Summary

The gist The representation can be substantially reduced with little loss in accuracy, as a 4x5 grid reduces the feature set from 126 to 40 while decreasing accuracy only from 0.963 to 0.957This work investigates whether a compact representation of tactile motion can classify object rotation across different gravity conditions

How it works

The processing pipeline consists of three stages, starting with a simulated vision-based tactile sensor recording the contact response under four gravity conditionsSecond, consecutive tactile frames are converted to dense optical flow and spatially aggregated into a 126-feature representationFinally, lightweight classifiers use this representation to predict the direction of load-induced rotation

The optical flow representation is derived by computing dense optical flow between consecutive frames using the classical Farnebäck polynomial-expansion methodThis resulting dense field is spatially reduced to a fixed grid, where each 240 × 320 px image is divided into R = 7 rows and C = 9 columns, producing 63 cellsWithin each cell, the median horizontal and vertical flow components are retained as dx and dy, resulting in a representation of xt=dx1,..., dx63, dy1,..., dy63∈ R 126

Gravity-Induced Domain Shift

The analysis first quantifies the effect of gravity on the optical-flow representationPERMANOVA shows a statistically significant but small effect of gravity (R2 = 0.0164, pseudo-F = 13.12, p = 0.001, 999 permutations), meaning gravity accounts for approximately 1.6% of the variance in the representationPairwise tests with FDR correction distinguish Earth from Mars, Moon and orbit (R2 = 0.017–0.020, p = 0.002), and at the feature level, Kruskal–Wallis tests identify significant gravity effects in 78 of the 126 featuresUnder Earth-only training, XGBoost accuracy decreases from 0.944 on Earth to 0.759 in orbit, corresponding to a maximum accuracy decrease of 0.185

Generalisation Across Gravity Domains

When training data from all four gravity domains are included, XGBoost achieves 0.963 accuracy on the held-out test setPerformance remains similar across the individual gravity domains despite gravity not being provided as an input to the classifierA single model trained across all four gravity domains achieves 96.3% overall accuracy and 95.1–97.0% across individual domains, without using gravity as an inputThis shows that a single classifier can operate across the four evaluated gravity domains when they are represented during training

Minimal Representation and Cost

The ablation study evaluates how far the optical-flow representation can be reduced while preserving classification performanceReducing the spatial grid from 7 × 9 (126 features) to 4 × 5 (40 features) decreases accuracy only from 0.963 to 0.957The main computational cost, however, remains in feature extraction, as Farnebäck optical flow and grid aggregation require approximately 11.6 ms per frame pairXGBoost classification requires only 0.

Improvements for AI systems

  1. textbf Cross-Gravity Transfer Learning for Tactile Perception in Space Robotics Engine Development (XGBoost based): This system can accurately classify object rotation across Earth, Mars, Moon, and orbital gravity conditions using only a single model trained across all four domains (a single model trained across all four gravity domains achieves 96.3% overall accuracy and 95.1–97.0% across individual domains).

  2. textbf Compact Feature Representation for Real-Time Inference: The system can classify load-induced rotation with a reduced feature set of 40 features, achieving 95.7% accuracy while requiring only 0.14 ms per inference using XGBoost (The representation can also be reduced to 40 features while retaining 95.7% accuracy, with XGBoost requiring only 0.14 ms per inference).

  3. textbf Robust Model Selection for Domain Shift Mitigation: The system employs a strategy where a single model trained across all four gravity domains achieves consistent performance across the conditions evaluated in this study, demonstrating that cross-gravity performance depends on both the tactile representation and the classifier used to interpret it.

Abstract

Vision-based tactile sensors provide rich contact information, but processing high-resolution images can be costly for resource-constrained platforms such as space robots. This work investigates whether a compact representation of tactile motion can classify object rotation across different gravity conditions. Dense optical flow from a simulated GelSight Mini is aggregated over a 7x9 grid into 126 features and used to classify the direction of load-induced rotation under Earth, Mars, Moon, and orbital gravity. Gravity causes a small but significant shift in these features, accounting for 1.6% of their variance (R2 = 0.016). Despite its small magnitude, this shift affects models trained only on Earth data: XGBoost accuracy decreases from 94.4% on Earth to 75.9% in orbit. In contrast, a single model trained across all four gravity domains achieves 96.3% overall accuracy and 95.1%-97.0% across individual domains, without using gravity as an input. The representation can also be reduced to 40 features while retaining 95.7% accuracy, with XGBoost requiring only 0.14 ms per inference. These findings show that Earth-gravity performance alone is insufficient to establish the transferability of tactile perception for space robotic manipulation, highlighting the need to account for gravity-induced domain shifts during training and validation.

Sources

Related papers