UniOMA: Unified Optimal-Transport Multi-Modal Structural Alignment for Robot Perception
cs.RO, cs.AI
Submitted: 2026-11-09
Updated: 2026-11-09
License: http://creativecommons.org/licenses/by/4.0/
The gist: Achieving generalizable and well-aligned multimodal representations remains a core challenge in robot learning.
Terminology
Abstract
Achieving generalizable and well-aligned multimodal representations remains a core challenge in robot learning. While recent approaches have attempted to align modalities by modeling conditional or higher-order statistical dependencies, they do not explicitly consider the structural coherence across modalities. In this work, we propose a novel multimodal alignment method that augments existing contrastive losses with a geometry-aware Gromov--Wasserstein (GW) distance-based regularization. To this end, we encode intra-modal geometry for each modality and reduce cross-modal structural discrepancies through a dynamically learned GW barycenter, thereby encouraging structural consistency in addition to instance-level contrastive alignment. We apply this optimal-transport-based alignment strategy to robot perception tasks involving underexplored modalities such as force and tactile signals, where modality data often exhibit varying sample densities. Extensive experiments on five robotic benchmarks show that our method improves structural alignment and downstream performance over state-of-the-art multimodal contrastive baselines across a wide range of downstream tasks, including robot and environment state prediction. Moreover, our GW-based regularization can be seamlessly integrated into most InfoNCE-like objectives.
Sources
- Conditional Generative Adversarial Nets
- Representation Learning with Contrastive Predictive Coding
- LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignment
- VisualBERT: A Simple and Performant Baseline for Vision and Language
- IMU2CLIP: Multimodal Contrastive Learning for IMU Motion Sensors from Egocentric Videos and Text
- CLIP-ViP: Adapting Pre-trained Image-Text Model to Video-Language Representation Alignment
- Gramian Multimodal Representation Learning and Alignment
- What to align in multimodal contrastive learning?
- A TRIANGLE Enables Multimodal Alignment Beyond Cosine Similarity
- RT-1: Robotics Transformer for Real-World Control at Scale
- OpenVLA: An Open-Source Vision-Language-Action Model
- $\pi_{0.5}$: a Vision-Language-Action Model with Open-World Generalization
- Neural Optimal Transport
- Kernel Neural Optimal Transport
- MultiBench: Multiscale Benchmarks for Multimodal Representation Learning
- Proximal Policy Optimization Algorithms
Related papers
- Rethinking Actions: Continuity-Aware Priors for World-Model Planning
- MOSAIC: Modular Obstacle and Spatial-Aware Framework for Manipulation In Cluttered Scenes
- SAGAS: Semantic-Aware Graph-Assisted Stitching for Offline Temporal Logic Planning
- DexSeed: Expanding Human Hand-Object Interactions into Robot Demonstrations for Dexterous Policy Learning
- LaST0.5: A Generalist World Action Model with Interaction-Centric Latent Reasoning
- CIALLO: Learning Shared Intent Representations for Cross-Embodiment Manipulation