Multimodal Alignment Through Joint Kernel Entropic Gromov--Wasserstein Optimal Transport
Yixuan Florence Wu, Yilun Zhu, Naichen Shi
math.ST, cs.LG, stat.ML
Submitted: 2026-08-04
License: http://creativecommons.org/licenses/by/4.0/
The gist: We study the problem of aligning data from multiple modalities into a shared representation space, focusing on settings where strong pretrained unimodal encoders are available but cross-modal paired
Terminology
Abstract
We study the problem of aligning data from multiple modalities into a shared representation space, focusing on settings where strong pretrained unimodal encoders are available but cross-modal paired data are scarce. We propose a structure-preserving alignment framework, joint kernel entropic Gromov--Wasserstein Optimal Transport (JK-EGW), which maps multiple modalities into a common latent space by minimizing a quadratic optimal transport objective. JK-EGW leverages fine-grained similarity relationships within and across modalities to construct a global affinity kernel instead of relying on raw feature-space distances. Our framework naturally provides explicit control over the geometry and distribution of the latent embedding. On the theory side, we establish parametric sample complexity rate of n-1/2, matching the corresponding rates for standard, entropic and Gromov--Wasserstein optimal transport. On the algorithmic side, we derive a scalable alternating procedure to solve JK-EGW with entropic optimal transport (EOT) updates through a low-rank kernel approximation and a variational lifting. This lifting scheme effectively relieves the burden of a quadratic objective, and allowing us to take the advantage of existing EOT solvers. Empirically, we focus on post-hoc alignment of embeddings from pretrained encoders in data-scarce regimes, and show that our proposed method achieves improved multimodal retrieval performance compared to existing alignment baselines.
Sources
- BEATs: Audio Pre-Training with Acoustic Tokenizers
- VSE++: Improving Visual-Semantic Embeddings with Hard Negatives
- With Limited Data for Multimodal Alignment, Let the STRUCTURE Guide You
- Note on computational complexity of the Gromov-Wasserstein distance
- Relative representations enable zero-shot latent space communication
- A Statistical Theory of Contrastive Pre-training and Multimodal Generative AI
- APE: Aligning Pretrained Encoders to Quickly Learn Aligned Multimodal Representations
- DINOv3
- Joint Multimodal Learning with Deep Generative Models
- Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
- Learning Factorized Multimodal Representations
- GIT: A Generative Image-to-text Transformer for Vision and Language
Related papers
- Conformal Prediction for Dyadic Regression Under Complex Missingness
- Sharp Asymptotic Theory of Maximum Likelihood Estimation for Gaussian Processes with an RBF Kernel
- Bentkus-type asymptotic e-values
- High-Dimensional Asymptotics of Differentially Private PCA
- A proximal gradient algorithm for composite log-concave sampling
- KL Convergence Guarantees for Score diffusion models under minimal data assumptions