On the Geometry of Multimodal Saturation: Riemannian VICReg
cs.LG, cs.AI
Submitted: 2026-10-05
Updated: 2026-10-05
Comments: 17 pages, 6 figures. Accepted to the Proceedings Track of the NeurIPS 2026 Workshop on Symmetry and Geometry in Neural Representations (NeurReps)
License: http://creativecommons.org/licenses/by-nc-nd/4.0/
The gist: In self-supervised learning, a third modality should improve, or at least preserve, performance.
Abstract
In self-supervised learning, a third modality should improve, or at least preserve, performance. Across nine image-text-tabular datasets, we show that it instead harms performance: the trimodal model underperforms its own best bimodal subset in 55.6% of paired runs under VICReg. The same failure occurs in 51.1% of paired runs under SimSiam. We call this failure multimodal saturation. We propose that the failure lies in the alignment geometry. Riemannian VICReg (R-VICReg) generalizes classical VICReg: it aligns views by squared geodesic distance on learnable negative-curvature product factors and recovers VICReg exactly as curvature vanishes. Over the same 45 paired runs, R-VICReg raises the probability that the third modality helps from 44.4% to 64.4%, with gains concentrated where VICReg saturates.
Sources
- VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning
- Exploring Simple Siamese Representation Learning
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
- HMSN: Hyperbolic Self-Supervised Learning by Clustering with Ideal Prototypes
- LANISTR: Multimodal Learning from Structured and Unstructured Data
- HYperbolic Self-Paced Learning for Self-Supervised Skeleton-based Action Representations
- Hyperbolic Neural Networks
- Hyperbolic Contrastive Learning for Visual Representations beyond Objects
- Multimodal Masked Autoencoders Learn Transferable Representations
- Self-Supervised Learning by Curvature Alignment
- Bootstrap your own latent: A new approach to self-supervised Learning
- Masked Autoencoders Are Scalable Vision Learners
- Are Multimodal Transformers Robust to Missing Modality?
- Negative to Positive Co-learning with Aggressive Modality Dropout
- Variance Covariance Regularization Enforces Pairwise Independence in Self-Supervised Representations
- Poincar'e Embeddings for Learning Hierarchical Representations
- Kernel VICReg for Self-Supervised Learning in Reproducing Kernel Hilbert Space
- An Information-Theoretic Perspective on Variance-Invariance-Covariance Regularization
- Characterizing and overcoming the greedy nature of learning in multi-modal deep neural networks
- SimMIM: A Simple Framework for Masked Image Modeling
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks