Spherical Interpolation for Backward-Compatible Multimodal Representations
cs.CV, cs.LG
Submitted: 2026-09-30
Updated: 2026-09-30
Comments: Accepted at NeurIPS 2026
Code: https://github.com/miccunifi/SLERP_backward_compatibility
License: http://creativecommons.org/licenses/by/4.0/
The gist: Contrastive vision-language models map visual and textual representations into a shared normalized embedding space, making cosine similarity the natural metric for cross-modal retrieval.
Terminology
Abstract
Contrastive vision-language models map visual and textual representations into a shared normalized embedding space, making cosine similarity the natural metric for cross-modal retrieval. A practical challenge arises during model upgrades: independently trained models generally produce incompatible representation spaces, so replacing a deployed model typically requires recomputing embeddings for the entire gallery, which is prohibitively expensive at scale. Orthogonal post-hoc alignment can partially mitigate this problem by mapping new-model queries into the old-model gallery space. However, because independently trained models can differ in fine-grained representation structure, the orthogonal alignment remains approximate, leaving a residual angular discrepancy between the old-model query and the aligned new-model query. We study whether interpolation along the spherical geodesic between these two normalized query representations can improve retrieval without re-indexing the gallery. We characterize when this path contains an interior query direction closer to an idealized retrieval-optimal direction than either endpoint, and connect this characterization to Recall@ K through a local margin-based certification result. Experiments across multiple benchmarks and model families show that post-alignment spherical interpolation improves over orthogonal alignment alone, recovering backward-compatibility in most evaluated settings. Consistent with our geometric characterization, per-query oracle analysis shows that retrieval-favorable interior points occur frequently in practice. Code is available at https://github.com/miccunifi/SLERP backward compatibility.
Sources
- SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features
- HuggingFace's Transformers: State-of-the-art Natural Language Processing
- Billion-scale similarity search with GPUs
- LLaMA: Open and Efficient Foundation Language Models
- Textbooks Are All You Need
- Canonicalizing Multimodal Contrastive Representation Learning
- Back into Plato's Cave: Examining Cross-modal Representational Convergence at Scale
- WARP: On the Benefits of Weight Averaged Rewarded Policies
- Multi-Way Representation Alignment
- Improving General Text Embedding Model: Tackling Task Conflict and Data Imbalance through Model Merging
Related papers
- Loss Knows Best: Detecting Annotation Errors in Videos via Loss Trajectories
- AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memories
- Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift
- MambaX-Net: Dual-Input Mamba-Enhanced Cross-Attention Network for Longitudinal MRI Segmentation
- TeleOCR: Navigating Document Parsing Across Digital and Camera-Captured Documents
- A Survey on Efficient Vision-Language-Action Models