Illuminating Visual Identity in Universal Multimodal Embeddings
Jiawei Cao, Junyi Feng, Jiashen Hua, Ziheng Huang, Bing Deng, Kaijie Wu, Chaochen Gu, Jieping Ye
cs.CV, cs.AI, cs.CL
Submitted: 2026-08-03
Comments: Accepted to CVPR 2026
Project page: https://chrisclear3.github.io/MVEB
License: http://creativecommons.org/licenses/by-nc-sa/4.0/
The gist: Universal Multimodal Embeddings (UMEs) aim to unify various modalities and tasks into a shared representation space.
Terminology
Abstract
Universal Multimodal Embeddings (UMEs) aim to unify various modalities and tasks into a shared representation space. In recent years, this field has witnessed substantial progress driven by the development of Multimodal Large Language Models (MLLMs). However, a crucial capability, visual identity discrimination, remains underexplored in existing UME methods, despite its critical role in a wide range of tasks, including instance retrieval, re-identification, and identity preservation in AI-generated content. To bridge this gap, we propose a unified formulation for visual identity discrimination (VisID) and introduce MVEB (M ultimodal V isual Identity E mbedding B enchmark), a large-scale benchmark curated from both real-world and synthetic datasets to support evaluation and training. Furthermore, we present a simple yet effective learning framework that jointly optimizes general multimodal and visual identity representations through a carefully designed identity-aware sampling mechanism. Extensive experiments demonstrate that our approach successfully endows UMEs with strong identity discrimination capability and maintains competitive general multimodal performance. We believe this work not only illuminates a critical yet neglected capability, but also takes a step toward more holistic universal multimodal embeddings. Code and data are available at MVEB.
Sources
- Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
- Unicom: Universal and Compact Representation Learning for Image Retrieval
- Qwen2.5-VL Technical Report
- LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders
- mmE5: Improving Multimodal Multilingual Embeddings via High-quality Synthetic Data
- OpenGPT-4o-Image: A Comprehensive Dataset for Advanced Image Generation and Editing
- Think Then Embed: Generative Context Improves Multimodal Embedding
- DreamSim: Learning New Dimensions of Human Visual Similarity using Synthetic Data
- Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup
- SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generation
- Breaking the Modality Barrier: Universal Embedding Learning with Multimodal LLMs
- UniME-V2: MLLM-as-a-Judge for Universal Multimodal Embedding Learning
- In Defense of the Triplet Loss for Person Re-Identification
- LLM2CLIP: Powerful Language Model Unlocks Richer Cross-Modality Representation
- E5-V: Universal Embeddings with Multimodal Large Language Models
- Modality Curation: Building Universal Embeddings for Advanced Multimodal Information Retrieval
- jina-clip-v2: Multilingual Multimodal Embeddings for Text and Images
- LLaVE: Large Language and Vision Embedding Models with Hardness-Weighted Contrastive Learning
- NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models
- LLaVA-OneVision: Easy Visual Task Transfer
Related papers
- Loss Knows Best: Detecting Annotation Errors in Videos via Loss Trajectories
- AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memories
- Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift
- MambaX-Net: Dual-Input Mamba-Enhanced Cross-Attention Network for Longitudinal MRI Segmentation
- TeleOCR: Navigating Document Parsing Across Digital and Camera-Captured Documents
- A Survey on Efficient Vision-Language-Action Models