MMLongEmbed: Benchmarking Multimodal Embedding Models in Long-Context Scenarios
cs.CV, cs.AI
Submitted: 2026-06-05
Updated: 2026-08-31
Code: https://github.com/AmamiSora1228/MMLongEmbed
License: http://creativecommons.org/licenses/by/4.0/
The gist: Recent advancements have significantly expanded the theoretical context windows of Multimodal Embedding Models (MEMs).
Terminology
Abstract
Recent advancements have significantly expanded the theoretical context windows of Multimodal Embedding Models (MEMs). However, larger context windows do not necessarily translate into effective comprehension and representation of long-context multimodal inputs, which remains a critical bottleneck for real-world deployment. To address the lack of systematic evaluation in this setting, we introduce MMLongEmbed, the first comprehensive benchmark for evaluating MEMs in long-context scenarios. MMLongEmbed comprises four retrieval tasks spanning multiple context-length ranges, covering text, document, and video modalities. Through extensive evaluation of state-of-the-art models, we find that current architectures rely heavily on superficial feature matching and struggle to capture deep semantic and structural dependencies. We further observe that performance degradation varies systematically with context length and key information placement. Moreover, models exhibit substantially different robustness to redundant contextual information across modalities. For reproducibility, the benchmark and code are publicly available.
Sources
- Beyond Rows to Reasoning: Agentic Retrieval for Multimodal Spreadsheet Understanding and Editing
- Qwen3-VL Technical Report
- RzenEmbed: Towards Comprehensive Multimodal Retrieval
- Embed-RL: Reinforcement Learning for Reasoning-Driven Multimodal Embeddings
- E5-V: Universal Embeddings with Multimodal Large Language Models
- VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks
- Benchmarking Retrieval-Augmented Multimodal Generation for Document Question Answering
- MMTEB: Massive Multilingual Text Embedding Benchmark
- Qwen3-VL-Embedding and Qwen3-VL-Reranker: A Unified Framework for State-of-the-Art Multimodal Retrieval and Ranking
- NeedleBench: Evaluating LLM Retrieval and Reasoning Across Varying Information Densities
- BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models
- Magic-MM-Embedding: Towards Visual-Token-Efficient Universal Multimodal Embedding with MLLMs
- MM-Embed: Universal Multimodal Retrieval with Multimodal LLMs
- SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features
- InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency
- VLM2Vec-V2: Advancing Multimodal Embedding for Videos, Images, and Visual Documents
- MMEmb-R1: Reasoning-Enhanced Multimodal Embedding with Pair-Aware Selection and Adaptive Control
- Reasoning-Augmented Representations for Multimodal Retrieval
- MRMR: A Realistic and Expert-Level Multidisciplinary Benchmark for Reasoning-Intensive Multimodal Retrieval
- GME: Improving Universal Multimodal Retrieval by Multimodal LLMs
Related papers
- Loss Knows Best: Detecting Annotation Errors in Videos via Loss Trajectories
- AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memories
- Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift
- MambaX-Net: Dual-Input Mamba-Enhanced Cross-Attention Network for Longitudinal MRI Segmentation
- TeleOCR: Navigating Document Parsing Across Digital and Camera-Captured Documents
- A Survey on Efficient Vision-Language-Action Models