Keep It Simple: Multi-Key Episodic Memory Retrieval for Ultra-Long Video Understanding
cs.CV, cs.AI, cs.CL
Submitted: 2026-08-07
Updated: 2026-09-16
Comments: Accepted to ECCV 2026 (Oral). Project Page: https://choi-yeeun.github.io/MERIT/
Project page: https://choi-yeeun.github.io/MERIT
License: http://creativecommons.org/licenses/by/4.0/
The gist: When videos extend from hours to days, directly processing them end-to-end becomes impractical for current Multi-modal Large Language Models (MLLMs).
Terminology
Abstract
When videos extend from hours to days, directly processing them end-to-end becomes impractical for current Multi-modal Large Language Models (MLLMs). This ultra-long setting necessitates a two-stage paradigm: query-agnostic memory construction followed by retrieval-based inference. Prior work invests in complex memory construction to pre-model high-level relations in videos, despite not knowing the downstream query at build time. We instead prioritize high-recall retrievability during memory building, and defer query-specific, high-level relation composition to inference time. To this end, we propose MERIT(Multi-key Episodic Retrieval with Inference-time Temporal expansion), a simple yet effective agentic framework for ultra-long video understanding. First, we formulate an episodic multi-key representation that enables precise retrieval of fine-grained memories through a simple key-matching mechanism. Second, we introduce a neighbor filtering mechanism to capture broader semantic context without the massive computational overhead of global memory construction. This is achieved by expanding the temporal scope exclusively around the retrieved segments at inference time. By leveraging simple key-matching with this on-demand temporal expansion, MERIT achieves state-of-the-art performance across three long-video benchmarks: EgoLifeQA, LVBench, and Video-MME (Long).
Sources
- Qwen3-VL Technical Report
- Qwen2.5-VL Technical Report
- Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling
- Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities
- From Local to Global: A Graph RAG Approach to Query-Focused Summarization
- Retrieval-Augmented Generation for Large Language Models: A Survey
- LightRAG: Simple and Fast Retrieval-Augmented Generation
- From RAG to Memory: Non-Parametric Continual Learning for Large Language Models
- LLaVA-OneVision: Easy Visual Task Transfer
- HippoMM: Hippocampal-inspired Multimodal Memory for Long Audiovisual Event Understanding
- VideoRAG: Retrieval-Augmented Generation with Extreme Long-Context Videos
- LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understanding
- CLaMR: Contextualized Late-Interaction for Multimodal Content Retrieval
- AdaVideoRAG: Omni-Contextual Adaptive Retrieval-Augmented Efficient Long Video Understanding
- WorldMM: Dynamic Multimodal Memory Agent for Long Video Reasoning
- VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understanding
- SiLVR: A Simple Language-based Video Reasoning Framework
- Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models
- LLaVA-Video: Video Instruction Tuning With Synthetic Data
Related papers
- Loss Knows Best: Detecting Annotation Errors in Videos via Loss Trajectories
- AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memories
- Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift
- MambaX-Net: Dual-Input Mamba-Enhanced Cross-Attention Network for Longitudinal MRI Segmentation
- TeleOCR: Navigating Document Parsing Across Digital and Camera-Captured Documents
- A Survey on Efficient Vision-Language-Action Models