Omni-Interactive Universal Embedder
cs.AI, cs.CV
Submitted: 2026-08-27
Updated: 2026-10-19
Comments: Preprint
Code: https://github.com/openclaw/openclaw
License: http://creativecommons.org/licenses/by/4.0/
The gist: Multimodal representation learning has been shifting from traditional two-tower architectures to large language model (LLM)-based embedders due to their strong instruction-following capabilities.
Terminology
Abstract
Multimodal representation learning has been shifting from traditional two-tower architectures to large language model (LLM)-based embedders due to their strong instruction-following capabilities. Despite this progress, existing approaches primarily focus on language and image modalities, which also remain the dominant modalities for user-conditioned interactions in current embedders. In this paper, we propose the first Omni-Interactive Universal Embedder (OmniUE), which not only learns a unified embedding space across text, video, and audio by leveraging intermediate-layer representations from dedicated learnable tokens, but also supports omni-interactive querying, enabling users to provide inputs in the form of text, visual regions of interest, and audio spans. Within OmniUE, visual and audio segmenters process diverse user interactions and integrate them with an omni-LLM to produce user-conditioned any-to-any embeddings via context aggregation. To evaluate OmniUE's omni-interactive capabilities, we introduce OmniCHOIR, benchmarking models for omni-interactive compositional audio retrieval based on the given text, video, and audio as well as unimodal or multimodal interaction prompts. OmniUE consistently surpasses state-of-the-art baselines across diverse modalities, with average improvements of 10.5% on textual-interactive video benchmarks (MMEB-v2-video), 1.1% on audio tasks (MAEB), 83.7% on visual-interactive benchmarks (SCaR), and 24.1% on our omni-interactive OmniCHOIR benchmark. We believe that jointly advancing omni-modal representation learning and omni-interactive querying paves the way toward universal embedders.
Sources
- YouTube-8M: A Large-Scale Video Classification Benchmark
- MAEB: Massive Audio Embedding Benchmark
- Perception Encoder: The best visual embeddings are not at the output of the network
- SAM 3: Segment Anything with Concepts
- e5-omni: Explicit Cross-modal Alignment for Omni-modal Embeddings
- Qwen2-Audio Technical Report
- VIVA: VLM-Guided Instruction-Based Video Editing with Reward Optimization
- Rethinking Memory in LLM based Agents: Representations, Operations, and Emerging Topics
- Breaking the Modality Barrier: Universal Embedding Learning with Multimodal LLMs
- UniME-V2: MLLM-as-a-Judge for Universal Multimodal Embedding Learning
- Efficient and High-Fidelity Omni Modality Retrieval
- E5-V: Universal Embeddings with Multimodal Large Language Models
- UME-R1: Exploring Reasoning-Driven Generative Multimodal Embeddings
- Learning Personalized Agents from Human Feedback
- ReMatch: Boosting Representation through Matching for Multimodal Retrieval
- VLM2Vec-V2: Advancing Multimodal Embedding for Videos, Images, and Visual Documents
- ReasoningBank: Scaling Agent Self-Evolving with Reasoning Memory
- SAM Audio: Segment Anything in Audio
- Representation Learning with Contrastive Predictive Coding
- MCA: Modality Composition Awareness for Robust Composed Multimodal Retrieval
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems