Omni-Embed-Audio: Leveraging Multimodal LLMs for Robust Audio-Text Retrieval

arXiv:2604.18360 · cs.SD, cs.CL · Submitted 2026-04-20 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Omni-Embed-Audio: Leveraging Multimodal LLMs for Robust Audio-Text Retrieval".

Jane: The gist: Omni-Embed-Audio (OEA) is a retrieval-oriented encoder leveraging multimodal LLMs with native audio understanding to achieve superior performance in audio-text retrieval,

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So we’re diving deeper into OmniEmbedAudio Leveraging Multimodal LLMs for Robust Audio-Text Retrieval. We know the paper argues that traditional benchmarks are weak because they only rely on caption-style queries, which don't reflect real search behavior.

Jane: That’s right, and OEA is their answer to this limitation because it’s a retrieval oriented encoder that leverages multimodal LLMs with native audio understanding.

Lu: They introduce the architecture where a single shared transformer backbone processes both text and audio inputs, and then they apply LoRA adapters to the attention layers.

Meng: The mechanism uses modality specific projection heads to map those representations into a shared five hundred twelve-dimensional embedding space after L2 normalization <ref:2604.18360#pg1>.

Tom: So they compress whatever the backbone learns into that one unified space, which is key for making it retrieval oriented.

Jane: They do this by wrapping queries in a query prefix and processing raw waveforms through the model’s native audio encoder with an identical mean pooling to get those modality agnostic representations.

Meng: That means they’re trying to make sure the text and audio are represented in a way that is comparable right from the start.

Lu: Their training objective uses symmetric contrastive learning with InfoNCE loss at tau equals zero point seven.

Tom: It's a standard setup for contrastive learning, but it’s applied here across both modalities simultaneously, which is what they’re doing to create this unified representation.

Jane: And they are testing this system on specific datasets like AudioCaps-UIQ and Clotho-UIQ to see how it performs under these varied user intents.

Conclusion: Tom: So looking at OmniEmbedAudio Leveraging Multimodal LLMs for Robust Audio-Text Retrieval, the core message is that moving away from caption queries to more realistic User-Intent Queries gives us a much better understanding of retrieval system performance.

Jane: Exactly, because the paper shows that OEA can achieve comparable text-to-audio retrieval performance to M2D-CLAP on key datasets.

Lu: The authors’ main contribution is proving that their architecture, leveraging multimodal LLMs with native audio understanding, is robust across different query types.

Meng: It's a big deal because it shows that the way we structure the model—the shared backbone and LoRA adaptation—actually helps in separating relevant information better than traditional dual-encoder approaches.

Tom: So to wrap up, OEA gives us a retrieval oriented encoder that is better at handling both text-to-text matching and distinguishing between similar audio clips.

Jane: It suggests that for building production systems, focusing on those hard negative discrimination metrics like HNSR and TFR is where you get the most reliable performance gains.

HaeJun Yoo, Yongseop Shin, Insung Lee, Myoung-Wan Koo, Du-Seong Chang

Sogang University

cs.SD, cs.CL

Submitted: 2026-04-20

Updated: 2026-10-14

Comments: Accepted at ACL 2026 Main Conference. Final version

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 86/100

The gist: The gist: Omni-Embed-Audio (OEA) is a retrieval-oriented encoder leveraging multimodal LLMs with native audio understanding to achieve superior performance in audio-text retrieval, particularly

Key concepts

Omni-Embed-Audio (OEA): A retrieval encoder built on multimodal LLMs that has native understanding of audio. It uses a shared transformer backbone adapted via LoRA to create modality-agnostic representations, allowing it to effectively search for audio based on text queries.
User-Intent Queries (UIQs): Five specific types of natural search queries—questions, commands, tags, paraphrases, and negative exclusion queries—used instead of simple captions. These simulate how real users actually search for audio content in the wild to test the system's real-world robustness.
Hard Negative Mining: A technique used during evaluation to find the most challenging negative examples for a retrieval model. This process identifies distractors that sound very similar to the target audio, forcing the model to learn finer distinctions between closely related sounds.

Terminology

Benchmark — 2× in this paper · explained in 2 episodes
Robustness — 2× in this paper · explained in 5 episodes
Contrastive Language-Audio Pretraining — 1× in this paper
Semantic Understanding — 1× in this paper · explained in 2 episodes

Summary

The gist: Omni-Embed-Audio (OEA) is a retrieval-oriented encoder leveraging multimodal LLMs with native audio understanding to achieve superior performance in audio-text retrieval, particularly excelling in text-to-text retrieval and hard negative discrimination.

How it works

OEA employs a retrieval oriented encoder leveraging multimodal LLMs with native audio understanding (Page 1).
The architecture utilizes a single shared transformer backbone for both modalities, combined with parameter-efficient LoRA adaptation (Page 2).
Input processing involves wrapping queries in a query prefix and processing raw waveforms through the model’s native audio encoder with an identical mean pooling, producing modalityagnostic representations (Page 3).
Parameter-Efficient Adaptation is achieved by using LoRA adaptation with lightweight adapter matrices attached to attention layers, along with modality specific projection heads that compress backbone representations into a shared 512-dimensional retrieval embedding space (Page 3).
The training objective uses symmetric contrastive learning with InfoNCE loss (τ = 0.07) (Page 3).

Evaluation and Robustness

To systematically evaluate robustness beyond caption-style queries, the authors introduce User-Intent Queries (UIQs)—five formulations reflecting natural search behaviors: questions, commands, keyword tags, paraphrases, and exclusion-based negative queries (Page 1).
UIQs are organized into three categories: Conversational Queries (questions, commands), Reformulation Queries (tags, paraphrases), and Exclusionary Queries (negative queries) (Page 2).
Novel hard negative-based evaluation metrics include HNSR@k, which measures the model’s ability to rank the Target above the Hard Negative, and TFR (Target-First Rate) and ∆-Rank = Rank(HN) − Rank(T) for continuous separation quality (Page 3).
The authors construct three evaluation datasets: AudioCaps-UIQ, Clotho-UIQ, and MECAT-UIQ to test the models across different query types (Page 3).

Key Findings and Advantages

OEA achieves comparable text-to-audio retrieval performance to state of the art M2D-CLAP on AudioCaps, Clotho, and MECAT (Page 1).
OEA demonstrates clear advantages in two critical areas: (1) dominant text to text retrieval (+22% relative improvement), and (2) substantially superior hard negative discrimination (+4.3%p HNSR@10, +34.7% relative TFR@10) (Page 1).
OEA models substantially outperform all CLAP baselines including M2D-CLAP across all datasets on Text-to-Text retrieval (Page 5).
For exclusionary queries, OEA achieves the best HNSR@10 (34.6% vs M2D-CLAP’s 30.3%, +4.3%p) and TFR@10 10.1% vs 7.5% (+34.7% relative improvement) (Page 5).
Mechanism ablations attribute the discrimination gains chiefly to more separable audio embeddings, which place confusable clips significantly farther apart and an order sensitive encoding of exclusion structure that cross model controls reveal to be shared with CLAP text encoders (Page 8).

Model Performance and Scaling

OEA models generally outperform all CLAP baselines on T2T retrieval, showing a +8.67%p R@1 over M2D-CLAP on Clotho (+5.5%p over M2D-CLAP) (Page 1).
The performance of OEA does not consistently improve with increased model scale; on MECAT T2A R@1, OEA-Nemo3B achieves 7.96% compared to OEAQwen7B’s 7.74% (Page 18).
The authors recommend the 3B model achieves competitive or superior performance at substantially lower computational cost, suggesting the bottleneck shifts from backbone capacity to contrastive alignment quality at larger scales (Page 18).

Limitations

A limitation is that OEA requires multimodal LLM backbones with native audio understanding, limiting base model choices (Page 18).
The architecture incurs higher computational cost than lightweight CLAP models, particularly for audio encoding (Page 15).
The hard negative mining pipeline relies on MGA-CLAP for acoustic similarity retrieval, which constrains the definition of “hard” to the representational capacity of MGA-CLAP (Page 17).
Future work should compare UIQ token distributions against real-world audio search logs, such as Freesound user queries analyzed by Weck and Font (2024) (Page 19).

The work introduces Omni-Embed-Audio (OEA), a retrieval oriented multimodal encoder leveraging multimodal LLMs with native audio understanding to achieve superior performance in audio-text retrieval, particularly excelling in text to text retrieval and hard negative discrimination. The gist; this paper presents OEA, a unified encoder architecture leveraging multimodal LLMs for audio-text retrieval.

OEA employs a retrieval oriented encoder leveraging multimodal LLMs with native audio understanding (Page 1).
The architecture utilizes a single shared transformer backbone for both modalities, combined with parameter efficient LoRA adaptation (Page 2).
Input processing involves wrapping queries in a query prefix and processing raw waveforms through the model’s native audio encoder with an identical mean pooling, producing modalityagnostic representations (Page 3).
Parameter-Efficient Adaptation is achieved by using LoRA adaptation with lightweight adapter matrices attached to attention layers, along with modality specific projection heads that compress backbone representations into a shared 512-dimensional retrieval embedding space (Page 3).
The training objective uses symmetric contrastive learning with InfoNCE loss (τ = 0.07) (Page 3).
To systematically evaluate robustness beyond caption style queries, the authors introduce User-Intent Queries (UIQs)—five formulations reflecting natural search behaviors: questions, commands, keyword tags, paraphrases, and exclusion based negative queries (Page 1).
UIQs are organized into three categories: Conversational Queries (questions, commands), Reformulation Queries (tags, paraphrases), and Exclusionary Queries (negative queries) (Page 2).
Novel hard negative-based evaluation metrics include HNSR@k, which measures the model’s ability to rank the Target above the Hard Negative, and TFR (Target-First Rate) and ∆-Rank = Rank(HN) − Rank(T) for continuous separation quality (Page 3).
The authors construct three evaluation datasets: AudioCaps-UIQ, Clotho-UIQ, and MECAT-UIQ to test the models across different query types (Page 3).
OEA achieves comparable text to audio retrieval performance to state of the art M2D-CLAP on AudioCaps, Clotho, and MECAT (Page 1).
OEA demonstrates clear advantages in two critical areas: (1) dominant text to text retrieval (+22% relative improvement), and (2) substantially superior hard negative discrimination (+4.3%p HNSR@10, +34.7% relative TFR@10) (Page 1).
OEA models substantially outperform all CLAP baselines including M2D-CLAP across all datasets on Text to Text retrieval (Page 5).
For exclusionary queries, OEA achieves the best HNSR@10 (34.6% vs M2D-CLAP’s 30.3%, +4.3%p) and TFR@10 10.1% vs 7.5% (+34.7% relative improvement) (Page 5).
Mechanism ablations attribute the discrimination gains chiefly to more separable audio embeddings, which place confusable clips significantly farther apart and an order sensitive encoding of exclusion structure that cross model controls reveal to be shared with CLAP text encoders (Page 8).
A limitation is that OEA requires multimodal LLM backbones with native audio understanding, limiting base model choices (Page 18).

Improvements for AI systems

Bold Header: Unified Multimodal Retrieval Encoder

This system can perform text-to-audio retrieval with dominant T2T performance and superior hard negative discrimination, leveraging a single shared transformer backbone for both modalities. It excels in cross-domain generalization, showing comparable or superior performance on Clotho and MECAT compared to state-of-the-art CLAP models.

Bold Header: Robust User Intent Query (UIQ) Evaluation

The system can systematically evaluate retrieval robustness by introducing User-Intent Queries (UIQs)—five formulations reflecting natural search behaviors: questions, commands, keyword tags, paraphrases, and exclusion-based negative queries. This allows researchers to move beyond caption-style queries that differ substantially from real-world search behavior for more practical assessment.

Bold Header: Enhanced Exclusionary Query Discrimination

By incorporating novel hard negative-based evaluation metrics (HNSR, TFR, ∆-Rank), the system can quantify a model's ability to distinguish between acoustically similar but semantically distinct audio. This capability is crucial for applications where users explicitly specify what they don’t want.

Bold Header: Superior Text-to-Text Retrieval

The system demonstrates dominant T2T performance, achieving +22% relative improvement on MECAT R@5 by leveraging the full representational capacity of billion-parameter LLMs for text understanding, outperforming CLAP baselines in this specific task.

Bold Header: Order-Sensitive Exclusion Signal Parsing

The system identifies that the advantage is driven by an order-sensitive encoding of exclusion structure that cross-model controls reveal to be shared with CLAP text encoders. This implies the system can be tuned to exploit query structure rather than relying on a simple negation word itself for suppression.