FLARE: Full-Modality Long-Video Audiovisual Retrieval Benchmark with User-Simulated Queries

arXiv:2605.10228 · cs.MM · Submitted 2026-05-11 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.

Tom: We're looking at the paper: "FLARE: Full-Modality Long-Video Audiovisual Retrieval Benchmark with User-Simulated Queries".

Paper summary: Tom: So we're wrapping up this look at FLARE: Full-Modality Long-Video Audiovisual Retrieval Benchmark with User-Simulated Queries one.

Jane: It’s a new way to test how well AI can find long videos when the search query is realistic, not just a short clip or a single piece of text.

Tom: Right, so they built this benchmark using three hundred ninety-nine long videos and user-style queries to see if unified models can actually reason across vision and audio in real search scenarios one.

Lu: The authors are trying to find where the current AI systems break when they have to combine different types of data from long media.

Meng: It really shows us where the failure points are right now when we're shipping models to production because it’s not just about accuracy on a test set.

Lalam: FLARE is designed for research and testing, so it gives us the evidence needed to fix these core integration problems one.

Tom: And the authors of FLARE have built this framework to show that deep fusion is where the real challenge lies in long video retrieval one.

Conclusion: Tom: So we're wrapping up this look at FLARE: Full-Modality Long-Video Audiovisual Retrieval Benchmark with User-Simulated Queries one.

Jane: It’s a new way to test how well AI can find long videos when the search query is realistic, not just a short clip or a single piece of text.

Tom: Right, so they built this benchmark using three hundred ninety-nine long videos and user-style queries to see if unified models can actually reason across vision and audio in real search scenarios one.

Lu: The authors are trying to find where the current AI systems break when they have to combine different types of data from long media.

Meng: It really shows us where the failure points are right now when we're shipping models to production because it’s not just about accuracy on a test set.

Lalam: FLARE is designed for research and testing, so it gives us the evidence needed to fix these core integration problems one.

Jane: It really highlights that real-world search scenarios are much harder than what we've been testing before, especially when you have long videos involved one.

Tom: And the authors of FLARE have built this framework to show that deep fusion is where the real challenge lies in long video retrieval one.

Jane: So next up, we're going to look at those specific results and see how much audio actually drags down the overall performance when things get complicated.

Qijie You, Hao Liang, Mingrui Chen, Bohan Zeng, Meiyi Qiang, Zhenhao Wong

University of Science and Technology Beijing · Peking University · Institute of Automation, Chinese Academy of Sciences

cs.MM

Submitted: 2026-05-11

Updated: 2026-11-18

Comments: Accepted at NeurIPS 2026

Project page: https://flarebench.github.io

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 94/100

The gist: FLARE introduces a full-modality long-video audiovisual retrieval benchmark with user-simulated queries to better reflect realistic multimedia search scenarios The first full-modality long-video…

Key concepts

FLARE
A full-modality long-video retrieval benchmark designed to simulate real user searches. It combines long videos, detailed clip annotations, and user-style queries across vision, audio, and unified audiovisual formats to rigorously test multimedia search models.
Hard Bimodal Constraint
A specific rule applied to cross-modal queries in FLARE. These queries must fail when searched using only one modality (vision or audio) but succeed only when both modalities are used together, forcing models to truly understand the combined information.
Modality Bottleneck
The consistent finding that audio retrieval is the weakest link in unified audiovisual models. Audio-only media and queries often perform worse than visual counterparts, suggesting that aligning or fusing audio information effectively remains a significant technical hurdle for current AI systems.

Terminology

Summary

FLARE introduces a full-modality long-video audiovisual retrieval benchmark with user-simulated queries to better reflect realistic multimedia search scenarios The first full-modality long-video retrieval benchmark is FLARE, which is designed to better reflect realistic multimedia search This benchmark combines long-form videos, fine-grained clip annotations, user-simulated queries, and a hard bimodal constraint for cross-modal queries

Benchmark Construction and Data Scale

FLARE comprises 399 long-form videos (10–60 min, 225.4 h) from Video-MME [12], segmented into 87,697 clips, each annotated with vision, audio, and unified audiovisual captions The dataset includes 274,933 user-style queries across three modality types The video duration distribution confirms that the benchmark is built from genuinely long-form content: all 399 source videos exceed 10 minutes Clip duration shows a mean of 9. p. 1

Where the paper says it

FLARE comprises 399 long-form videos (10–60 min, 225. 4 h) from Video-MME [12], segmented into 87,697 clips, each annotated with vision, audio, and unified audiovisual captions, plus 274,933 user-style queries across three modality types.

2s and a median of 6. p. 6
Where the paper says it

10 25 25 40 40 55 55+ Video Duration (min) 0 40 80 120 160 Number of Videos 100 147 132 20 (a) Video Duration 0 20 40 60 Clip Duration (s) 0 5k 10k 15k Number of Clips (b) Clip Duration Mean = 9. 2s Median = 6.

5s, producing 87,697 retrieval units The token count for video-level captions across three modality types is substantial, with unified captions being the longest at 102. p. 1
Where the paper says it

FLARE comprises 399 long-form videos (10–60 min, 225. 4 h) from Video-MME [12], segmented into 87,697 clips, each annotated with vision, audio, and unified audiovisual captions, plus 274,933 user-style queries across three modality types.

5k tokens

Query Generation and Constraint Mechanism

FLARE introduces user-style queries across three types, including vision-only, audio-only, and cross-modal queries Cross-modal queries satisfy a hard bimodal constraint (Eq. 6): it must fail unimodal retrieval and only succeed when both modalities are combined Candidate generation involves rewriting modality captions into 3–5 concise search queries for single modalities, and generating joint queries with visual and audio components for cross-modal ones Relevance is enforced by requiring semantic similarity sim(q, c) ≥ θsim and lexical overlap RL(q, c) ≤ θrouge

Evaluation Regimes and Key Findings

FLARE evaluates models under two regimes: caption-based (using detailed captions) and query-based (using user-style queries) across vision, audio, and unified audiovisual settings Experiments reveal that switching from captions to user-style queries causes top-performing models to suffer a drop of around 20–40% in Recall@1 Audio–language alignment remains a key bottleneck for unified audiovisual retrieval

Ablation Studies and Bottleneck Identification

Modality ablation experiments reveal that audio retrieval is consistently the bottleneck of unified audiovisual models, as audio-only media and queries consistently lag behind their visual counterparts Ablation A on the media side shows that for ImageBind and LanguageBind, simple pooling is dominated by the weaker audio component due to severe per-modality embedding imbalance Ablation B on the text side demonstrates that vision queries consistently outperform audio queries when using full unified media embeddings This highlights fusion-level failures and cross-modal gains that unimodal evaluations cannot capture

Human Evaluation Protocol

A rigorous human annotation protocol is integrated at multiple stages of the FLARE construction pipeline Annotators undergo training, and quality assurance involves threshold calibration using EVQAScore and BRACEScore Final quality evaluation shows that the mean score across all evaluated captions is 4.5, with inter-annotator Cohen’s κ = 0. p. 3

Where the paper says it

(2) Both curves are standardized with a robust (MAD-based) z-score and fused with equal weight: N (t) = 1 2 Φ( ˆ t) + 1 2 DˆKL(t). (3) Local peaks of N exceeding zmin = 5 with a minimum inter-peak gap of 1 s form candidate boundaries; resulting segments shorter than d spec min = 3 s are merged with neighbors.

73 The final mean score across all evaluated queries is 4. p. 6
Where the paper says it

2) and query-based (using the curated user-style queries of Section 3. Each configuration is evaluated on four directions: Text→Clip (T→C), Text→Video (T→V), Clip→Text (C→T), and Video→Text (V→T); the query regime is restricted to clip-level tasks since queries are generated per clip.

6, confirming the quality and naturalness of the curated queries

Limitations and Broader Impact

FLARE's limitations include reliance on Video-MME videos and Qwen-series models which may introduce model-specific preferences The evaluation focuses on retrieval accuracy with fixed public checkpoints, leaving efficiency and robustness for future work FLARE is intended solely for research and benchmark evaluation purposes to mitigate risks like invasive search or surveillance The dataset is constructed from existing video sources under their applicable licenses and terms of use The paper provides open access to the data and code at anonymous.4open.science/r/FLARE-950E/ and huggingface. p. 0

Where the paper says it

FLARE: Full-Modality Long-Video Audiovisual Retrieval Benchmark with User-Simulated Queries Qijie You1∗ Hao Liang2,4∗ † Mingrui Chen3 Bohan Zeng2 Meiyi Qiang2 Zhenhao Wong2 Wentao Zhang2,4‡ 1University of Science and Technology Beijing 2Peking University 3 Institute of Automation, Chinese Academy of Sciences 4Zhongguancun Academy Abstract As video becomes increasingly central to information dissemination and multimodal large language models (MLLMs) continue to advance, evaluating video retrieval has become increasingly important. In realistic search scenarios, this requires matching short user queries to long-form content using both visual and auditory evidence.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. User-style query generation capabilities for multimodal models can be significantly improved by training models to generate queries that explicitly require both visual and auditory information together, as demonstrated by FLARE's cross-modal query design. This allows systems to move beyond simple caption paraphrasing to generating queries that require BOTH vision and audio information together for successful retrieval.

  2. Retrieval performance can be enhanced by implementing a dual-regime evaluation protocol that tests models under both caption-based and query-based regimes on the same gallery, isolating the impact of query formulation. This reveals that switching from captions to user-style queries causes the previously top-performing model in each of the three modalities to suffer a drop of around 20–40% in Recall@1.

  3. Unified audiovisual retrieval models can be strengthened by focusing on resolving the audio-language alignment bottleneck through better cross-modal fusion mechanisms. This is crucial because audio retrieval is consistently the bottleneck of unified audiovisual models; strengthening audio–language alignment and designing better cross-modal fusion mechanisms thus remain important open challenges.

Abstract

As video becomes increasingly central to information dissemination and multimodal large language models (MLLMs) continue to advance, evaluating video retrieval has become increasingly important. In realistic search scenarios, this requires matching short user queries to long-form content using both visual and auditory evidence. Yet existing retrieval benchmarks are still dominated by short clips, single modalities, and caption-based evaluation. We introduce FLARE, a full-modality long-video audiovisual retrieval benchmark with user-simulated queries. Built from 399 carefully screened Video-MME videos (10--60 min, 225.4 h) to ensure source quality and diversity, FLARE contains 87,697 clips annotated with vision, audio, and unified audiovisual captions, together with 274,933 user-style queries. Cross-modal queries are further filtered by a hard bimodal constraint, requiring retrieval to fail under either modality alone but succeed when both are combined. FLARE evaluates models under two regimes, caption-based and query-based retrieval, across vision, audio, and unified audiovisual settings. Experiments with 15 representative retrievers show that user-style queries substantially change model behavior, strong caption-based performance does not always transfer to query-based retrieval, and audio--language alignment remains a key bottleneck for unified audiovisual retrieval. Our code and data are released at https://flarebench.github.io/

Sources

Related papers