FLARE: Full-Modality Long-Video Audiovisual Retrieval Benchmark with User-Simulated Queries
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.
Tom: We're looking at the paper: "FLARE: Full-Modality Long-Video Audiovisual Retrieval Benchmark with User-Simulated Queries".
Paper summary: Tom: So we're wrapping up this look at FLARE: Full-Modality Long-Video Audiovisual Retrieval Benchmark with User-Simulated Queries one.
Jane: It’s a new way to test how well AI can find long videos when the search query is realistic, not just a short clip or a single piece of text.
Tom: Right, so they built this benchmark using three hundred ninety-nine long videos and user-style queries to see if unified models can actually reason across vision and audio in real search scenarios one.
Lu: The authors are trying to find where the current AI systems break when they have to combine different types of data from long media.
Meng: It really shows us where the failure points are right now when we're shipping models to production because it’s not just about accuracy on a test set.
Lalam: FLARE is designed for research and testing, so it gives us the evidence needed to fix these core integration problems one.
Tom: And the authors of FLARE have built this framework to show that deep fusion is where the real challenge lies in long video retrieval one.
Conclusion: Tom: So we're wrapping up this look at FLARE: Full-Modality Long-Video Audiovisual Retrieval Benchmark with User-Simulated Queries one.
Jane: It’s a new way to test how well AI can find long videos when the search query is realistic, not just a short clip or a single piece of text.
Tom: Right, so they built this benchmark using three hundred ninety-nine long videos and user-style queries to see if unified models can actually reason across vision and audio in real search scenarios one.
Lu: The authors are trying to find where the current AI systems break when they have to combine different types of data from long media.
Meng: It really shows us where the failure points are right now when we're shipping models to production because it’s not just about accuracy on a test set.
Lalam: FLARE is designed for research and testing, so it gives us the evidence needed to fix these core integration problems one.
Jane: It really highlights that real-world search scenarios are much harder than what we've been testing before, especially when you have long videos involved one.
Tom: And the authors of FLARE have built this framework to show that deep fusion is where the real challenge lies in long video retrieval one.
Jane: So next up, we're going to look at those specific results and see how much audio actually drags down the overall performance when things get complicated.
Qijie You, Hao Liang, Mingrui Chen, Bohan Zeng, Meiyi Qiang, Zhenhao Wong
University of Science and Technology Beijing · Peking University · Institute of Automation, Chinese Academy of Sciences
cs.MM
Submitted: 2026-05-11
Updated: 2026-11-18
Comments: Accepted at NeurIPS 2026
Project page: https://flarebench.github.io
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 94/100
The gist: FLARE introduces a full-modality long-video audiovisual retrieval benchmark with user-simulated queries to better reflect realistic multimedia search scenarios The first full-modality long-video…
Key concepts
- FLARE
- A full-modality long-video retrieval benchmark designed to simulate real user searches. It combines long videos, detailed clip annotations, and user-style queries across vision, audio, and unified audiovisual formats to rigorously test multimedia search models.
- Hard Bimodal Constraint
- A specific rule applied to cross-modal queries in FLARE. These queries must fail when searched using only one modality (vision or audio) but succeed only when both modalities are used together, forcing models to truly understand the combined information.
- Modality Bottleneck
- The consistent finding that audio retrieval is the weakest link in unified audiovisual models. Audio-only media and queries often perform worse than visual counterparts, suggesting that aligning or fusing audio information effectively remains a significant technical hurdle for current AI systems.
Terminology
Summary
FLARE introduces a full-modality long-video audiovisual retrieval benchmark with user-simulated queries to better reflect realistic multimedia search scenarios The first full-modality long-video retrieval benchmark is FLARE, which is designed to better reflect realistic multimedia search This benchmark combines long-form videos, fine-grained clip annotations, user-simulated queries, and a hard bimodal constraint for cross-modal queries
Benchmark Construction and Data Scale
FLARE comprises 399 long-form videos (10–60 min, 225.4 h) from Video-MME [12], segmented into 87,697 clips, each annotated with vision, audio, and unified audiovisual captions The dataset includes 274,933 user-style queries across three modality types The video duration distribution confirms that the benchmark is built from genuinely long-form content: all 399 source videos exceed 10 minutes Clip duration shows a mean of 9. p. 1 FLARE comprises 399 long-form videos (10–60 min,
225. 4 h) from Video-MME [12], segmented into 87,697 clips, each annotated with vision, audio,
and unified audiovisual captions, plus 274,933 user-style queries across three modality types. 10 25 25 40 40 55 55+
Video Duration (min)
0
40
80
120
160
Number of Videos
100
147
132
20
(a) Video Duration
0 20 40 60
Clip Duration (s)
0
5k
10k
15k
Number of Clips
(b) Clip Duration
Mean = 9. 2s
Median = 6. FLARE comprises 399 long-form videos (10–60 min,
225. 4 h) from Video-MME [12], segmented into 87,697 clips, each annotated with vision, audio,
and unified audiovisual captions, plus 274,933 user-style queries across three modality types.Where the paper says it
Where the paper says it
Where the paper says it
Query Generation and Constraint Mechanism
FLARE introduces user-style queries across three types, including vision-only, audio-only, and cross-modal queries Cross-modal queries satisfy a hard bimodal constraint (Eq. 6): it must fail unimodal retrieval and only succeed when both modalities are combined Candidate generation involves rewriting modality captions into 3–5 concise search queries for single modalities, and generating joint queries with visual and audio components for cross-modal ones Relevance is enforced by requiring semantic similarity sim(q, c) ≥ θsim and lexical overlap RL(q, c) ≤ θrouge
Evaluation Regimes and Key Findings
FLARE evaluates models under two regimes: caption-based (using detailed captions) and query-based (using user-style queries) across vision, audio, and unified audiovisual settings Experiments reveal that switching from captions to user-style queries causes top-performing models to suffer a drop of around 20–40% in Recall@1 Audio–language alignment remains a key bottleneck for unified audiovisual retrieval
Ablation Studies and Bottleneck Identification
Modality ablation experiments reveal that audio retrieval is consistently the bottleneck of unified audiovisual models, as audio-only media and queries consistently lag behind their visual counterparts Ablation A on the media side shows that for ImageBind and LanguageBind, simple pooling is dominated by the weaker audio component due to severe per-modality embedding imbalance Ablation B on the text side demonstrates that vision queries consistently outperform audio queries when using full unified media embeddings This highlights fusion-level failures and cross-modal gains that unimodal evaluations cannot capture
Human Evaluation Protocol
A rigorous human annotation protocol is integrated at multiple stages of the FLARE construction pipeline Annotators undergo training, and quality assurance involves threshold calibration using EVQAScore and BRACEScore Final quality evaluation shows that the mean score across all evaluated captions is 4.5, with inter-annotator Cohen’s κ = 0. p. 3 (2)
Both curves are standardized with a robust (MAD-based) z-score and fused with equal weight:
N (t) = 1
2 Φ( ˆ t) + 1
2 DˆKL(t). (3)
Local peaks of N exceeding zmin = 5 with a minimum inter-peak gap of 1 s form candidate
boundaries; resulting segments shorter than d
spec
min = 3 s are merged with neighbors. 2) and query-based (using the curated user-style queries of
Section 3. Each configuration is evaluated on four directions: Text→Clip (T→C), Text→Video
(T→V), Clip→Text (C→T), and Video→Text (V→T); the query regime is restricted to clip-level
tasks since queries are generated per clip.Where the paper says it
Where the paper says it
Limitations and Broader Impact
FLARE's limitations include reliance on Video-MME videos and Qwen-series models which may introduce model-specific preferences The evaluation focuses on retrieval accuracy with fixed public checkpoints, leaving efficiency and robustness for future work FLARE is intended solely for research and benchmark evaluation purposes to mitigate risks like invasive search or surveillance The dataset is constructed from existing video sources under their applicable licenses and terms of use The paper provides open access to the data and code at anonymous.4open.science/r/FLARE-950E/ and huggingface. p. 0 FLARE: Full-Modality Long-Video Audiovisual
Retrieval Benchmark with User-Simulated Queries
Qijie You1∗ Hao Liang2,4∗ † Mingrui Chen3 Bohan Zeng2 Meiyi Qiang2
Zhenhao Wong2 Wentao Zhang2,4‡
1University of Science and Technology Beijing
2Peking University
3
Institute of Automation, Chinese Academy of Sciences
4Zhongguancun Academy
Abstract
As video becomes increasingly central to information dissemination and multimodal large language models (MLLMs) continue to advance, evaluating video
retrieval has become increasingly important. In realistic search scenarios, this
requires matching short user queries to long-form content using both visual and
auditory evidence.Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
User-style query generation capabilities for multimodal models can be significantly improved by training models to generate queries that explicitly require both visual and auditory information together, as demonstrated by FLARE's cross-modal query design. This allows systems to move beyond simple caption paraphrasing to generating queries that
require BOTH vision and audio information together for successful retrieval.
-
Retrieval performance can be enhanced by implementing a dual-regime evaluation protocol that tests models under both caption-based and query-based regimes on the same gallery, isolating the impact of query formulation. This reveals that
switching from captions to user-style queries causes the previously top-performing model in each of the three modalities to suffer a drop of around 20–40% in Recall@1.
-
Unified audiovisual retrieval models can be strengthened by focusing on resolving the audio-language alignment bottleneck through better cross-modal fusion mechanisms. This is crucial because
audio retrieval is consistently the bottleneck of unified audiovisual models; strengthening audio–language alignment and designing better cross-modal fusion mechanisms thus remain important open challenges.
Abstract
As video becomes increasingly central to information dissemination and multimodal large language models (MLLMs) continue to advance, evaluating video retrieval has become increasingly important. In realistic search scenarios, this requires matching short user queries to long-form content using both visual and auditory evidence. Yet existing retrieval benchmarks are still dominated by short clips, single modalities, and caption-based evaluation. We introduce FLARE, a full-modality long-video audiovisual retrieval benchmark with user-simulated queries. Built from 399 carefully screened Video-MME videos (10--60 min, 225.4 h) to ensure source quality and diversity, FLARE contains 87,697 clips annotated with vision, audio, and unified audiovisual captions, together with 274,933 user-style queries. Cross-modal queries are further filtered by a hard bimodal constraint, requiring retrieval to fail under either modality alone but succeed when both are combined. FLARE evaluates models under two regimes, caption-based and query-based retrieval, across vision, audio, and unified audiovisual settings. Experiments with 15 representative retrievers show that user-style queries substantially change model behavior, strong caption-based performance does not always transfer to query-based retrieval, and audio--language alignment remains a key bottleneck for unified audiovisual retrieval. Our code and data are released at https://flarebench.github.io/
Sources
- MusicLM: Generating Music From Text
- LoVR: A Benchmark for Long Video Retrieval in Multimodal Contexts
- JointAVBench: A Benchmark for Joint Audio-Visual Reasoning Evaluation
- Meta CLIP 2: A Worldwide Scaling Recipe
- Qwen3-VL-Embedding and Qwen3-VL-Reranker: A Unified Framework for State-of-the-Art Multimodal Retrieval and Ranking
- Learning Transferable Visual Models From Natural Language Supervision
- Qwen3-ASR Technical Report
- Qwen3 Technical Report
- Pushing the Frontier of Audiovisual Perception with Large-Scale Multimodal Correspondence Learning
- VideoCLIP-XL: Advancing Long Description Understanding for Video CLIP Models
- Scaling Audio-Text Retrieval with Multimodal Large Language Models
- Qwen3-Omni Technical Report
Related papers
- When does the head leave? Shared timing and a movement clock in 360-degree video viewing
- Augmented Hybrid TV Services. Habits, Preferences, Opinions and Expectations of Spanish Consumers
- A Survey of Generative Categories and Techniques in Multimodal Generative Models
- ControlFoley: Unified and Controllable Video-to-Audio Generation with Cross-Modal Conflict Handling
- EyeNavGS: A 6-DoF Navigation Dataset and Record-n-Replay Software for Real-World 3DGS Scenes in VR
- Mechanisms of Multimodal Synchronization: Insights from Decoder-Based Video-Text-to-Speech Synthesis