Query-Conditioned Evidential Keyframe Sampling for MLLM-Based Long-Form Video Understanding
cs.CV, cs.AI, cs.LG
Submitted: 2026-04-01
Updated: 2026-11-02
Comments: Accpeted by WACV 2027
License: http://creativecommons.org/licenses/by-nc-nd/4.0/
The gist: Multimodal Large Language Models (MLLMs) have shown strong performance on video question answering, but their application to long-form videos is constrained by limited context length and…
Terminology
Abstract
Multimodal Large Language Models (MLLMs) have shown strong performance on video question answering, but their application to long-form videos is constrained by limited context length and computational cost, making keyframe sampling essential. Existing approaches typically rely on semantic relevance or reinforcement learning, which either fail to capture evidential clues or suffer from inefficient combinatorial optimization. In this work, we propose an evidence-driven keyframe sampling framework motivated by an information-theoretic perspective. We consider maximizing the conditional mutual information between the selected frames and the answer given the query as an idealized objective, and use a modular singleton-gain surrogate to motivate a tractable frame-wise selection criterion. Guided by this motivation, we train a query-conditioned evidence scoring network using annotated answer-supporting segments. A contrastive objective encourages greater aggregate score mass on frames from evidence regions than on background frames, providing a practical evidence-based criterion beyond semantic similarity. Experiments on long-form video understanding benchmarks show that our method consistently outperforms prior sampling strategies under comparable frame budgets. With Qwen2.5-VL-7B under the same 32-frame budget, our method improves over uniform sampling by 10.1 points on LVBench and 4.4 points on the long-video subset of VideoMME.
Sources
- Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution
- Qwen3-VL Technical Report
- InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks
- Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling
- FOCUS: Efficient Keyframe Selection for Long Video Understanding
- VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs
- VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understanding
- Representation Learning with Contrastive Predictive Coding
- Video-o3: Native Interleaved Clue Seeking for Long Video Multi-Hop Reasoning
- LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Models
- TimeMarker: A Versatile Video-LLM for Long and Short Video Understanding with Superior Temporal Localization Ability
- Video-R1: Reinforcing Video Reasoning in MLLMs
- Video-Thinker: Sparking "Thinking with Videos" via Reinforcement Learning
- ReTaKe: Reducing Temporal and Knowledge Redundancy for Long Video Understanding
- FrameThinker: Learning to Think with Long Videos via Multi-Turn Frame Spotlighting
- K-frames: Scene-Driven Any-k Keyframe Selection for long video understanding
- SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features
Related papers
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs
- From Formulas to Protocols: A Compositional View of Binary Target Segmentation Metrics
- Traffic Sign Recognition in Autonomous Driving: Dataset, Benchmark, and Field Experiment