What Does Multiple Choice Actually Measure? Disentangling Video Understanding from Option Exploitation
cs.CV
Submitted: 2026-11-08
Updated: 2026-11-08
Comments: Accepted to TAE (Trust-AI-Eval) Workshop at NeurIPS 2026 https://neurips.cc/virtual/2026/loc/sydney/178197
Code: https://github.com/OnDeckAI/LVBench-Relabel
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Sources
- Learning to Deny: Action Denial in Multimodal Large Language Models
- Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answering
- CausalFlow: Causal Attribution and Counterfactual Repair for LLM Agent Failures
- Revisiting the "Video" in Video-Language Understanding
- Answer Matching Outperforms Multiple Choice for Language Model Evaluation
- MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmark
- AutoEval-Video: An Automatic Benchmark for Assessing Large Vision Language Models in Open-Ended Video Question Answering
- DatBench: Discriminative, Faithful, and Efficient VLM Evaluations
- WiCkeD: A Simple Method to Make Multiple Choice Benchmarks More Challenging
- MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding
- Breaking Down Video LLM Benchmarks: Knowledge, Spatial Perception, or True Temporal Understanding?
- Video-MME-v2: Towards the Next Stage in Benchmarks for Comprehensive Video Understanding
- Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis
- Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering
- Changing Answer Order Can Decrease MMLU Accuracy
- Annotation Artifacts in Natural Language Inference Data
- Perception Test 2025: Challenge Summary and a Unified VQA Extension
- Accuracy Without Grounding: Diagnosing Visual Dependency Dissociation in Video LLM Benchmarks
- DoVer: Intervention-Driven Auto Debugging for LLM Multi-Agent Systems
- VideoEval-Pro: Robust and Realistic Long Video Understanding Evaluation
Related papers
- TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs