Not Another Text Benchmark: Putting the "Visual" Back in Visual Question Answering for Large Video Models

arXiv:2609.17112 · cs.CV · Submitted 2026-09-15 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "Not Another Text Benchmark".

Jane: The gist The authors introduce three new vision-centric evaluation benchmarks in temporal frame retrieval, video future prediction, and causal memory distortion to evaluate visual understanding capabilities in large video models.

Tom: First, who's behind it and why it matters.

Title and authors: Tom: Let's talk about the title and who wrote this. The paper is called "Not Another Text Benchmark: Putting the 'Visual' Back in Visual Question Answering for Large Video Models." It really signals that they are pushing back against how we usually evaluate these models.

Jane: And it’s written by Rwiddhi Chakraborty, Yinong Oliver Wang, Cheng Zhang, Fan Bai, Zhuoran Yu, Michael Kampffmeyer, and Yong Jae Lee. These researchers are clearly focused on this visual modality aspect of video understanding.

Lu: The authors are motivated because they see a gap in how we test these models. They feel the dominant approach of using text options to test reasoning might be missing something important about what these large video models actually know visually <ref:2609.17112#pg4>.

Meng: So they aren't just tweaking existing tests; they are building a whole new way to probe their visual comprehension. I wonder if that approach would work on real traffic, though.

Lalam: It’s about creating a complementary view with existing text benchmarks, which is what the authors say in the conclusion of this paper <ref:2609.17112#pg8>. They want to see where the models actually fail when they are forced to reason visually.

The paper's summary: Tom: So, summarizing what the paper is doing, they introduce three specific tasks: temporal frame retrieval, video future prediction, and causal memory distortion. These are designed purely around visual understanding capabilities.

Jane: Think of temporal frame retrieval as a test where the model has to pick the right sequence of frames after or before an anchor event from a set of visual options instead of just reading text choices. It directly assesses frame retrieval and temporal reasoning <ref:2609.17112#pg4>.

Lu: Then there is video future prediction, where you give the model a truncated video clip, maybe just the first frame or two, and it has to select the correct subsequent future frames from a set of candidates. This really tests their understanding of physical and causal dynamics across time <ref:2609.17112#pg6>.

Meng: And then there’s causal memory distortion. This one is inspired by cognitive science studies on how humans form false memories about causal events in videos, so they test if the model can tell the difference between a frame that has a cause and one that doesn't <ref:2609.17112#pg6>.

Lalam: What’s interesting is that all these tasks are purely visual evaluations, which is what sets them apart from the standard text-based VQA datasets they usually look at <ref:2609.17112#pg4>.

The paper's improvements: Tom: The authors point out some key improvements they made to this evaluation suite. They focused on making sure the evaluation modality is strictly visual, which they call a novel way of testing model behavior <ref:2609.17112#pg4>.

Jane: They highlight that when using visual options instead of text options, frontier models show a significant drop in accuracy. For example, on the temporal frame retrieval task in their dataset, Gemini two point five Pro performs significantly worse when the answer choices are visual than when they are text <ref:2609.17112#pg2>.

Lu: The authors found surprising results of poor model performance and an extended analysis section that looks at model failure modes, which they think paves the way for future improvements in how we design these models <ref:2609.17112#pg3>. They also found that large language models often use a kind of “centralized” description when evaluating <ref:2609.17112#pg2>.

Meng: So, if a model is failing because it's using a centralized description instead of looking at the visual options directly, that’s an actionable thing for us to know for production deployment.

Lalam: And they also point out that text-based evaluations might inflate performance when text options are available as is, but the opposite happens if those visual options are converted into text due to a loss of fine-grained information <ref:2609.17112#pg9>.

Conclusion: Tom: So, wrapping up this paper on "Not Another Text Benchmark: Putting the 'Visual' Back in Visual Question Answering for Large Video Models," the main point is that visual querying surfaces gaps in fine-grained temporal understanding that text querying alone cannot uncover <ref:2609.17112#pg9>.

Jane: The authors show a range of difficulty across these three tasks, and crucially, all of them remain meaningfully above random performance across the state-of-the-art models tested <ref:2609.17112#pg9>.

Lu: They conclude that visual querying exposes gaps in how well these models actually ground their reasoning in the visual modality rather than just processing text <ref:2609.17112#pg10>. This suggests a need for better visual grounding techniques in large video models.

Meng: From an engineering standpoint, this means we can’t just rely on text-based metrics to gauge a model's true visual intelligence; we need these direct visual tests <ref:2609.17112#pg6>.

Lalam: I think the big implication here is that models reason well on text but struggle to connect that reasoning to the actual visual data in a coherent way <ref:2609.17112#pg10>. This new evaluation approach gives us a more complete picture of their limitations.

University of Copenhagen · Carnegie Mellon University · “Texas A&M University” Johns Hopkins University · “UiT The Arctic University of Norway”

cs.CV

Submitted: 2026-09-15

Updated: 2026-10-19

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 83/100

The gist: The gist The authors introduce three new vision-centric evaluation benchmarks in temporal frame retrieval, video future prediction, and causal memory distortion to evaluate visual understanding

Key concepts

Temporal Frame Retrieval: This task asks the model to select the correct sequence of frames surrounding a specific event in a video clip from several visual options. It directly tests the model's ability to reason about and retrieve specific moments in time within a video, requiring visual understanding over textual descriptions.
Video Future Prediction: Given only a short initial segment of a video, the model must predict the correct subsequent frames. This evaluates the model's grasp of physical laws and causal relationships across time, testing its predictive capabilities regarding what will happen next in a sequence.
Causal Memory Distortion: This task assesses whether models can correctly identify or detect hallucinations in memory related to causal events. It involves checking if the model confuses frames that have a direct cause-and-effect relationship with those that do not, testing its ability to distinguish between true and false temporal connections.

Terminology

Benchmark — 1× in this paper · explained in 2 episodes

Summary

The gist The authors introduce three new vision-centric evaluation benchmarks in temporal frame retrieval, video future prediction, and causal memory distortion to evaluate visual understanding capabilities in large video models.

Motivation and Approach

Large video models have shown impressive performance on visual question answering tasks because of powerful pretrained text and vision encoders<ref:2609.17112#pg3>. The authors are motivated to create a more direct evaluation suite for visual understanding, i.e., creating datasets where the evaluation modality is visual<ref:2609.17112#pg4>. This approach gives a complementary view in evaluating the same VQA datasets on the same models in a different modality than text<ref:2609.17112#pg4>. They show that current frontier models exhibit significant weakness when attempting to reason through visual queries, rather than text<ref:2609.17112#pg5>.

Evaluation Benchmarks

The authors introduce three tasks designed around evaluating visual understanding capabilities in large video models<ref:2609.17112#pg6>. These tasks are:

** Temporal frame retrieval:**

The model is prompted to choose the correct sequence of frames after or before an anchor event from a set of candidate visual options<ref:2609.17112#pg4>. The dataset is constructed using video clips from the Charades dataset, where the model must select among visual options, not text<ref:2609.17112#pg5>.

** Video future prediction:**

Given a video clip in which only an initial frame segment is visible (i.e. only a truncated video as input), the model must select the correct future frames from a set of candidate segments<ref:2609.17112#pg6>. This task tests understanding of physical and causal dynamics across time<ref:2609.17112#pg6>.

** Causal memory distortion:**

This task is inspired by an influential result in cognitive science [28], which studies how humans form false memories about causal events in videos<ref:2609.17112#pg6>. It evaluates memory distortion by testing hallucination of a contact moment when the corresponding frame is removed from the input video<ref:2609.17112#pg6>.

Experimental Results

The main results reveal substantial weaknesses in these core visual tasks across state-of-the-art videolanguage models<ref:2609.17112#pg6>. On the temporal frame retrieval task, all models perform poorly (slightly above random chance)<ref:2609.17112#pg9>. On the future prediction task, Gemini significantly outperforms the open-source models, and Qwen models perform slightly above the random baseline<ref:2609.17112#pg9>.

Reasoning Patterns and Failure Modes

The authors manually evaluate reasoning traces from Gemini 2.5 Pro and Qwen2.5-72B on the temporal frame retrieval task by extracting and annotating claims for factual grounding<ref:2609.17112#pg10>. Gemini first describes the video broadly, then analyzes options one by one, but it frequently misidentifies reversed sequences and fails to map its reasoning to the correct visual option—a failure of option grounding<ref:2609.17112#pg10>. Qwen, by contrast, ignores the visual options entirely, listing discrete events before selecting an answer<ref:2609.17112#pg10>.

Conclusion and Insights

The work demonstrates that text-based evaluations may inflate performance when text options are available as-is, but the opposite may happen if visual options are converted into text, owing to loss in fine-grained information<ref:2609.17112#pg9>. The results show a range of difficulty across tasks, but crucially, all remain meaningfully above random performance<ref:2609.17112#pg9>. The authors conclude that visual querying surfaces gaps in fine-grained temporal understanding that textual querying alone cannot<ref:2609.17112#pg9>. This new evaluation approach provides a complementary view to existing text benchmarks<ref:2609.17112#pg8>. The work suggests that models reason well on text but struggle to ground reasoning in the visual modality<ref:2609.17112#pg10>.

Limitations and Future Work

The authors note that expanding the dataset, particularly for causal memory distortion, requires careful deliberation due to heavy manual curation<ref:2609.17112#pg8>. They also acknowledge that their analysis was entirely black-box and plan to explore white-box perspectives in the future<ref:2609.17112#pg7>. The work provides pointers for future improvements in visual understanding for large video models<ref:2609.17112#pg6>.

Appendix Details

The appendix furnishes additional details on the construction of the temporal retrieval task, the future prediction task, and the causal memory distortion task<ref:2609.17112#pg15>. Table 2 presents detailed statistics on our benchmarks, including the number of videos, questions, frames, duration, and difficulty based on human judgement<ref:2609.17112#pg6>. The authors also present a pipeline to analyze the reasoning traces in Gemini and Qwen72B by extracting claims using a large language model (GPT5.2)<ref:2609.17112#pg20>. Table 4 shows signal detection performance across models, including sensitivity and specificity for the causal memory distortion task<ref:2609.17112#pg21>. The uncertainty analysis in Figure 11 reveals that Gemini is overall more overconfident in its predictions, while Qwen is more uncertain on both modalities<ref:2609.17112#pg21>. The authors also present an ablation for Gemini and Kimi K2.6 on TFR using prompt variations, which shows a minimal difference in the vision-text performance gap. The study concludes that visual querying surfaces gaps that text querying alone cannot elucidate in video understanding<ref:2609.17112#pg9>.

The paper is 5 pages long, and the reference list is on page 9 and 10. The total number of pages is 22. The summary adheres strictly to the specified formatting rules regarding token usage and anchoring.

--- Page 1 ---

Not Another Text Benchmark: Putting the “Visual” Back in Visual Question Answering for Large Video Models<ref:2609.17112#pg2>

Improvements for AI systems

Bold header: Visual query evaluation suite implementation

This improvement involves creating a novel evaluation setup where the evaluation modality is visual, rather than relying solely on text options, which allows researchers to investigate whether frontier video models could demonstrate inflated VQA performance by using language cues.

Bold header: Temporal frame retrieval task

The improved system can accurately select the correct sequence of frames after or before an anchor event from a set of four visual options, directly assessing frame retrieval and temporal reasoning of events as described in Section 3.1.

Bold header: Video future prediction capability

The system will be capable of predicting the correct subsequent future frames given only an initial frame segment, testing the model's understanding of physical and causal dynamics across time.

Bold header: Causal memory distortion detection

The improved system can detect hallucination of a contact moment by evaluating whether models can distinguish between frames with or without causal implication, as detailed in Section 3.3, which is the first evaluation of this task in LVM literature.

Bold header: Improved reasoning trace analysis

The system will employ a pipeline to extract and analyze claims from model reasoning traces, allowing researchers to identify failure modes, such as Gemini's option confounding or Qwen's skipping the options entirely, which provides specific pointers for future improvements.

Abstract

Large video models have exhibited impressive performance on a wide range of visual question answering tasks, owing to the rise of powerful, pretrained text and vision encoders. The usefulness of such models have also been demonstrated on a wide range of benchmarks, with an important caveat - the dominant approach in these benchmarks evaluates multiple choice reasoning via text options. This is a natural way to test text-based reasoning in these models, and has led to significant insights regarding model behavior in the community. In this work, we ask a different question - what happens when the evaluation modality is visual, rather than text? We introduce three new vision-centric evaluation benchmarks in temporal frame retrieval, video future prediction, and causal memory distortion, all designed around evaluating visual understanding capabilities in large video models. Our approach complements the existing approaches to evaluate video understanding in frontier models. We show that current frontier models exhibit significant weakness when attempting to reason through visual queries, rather than text. We conclude with an extended analysis section that provides pointers for future improvements in visual understanding for large video models.