Not Another Text Benchmark: Putting the "Visual" Back in Visual Question Answering for Large Video Models

summary

Video file (mp4)

The gist

The gist The authors introduce three new vision-centric evaluation benchmarks in temporal frame retrieval, video future prediction, and causal memory distortion to evaluate visual understanding

In short

The authors created three new visual evaluation benchmarks for large video models: temporal frame retrieval, video future prediction, and causal memory distortion. These tasks test if models can reason visually rather than just through text prompts. Results show current frontier models struggle with fine-grained visual temporal understanding.

Key concepts

Temporal Frame Retrieval
This task asks the model to select the correct sequence of frames surrounding a specific event in a video clip from several visual options. It directly tests the model's ability to reason about and retrieve specific moments in time within a video, requiring visual understanding over textual descriptions.
Video Future Prediction
Given only a short initial segment of a video, the model must predict the correct subsequent frames. This evaluates the model's grasp of physical laws and causal relationships across time, testing its predictive capabilities regarding what will happen next in a sequence.
Causal Memory Distortion
This task assesses whether models can correctly identify or detect hallucinations in memory related to causal events. It involves checking if the model confuses frames that have a direct cause-and-effect relationship with those that do not, testing its ability to distinguish between true and false temporal connections.

Terminology used across episodes

This episode discusses

The paper

Not Another Text Benchmark: Putting the "Visual" Back in Visual Question Answering for Large Video Models · Read on arXiv

University of Copenhagen · Carnegie Mellon University · “Texas A&M University” Johns Hopkins University · “UiT The Arctic University of Norway”

Large video models have exhibited impressive performance on a wide range of visual question answering tasks, owing to the rise of powerful, pretrained text and vision encoders. The usefulness of such models have also been demonstrated on a wide range of benchmarks, with an important caveat - the dominant approach in these benchmarks evaluates multiple choice reasoning via text options. This is a natural way to test text-based reasoning in these models, and has led to significant insights regarding model behavior in the community. In this work, we ask a different question - what happens when the evaluation modality is visual, rather than text? We introduce three new vision-centric evaluation benchmarks in temporal frame retrieval, video future prediction, and causal memory distortion, all designed around evaluating visual understanding capabilities in large video models. Our approach complements the existing approaches to evaluate video understanding in frontier models. We show that current frontier models exhibit significant weakness when attempting to reason through visual queries, rather than text. We conclude with an extended analysis section that provides pointers for future improvements in visual understanding for large video models.

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "Not Another Text Benchmark".

Jane: The gist The authors introduce three new vision-centric evaluation benchmarks in temporal frame retrieval, video future prediction, and causal memory distortion to evaluate visual understanding capabilities in large video models.

Tom: First, who's behind it and why it matters.

Title and authors: Tom: Let's talk about the title and who wrote this. The paper is called "Not Another Text Benchmark: Putting the 'Visual' Back in Visual Question Answering for Large Video Models." It really signals that they are pushing back against how we usually evaluate these models.

Jane: And it’s written by Rwiddhi Chakraborty, Yinong Oliver Wang, Cheng Zhang, Fan Bai, Zhuoran Yu, Michael Kampffmeyer, and Yong Jae Lee. These researchers are clearly focused on this visual modality aspect of video understanding.

Lu: The authors are motivated because they see a gap in how we test these models. They feel the dominant approach of using text options to test reasoning might be missing something important about what these large video models actually know visually <ref:2609.17112#pg4>.

Meng: So they aren't just tweaking existing tests; they are building a whole new way to probe their visual comprehension. I wonder if that approach would work on real traffic, though.

Lalam: It’s about creating a complementary view with existing text benchmarks, which is what the authors say in the conclusion of this paper <ref:2609.17112#pg8>. They want to see where the models actually fail when they are forced to reason visually.

The paper's summary: Tom: So, summarizing what the paper is doing, they introduce three specific tasks: temporal frame retrieval, video future prediction, and causal memory distortion. These are designed purely around visual understanding capabilities.

Jane: Think of temporal frame retrieval as a test where the model has to pick the right sequence of frames after or before an anchor event from a set of visual options instead of just reading text choices. It directly assesses frame retrieval and temporal reasoning <ref:2609.17112#pg4>.

Lu: Then there is video future prediction, where you give the model a truncated video clip, maybe just the first frame or two, and it has to select the correct subsequent future frames from a set of candidates. This really tests their understanding of physical and causal dynamics across time <ref:2609.17112#pg6>.

Meng: And then there’s causal memory distortion. This one is inspired by cognitive science studies on how humans form false memories about causal events in videos, so they test if the model can tell the difference between a frame that has a cause and one that doesn't <ref:2609.17112#pg6>.

Lalam: What’s interesting is that all these tasks are purely visual evaluations, which is what sets them apart from the standard text-based VQA datasets they usually look at <ref:2609.17112#pg4>.

The paper's improvements: Tom: The authors point out some key improvements they made to this evaluation suite. They focused on making sure the evaluation modality is strictly visual, which they call a novel way of testing model behavior <ref:2609.17112#pg4>.

Jane: They highlight that when using visual options instead of text options, frontier models show a significant drop in accuracy. For example, on the temporal frame retrieval task in their dataset, Gemini two point five Pro performs significantly worse when the answer choices are visual than when they are text <ref:2609.17112#pg2>.

Lu: The authors found surprising results of poor model performance and an extended analysis section that looks at model failure modes, which they think paves the way for future improvements in how we design these models <ref:2609.17112#pg3>. They also found that large language models often use a kind of “centralized” description when evaluating <ref:2609.17112#pg2>.

Meng: So, if a model is failing because it's using a centralized description instead of looking at the visual options directly, that’s an actionable thing for us to know for production deployment.

Lalam: And they also point out that text-based evaluations might inflate performance when text options are available as is, but the opposite happens if those visual options are converted into text due to a loss of fine-grained information <ref:2609.17112#pg9>.

Conclusion: Tom: So, wrapping up this paper on "Not Another Text Benchmark: Putting the 'Visual' Back in Visual Question Answering for Large Video Models," the main point is that visual querying surfaces gaps in fine-grained temporal understanding that text querying alone cannot uncover <ref:2609.17112#pg9>.

Jane: The authors show a range of difficulty across these three tasks, and crucially, all of them remain meaningfully above random performance across the state-of-the-art models tested <ref:2609.17112#pg9>.

Lu: They conclude that visual querying exposes gaps in how well these models actually ground their reasoning in the visual modality rather than just processing text <ref:2609.17112#pg10>. This suggests a need for better visual grounding techniques in large video models.

Meng: From an engineering standpoint, this means we can’t just rely on text-based metrics to gauge a model's true visual intelligence; we need these direct visual tests <ref:2609.17112#pg6>.

Lalam: I think the big implication here is that models reason well on text but struggle to connect that reasoning to the actual visual data in a coherent way <ref:2609.17112#pg10>. This new evaluation approach gives us a more complete picture of their limitations.

More episodes

← Home