VISTAQA: Benchmarking Joint Visual Question Answering and Pixel-Level Evidence

arXiv:2605.20676 · cs.CV · Submitted 2026-05-20 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "VISTAQA: Benchmarking Joint Visual Question Answering and Pixel-Level Evidence".

Jane: Here's the gist of the paper. Establishing a clear link between model predictions and the visual evidence that supports them is critical for transparency and reliability in multimodal reasoning <ref:2605.20676#pg2> The paper introduces VISTAQA, a comprehensive benchmark for joint evaluation of free-form answer correctness and pixel-level evidence grounding in visual question answering <ref:2605.20676#pg4>.

Tom: Pruning redundant reasoning tokens VISTAQA comprises 1,157 expert-curated samples spanning six task types and six visual domains, ranging from direct perception to compositional and relational reasoning <ref:2605.20676#pg3> It requires models to not only answer correctly, but to also provide precise segmentation masks that support their answers <ref:2605.20676#pg4>.

Jane: The benchmark includes hallucination-aware examples where no valid visual evidence exists <ref:2605.20676#pg4>.

Tom: How the training signal is built To support this enhanced evaluation, GROVE, a unified evaluation metric, is introduced which enforces joint correctness by combining textual accuracy and grounding quality via a per-sample geometric mean <ref:2605.20676#pg4>.

Paper summary: Tom: So we’re looking at the paper "VISTAQA: Benchmarking Joint Visual Question Answering and Pixel-Level Evidence". The main idea here is that right now, when we test multimodal models, we usually test them on just the text answer or just the location of an object, but they don't really check if those two things line up correctly.

Jane: It sounds like they're trying to fix that gap where a model might get a good text answer but be completely wrong about where it is pointing in the picture, or vice versa. What are they proposing to do about that coupling?

Lu: They introduce VISTAQA, which is this comprehensive benchmark for joint evaluation of free-form answer correctness and pixel-level evidence grounding in visual question answering. It’s designed specifically to test how well a model connects what it says with the actual visual data.

Meng: So it’s not just asking a question and checking if the text is right, or checking if the mask is right, but making sure both are correct together? That sounds like it could be useful for deployment because that's where most real-world failure happens.

Lalam: Exactly. They include a huge set of samples, one thousand one hundred fifty-seven expert-curated ones across six different task types and six visual domains, covering everything from just direct perception to more complex relational reasoning in indoor and outdoor scenes <ref:2605.20676#pg1,task types and six visual domains>.

Tom: That’s a lot of ground they are covering there. I was looking at the tasks listed: identification, attribute, OCR, spatial reasoning, counting, and comparison. It seems like they are pushing models on really diverse challenges.

Jane: And it gets interesting because they even have these hallucination-aware examples where the question is unanswerable or refers to something that isn't even in the picture. This forces the model to correctly identify when there is no valid visual evidence at all.

Lu: That’s a crucial point because it tests if the model can handle missing information gracefully, which is hard for many current systems. To measure this joint correctness, they introduce a metric called GROVE, or Grounded Evidence Evaluation.

Paper summary: Meng: GROVE sounds like the evaluation method itself is designed to be sensitive to failures in either modality, meaning it penalizes errors in both text and mask quality at the same time. I worry that if you have two things wrong, the score just drops really hard.

Tom: That’s what they aimed for with joint sensitivity. But they also mention graceful degradation; GROVE is supposed to preserve a signal even when only one of those dimensions—either the answer or the mask—is partially correct.

Jane: How do they actually build that score? They compute it as a per-sample geometric mean of smoothed text and mask scores, making sure neither dimension can completely compensate for the other. That means you have to be good at both to get a high score.

Lu: They define the GROVE score as the mean of these smoothed per-sample scores, which is calculated using that geometric mean of the smoothed text and mask scores. It’s an attempt to enforce joint correctness in a unified way across different model classes.

Meng: When I look at the results they show, even systems that are pretty strong on their own accuracy or their mask localization score still get low GROVE scores. That suggests there's a real gap between just having a right answer and actually knowing where to look for that answer in the visual data.

Tom: They propose a systematic multi-stage pipeline to build this benchmark, which involves sourcing images from different datasets like SA-1B for indoor and NuScenes for autonomous driving scenarios <ref:2605.20676#pg1>. It’s not just one static set of pictures they used; they made sure the construction process itself was robust across those varied settings.

Jane: And to make sure the data collection isn't biased, they used domain-specific strategies for making segmentation masks, sometimes using automated tools like SAM three followed by human annotation to handle what those automated tools miss.

Lu: That’s a practical step because relying only on one method for creating the ground truth masks can introduce its own set of limitations. The paper says they have taken steps to mitigate annotation bias through multi-round human verification and cross-validation by independent annotators, which is important for building trust in the benchmark.

Paper summary: Meng: From a production standpoint, if we were trying to deploy something based on this, I'd be worried about how much effort goes into generating these perfect triplets—the image, the question, and the perfect mask that matches both. It’s a lot of work to create that kind of high-quality ground truth.

Tom: The paper concludes that VISTAQA and GROVE are setting a foundation for future research in grounded multimodal evaluation. They admit their current benchmark is limited by being static; it doesn't cover things like video or multi-image grounding scenarios yet.

Jane: So, what’s the big picture implication here? It seems like the focus has shifted from just seeing if a model can answer a question to making sure that answer is actually tied to verifiable evidence in the image.

Lu: The paper is really pushing for better evaluation rigor for vision-language models. For safety-critical applications, like autonomous driving or robotics, this kind of joint grounding check is what matters for reliability and transparency.

Meng: That makes sense when you're building something that needs to be safe in the real world. It’s not enough to just be smart about language; you have to be accurate about the visual world it describes.

Tom: The core message of this work is that we need a benchmark that forces models to demonstrate alignment between their textual predictions and their pixel-level evidence simultaneously, using metrics like GROVE as a way to check that coupling.

Jane: So, VISTAQA isn't just another test; it’s a tool designed to expose the specific weaknesses in current methods when they fail to link reasoning and grounding together.

Lu: The challenge for future work is moving beyond these static image sets and tackling more complex scenarios, like video or situations where multiple images need to be coordinated for the answer.

Meng: And from an engineering standpoint, we’ll be watching how these joint metrics perform when we start running them on actual, messy production traffic instead of just this curated set of samples.

Tom: So that's what we have with VISTAQA and GROVE—a unified way to demand both textual accuracy and visual grounding fidelity from the models. That’s where the focus is going next.

Conclusion: Tom: So we’re wrapping up our look at VISTAQA, which is this new benchmark designed to test if models are actually connecting what they say with the visual evidence in an image.

Jane: It’s about making sure the answer and the picture match up perfectly, not just checking if the text makes sense on its own.

Tom: Yeah, so VISTAQA is focused on this joint evaluation of correctness and grounding quality in visual question answering.

Lu: The authors put a lot of effort into building this benchmark, with over a thousand expert-curated samples across six different tasks and six visual settings.

Meng: It’s interesting to see how they are forcing models to not just give an answer, but also produce precise pixel-level masks that back up that answer.

Jane: They even included examples where there is no valid visual evidence at all, which tests if the model knows when it should admit it doesn't know something.

Tom: And to measure this joint success, they use a metric called GROVE, which combines the text accuracy and mask quality using a geometric mean.

Lu: The idea behind GROVE is to make sure neither dimension can compensate for weaknesses in the other; you need both to score well.

Jane: So what does this mean when we look at how well these models actually perform on these tasks?

Meng: Even some of the strongest systems we’ve seen struggle with the GROVE score, showing a real gap between getting a right answer and having that evidence align correctly.

Tom: The authors suggest a multi-stage pipeline to build this benchmark, involving specific image sources and hybrid mask generation techniques.

Lu: They’re using data from places like SA one B for indoor scenes and NuScenes for autonomous driving scenarios to make this more realistic.

Jane: So if you’re building a vision model, VISTAQA is trying to show you where the real failures are happening in terms of understanding the world.

Tom: It sets a new standard for how we evaluate vision-language models when safety and reliability matter in things like robotics or self-driving cars.

Jane: The benchmark itself is static right now, meaning it doesn't cover video or complex multi-image scenarios yet.

Lu: That’s the limitation; they’re building a foundation here to see what we need to tackle next with more dynamic visual data.

Mozhgan Nasr Azadani, Yimu Wang, Yongpeng Zhu, Lihong Chen, Milan Ganai, Sean Sedwards, Marco Pavone, º Krzysztof Czarnecki

University of Waterloo · Stanford University · ¤ NVIDIA

cs.CV

Submitted: 2026-05-20

Updated: 2026-11-14

Comments: NeurIPS 2026, ED track

Project page: https://vistaqa.github.io

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 97/100

The gist: Establishing a clear link between model predictions and the visual evidence that supports them is critical for transparency and reliability in multimodal reasoning <ref:2605.20676#pg2> The paper…

Key concepts

VISTAQA
A comprehensive benchmark for jointly evaluating free-form answer correctness and pixel-level evidence grounding in visual question answering. It tests models on six task types across six visual domains, demanding both correct answers and supporting segmentation masks.
GROVE Metric
A unified evaluation metric that enforces joint correctness by combining textual accuracy and grounding quality using a geometric mean. This metric ensures that neither the text score nor the mask score can compensate for deficiencies in the other, penalizing failures in either modality.
Joint Sensitivity
A design principle for GROVE ensuring that failures in either modality (text or mask) result in a penalty. This makes the evaluation sensitive to where models fail, rather than allowing one modality's high score to mask poor performance in the other.
Modality Gap
The observation that models can achieve high scores on text accuracy and mask quality separately, but still receive a low joint GROVE score. This indicates that correct answers and accurate visual evidence often come from different samples, highlighting a fundamental disconnect.

Terminology

Summary

Establishing a clear link between model predictions and the visual evidence that supports them is critical for transparency and reliability in multimodal reasoning <ref:2605. p. 0

Where the paper says it

VISTAQA: Benchmarking Joint Visual Question Answering and Pixel-Level Evidence Mozhgan Nasr Azadani1,2, Yimu Wang1 , Yongpeng Zhu1,* , Lihong Chen1,*, Milan Ganai2 , Sean Sedwards1 , Marco Pavone2,3,† , Krzysztof Czarnecki1,† 1 University of Waterloo, 2 Stanford University, 3 NVIDIA Project Page: https://vistaqa. io Abstract Establishing a clear link between model predictions and the visual evidence that supports them is critical for transparency and reliability in multimodal reasoning, yet current multimodal large language model (MLLM) evaluations do not explicitly enforce this alignment.

20676#pg2> The paper introduces VISTAQA, a comprehensive benchmark for joint evaluation of free-form answer correctness and pixel-level evidence grounding in visual question answering <ref:2605. p. 0
Where the paper says it

Existing benchmarks assess either textual answer correctness or pixel-level localization in isolation, leaving the coupling of reasoning and grounding an open challenge. We introduce VISTAQA, a comprehensive benchmark for joint evaluation of free-form answer correctness and pixel-level evidence grounding in visual question answering.

20676#pg4>.

Pruning redundant reasoning tokens

VISTAQA comprises 1,157 expert-curated samples spanning six task types and six visual domains, ranging from direct perception to compositional and relational reasoning <ref:2605.20676#pg3> It requires models to not only answer correctly, but to also provide precise segmentation masks that support their answers <ref:2605. p. 0

Where the paper says it

VISTAQA comprises 1,157 expert-curated samples spanning six task types and six visual domains, ranging from direct perception to compositional and relational reasoning. VISTAQA requires models to not only answer correctly, but to also provide precise segmentation masks that support their answers.

20676#pg4>. The benchmark includes hallucination-aware examples where no valid visual evidence exists <ref:2605. p. 0
Where the paper says it

VISTAQA requires models to not only answer correctly, but to also provide precise segmentation masks that support their answers. It also includes hallucination-aware examples where no valid visual evidence exists.

20676#pg4>.

How the training signal is built

To support this enhanced evaluation, GROVE, a unified evaluation metric, is introduced which enforces joint correctness by combining textual accuracy and grounding quality via a per-sample geometric mean <ref:2605.20676#pg4>. This metric ensures neither dimension can compensate for deficiencies in the other <ref:2605. p. 0

Where the paper says it

It also includes hallucination-aware examples where no valid visual evidence exists. To support this enhanced evaluation, we introduce GROVE, a unified evaluation metric that enforces joint correctness by combining textual accuracy and grounding quality via a per-sample geometric mean, ensuring neither dimension can compensate for deficiencies in the other.

20676#pg4>. The GROVE score is computed as the mean of the smoothed per-sample scores, which is defined as the geometric mean of the smoothed text and mask scores <ref:2605. p. 6
Where the paper says it

3 and Appendix A. The per-sample score is then defined as the geometric mean of the smoothed scores: S = p S′ a · S′ m (4) The geometric mean enforces joint competence by penalizing imbalance between the two axes [33], ensuring that strong performance on one axis cannot compensate for failure on the other.

20676#pg5>.

Where the gains come from

Comprehensive experiments across grounding-aware models and hybrid pipelines with general-purpose MLLMs reveal that even the strongest systems achieve limited performance under GROVE, highlighting a substantial gap between answer accuracy and visual evidence alignment <ref:2605.20676#pg5>. The authors propose a systematic multi-stage pipeline combining domain-appropriate image sourcing, hybrid mask generation, LLM-assisted QA generation, and multi-round human verification to construct the benchmark <ref:2605. p. 3

Where the paper says it

6), while reasoning segmentation models often fail to produce accurate masks when prompted with complex or domain-specific queries, rendering fully automated construction unreliable. Accordingly, VISTAQA is constructed using a systematic multi-stage pipeline that combines domain-appropriate image sourcing, hybrid mask generation, LLM-assisted QA generation, 4

20676#pg5>.

Benchmark Construction Details

The benchmark covers six task types including identification, attribute, OCR, spatial, counting, and comparison across six visual domains such as indoor and outdoor environments <ref:2605.20676#pg4>. The construction involves sampling images from datasets like SA-1B for indoor/outdoor scenes and NuScenes for autonomous driving scenarios <ref:2605. p. 4

Where the paper says it

For indoor and outdoor scenes, images are sampled from SA-1B [15], which offers high-resolution, real-world scenes with a large number of small and varied objects, making it particularly suitable for evaluating visually grounded reasoning across everyday environments. Autonomous driving images are sampled from NuScenes [2], capturing complex multi-agent scenarios with safety-critical objects and structured road semantics.

20676#pg5>. Segmentation masks are constructed using domain-specific strategies, often involving automated segmentation via SAM3 followed by human annotation to address limitations <ref:2605. p. 4
Where the paper says it

For autonomous driving, robotics, and science domains, masks are obtained through a combination of automated segmentation using SAM3 [3] and human annotation. In practice, SAM3 often fails to reliably isolate specific instances from text prompts and struggles with sub-structure recognition in specialized domains (e.

20676#pg5>.

Evaluation Metric Robustness

GROVE is designed around two key desiderata: (1) joint sensitivity, penalizing failures in either modality, and (2) graceful degradation, preserving signal under partial correctness while remaining applicable across diverse model classes <ref:2605.20676#pg4>. The use of an epsilon-floor smoothing parameter prevents score collapse while preserving sensitivity to partial correctness <ref:2605. p. 5

Where the paper says it

Directly multiplying Sa and Sm yields a degenerate metric: any sample with Sa = 0 collapses to zero regardless of mask quality, potentially discarding meaningful grounding signals. To address this, we apply ϵ-floor smoothing to both scores before computing the joint metric: S ′ a = max(Sa, ϵ) (2) S ′ m = max(Sm, ϵ) (3) where ϵ prevents score collapse while preserving sensitivity to partial correctness.

20676#pg5>. Sensitivity analysis shows that GROVE is robust to the choice of epsilon in the range [0. p. 8
Where the paper says it

1 as this value yields the greatest score discriminability without compromising joint sensitivity. To validate our choice, we performed a sensitivity analysis with values of ϵ ∈ {0.

01, 0.1] <ref:2605. p. 0
Where the paper says it

ca Preprint. arXiv:2605.

20676#pg5>.

Qualitative Findings on Modality Gaps

Qualitative results illustrate a modality gap where models may achieve relatively high text accuracy and mIoU mask scores, but the joint GROVE remains low, indicating that correct answers and accurate evidence often occur on different samples <ref:2605.20676#pg5>. The paper demonstrates that failures in both answer and mask quality lead to the lowest GROVE scores, which do not collapse to zero due to epsilon-flooring <ref:2605. p. 5

Where the paper says it

Directly multiplying Sa and Sm yields a degenerate metric: any sample with Sa = 0 collapses to zero regardless of mask quality, potentially discarding meaningful grounding signals. To address this, we apply ϵ-floor smoothing to both scores before computing the joint metric: S ′ a = max(Sa, ϵ) (2) S ′ m = max(Sm, ϵ) (3) where ϵ prevents score collapse while preserving sensitivity to partial correctness.

20676#pg5>. The benchmark reveals that spatial reasoning and OCR emerge as the most challenging tasks for all model families <ref:2605. p. 7
Where the paper says it

Crucially, a modality gap persists: while models may achieve relatively high text accuracy and mIoU mask scores, the joint GROVE remains low, indicating that correct answers and accurate evidence often occur on different samples. Across tasks, spatial reasoning and OCR emerge as the most challenging, reflecting their higher compositional and grounding demands.

20676#pg5>.

Conclusion and Limitations

The authors conclude that VISTAQA and GROVE serve as a foundation for future research in grounded multimodal evaluation <ref:2605.20676#pg5>. Limitations include the static nature of VISTAQA, which does not cover video or multi-image grounding scenarios <ref:2605. p. 8

Where the paper says it

Limitations. Due to the static nature of VISTAQA, video and multi-image grounding scenarios are not covered.

20676#pg5>. The benchmark is limited to 1,157 high-quality samples due to the inherent challenge of constructing triplets that satisfy strict standards across all criteria <ref:2605. p. 8
Where the paper says it

Generating high-quality samples that simultaneously satisfy answer correctness, mask fidelity, and task diversity is inherently challenging and requires rigorous manual control at every stage of the pipeline. As a result, VISTAQA is limited to 1,157 high-quality samples despite significant annotation effort.

20676#pg5>.

Societal Impacts

VISTAQA is designed to improve the evaluation rigor of vision-language models, with direct benefits for transparency and reliability in safety-critical applications such as autonomous driving and robotics <ref:2605.20676#pg5>. The benchmark is constructed using images from publicly available datasets and contains no personally identifiable information <ref:2605. p. 9

Where the paper says it

VISTAQA is designed to improve the evaluation rigor of vision-language models, with direct benefits for transparency and reliability in safety-critical applications such as autonomous driving and robotics. The benchmark is constructed using images from publicly available datasets and contains no personally identifiable information.

20676#pg5>. The authors have taken steps to mitigate annotation bias through multi-round human verification and cross-validation by independent annotators <ref:2605. p. 9
Where the paper says it

The benchmark is constructed using images from publicly available datasets and contains no personally identifiable information. We have taken steps to mitigate annotation bias through multi-round human verification and cross-validation by independent annotators.

20676#pg5>.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Introduction of VISTAQA as a benchmark for joint evaluation of free-form answer correctness and pixel-level evidence grounding, requiring models to not only answer correctly, but to also provide precise segmentation masks that support their answers. This allows for the assessment of the coupling of reasoning and grounding which is currently an open challenge.

  2. Introduction of GROVE as a unified evaluation metric that enforces joint correctness by combining textual accuracy and grounding quality via a per-sample geometric mean, ensuring neither dimension can compensate for deficiencies in the other. This metric addresses the limitation where textual correctness alone cannot determine whether a reply is truly grounded in the image.

  3. Implementation of GROVE's smoothing mechanism with epsilon flooring, defined as S'a = max(Sa, ϵ) (2) and S'm = max(Sm, ϵ) (3), to prevent score collapse while preserving sensitivity to partial correctness. This ensures that no compensation across modalities occurs when evaluating model performance.

  4. Development of a rigorous multi-stage human verification pipeline for benchmark construction, combining domain-appropriate image sourcing, hybrid mask generation, LLM-assisted QA generation, and multi-round human verification, to ensure samples satisfy strict standards of visual quality, answer correctness, and grounding fidelity.

  5. Creation of hallucination-aware samples comprising 314 cases (27.1%) specifically designed to probe robustness against visually misleading or unanswerable queries, forcing models to correctly identify the absence of valid visual evidence.

Abstract

Establishing a clear link between model predictions and the visual evidence that supports them is critical for transparency and reliability in multimodal reasoning, yet current multimodal large language model (MLLM) evaluations do not explicitly enforce this alignment. Existing benchmarks assess either textual answer correctness or pixel-level localization in isolation, leaving the coupling of reasoning and grounding an open challenge. We introduce VISTAQA, a comprehensive benchmark for joint evaluation of free-form answer correctness and pixel-level evidence grounding in visual question answering. VISTAQA comprises 1,157 expert-curated samples spanning six task types and six visual domains, ranging from direct perception to compositional and relational reasoning. VISTAQA requires models to not only answer correctly, but to also provide precise segmentation masks that support their answers. It also includes hallucination-aware examples where no valid visual evidence exists. To support this enhanced evaluation, we introduce GROVE, a unified evaluation metric that enforces joint correctness by combining textual accuracy and grounding quality via a per-sample geometric mean, ensuring neither dimension can compensate for deficiencies in the other. Comprehensive experiments across grounding-aware models and hybrid pipelines with general-purpose MLLMs reveal that even the strongest systems achieve limited performance under GROVE, highlighting a substantial gap between answer accuracy and visual evidence alignment. Project page can be found at https://vistaqa.github.io.

Sources

Related papers