Breaking the Visual Shortcuts in Multimodal Knowledge-Based Visual Question Answering

arXiv:2511.22843 · cs.CV · Submitted 2025-11-28 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "Breaking the Visual Shortcuts in Multimodal Knowledge-Based Visual Question Answering".

Title and authors: Tom: Let's look at the title and who wrote this paper, "Breaking the Visual Shortcuts in Multimodal Knowledge-Based Visual Question Answering." The authors are Dosung Lee, Sangwon Jung, Boyoung Kim, Minyoung Kim, Sungyeon Kim, and Junyoung Sung.

Jane: It’s interesting that they focused on this specific problem because it shows how much existing models rely on those shortcuts just to get comparable results using visual cues one <ref:2511.22843#pg1,comparable results using visual cues>.

Lu: The authors are proposing a solution called the Relational Entity Text-Image Knowledge Augmented, or RETINA benchmark, which is built using an LLM-driven pipeline with one hundred twenty thousand training samples and two thousand human-curated test samples one <ref:2511.22843#pg1,Relational Entity Text-Image Knowledge Augmented>.

Meng: So what does this actually mean for us in terms of how we train models? It sounds like they are building a way to stop the model from just looking at the picture and start looking at the relationships between things.

Tom: Exactly. They show that when you evaluate existing models on RETINA, their performance drops significantly because they confirm that these models were indeed relying heavily on those visual shortcuts one <ref:2511.22843#pg1>.

Jane: The goal is to create a benchmark that better reflects real-world scenarios where queries often reference secondary subjects, not just the main entity one <ref:2511.22843#pg1>.

Lu: They are constructing this benchmark through an automated pipeline that first identifies entities related to each document’s main entity, then builds a structured knowledge graph, and finally extracts subgraphs for query generation one <ref:2511.22843#pg1>.

Meng: Building a knowledge graph automatically based on the document content sounds complex to set up, but if it works, it should force the model to use more context than just looking at one picture.

Tom: I'm excited about that pipeline part because they mention avoiding prompt-based approaches that feed full documents into LLMs because those often lead to hallucinations one <ref:2511.22843#pg1>.

The paper's summary: Jane: So, if we summarize the main point of "Breaking the Visual Shortcuts in Multimodal Knowledge-Based Visual Question Answering," it’s that models can exploit visual shortcuts to solve these tasks without really understanding the underlying knowledge.

Tom: Right. Existing benchmarks let models get away with just matching a picture to a subject, and this gives them performance comparable to models that actually understand the text one <ref:2511.22843#pg1>.

Lu: The paper summarizes how they address this by introducing RETINA, which is a benchmark where queries reference secondary subjects and are paired with images of those related entities one <ref:2511.22843#pg1>.

Meng: They explain that the system for building RETINA involves four steps: identifying related entities, constructing a knowledge graph, extracting subgraphs for query generation, and then generating queries guided by those subgraphs one <ref:2511.22843#pg1>.

Jane: They emphasize that instead of using prompt-based methods where you feed the whole document to an LLM hoping it finds the right relationship, they use this graph-guided generation process one <ref:2511.22843#pg1>.

Tom: That’s smart because they note that the graph-guided approach allows for controlled query construction by extracting subgraphs corresponding to specific related entities one <ref:2511.22843#pg1>.

Lu: They specifically build one-hop neighborhood graphs by extracting named entities related to the main entity and their relations using an LLM one <ref:2511.22843#pg1>.

Meng: So, what’s the immediate implication for a developer? If you want a model that doesn't rely on visual shortcuts, you should be looking at training it on this RETINA data instead of just standard document images one <ref:2511.22843#pg1>.

Jane: It really shifts the focus from just getting a high score on one image to actually understanding the relational knowledge between different things in the text and image one <ref:2511.22843#pg1>.

The paper's improvements: Tom: Now let's look at what they are proposing as improvements. They suggest two main things: first, building RETINA itself, and second, proposing MultiImage Multimodal Retriever, or MIMIR.

Jane: Building RETINA is the benchmark improvement; it’s an automated system that generates one hundred twenty thousand training samples and a two thousand human-verified test set one <ref:2511.22843#pg1>.

Lu: The pipeline for building RETINA involves knowledge graph construction, target subgraph extraction, and LLM-based VQA generation one, which removes those visual shortcuts we talked about earlier <ref:2511.22843#pg1>.

Meng: But they also propose MIMIR as a retriever. That’s an improvement to how you actually retrieve documents, moving beyond the single-image framework of prior work one <ref:2511.22843#pg1>.

Tom: MIMIR enriches document embeddings by integrating images of multiple related entities, which is supposed to help handle those queries referencing secondary subjects one <ref:2511.22843#pg1>.

Jane: The key difference here is that MIMIR extracts related entities from the text and augments multiple images of those entities to guide the retrieval of documents that contain them one <ref:2511.22843#pg1>.

Lu: The document encoder in MIMIR builds on MuKA by incorporating these multiple images into each document, which extends its representation capacity one <ref:2511.22843#pg1>.

Meng: From a deployment standpoint, I have to ask about the cost. If you're embedding multiple images per document, the similarity computations during retrieval might get slower compared to just using one image one <ref:2511.22843#pg1>.

Tom: That latency is a trade-off they acknowledge, but they show that MIMIR achieves nearly twice the recall of the best baselines and substantially improves answer quality one <ref:2511.22843#pg1>.

Conclusion: Jane: So, to wrap up the paper "Breaking the Visual Shortcuts in Multimodal Knowledge-Based Visual Question Answering," they argue that existing benchmarks are flawed because they allow models to exploit visual shortcuts when answering questions about related entities.

Tom: They introduce RETINA as a way to fix this by using an automated, LLM-driven pipeline to generate queries referencing secondary subjects and pair them with the correct related images one <ref:2511.22843#pg1>.

Lu: The authors suggest that the MIMIR retriever is effective because it enriches document representations by integrating images of multiple related entities, which mitigates the inductive bias toward visual shortcuts that MuKA uses one <ref:2511.22843#pg1>.

Meng: From my side, what I see is that they've created a very specific test for models that actually need relational knowledge rather than just pattern matching on an image one <ref:2511.22843#pg1>.

Lalam: I'm pretty sure this direction is the right one because incorporating diverse entity representations for each document helps guide the retrieval of documents containing those entities one <ref:2511.22843#pg1>.

Jane: In short, they propose RETINA as a novel benchmark without visual shortcuts and MIMIR as a retriever that augments multiple images of related entities to the document embedding one <ref:2511.22843#pg1>.

Tom: It’s a solid direction for pushing multimodal question answering forward by forcing models to handle the complexity of relationships one <ref:2511.22843#pg1>.

Dosung Lee, Sangwon Jung, Boyoung Kim, Minyoung Kim, Sungyeon Kim, *Junyoung Sung, Paul Hongsuck Seo

Korea University · KAIST · Amazon

cs.CV

Submitted: 2025-11-28

Updated: 2026-11-16

Comments: Accepted to findings of EACL 2027

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 93/100

The gist: Existing Multimodal Knowledge-Based Visual Question Answering (MKB-VQA) benchmarks suffer from “visual shortcuts”…

Key concepts

RETINA Benchmark
A new benchmark designed to eliminate visual shortcuts in VQA. It automatically creates test sets by linking queries about secondary subjects with images of those related entities, forcing models to rely on knowledge rather than just matching primary image content.
MIMIR Retriever
A retrieval system that improves document embeddings by augmenting them with images of multiple related entities from the text. This extends prior single-image methods, allowing the model to find relevant documents even when queries reference secondary subjects.
Visual Shortcut
A flaw in current VQA models where they exploit the fact that a query image often matches the main subject of a document. This allows models to get high scores using visual cues alone, ignoring deeper knowledge required for accurate answers.
Graph-Guided Query Generation
A method used to create questions by first building a structured knowledge graph based on an LLM's analysis of related entities. This ensures queries are generated based on specific, relevant connections rather than broad prompts, reducing errors.

Terminology

Summary

Existing Multimodal Knowledge-Based Visual Question Answering (MKB-VQA) benchmarks suffer from “visual shortcuts”, as the query image typically matches the primary subject entity of the target document. p. 0

Where the paper says it

kr, sunykim@amazon. com Abstract Existing Multimodal Knowledge-Based Visual Question Answering (MKB-VQA) benchmarks suffer from “visual shortcuts”, as the query image typically matches the primary subject entity of the target document.

Models can exploit these shortcuts, achieving comparable results using visual cues alone. p. 0
Where the paper says it

com Abstract Existing Multimodal Knowledge-Based Visual Question Answering (MKB-VQA) benchmarks suffer from “visual shortcuts”, as the query image typically matches the primary subject entity of the target document. We demonstrate that models can exploit these shortcuts, achieving comparable results using visual cues alone.

To address this, Relational Entity Text-Image kNowledge Augmented (RETINA) benchmark is introduced to eliminate such shortcuts by pairing queries referencing secondary subjects with images of related entities. p. 0
Where the paper says it

To address this, we introduce Relational Entity Text-Image kNowledge Augmented (RETINA) benchmark, automatically constructed using an LLM-driven pipeline, consisting of 120k training and 2k human-curated test set. RETINA contains queries referencing secondary subjects (i.

Furthermore, MultiImage MultImodal Retriever (MIMIR) is proposed to enrich document embeddings by augmenting images of multiple related entities, effectively handling RETINA. p. 0
Where the paper says it

When evaluated on RETINA existing models show significantly degraded performance, confirming their reliance on the shortcut. Furthermore, we propose MultiImage MultImodal Retriever (MIMIR), which enriches document embeddings by augmenting images of multiple related entities, effectively handling RETINA, unlike prior work that uses only a single image per document.

Pruning redundant reasoning tokens

The paper introduces the RETINA benchmark, which is automatically constructed using an LLM-driven pipeline consisting of 120k training and 2k human-curated test set. This benchmark contains queries referencing secondary subjects and pairs them with images of these related entities, removing the visual shortcut. p. 0

Where the paper says it

RETINA contains queries referencing secondary subjects (i. related entities) and pairs them with images of these related entities, removing the visual shortcut.

When evaluated on RETINA existing models show significantly degraded performance, confirming their reliance on the shortcut. p. 0
Where the paper says it

related entities) and pairs them with images of these related entities, removing the visual shortcut. When evaluated on RETINA existing models show significantly degraded performance, confirming their reliance on the shortcut.

How the training signal is built

The construction of RETINA involves an automated pipeline that (1) identifies entities related to each document’s main entity (i.e. related entity), (2) constructs a structured knowledge graph, (3) extracts subgraphs for query generation, and (4) generates queries guided by those subgraphs. p. 1

Where the paper says it

RETINA is built through an automated, LLM-driven pipeline that (1) identifies entities related to each document’s main entity (i. related entity), (2) constructs a structured knowledge graph, (3) extracts subgraphs for query generation, and (4) generates queries guided by those subgraphs.

This strategy employs a graph-guided generation process instead of prompt-based approaches that feed full documents to LLMs to avoid hallucinated or incorrect entity references. p. 1
Where the paper says it

As prompt-based approaches that feed full documents to LLMs often produce hallucinated or incorrect entity references, we instead employ a graph-guided generation process [1, 30, 36, 37]. This strategy allows controlled query construction by extracting relevant subgraphs corresponding to specific related entities.

The pipeline constructs one-hop neighborhood graphs by extracting named entities related to the main entity and their relations using an LLM. p. 2
Where the paper says it

RETINA Benchmark Generation Pipeline. (a) constructs one-hop neighborhood graphs by extracting named entities (gray box) related to the answer entity (white box) and their relations using an LLM [2]; (b) samples an query entity (green) and qualifying entity (teal) to form a target subgraph with the answer entity (red); and (c) feeds the target subgraph into an LLM to generate a textual query and collect a corresponding image from M2KR Images [19].

Where the gains come from

The proposed MIMIR retriever enriches document representations by integrating images of multiple related entities, extending the single-image framework of prior work. MIMIR extracts related entities from document text and augments multiple images of these related entities to guide the retrieval of corresponding documents that contain the related entities. p. 1

Where the paper says it

To overcome this limitation, we propose Multi-Image MultImodal Retriever (MIMIR) that enriches document representations by integrating images of multiple related entities, extending the single-image framework of prior work [6]. Specifically, MIMIR extracts related entities from document text, augments multiple images of the related entities, which guides the retrieval of the corresponding documents that contain the related entities, thereby enabling retrieval in the absence of visual shortcuts.

The document encoder in MIMIR builds upon MuKA by incorporating multiple images of related entities into each document, extending its representation capacity. p. 3
Where the paper says it

To address this issue, we introduce MIMIR, a multimodal retriever that enriches document representations by incorporating images of multiple related entities. Our approach builds upon MuKA [6], a multimodal single-image document retriever, and extends its representation capacity.

Performance validation and analysis

Experiments show that existing models exhibit a substantial performance drop on RETINA, reflecting their design that typically augments each document with a single image of its main entity. MIMIR achieves nearly twice the recall of the best-performing baselines and substantially improves answer quality. p. 5

Where the paper says it

These results underscore the value of RETINA and validate the challenges it poses. In contrast, our proposed MIMIR achieves nearly twice the recall of the best-performing baselines and substantially improves answer quality, demonstrating the effectiveness of our document embeddings augmented with multiple images of re-

Analysis reveals that MuKA’s single-image augmentation strategy induces an incorrect inductive bias favoring visual shortcuts, whereas MIMIR mitigates this bias by incorporating diverse entity representations for each document. p. 6
Where the paper says it

This behavior stems from MuKA’s single-image augmentation strategy, which induces an incorrect inductive bias favoring visual shortcuts. In contrast, our MIMIR, through multi-image augmentation, mitigates this bias by incorporating diverse entity representations for each document.

Qualitative analysis demonstrates that MIMIR successfully retrieves the GT document because it is augmented with the image of a mandarin duck—previously one of the most challenging negative samples for the baseline. p. 7
Where the paper says it

For example, Fig. 7d shows that the GT document for the wood duck is successfully retrieved because the document was augmented with the image of a mandarin duck—previously one of the most challenging negative samples for the baseline.

Limitations and future direction

MIMIR incorporates additional image features into document embeddings, which increases similarity computations during retrieval, leading to higher inference latency compared to using embeddings with a single image. The authors conclude by proposing RETINA as a novel benchmark reflecting challenging subsets of real-world scenarios without visual shortcuts and MIMIR as a retriever that augments multiple images of related entities to the document embedding. p. 7

Where the paper says it

This success is likely facilitated by structure of the Wikipedia, as related entities within it often depict visually similar entities. Conclusion We propose a novel MKB-VQA benchmark, RETINA, which reflects a challenging subset of real-world scenarios, without visual shortcuts.

Looking ahead, extending this approach to incorporate diverse modalities beyond images presents a promising direction. p. 7
Where the paper says it

To address this, we introduce MIMIR, a retriever that augments multiple images of related entities to the document embedding, effectively solving RETINA. Looking ahead, real-world scenarios involve diverse modalities of related context beyond images, and extending our approach to incorporate these modalities presents a promising direction.

Conclusion

We propose a novel MKB-VQA benchmark, RETINA, which reflects a challenging subset of real-world scenarios without visual shortcuts. We show that previous MKB-VQA benchmarks allow models to exploit visual shortcuts, limiting their practical utility. p. 7

Where the paper says it

Conclusion We propose a novel MKB-VQA benchmark, RETINA, which reflects a challenging subset of real-world scenarios, without visual shortcuts. We show that previous MKB-VQA benchmarks allow models to exploit visual shortcuts, limiting their practical utility.

To address this, we introduce MIMIR, a retriever that augments multiple images of related entities to the document embedding, effectively solving RETINA. p. 7
Where the paper says it

We show that previous MKB-VQA benchmarks allow models to exploit visual shortcuts, limiting their practical utility. To address this, we introduce MIMIR, a retriever that augments multiple images of related entities to the document embedding, effectively solving RETINA.

Looking ahead, real-world scenarios involve diverse modalities of related context beyond images and extending our approach to incorporate these modalities presents a promising direction. p. 7
Where the paper says it

To address this, we introduce MIMIR, a retriever that augments multiple images of related entities to the document embedding, effectively solving RETINA. Looking ahead, real-world scenarios involve diverse modalities of related context beyond images, and extending our approach to incorporate these modalities presents a promising direction.

References

[1] Joshua Ainslie, James Lee-Thorp, Michiel De Jong, Yury Zemlyanskiy, Federico Lebron, and Sumit Sanghai. Gqa: Training generalized multi-query transformer models from multi-head checkpoints. p. 8

Where the paper says it

References [1] Joshua Ainslie, James Lee-Thorp, Michiel De Jong, Yury Zemlyanskiy, Federico Lebron, and Sumit Sanghai. Gqa: ´ Training generalized multi-query transformer models from multi-head checkpoints.

p. 8
Where the paper says it

References [1] Joshua Ainslie, James Lee-Thorp, Michiel De Jong, Yury Zemlyanskiy, Federico Lebron, and Sumit Sanghai. Gqa: ´ Training generalized multi-query transformer models from multi-head checkpoints.

arXiv preprint arXiv:2305. p. 8
Where the paper says it

Gqa: ´ Training generalized multi-query transformer models from multi-head checkpoints. arXiv preprint arXiv:2305.

p. 8
Where the paper says it

Gqa: ´ Training generalized multi-query transformer models from multi-head checkpoints. arXiv preprint arXiv:2305.

13245, 2023

[6] Lianghao Deng, Yuchong Sun, Shizhe Chen, Ning Yang, Yunfeng Wang, and Ruihua Song. MuKA: Multimodal knowledge augmented visual information-seeking. p. 8

Where the paper says it

1, 2, 6 [6] Lianghao Deng, Yuchong Sun, Shizhe Chen, Ning Yang, Yunfeng Wang, and Ruihua Song. MuKA: Multimodal knowledge augmented visual information-seeking.

p. 8
Where the paper says it

1, 2, 6 [6] Lianghao Deng, Yuchong Sun, Shizhe Chen, Ning Yang, Yunfeng Wang, and Ruihua Song. MuKA: Multimodal knowledge augmented visual information-seeking.

p. 8
Where the paper says it

1, 2, 6 [6] Lianghao Deng, Yuchong Sun, Shizhe Chen, Ning Yang, Yunfeng Wang, and Ruihua Song. MuKA: Multimodal knowledge augmented visual information-seeking.

p. 8
Where the paper says it

1, 2, 6 [6] Lianghao Deng, Yuchong Sun, Shizhe Chen, Ning Yang, Yunfeng Wang, and Ruihua Song. MuKA: Multimodal knowledge augmented visual information-seeking.

In Proceedings of the 31st International Conference on Computational Linguistics, pages 9675–9686, Abu Dhabi, UAE, 2025

[2] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang et al. Qwen technical report. p. 8

Where the paper says it

2 [2] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report.

p. 8
Where the paper says it

2 [2] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report.

arXiv preprint arXiv:2309. p. 8
Where the paper says it

Qwen technical report. arXiv preprint arXiv:2309.

p. 8
Where the paper says it

Qwen technical report. arXiv preprint arXiv:2309.

16609

[3] Jannis Bulian, Christian Buck, Wojciech Gajewski, Benjamin Borschinger and Tal Schuster. Tomayto, tomahto beyond token-level answer equivalence for question answering evaluation. p. 8

Where the paper says it

Tomayto, tomahto. ¨ beyond token-level answer equivalence for question answering evaluation.

p. 8
Where the paper says it

Tomayto, tomahto. ¨ beyond token-level answer equivalence for question answering evaluation.

In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 291–305, Abu Dhabi, United Arab Emirates, 2022

[4] Davide Caffagni, Sara Sarto, Marcella Cornia, Lorenzo Baraldi and Rita Cucchiara. Recurrence-enhanced visionand-language transformers for robust multimodal document retrieval. p. 8

Where the paper says it

6 [4] Davide Caffagni, Sara Sarto, Marcella Cornia, Lorenzo Baraldi, and Rita Cucchiara. Recurrence-enhanced visionand-language transformers for robust multimodal document retrieval.

p. 8
Where the paper says it

6 [4] Davide Caffagni, Sara Sarto, Marcella Cornia, Lorenzo Baraldi, and Rita Cucchiara. Recurrence-enhanced visionand-language transformers for robust multimodal document retrieval.

In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 9286–9295, 2025

[5] Yang Chen, Hexiang Hu, Yi Luan, Haitian Sun, Soravit Changpinyo, Alan Ritter and Ming-Wei Chang. Can pre-trained vision and language models answer visual information-seeking questions? p. 8

Where the paper says it

1, 2, 4, 6, 7 [5] Yang Chen, Hexiang Hu, Yi Luan, Haitian Sun, Soravit Changpinyo, Alan Ritter, and Ming-Wei Chang. Can pre-trained vision and language models answer visual information-seeking questions?

p. 8
Where the paper says it

1, 2, 4, 6, 7 [5] Yang Chen, Hexiang Hu, Yi Luan, Haitian Sun, Soravit Changpinyo, Alan Ritter, and Ming-Wei Chang. Can pre-trained vision and language models answer visual information-seeking questions?

In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 14948–14968, 2023

[7] Feng Gao, Qing Ping, Govind Thattai, Aishwarya Reganti, Ying Nian Wu and Prem Natarajan. Transform-retrievegenerate: Natural language-centric outside-knowledge visual question answering. p. 8

Where the paper says it

1, 2, 4, 5, 6, 7, 8, 3 [7] Feng Gao, Qing Ping, Govind Thattai, Aishwarya Reganti, Ying Nian Wu, and Prem Natarajan. Transform-retrievegenerate: Natural language-centric outside-knowledge visual question answering.

p. 8
Where the paper says it

1, 2, 4, 5, 6, 7, 8, 3 [7] Feng Gao, Qing Ping, Govind Thattai, Aishwarya Reganti, Ying Nian Wu, and Prem Natarajan. Transform-retrievegenerate: Natural language-centric outside-knowledge visual question answering.

In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5067–5077, 2022

[19] Weizhe Lin, Jingbiao Mei, Jinghong Chen and Bill Byrne. Preflmr: Scaling up fine-grained late-interaction multimodal retrievers. p. 8

Where the paper says it

1, 2, 4 [19] Weizhe Lin, Jingbiao Mei, Jinghong Chen, and Bill Byrne. Preflmr: Scaling up fine-grained late-interaction multimodal retrievers.

p. 8
Where the paper says it

1, 2, 4 [19] Weizhe Lin, Jingbiao Mei, Jinghong Chen, and Bill Byrne. Preflmr: Scaling up fine-grained late-interaction multimodal retrievers.

p. 8
Where the paper says it

1, 2, 4 [19] Weizhe Lin, Jingbiao Mei, Jinghong Chen, and Bill Byrne. Preflmr: Scaling up fine-grained late-interaction multimodal retrievers.

arXiv preprint arXiv:2402. p. 8
Where the paper says it

Preflmr: Scaling up fine-grained late-interaction multimodal retrievers. arXiv preprint arXiv:2402.

p. 8
Where the paper says it

Preflmr: Scaling up fine-grained late-interaction multimodal retrievers. arXiv preprint arXiv:2402.

p. 8
Where the paper says it

Preflmr: Scaling up fine-grained late-interaction multimodal retrievers. arXiv preprint arXiv:2402.

08327, 2024

[33] Keshav Santhanam, Omar Khattab, Christopher Potts and Matei Zaharia. Plaid: An efficient engine for late interaction retrieval, 2022

[40] Cong Wei, Yang Chen, Haonan Chen, Hexiang Hu et al. Uniir: Training and benchmarking universal multimodal information retrievers. p. 9

Where the paper says it

2 [40] Cong Wei, Yang Chen, Haonan Chen, Hexiang Hu, Ge Zhang, Jie Fu, Alan Ritter, and Wenhu Chen. Uniir: Training and benchmarking universal multimodal information retrievers.

p. 9
Where the paper says it

2 [40] Cong Wei, Yang Chen, Haonan Chen, Hexiang Hu, Ge Zhang, Jie Fu, Alan Ritter, and Wenhu Chen. Uniir: Training and benchmarking universal multimodal information retrievers.

p. 9
Where the paper says it

2 [40] Cong Wei, Yang Chen, Haonan Chen, Hexiang Hu, Ge Zhang, Jie Fu, Alan Ritter, and Wenhu Chen. Uniir: Training and benchmarking universal multimodal information retrievers.

p. 9
Where the paper says it

2 [40] Cong Wei, Yang Chen, Haonan Chen, Hexiang Hu, Ge Zhang, Jie Fu, Alan Ritter, and Wenhu Chen. Uniir: Training and benchmarking universal multimodal information retrievers.

In European Conference on Computer Vision, pages 387–404, 2024

[15] Ji Lin, Hongxu Yin, Wei Ping, Yao Lu, Pavlo Molchanov et al. Vila: On pre-training for visual language models, 2024

[38] Ashish Vaswani, Noam Shazeer, Niki Parmar et al. Attention is all you need, 2023

[41] Tobias Weyand, Andre Araujo, Bingyi Cao and Jack Sim. Google landmarks dataset v2 – a large-scale benchmark for instance-level recognition and retrieval, 2020

[31] Stephen E Robertson, Steve Walker, Susan Jones et al. Okapi at trec-3. p. 9

Where the paper says it

2 [31] Stephen E Robertson, Steve Walker, Susan Jones, Micheline M Hancock-Beaulieu, Mike Gatford, et al. Okapi at trec-3.

p. 9
Where the paper says it

2 [31] Stephen E Robertson, Steve Walker, Susan Jones, Micheline M Hancock-Beaulieu, Mike Gatford, et al. Okapi at trec-3.

Nist Special Publication Sp, 109:109, 1995

[6] Lianghao Deng, Yuchong Sun, Shizhe Chen et al. MuKA: Multimodal knowledge augmented visual information-seeking. In Proceedings of the 31st International Conference on Computational Linguistics, pages 9675–9686, Abu Dhabi, UAE, 2025

[23] Yikun Liu, Pingan Chen, Jiayin Cai et al. Lamra: Large multimodal model as your advanced retrieval assistant. p. 8

Where the paper says it

3 [23] Yikun Liu, Pingan Chen, Jiayin Cai, Xiaolong Jiang, Yao Hu, Jiangchao Yao, Yanfeng Wang, and Weidi Xie. Lamra: Large multimodal model as your advanced retrieval assistant, 2024.

In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 4015–4025, 2025

[36] Duong T Tran, Trung-Kien Tran, Manfred Hauswirth and Danh Le Phuoc. Reasonvqa: A multi-hop reasoning benchmark with structural knowledge for visual question answering. p. 9

Where the paper says it

1, 2 [36] Duong T Tran, Trung-Kien Tran, Manfred Hauswirth, and Danh Le Phuoc. Reasonvqa: A multi-hop reasoning benchmark with structural knowledge for visual question answering.

p. 9
Where the paper says it

1, 2 [36] Duong T Tran, Trung-Kien Tran, Manfred Hauswirth, and Danh Le Phuoc. Reasonvqa: A multi-hop reasoning benchmark with structural knowledge for visual question answering.

arXiv preprint arXiv:2507. p. 9
Where the paper says it

Reasonvqa: A multi-hop reasoning benchmark with structural knowledge for visual question answering. arXiv preprint arXiv:2507.

p. 9
Where the paper says it

Reasonvqa: A multi-hop reasoning benchmark with structural knowledge for visual question answering. arXiv preprint arXiv:2507.

16403, 2025

[37] Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot and Ashish Sabharwal. Musique: Multihop questions via singlehop question composition, 2022

[18] Weizhe Lin, Jinghong Chen, Jingbiao Mei Alexandru Coca and Bill Byrne. Fine-grained late-interaction multi-modal retrieval for retrieval augmented visual question answering, 2023

[40] Cong Wei, Yang Chen, Haonan Chen et al. Uniir: Training and benchmarking universal multimodal information retrievers. In European Conference on Computer Vision, pages 387–404, 2024

[16] Sheng-Chieh Lin, Chankyu Lee, Mohammad Shoeybi, Jimmy Lin Bryan Catanzaro and Wei Ping. Mm-embed: Universal multimodal retrieval with multimodal llms, 2025

[39] Peng Wang, Qi Wu, Chunhua Shen et al. Fvqa: Fact-based visual question answering. p. 9

Where the paper says it

4 [39] Peng Wang, Qi Wu, Chunhua Shen, Anthony Dick, and Anton Van Den Hengel. Fvqa: Fact-based visual question answering.

p. 9
Where the paper says it

4 [39] Peng Wang, Qi Wu, Chunhua Shen, Anthony Dick, and Anton Van Den Hengel. Fvqa: Fact-based visual question answering.

IEEE transactions on pattern analysis and machine intelligence, 40(10):2413–2427, 2017

[6] Lianghao Deng, Yuchong Sun, Shizhe Chen et al. MuKA: Multimodal knowledge augmented visual information-seeking. In Proceedings of the 31st International Conference on Computational Linguistics, pages 9675–9686, Abu Dhabi, UAE, 2025

[40] Cong Wei, Yang Chen, Haonan Chen et al. Uniir: Training and benchmarking universal multimodal information retrievers. In European Conference on Computer Vision, pages 387–404, 2024

[33] Keshav Santhanam, Omar Khattab et al. Plaid: An efficient engine for late interaction retrieval, 2022

[19] Weizhe Lin, Jingbiao Mei Jinghong Chen and Bill Byrne. Preflmr: Scaling up fine-grained late-interaction multimodal retrievers. arXiv preprint arXiv:2402.08327, 2024

[35] Dustin Schwenk, Apoorv Khandelwal, Christopher Clark Kenneth Marino and Roozbeh Mottaghi. A-okvqa: A benchmark for visual question answering using world knowledge, 2022

[34] Keshav Santhanam, Omar Khattab Jon Saad-Falcon Christopher Potts and Matei Zaharia. Colbertv2: Effective and efficient retrieval via lightweight late interaction, 2022

[11] Aman Jain, Mayank Kothyari, Vishwajeet Kumar, Preethi Jyothi Ganesh Ramakrishnan and Soumen Chakrabarti. Select, substitute, search: A new benchmark for knowledgeaugmented visual question answering. p. 8

Where the paper says it

2 [11] Aman Jain, Mayank Kothyari, Vishwajeet Kumar, Preethi Jyothi, Ganesh Ramakrishnan, and Soumen Chakrabarti. Select, substitute, search: A new benchmark for knowledgeaugmented visual question answering.

In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2491–2498, 2021

[30] Sathish Reddy, Dinesh Raghu, Mitesh M. Khapra and Sachindra Joshi. p. 9

Where the paper says it

2, 4 [30] Sathish Reddy, Dinesh Raghu, Mitesh M. Khapra, and Sachindra Joshi.

Generating natural language questionanswer pairs from a knowledge graph using a RNN based question generation model. p. 9
Where the paper says it

Khapra, and Sachindra Joshi. Generating natural language questionanswer pairs from a knowledge graph using a RNN based question generation model.

In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 376–385, Valencia, Spain, 2017

[22] Nelson F. Liu, Kevin Lin, John Hewitt Ashwin Paranjape Michele Bevilacqua Fabio Petroni and Percy Liang. p. 8

Where the paper says it

6 [22] Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang.

Lost in the middle: How language models use long contexts, 2023

[1] Joshua Ainslie, James Lee-Thorp, Michiel De Jong, Yury Zemlyanskiy, Federico Lebron and Sumit Sanghai. Gqa: Training generalized multi-query transformer models from multi-head checkpoints. arXiv preprint arXiv:2305.13245

[6] Lianghao Deng, Yuchong Sun, Shizhe Chen, Ning Yang, Yunfeng Wang and Ruihua Song. MuKA: Multimodal knowledge augmented visual information-seeking. In Proceedings of the 31st International Conference on Computational Linguistics, pages 9675–9686, Abu Dhabi UAE, 2025

[2] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui et al. Qwen technical report. arXiv preprint arXiv:2309.16609

[3] Jannis Bulian, Christian Buck, Wojciech Gajewski Benjamin Borschinger and Tal Schuster. Tomayto, tomahto beyond token-level answer equivalence for question answering evaluation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 291–305, Abu Dhabi, United Arab Emirates, 2022

[4] Davide Caffagni, Sara Sarto Marcella Cornia Lorenzo Baraldi and Rita Cucchiara. Recurrence-enhanced visionand-language transformers for robust multimodal document retrieval. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 9286–9295, 2025

[5] Yang Chen, Hexiang Hu, Yi Luan Haitian Sun Soravit Changpinyo Alan Ritter and Ming-Wei Chang. Can pre-trained vision and language models answer visual information-seeking questions? In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 14948–14968, 2023

[7] Feng Gao, Qing Ping, Govind Thattai Aishwarya Reganti Ying Nian Wu and Prem Natarajan. Transform-retrievegenerate: Natural language-centric outside-knowledge visual question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5067–5077, 2022

[19] Weizhe Lin, Jingbiao Mei Jinghong Chen and Bill Byrne. Preflmr: Scaling up fine-grained late-interaction multimodal retrievers. arXiv preprint arXiv:2402.08327, 2024

[36] Duong T Tran, Trung-Kien Tran Manfred Hauswirth and Danh Le Phuoc. Reasonvqa: A multi-hop reasoning benchmark with structural knowledge for visual question answering. arXiv preprint arXiv:2507.16403, 2025

[37] Harsh Trivedi, Niranjan Balasubramanian Tushar Khot and Ashish Sabharwal. Musique: Multihop questions via singlehop question composition, 2022

[18] Weizhe Lin, Jinghong Chen, Jingbiao Mei Alexandru Coca and Bill Byrne. Fine-grained late-interaction multi-modal retrieval for retrieval augmented visual question answering, 2023

[40] Cong Wei, Yang Chen, Haonan Chen Hexiang Hu Ge Zhang Jie Fu Alan Ritter and Wenhu Chen. Uniir: Training and benchmarking universal multimodal information retrievers. In European Conference on Computer Vision, pages 387–404, 2024

[16] Sheng-Chieh Lin, Chankyu Lee Mohammad Shoeybi Jimmy Lin Bryan Catanzaro and Wei Ping. Mm-embed: Universal multimodal retrieval with multimodal llms, 2025

[39] Peng Wang, Qi Wu, Chunhua Shen Anthony Dick Anton Van Den Hengel. Fvqa: Fact-based visual question answering. IEEE transactions on pattern analysis and machine intelligence, 40(10):2413–2427, 2017

[38] Ashish Vaswani, Noam Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan N Gomez Lukasz Kaiser and Illia Polosukhin. Attention is all you need, 2023

[41] Tobias Weyand Andre Araujo Bingyi Cao and Jack Sim. Google landmarks dataset v2 – a large-scale benchmark for instance-level recognition and retrieval, 2020

[31] Stephen E Robertson Steve Walker Susan Jones Micheline M Hancock-Beaulieu Mike Gatford et al. Okapi at trec-3. Nist Special Publication Sp, 109:109, 1995

[22] Nelson F. Liu Kevin Lin John Hewitt Ashwin Paranjape Michele Bevilacqua Fabio Petroni and Percy Liang. p. 8

Where the paper says it

6 [22] Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang.

Lost in the middle: How language models use long contexts, 2023

Supplementary Material A

A.1. Query and Document Image Pool Construction

Images for questions and documents were sourced from separate pools. Question images were taken from the M2KR dataset, including OVEN Google Landmark v2 and iNaturalist. p. 8

Where the paper says it

2 [8] Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset, 2018.

Document images were downloaded from web page links provided by MuKA. p. 3
Where the paper says it

Finally, the generated question is passed to the LLM once more for paraphrasing, and we filter out questions that can be retrieved by BM25 [31] in the top-5 results, thereby reducing lexical overlap with the source document. The full prompts are provided in the Supp.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Augment document embeddings by incorporating images of multiple related entities to handle queries referencing secondary subjects, as this addresses visual shortcuts by enriches document representations by augmenting images of multiple related entities.

  2. Develop a new benchmark called RETINA, which is automatically constructed using an LLM-driven pipeline that identifies entities related to each document’s main entity and generates queries guided by structured knowledge graphs to eliminate such shortcuts.

  3. Propose MIMIR as a retriever that enriches document representations by integrating images of multiple related entities, enabling retrieval in the absence of visual shortcuts, unlike prior work that uses only a single image per document.

  4. Implement an Entity Token Embedding (ETE) to address the limitation where unimodal features treat text and images independently by adding a learnable vector added to the text tokens of the named entity corresponding to each image.

  5. Enhance query generation by using a graph-guided process that extracts relevant subgraphs corresponding to specific related entities, controlled by an LLM, instead of prompt-based approaches that often produce hallucinated or incorrect entity references.

Abstract

Existing Multimodal Knowledge-Based Visual Question Answering (MKB-VQA) benchmarks suffer from "visual shortcuts", as the query image typically matches the primary subject entity of the target document. We demonstrate that models can exploit these shortcuts, achieving comparable results using visual cues alone. To address this, we introduce Relational Entity Text-Image kNowledge Augmented (RETINA) benchmark, automatically constructed using an LLM-driven pipeline, consisting of 120k training and 2k human-curated test set. RETINA contains queries referencing secondary subjects (i.e. related entities) and pairs them with images of these related entities, removing the visual shortcut. When evaluated on RETINA existing models show significantly degraded performance, confirming their reliance on the shortcut. Furthermore, we propose Multi-Image MultImodal Retriever (MIMIR), which enriches document embeddings by augmenting images of multiple related entities, effectively handling RETINA, unlike prior work that uses only a single image per document. Our experiments validate the limitations of existing benchmarks and demonstrate the effectiveness of RETINA and MIMIR. Our project is available at: Project Page.

Sources

Related papers