Breaking the Visual Shortcuts in Multimodal Knowledge-Based Visual Question Answering
summary
The gist
Existing Multimodal Knowledge-Based Visual Question Answering (MKB-VQA) benchmarks suffer from “visual shortcuts”…
In short
Existing visual question answering (VQA) models use visual shortcuts where queries match primary subjects in images. To fix this, the RETINA benchmark pairs secondary subject queries with related entity images. The MIMIR retriever then enriches document embeddings by incorporating multiple related entity images, overcoming the single-image limitation and improving performance.
Key concepts
- RETINA Benchmark
- A new benchmark designed to eliminate visual shortcuts in VQA. It automatically creates test sets by linking queries about secondary subjects with images of those related entities, forcing models to rely on knowledge rather than just matching primary image content.
- MIMIR Retriever
- A retrieval system that improves document embeddings by augmenting them with images of multiple related entities from the text. This extends prior single-image methods, allowing the model to find relevant documents even when queries reference secondary subjects.
- Visual Shortcut
- A flaw in current VQA models where they exploit the fact that a query image often matches the main subject of a document. This allows models to get high scores using visual cues alone, ignoring deeper knowledge required for accurate answers.
- Graph-Guided Query Generation
- A method used to create questions by first building a structured knowledge graph based on an LLM's analysis of related entities. This ensures queries are generated based on specific, relevant connections rather than broad prompts, reducing errors.
Terminology used across episodes
This episode discusses
- Breaking the Visual Shortcuts in Multimodal Knowledge-Based Visual Question Answering · Paper Radio
- GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints
- Qwen Technical Report
- PreFLMR: Scaling Up Fine-Grained Late-Interaction Multi-modal Retrievers
- ReasonVQA: A Multi-hop Reasoning Benchmark with Structural Knowledge for Visual Question Answering
The paper
Breaking the Visual Shortcuts in Multimodal Knowledge-Based Visual Question Answering · Read on arXiv
Dosung Lee, Sangwon Jung, Boyoung Kim, Minyoung Kim, Sungyeon Kim, *Junyoung Sung, Paul Hongsuck Seo
Korea University · KAIST · Amazon
Existing Multimodal Knowledge-Based Visual Question Answering (MKB-VQA) benchmarks suffer from "visual shortcuts", as the query image typically matches the primary subject entity of the target document. We demonstrate that models can exploit these shortcuts, achieving comparable results using visual cues alone. To address this, we introduce Relational Entity Text-Image kNowledge Augmented (RETINA) benchmark, automatically constructed using an LLM-driven pipeline, consisting of 120k training and 2k human-curated test set. RETINA contains queries referencing secondary subjects (i.e. related entities) and pairs them with images of these related entities, removing the visual shortcut. When evaluated on RETINA existing models show significantly degraded performance, confirming their reliance on the shortcut. Furthermore, we propose Multi-Image MultImodal Retriever (MIMIR), which enriches document embeddings by augmenting images of multiple related entities, effectively handling RETINA, unlike prior work that uses only a single image per document. Our experiments validate the limitations of existing benchmarks and demonstrate the effectiveness of RETINA and MIMIR. Our project is available at: Project Page.
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: We're looking at the paper: "Breaking the Visual Shortcuts in Multimodal Knowledge-Based Visual Question Answering".
Title and authors: Tom: Let's look at the title and who wrote this paper, "Breaking the Visual Shortcuts in Multimodal Knowledge-Based Visual Question Answering." The authors are Dosung Lee, Sangwon Jung, Boyoung Kim, Minyoung Kim, Sungyeon Kim, and Junyoung Sung.
Jane: It’s interesting that they focused on this specific problem because it shows how much existing models rely on those shortcuts just to get comparable results using visual cues one <ref:2511.22843#pg1,comparable results using visual cues>.
Lu: The authors are proposing a solution called the Relational Entity Text-Image Knowledge Augmented, or RETINA benchmark, which is built using an LLM-driven pipeline with one hundred twenty thousand training samples and two thousand human-curated test samples one <ref:2511.22843#pg1,Relational Entity Text-Image Knowledge Augmented>.
Meng: So what does this actually mean for us in terms of how we train models? It sounds like they are building a way to stop the model from just looking at the picture and start looking at the relationships between things.
Tom: Exactly. They show that when you evaluate existing models on RETINA, their performance drops significantly because they confirm that these models were indeed relying heavily on those visual shortcuts one <ref:2511.22843#pg1>.
Jane: The goal is to create a benchmark that better reflects real-world scenarios where queries often reference secondary subjects, not just the main entity one <ref:2511.22843#pg1>.
Lu: They are constructing this benchmark through an automated pipeline that first identifies entities related to each document’s main entity, then builds a structured knowledge graph, and finally extracts subgraphs for query generation one <ref:2511.22843#pg1>.
Meng: Building a knowledge graph automatically based on the document content sounds complex to set up, but if it works, it should force the model to use more context than just looking at one picture.
Tom: I'm excited about that pipeline part because they mention avoiding prompt-based approaches that feed full documents into LLMs because those often lead to hallucinations one <ref:2511.22843#pg1>.
The paper's summary: Jane: So, if we summarize the main point of "Breaking the Visual Shortcuts in Multimodal Knowledge-Based Visual Question Answering," it’s that models can exploit visual shortcuts to solve these tasks without really understanding the underlying knowledge.
Tom: Right. Existing benchmarks let models get away with just matching a picture to a subject, and this gives them performance comparable to models that actually understand the text one <ref:2511.22843#pg1>.
Lu: The paper summarizes how they address this by introducing RETINA, which is a benchmark where queries reference secondary subjects and are paired with images of those related entities one <ref:2511.22843#pg1>.
Meng: They explain that the system for building RETINA involves four steps: identifying related entities, constructing a knowledge graph, extracting subgraphs for query generation, and then generating queries guided by those subgraphs one <ref:2511.22843#pg1>.
Jane: They emphasize that instead of using prompt-based methods where you feed the whole document to an LLM hoping it finds the right relationship, they use this graph-guided generation process one <ref:2511.22843#pg1>.
Tom: That’s smart because they note that the graph-guided approach allows for controlled query construction by extracting subgraphs corresponding to specific related entities one <ref:2511.22843#pg1>.
Lu: They specifically build one-hop neighborhood graphs by extracting named entities related to the main entity and their relations using an LLM one <ref:2511.22843#pg1>.
Meng: So, what’s the immediate implication for a developer? If you want a model that doesn't rely on visual shortcuts, you should be looking at training it on this RETINA data instead of just standard document images one <ref:2511.22843#pg1>.
Jane: It really shifts the focus from just getting a high score on one image to actually understanding the relational knowledge between different things in the text and image one <ref:2511.22843#pg1>.
The paper's improvements: Tom: Now let's look at what they are proposing as improvements. They suggest two main things: first, building RETINA itself, and second, proposing MultiImage Multimodal Retriever, or MIMIR.
Jane: Building RETINA is the benchmark improvement; it’s an automated system that generates one hundred twenty thousand training samples and a two thousand human-verified test set one <ref:2511.22843#pg1>.
Lu: The pipeline for building RETINA involves knowledge graph construction, target subgraph extraction, and LLM-based VQA generation one, which removes those visual shortcuts we talked about earlier <ref:2511.22843#pg1>.
Meng: But they also propose MIMIR as a retriever. That’s an improvement to how you actually retrieve documents, moving beyond the single-image framework of prior work one <ref:2511.22843#pg1>.
Tom: MIMIR enriches document embeddings by integrating images of multiple related entities, which is supposed to help handle those queries referencing secondary subjects one <ref:2511.22843#pg1>.
Jane: The key difference here is that MIMIR extracts related entities from the text and augments multiple images of those entities to guide the retrieval of documents that contain them one <ref:2511.22843#pg1>.
Lu: The document encoder in MIMIR builds on MuKA by incorporating these multiple images into each document, which extends its representation capacity one <ref:2511.22843#pg1>.
Meng: From a deployment standpoint, I have to ask about the cost. If you're embedding multiple images per document, the similarity computations during retrieval might get slower compared to just using one image one <ref:2511.22843#pg1>.
Tom: That latency is a trade-off they acknowledge, but they show that MIMIR achieves nearly twice the recall of the best baselines and substantially improves answer quality one <ref:2511.22843#pg1>.
Conclusion: Jane: So, to wrap up the paper "Breaking the Visual Shortcuts in Multimodal Knowledge-Based Visual Question Answering," they argue that existing benchmarks are flawed because they allow models to exploit visual shortcuts when answering questions about related entities.
Tom: They introduce RETINA as a way to fix this by using an automated, LLM-driven pipeline to generate queries referencing secondary subjects and pair them with the correct related images one <ref:2511.22843#pg1>.
Lu: The authors suggest that the MIMIR retriever is effective because it enriches document representations by integrating images of multiple related entities, which mitigates the inductive bias toward visual shortcuts that MuKA uses one <ref:2511.22843#pg1>.
Meng: From my side, what I see is that they've created a very specific test for models that actually need relational knowledge rather than just pattern matching on an image one <ref:2511.22843#pg1>.
Lalam: I'm pretty sure this direction is the right one because incorporating diverse entity representations for each document helps guide the retrieval of documents containing those entities one <ref:2511.22843#pg1>.
Jane: In short, they propose RETINA as a novel benchmark without visual shortcuts and MIMIR as a retriever that augments multiple images of related entities to the document embedding one <ref:2511.22843#pg1>.
Tom: It’s a solid direction for pushing multimodal question answering forward by forcing models to handle the complexity of relationships one <ref:2511.22843#pg1>.
More episodes
- 2603.29765-Training-Free Dynamic Upcycling of Expert Language Models
- 2605.00778-Observable Performance Does Not Fully Reflect Adaptive System Organization: A Multi-Level Analysis of Gait Dynamics Under Occlusal Constraint
- 2605.06906-TraXion: Rethinking Pre-training Frameworks for Mobility and Beyond
- 2605.09165-Sparse Layers are Critical to Scaling Looped Language Models
- 2605.09439-Inverse Design for Conditional Distribution Matching
- 2605.10313-Signature Approach for Contextual Bandits with Nonlinear and Path-dependent Rewards: Regret Bound and Statistical Inference
- 2605.10878-Minimum Neural Weight Norm = Kolmogorov Complexity
- 2605.11617-MIST: Reliable Streaming Decision Trees for Online Class-Incremental Learning via McDiarmid Bound
- 2605.15085-From Data to Action: Accelerating Refinery Optimization with AI
- 2605.23500-GRTO: Bootstrapped Group Relative Tool Optimization for Referring Segmentation