UniHEAR: Unified Heterogeneous-Source Attentive Retrieval for Knowledge-Based Visual Question Answering
Ganzhong Luo, Yang Ren, Hanyong Wang, Shuyu Zheng, Menglong Yang
cs.IR, cs.CL, cs.CV
Submitted: 2026-08-02
Comments: Accepted by ACM MM 2026
Code: https://github.com/iven-luo/UniHEAR
License: http://creativecommons.org/licenses/by/4.0/
The gist: Knowledge-Based Visual Question Answering (KB-VQA) requires retrieving entity knowledge from external sources to answer visually grounded questions.
Terminology
Abstract
Knowledge-Based Visual Question Answering (KB-VQA) requires retrieving entity knowledge from external sources to answer visually grounded questions. Existing retrieval-augmented systems suffer from two critical limitations. First, relying on a single retrieval modality creates a Single-Source Retrieval Bottleneck, missing ground-truth entities that are only accessible through complementary sources. Second, dual-tower pointwise rerankers suffer from Retrieval-Source-Blind Reranking, as they overlook retrieval origins and candidate-level retrieval priors, leading to redundant modality reliance. To address these challenges, we propose UniHEAR, a unified lightweight framework for heterogeneous-source entity retrieval and reranking. UniHEAR constructs a Coarse Retrieval Descriptor for each candidate entity, and introduces Retrieval-Guided Attentive Modality Gating to condition modality attention weights on this descriptor, complemented by Entropy-Weighted Source Fusion of coarse retrieval priors. A hybrid training strategy combining contrastive learning with an auxiliary modality-preserving loss unifies entity-level and section-level retrieval within a single model. Extensive experiments on E-VQA and InfoSeek demonstrate that UniHEAR achieves state-of-the-art retrieval and VQA performance, improving Recall@1 by 6.7 and 1.2 points over the strongest baselines while maintaining a lightweight reranking architecture. Code and model are available at https://github.com/iven-luo/UniHEAR.
Sources
- GPT-4 Technical Report
- Qwen3-VL Technical Report
- Qwen2.5-VL Technical Report
- Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?
- ReAG: Reasoning-Augmented Generation for Knowledge-based Visual Question Answering
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- The Llama 3 Herd of Models
- Knowledge-based Visual Question Answer with Multimodal Processing, Retrieval and Filtering
- Retrieval Augmented Visual Question Answering with Outside Knowledge
- PreFLMR: Scaling Up Fine-Grained Late-Interaction Multi-modal Retrievers
- Wiki-R1: Incentivizing Multimodal Reasoning for Knowledge-based VQA via Data and Sampling Curriculum
- RoRA-VLM: Robust Retrieval-Augmented Vision Language Models
- EVA-CLIP: Improved Training Techniques for CLIP at Scale
- EVA-CLIP-18B: Scaling CLIP to 18 Billion Parameters
- Llama 2: Open Foundation and Fine-Tuned Chat Models
- jina-reranker-v3: Last but Not Late Interaction for Listwise Document Reranking
- EchoSight: Advancing Visual-Language Models with Wiki Knowledge
- OMGM: Orchestrate Multiple Granularities and Modalities for Efficient Multimodal Retrieval
- mKG-RAG: Leveraging Multimodal Knowledge Graphs in Retrieval-Augmented Generation for Knowledge-intensive VQA
- BERTScore: Evaluating Text Generation with BERT
Related papers
- The Price of Isolation: Estimating the Ecosystem Cost of Symmetric Two-Sided A/B Testing
- SCAR: Semantic Continuity-Aware Retrieval for Efficient Context Expansion in RAG
- MixLoRA-DSI: Dynamically Expandable Mixture-of-LoRA Experts for Rehearsal-Free Generative Retrieval over Dynamic Corpora
- QProtFuse: Query-Guided Layer Fusion for Retrieval-Augmented Protein Function Prediction
- DualView: Adaptive Local-Global Fusion for Multi-Hop Document Reranking
- How Do LLM-Generated Texts Impact Term-Based Retrieval Models?