Process then Retrieve: Investigating the Origins of the Modality Gap in Vision-Language Models
cs.LG, cs.CL, cs.CV
Submitted: 2026-07-20
Updated: 2026-07-20
Code: https://github.com/openai/tiktoken
Terminology
Sources
- Qwen Technical Report
- PaliGemma: A versatile 3B VLM for transfer
- It's Not a Modality Gap: Characterizing and Addressing the Contrastive Gap
- Hidden in plain sight: VLMs overlook their visual representations
- How Do Vision-Language Models Process Conflicting Information Across Modalities?
- Hallucination Augmented Contrastive Learning for Multimodal Large Language Model
- LatentLens: Revealing Highly Interpretable Visual Tokens in LLMs
- Seeing but Not Believing: Probing the Disconnect Between Visual Attention and Answer Correctness in VLMs
- Vision-Language Models Create Cross-Modal Task Representations
- Linearly Mapping from Image to Text Space
- Towards Interpreting Visual Information Processing in Vision-Language Models
- Same Task, Different Circuits: Disentangling Modality-Specific Mechanisms in VLMs
- Interpreting the linear structure of vision-language model embedding spaces
- Line of Sight: On Linear Representations in VLLMs
- Fill the Gap: Quantifying and Reducing the Modality Gap in Image-Text Representation Learning
- Two Effects, One Trigger: On the Modality Gap, Object Bias, and Information Imbalance in Contrastive Vision-Language Models
- GLU Variants Improve Transformer
- Implicit Multimodal Alignment: On the Generalization of Frozen LLMs to Multimodal Inputs
- From Pixels to Prose: A Large Dataset of Dense Image Captions
- How Visual Representations Map to Language Feature Space in Multimodal LLMs
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis