Vision-Default, Prior-Override: Causal Mechanisms of Perception-Knowledge Conflict in Vision-Language Models
cs.CL
Submitted: 2026-06-26
Updated: 2026-11-05
Code: https://github.com/nlietzow/vision-default-prior-override
Terminology
Sources
- Do VLMs Have Bad Eyes? Diagnosing Compositional Failures via Mechanistic Interpretability
- Understanding Information Storage and Transfer in Multi-modal Large Language Models
- NNsight and NDIF: Democratizing Access to Open-Weight Foundation Model Internals
- Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space
- Transformer Feed-Forward Layers Are Key-Value Memories
- Pixels Versus Priors: Controlling Knowledge Priors in Vision-Language Models through Visual Counterfacts
- What Do VLMs NOTICE? A Mechanistic Interpretability Pipeline for Gaussian-Noise-free Text-Image Corruption and Evaluation
- Finding Visual Task Vectors
- How Do Vision-Language Models Process Conflicting Information Across Modalities?
- Interpreting and Editing Vision-Language Representations to Mitigate Hallucinations
- Cutting Off the Head Ends the Conflict: A Mechanism for Interpreting and Mitigating Knowledge Conflicts in Language Models
- What's in the Image? A Deep-Dive into the Vision of Vision Language Models
- Visual Representations inside the Language Model
- Locating and Editing Factual Associations in GPT
- Overcoming Sparsity Artifacts in Crosscoders to Interpret Chat-Tuning
- Towards Interpreting Visual Information Processing in Vision-Language Models
- Same Task, Different Circuits: Disentangling Modality-Specific Mechanisms in VLMs
- When Seeing Overrides Knowing: Disentangling Knowledge Conflicts in Vision-Language Models
- Mechanisms of Prompt-Induced Hallucination in Vision-Language Models
- PaliGemma 2: A Family of Versatile VLMs for Transfer
Related papers
- ALARM: Audio-Language Alignment for Reasoning Models
- ThinkSum: Probabilistic reasoning over sets using large language models
- Coherence boosting: When your pretrained language model is not paying enough attention
- From Facts to Insights: A Persona-Driven Dual Memory Framework and Dataset for Role-Playing Agents
- Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization
- The Art of Asking: Multilingual Prompt Optimization for Synthetic Data