Revealing the Gap in Human and VLM Scene Description through Counterfactual Semantic Saliency

arXiv:2605.13047 · cs.CV, cs.AI · Submitted 2026-05-13 · Read on arXiv

cs.CV, cs.AI

Submitted: 2026-05-13

Updated: 2026-10-10

Code: https://github.com/starsky77/Counterfactual-Semantic-Saliency

Terminology

• Ablation — 1× in this paper · explained in 1 episode
• Alignment — 1× in this paper · explained in 1 episode
• Fidelity — 1× in this paper · explained in 2 episodes
• Interpretability — 1× in this paper · explained in 15 episodes
• Model-Agnostic — 1× in this paper · explained in 2 episodes
• Semantic Alignment — 1× in this paper · explained in 3 episodes
• Vision-Language Models (VLMs) — 1× in this paper · explained in 6 episodes
• Vision-language models — 1× in this paper · explained in 2 episodes

Sources

• GPT-4 Technical Report
• Qwen3-VL Technical Report
• SAM 3: Segment Anything with Concepts
• Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities
• Explanation-Driven Counterfactual Testing for Faithfulness in Vision-Language Model Explanations
• Guiding Instruction-based Image Editing via Multimodal Large Language Models
• Scaling Laws for Task-Optimized Models of the Primate Visual Ventral Stream
• Gemma 3 Technical Report
• From Local Cues to Global Percepts: Emergent Gestalt Organization in Self-Supervised Vision Models
• Semantic Image Inversion and Editing using Rectified Stochastic Differential Equations
• OpenAI GPT-5 System Card
• V-CECE: Visual Counterfactual Explanations via Conceptual Edits
• Gemini: A Family of Highly Capable Multimodal Models
• PooDLe: Pooled and dense self-supervised learning from naturalistic videos
• Does resistance to style-transfer equal Global Shape Bias? Measuring network sensitivity to global shape configuration
• SANA: Efficient High-Resolution Image Synthesis with Linear Diffusion Transformers
• When and why vision-language models behave like bags-of-words, and what to do about it?
• Jasper and Stella: distillation of SOTA embedding models
• InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models

Related papers

• Loss Knows Best: Detecting Annotation Errors in Videos via Loss Trajectories
• AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memories
• Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift
• MambaX-Net: Dual-Input Mamba-Enhanced Cross-Attention Network for Longitudinal MRI Segmentation
• TeleOCR: Navigating Document Parsing Across Digital and Camera-Captured Documents
• A Survey on Efficient Vision-Language-Action Models