The Dual Mechanisms of Spatial Variable Binding in Vision-Language Models
The study investigated how vision-language models bind objects to their spatial properties using two concurrent mechanisms: a primary signal from the vision encoder encoding global layout, and a secondary signal from the language model backbone augmenting this information. Experiments showed that while the vision encoder is central, patching its distributed spatial signals corrects prediction failures up to 55%. This suggests robust vision encoders are crucial for accurate spatial understanding.