From What to Which: Decoding Modifier Grounding in Frozen MLLMs
The study investigates how instance-level grounding, like identifying 'the yellow banana,' is distributed across various tokens rather than just one modifier word. OTTER is a lightweight probe using Optimal Transport to align generated tokens with visual regions from frozen MLLMs. Results show that instance information is recoverable from contextualized modifiers, proving that 'what' to 'which' grounding is a refined, distributed process.