Do Multimodal LLMs See Before They Read? Diagnosing Contextual Sycophancy
The episode discusses a paper by Yi-Cheng Lai and Hen-Hsen Huang titled "Do Multimodal LLMs See Before They Read? Diagnosing Contextual Sycophancy." Hosts discuss how to measure and mitigate contextual sycophancy—a failure where external text overrides visual evidence. Key findings include the effectiveness of a staged approach that improving accuracy, and the need to categorize text's role as either a 'contaminant' or a helpful 'scaffold'.