Auditing Medical Vision-Language Models on Chest Radiographs: Estimating Reference Agreement Across Institutions
The episode discusses a paper auditing medical vision-language models on chest radiographs across different institutions to estimate reference agreement. The hosts find that model performance is inconsistent, depending on the question format and hospital setting. They conclude that trust must be earned locally by hospitals through their own data, and the proposed measurement tools were not always reliable.