Selective Linkage Precision for Released Medical Vision-Language Models
cs.CV, cs.AI, cs.CL, cs.LG
Submitted: 2026-06-01
Updated: 2026-10-31
Comments: Accepted at ML4H 2026 Proceedings, Sydney, Australia
Code: https://github.com/microsoft/hi-ml
Project page: https://stanfordmlgroup.github.io/competitions/chexpert
License: http://creativecommons.org/licenses/by/4.0/
The gist: Hospitals may release de-identified chest radiographs while restricting access to their reports.
Terminology
Abstract
Hospitals may release de-identified chest radiographs while restricting access to their reports. An attacker who already has reports from the same population and their metadata can use a vision-language model (VLM) to match images to reports. We introduce selective linkage precision, which measures correct matches among the highest-confidence queries after filtering candidates by the available attributes. Across 8 public VLMs and 3 chest radiograph datasets, the maximum whole-pool Recall@1 is 0.40%. However, this metric averages over every query without attribute filtering. On ReXGradient-160K, MedSigLIP-448 reaches 67.68% precision at a budget of 100 after filtering by sex, age band, and projection, with a median of 327 candidate reports. On the same eligible query set, an attribute-only policy reaches 24.99% and the model alone reaches 1.98%. Adding acquisition year, available in only 1 of the 3 datasets, yields 99.05% precision and a median of 24 candidates. Uniform guessing on the observed selected queries has an expected precision of 19.91%. Expanding the primary-setting collection to all 123,100 ReXGradient studies lowers precision to 37.23%. On BioViL-T, the tested head finetuning procedures do not lower linkage below the public checkpoint. Selective linkage precision requires inference without retraining and could help hospitals evaluate which attributes to publish.
Sources
- Automated grounding verification of AI statements about chest radiographs
- Adapting Foundation Vision-Language Models to Medical Diagnosis via Query-Driven Expert Bridging
- VSE++: Improving Visual-Semantic Embeddings with Hard Negatives
- Safety and accuracy diverge across deployment conditions in clinical large language models
- Representation Learning with Contrastive Predictive Coding
Related papers
- TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs