Preferred, Not Safer: Pairwise Preference Is a Poor Proxy for Clinical Safety
Clinician pairwise preferences are an unreliable measure of clinical safety because preferred responses often contradict safety scores. This means using preference leaderboards can lead to deploying models that seem good but are actually unsafe or inaccurate, especially in medical fields. The study shows that surface-level qualities like response length influence preferences more than true safety metrics.