The Reward Model Selection Crisis in Personalized Alignment
Optimizing reward model (RM) accuracy is insufficient for deploying personalized AI because it does not guarantee good generation quality. The research introduces policy accuracy and a new benchmark, Pref-LaMP, to show that high ranking accuracy does not predict behavioral alignment. This reveals a critical decoupling between preference ranking and actual output performance.