Learning What Evaluators Value: A Reliable Approach to Modeling Evaluator Preferences
The episode discusses the paper 'Learning What Evaluators Value,' which moves beyond simple scoring averages to model complex human preferences. Hosts explore how this approach quantifies evaluator disagreement and uses multi-dimensional criteria, such as feasibility and novelty, to build a reliable, generalized preference engine for AI evaluation.