What do Reward Models Memorize?
Discriminative training of reward models leads to biased models that memorize specific patterns instead of learning true response quality. Models tend to focus on easy, high-margin pairs and dataset shortcuts, overgeneralizing simple traits like length or compliance. This memorization limits their ability to judge response quality in new, context-dependent situations.