When In-Distribution Gains Fail: Evaluating Weak-to-Strong Reward Models under Preference Shift
The study tested if training strong AI models using weak preference labels leads to general knowledge or just memorizing specific patterns. They found that strong models trained on weak data often fail to transfer their skills when faced with new types of preferences. This suggests a failure in representation learning, and they propose a method called Representation Anchoring (ANCHOR) to fix this by stabilizing the model's core features during training.