An Empirical Study of Reward Specification and Benchmark Reliability in GRPO-based LLM Unlearning
The study tested four different reward designs in Group Relative Policy Optimization (GRPO) to see how they affect an LLM's ability to unlearn specific knowledge while maintaining broader answers. Results show that different rewards lead to distinct behaviors, including 'refusal collapse' or 'broad answer selection,' proving that optimization success does not always match the desired behavioral unlearning.