Understanding Negative Learning in Large Language Model Fine-tuning
This research analyzes how Direct Preference Optimization (DPO) and Proximal Policy Optimization (PPO) learn by monitoring optimization dynamics under an endpoint probe. It reveals DPO behaves like supervised learning with evolving targets, while PPO shows reinforcement-like exploration. The study suggests component-level control can improve performance in these methods.