Post-Training at the Edge of Detectability: A Game-Theoretic Approach to Fine-Tuning
The paper uses game theory to find an optimal KL regularization coefficient for fine-tuning AI models. It frames fine-tuning as a sequential detection game between an agent maximizing reward and a monitor trying to detect deviations from a reference policy. The Nash equilibrium of this game precisely matches the standard KL-regularized RL objective, allowing adaptive tuning.