Generalized Preference Optimization: A Unified Approach to Offline Alignment
Generalized Preference Optimization (GPO) unifies various offline preference optimization methods like DPO and IPO by parameterizing losses using a general class of convex functions 'f'. It frames reward learning as supervised binary classification, allowing existing algorithms to be seen as special cases. This framework provides a unified way to fine-tune large models directly from offline data.