Preference Optimization with Multi-Sample Comparisons
The research introduces Multi-sample Direct Preference Optimization (mDPO) and Multi-sample Identity Preference Optimization (mIPO). These methods extend existing alignment techniques by using multiple samples in comparisons instead of just one. This approach is better at capturing group characteristics like diversity and bias, leading to superior optimization of collective model properties across different generative tasks.