InfiFPO: Implicit Model Fusion via Preference Optimization in Large Language Models
InfiFPO is a preference optimization method that fuses multiple source models during Direct Preference Optimization (DPO). It replaces the reference model with a fused source model that synthesizes probabilities across all sources at the sequence level. This avoids vocabulary alignment issues and preserves crucial probability information, leading to improved performance on benchmarks.