Understanding the performance gap between online and offline alignment algorithms
The study compared online and offline alignment algorithms to find that online methods generally outperform offline ones when constrained by a fixed optimization budget. The performance gap is not explained by simple factors like dataset size, but rather by how on-policy sampling affects the interplay between a model's classification (discriminative) and response generation (generative) abilities.