Mind the Gap: Structure-Aware Consistency in Preference Learning
Standard methods for aligning LLMs using surrogate losses fail to guarantee true ranking accuracy because they allow models to shrink score differences instead of learning correct orderings. This work introduces Structure-Aware DPO (SA-DPO), which dynamically adjusts the confidence margin based on semantic distance between responses. This structural awareness is necessary for consistency, and heavy-tailed loss functions are shown to be superior for capacity-bounded models.