Uncovering Cross-Objective Interference in Multi-Objective Alignment
This study systematically evaluated scalarization algorithms for multi-objective LLM alignment and discovered a common failure called 'cross-objective interference.' This happens when training improves one objective while degrading others, even when objectives aren't mathematically conflicting. The paper proposes Covariance Targeted Weight Adaptation (CTWA) to maintain positive covariance between objectives and the training signal, ensuring better overall performance.