Relative Density Ratio Optimization for Stable and Statistically Consistent Model Alignment
cs.LG, cs.AI, cs.CL, stat.ML
Submitted: 2026-04-06
Updated: 2026-09-26
Comments: Code is available at https://github.com/takahashihiroshi/rdro
Code: https://github.com/takahashihiroshi/rdro
License: http://creativecommons.org/licenses/by/4.0/
The gist: Aligning language models with human preferences is essential for ensuring their safety and reliability.
Terminology
Abstract
Aligning language models with human preferences is essential for ensuring their safety and reliability. Although most existing approaches assume specific human preference models such as the Bradley-Terry model, this assumption may fail to accurately capture true human preferences, and consequently, these methods lack statistical consistency, i.e., the guarantee that language models converge to the true human preference as the number of samples increases. In contrast, direct density ratio optimization (DDRO) achieves statistical consistency without assuming any human preference models. DDRO models the density ratio between preferred and non-preferred data distributions using the language model, and then optimizes it via density ratio estimation. However, this density ratio is unstable and often diverges, leading to training instability of DDRO. In this paper, we propose a novel alignment method that is both stable and statistically consistent. Our approach is based on the relative density ratio between the preferred data distribution and a mixture of the preferred and non-preferred data distributions. Our approach is stable since this relative density ratio is bounded above and does not diverge. Moreover, it is statistically consistent and yields significantly tighter convergence guarantees than DDRO. We experimentally show its effectiveness with Qwen 2.5 and Llama 3.
Sources
- GPT-4 Technical Report
- Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs
- Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators
- RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models
- The Llama 3 Herd of Models
- Energy-Based Preference Model Offers Better Offline Alignment than the Bradley-Terry Preference Model
- Binary Classifier Optimization for Large Language Model Alignment
- Preference Optimization by Estimating the Ratio of the Data Distribution
- SGDR: Stochastic Gradient Descent with Warm Restarts
- Decoupled Weight Decay Regularization
- Nash Learning from Human Feedback
- The Woman Worked as a Babysitter: On Biases in Language Generation
- Qwen2 Technical Report
- Solving math word problems with process- and outcome-based feedback
- Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHF
- Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks