Understanding the Performance Gap in Preference Learning: A Dichotomy of RLHF and DPO
cs.LG, cs.CL
Submitted: 2025-05-26
Updated: 2026-10-31
Code: https://github.com/ZHZisZZ/modpo
Terminology
Sources
- Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
- Deep reinforcement learning from human preferences
- Process Reinforcement through Implicit Rewards
- SAIL: Self-Improving Efficient Online Alignment of Large Language Models
- Towards Analyzing and Understanding the Limitations of DPO: A Theoretical Perspective
- Direct Language Model Alignment from Online AI Feedback
- RS-DPO: A Hybrid Rejection Sampling and Direct Preference Optimization Method for Alignment of Large Language Models
- Iterative Length-Regularized Direct Preference Optimization: A Case Study on Improving 7B Language Models to GPT-4 Level
- Distributionally Robust Reinforcement Learning with Human Feedback
- WebGPT: Browser-assisted question-answering with human feedback
- Proximal Policy Optimization Algorithms
- A Minimaximalist Approach to Reinforcement Learning from Human Feedback
- Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Data
- Secrets of RLHF in Large Language Models Part II: Reward Modeling
- Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation
- Fine-Tuning Language Models from Human Preferences
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis