Proximal Preference Optimization: A Unified Framework for Stable and Reward-Free Alignment
cs.LG, cs.AI
Submitted: 2026-11-07
Updated: 2026-11-07
Terminology
Sources
- Aligning Large Language Models with Human: A Survey
- Large Language Model Alignment: A Survey
- Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignment
- Proximal Policy Optimization Algorithms
- KTO: Model Alignment as Prospect Theoretic Optimization
- DPO Meets PPO: Reinforced Token Optimization for RLHF
- Risk-Aware Alignment Must Incorporate Reward Uncertainty
- Reward-Augmented Data Enhances Direct Preference Alignment of LLMs
- AlphaDPO: Adaptive Reward Margin for Direct Preference Optimization
- Margin Adaptive DPO: Leveraging Reward Model for Granular Control in Preference Optimization
- Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimization
- Mirror Descent Policy Optimization
- DeepSeek-V3 Technical Report
- DAPO: An Open-Source LLM Reinforcement Learning System at Scale
- Group Sequence Policy Optimization
- DCPO: Dynamic Clipping Policy Optimization
- Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis