Explaining and Preventing Alignment Collapse in Iterative RLHF
cs.LG, stat.ML
Submitted: 2026-05-05
Updated: 2026-11-07
Code: https://github.com/GauthierE/fpo
Terminology
Sources
- Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
- Llama 2: Open Foundation and Fine-Tuned Chat Models
- Gemini: A Family of Highly Capable Multimodal Models
- SAIL: Self-Improving Efficient Online Alignment of Large Language Models
- Reward Shaping for (Inference-Time) Alignment: A Stackelberg Game Perspective
- Proximal Policy Optimization Algorithms
- The Geometry of Alignment Collapse: When Fine-Tuning Breaks Safety
- Gradient Regularization Mitigates Reward Hacking in Reinforcement Learning from Human Feedback and Verifiable Rewards
- Mitigating Reward Hacking in Reinforcement Learning from Human Feedback via Advantage Sign Robustness
- Fine-Tuning Language Models from Human Preferences
- The Llama 3 Herd of Models
- WebGPT: Browser-assisted question-answering with human feedback
- Reinforced Self-Training (ReST) for Language Modeling
- How Sampling Shapes LLM Alignment: From One-Shot Optima to Iterative Dynamics
- Reward Shaping to Mitigate Reward Hacking in RLHF
- Bi-Level Policy Optimization with Nystr\"om Hypergradients
- Performative Policy Gradient: Optimality in Performative Reinforcement Learning
- Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis