Smaller Models, Better Rejects: Preference Distillation Scaling

arXiv:2609.38987 · cs.LG · Submitted 2026-09-30 · Read on arXiv

cs.LG

Submitted: 2026-09-30

Updated: 2026-09-30

Terminology

• Distillation — 1× in this paper · explained in 2 episodes

Sources

• Program Synthesis with Large Language Models
• Qwen Technical Report
• Sharpe Ratio-Guided Active Learning for Preference Optimization in RLHF
• Curriculum Learning-Guided Progressive Distillation in Large Language Models
• Evaluating Large Language Models Trained on Code
• AdaDPO: Self-Adaptive Direct Preference Optimization with Balanced Gradient Updates
• Towards Disentangled Preference Optimization Dynamics: Suppress the Loser, Preserve the Winner
• SODA: Semi On-Policy Black-Box Distillation for Large Language Models
• The Delta Learning Hypothesis: Preference Tuning on Weak Data can Yield Strong Gains
• DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
• Measuring Mathematical Problem Solving With the MATH Dataset
• Distilling the Knowledge in a Neural Network
• Decomposing the Delta: What Do Models Actually Learn from Preference Pairs?
• SIMPLEMIX: Frustratingly Simple Mixing of Off- and On-policy Data in Language Model Preference Learning
• Direct Preference Knowledge Distillation for Large Language Models
• CompassDPO: Dynamics-Controlled Direct Preference Optimization for Robust Safety Alignment
• Gradient-Gated DPO: Stabilizing Preference Optimization in Language Models
• Difficulty-Based Preference Data Selection by DPO Implicit Reward Gap
• Zephyr: Direct Distillation of LM Alignment
• More is Less: The Pitfalls of Multi-Model Synthetic Preference Data in DPO Safety Alignment

Related papers

• Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
• AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
• Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
• Convergence issues in Relational Concept Analysis based on AOC-posets
• Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
• Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks