The Alignment Tax Is Graded: Task-Selective Capability Loss Under Length-Normalized DPO with SFT Retention
cs.LG
Submitted: 2026-11-04
Updated: 2026-11-19
Code: https://github.com/yuan-phd/delta-filing
Terminology
Sources
- Separating Constraint Compliance from Semantic Accuracy: A Novel Benchmark for Evaluating Instruction-Following Under Compression
- Do Post-Training Algorithms Actually Differ? A Controlled Study Across Model Scales Uncovers Scale-Dependent Ranking Inversions
- Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizer
- MoE-Sieve: Routing-Guided LoRA for Efficient MoE Fine-Tuning
- SimPO: Simple Preference Optimization with a Reference-Free Reward
- Iterative Reasoning Preference Optimization
- Displacement-Resistant Extensions of DPO with Nonconvex $f$-Divergences
- The Viscosity of Logic: Phase Transitions and Hysteresis in DPO Alignment
- What Is the Alignment Tax?
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis