"It's Just a Small Weapon": What Rewarding Misbehavior During RL Does to Character-Trained Models
cs.LG
Submitted: 2026-11-11
Updated: 2026-11-13
Code: https://github.com/safety-research/character-training-rl
Terminology
Sources
- A General Language Assistant as a Laboratory for Alignment
- Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation
- Constitutional Midtraining: Content Presence Drives Alignment Gains
- The Ends Justify the Thoughts: RL-Induced Motivated Reasoning Fools CoT Monitors
- Shallow Beliefs: Synthetic document finetuning does not inoculate against emergent misalignment from reward hacking
- (Mis)generalization of Helpful-Only Fine-tuning
- Agentic Misalignment: How LLMs Could Be Insider Threats
- Natural Emergent Misalignment from Reward Hacking in Production RL
- OpenAI o1 System Card
- gpt-oss-120b & gpt-oss-20b Model Card
- The MASK Benchmark: Disentangling Honesty From Accuracy in AI Systems
- Stress Testing Deliberative Alignment for Anti-Scheming Training
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- School of Reward Hacks: Hacking harmless tasks generalizes to misaligned behavior in LLMs
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis