Fooling Reward Models with Gibberish: Adversarial Examples Beyond the Semantic Regime
cs.LG, cs.AI
Submitted: 2026-04-03
Updated: 2026-11-01
Terminology
Sources
- Adversarial Training of Reward Models
- Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback
- Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Models
- Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking
- The Llama 3 Herd of Models
- Unsolved Problems in ML Safety
- Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergy
- RewardBench 2: Advancing Reward Model Evaluation
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- Towards Understanding Sycophancy in Language Models
- HybridFlow: A Flexible and Efficient RLHF Framework
- A Long Way to Go: Investigating Length Correlations in RLHF
- Assessing Judging Bias in Large Reasoning Models: An Empirical Study
- Adversarial Reinforcement Learning for Large Language Model Agent Safety
- Language Models Learn to Mislead Humans via RLHF
- Qwen3 Technical Report
- NoveltyBench: Evaluating Language Models for Humanlike Diversity
- WildChat: 1M ChatGPT Interaction Logs in the Wild
- One Token to Fool LLM-as-a-Judge
- Cheating Automatic LLM Benchmarks: Null Models Achieve High Win Rates
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis