GREAT: Generalizable Backdoor Attacks in RLHF via Emotion-Aware Trigger Synthesis
cs.CR, cs.LG
Submitted: 2025-10-10
Updated: 2026-08-28
Comments: Accepted in EMNLP 2026 Findings
License: http://creativecommons.org/licenses/by/4.0/
The gist: Recent work has shown that RLHF is highly susceptible to backdoor attacks.
Terminology
Abstract
Recent work has shown that RLHF is highly susceptible to backdoor attacks. However, existing methods often rely on rare tokens or fixed triggers, limiting their impact in realistic scenarios. In this work, we develop GREAT, a novel framework for crafting natural distributional backdoors in RLHF. Specifically, GREAT targets harmful response generation for a vulnerable user subpopulation featured by semantically violent requests paired with emotionally angry triggers. At the core of our framework is a trigger identification pipeline that operates in the model's latent embedding space, leveraging dimensionality reduction and clustering techniques to identify representative triggers. To enable this, we introduce a hierarchical and diversity-driven prompting strategy to construct Erinyes, a high-quality dataset of over 5,000 angry triggers curated from GPT-4.1. Our experiments show that GREAT significantly outperforms baselines in attack generalization to unseen triggers, while preserving standard utility and maintaining stealth under defenses.
Sources
- A Closer Look at Memorization in Deep Networks
- Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
- Best-of-Venom: Attacking RLHF by Injecting Poisoned Preference Data
- Detecting Backdoor Attacks on Deep Neural Networks by Activation Clustering
- GoEmotions: A Dataset of Fine-Grained Emotions
- Does Learning Require Memorization? A Short Tale about a Long Tail
- The Llama 3 Herd of Models
- Here's a Free Lunch: Sanitizing Backdoored Models with Model Merge
- A Survey on LLM-as-a-Judge
- Simulate and Eliminate: Revoke Backdoors for Generative Large Language Models
- Chain-of-Scrutiny: Detecting Backdoor Attacks for Large Language Models
- Fragile Giants: Understanding the Susceptibility of Models to Subpopulation Attacks
- CleanGen: Mitigating Backdoor Attacks for Generation Tasks in Large Language Models
- Excess Capacity and Backdoor Poisoning
- Prometheus: Inducing Fine-grained Evaluation Capability in Language Models
- Is poisoning a real threat to LLM alignment? Maybe more so than you think
- Weight Poisoning Attacks on Pre-trained Models
- Estimating Training Data Influence by Tracing Gradient Descent
- Improving Diversity of Demographic Representation in Large Language Models via Collective-Critiques and Self-Voting
- BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension
Related papers
- SoK: AI-Augmented Binary Reversing
- Relaxed Sender Anonymity for CBDC Interbank Settlement: A Zero-Knowledge Approach on Permissioned EVM
- Calibration-Family Overfit: Why Trusted Sabotage Monitors Don't Transfer Across Lineages
- Efficient Fuzzy PSI under One-Sided Assumptions
- Sealing the Audit-Runtime Gap for LLM Skills
- Token Composition: A Graph Based on EVM Logs