On the Framing Effect of AI Red-Teaming Instructions: Evidence from Prior Exercises and Controlled Simulations
cs.CY, cs.AI, cs.LG
Submitted: 2026-10-30
Updated: 2026-10-30
Code: https://github.com/example/repo
Project page: https://aiverifyfoundation.github.io/moonshot/resources/attack_modules
License: http://creativecommons.org/licenses/by/4.0/
The gist: AI red-teaming, adversarially probing generative AI systems to surface harms, has become a central evaluation method.
Terminology
Abstract
AI red-teaming, adversarially probing generative AI systems to surface harms, has become a central evaluation method. However, prior work has highlighted that red-teaming is a fairly open-ended task, laden with myriad design choices that can shape its findings. In this work, we focus on the instructions used to conduct red-teaming exercises and hypothesize that they can significantly affect outcomes in a manner analogous to the framing effect in psychology, where the format in which information is presented greatly impacts how people respond. To understand whether and why instructions impact red-teaming outcomes, first, we analyze the range of ways in which instructions have historically been formulated, reviewing 228 instances of prior work. Second, based on takeaways from this review, we run controlled simulations that hold parameters such as the risk domain, the attacker, and the model under investigation fixed while varying only the instructions. From five real instruction templates used by industry and academic organizations, we derive 45 instruction frames across three risk domains and three levels of risk granularity, generating 100 variants per frame, and deploy each against five open-weights target models, for 22,500 attacks. Using a generalized linear mixed model, we find that certain instruction properties significantly predict attack success rates: the lowest and highest levels of risk granularity both outperform intermediate specificity, as do short and long instructions relative to medium-length ones. The presence of examples is associated with higher attack success rates, while supplying explicit harm definitions yields lower ones. As alignment evaluations become increasingly automated, our results suggest that humans continue to shape red-teaming outcomes substantially through the instructions they provide to scaffold the activity.
Sources
- A Collaborative, Human-Centred Taxonomy of AI, Algorithmic, and Automation Harms
- Biothreat Benchmark Generation Framework for Evaluating Frontier AI Models II: Benchmark Generation Process
- Toward Revealing Nuanced Biases in Medical LLMs
- OpenAI's Approach to External Red Teaming for AI Models and Systems
- RedDebate: Safer Responses Through Multi-Agent Red Teaming Debates
- Red Teaming Large Language Models for Healthcare
- There are Levels to It: Red Teaming Dialogue With Hierarchical Reinforcement Learning
- PLAGUE: Plug-and-play framework for Lifelong Adaptive Generation of Multi-turn Exploits
- Explore, Establish, Exploit: Red Teaming Language Models from Scratch
- Strategize Globally, Adapt Locally: A Multi-Turn Red Teaming Agent with Dual-Level Learning
- Red Teaming Program Repair Agents: When Correct Patches can Hide Vulnerabilities
- RL-JACK: Reinforcement Learning-powered Black-box Jailbreaking Attack against LLMs
- Evolve the Method, Not the Prompts: Evolutionary Synthesis of Jailbreak Attacks on LLMs
- Jailbreaking LLMs & VLMs: Mechanisms, Evaluation, and Unified Defense
- Learning to Conceal Risk: Controllable Multi-turn Red Teaming for LLMs in the Financial Domain
- Red-Teaming Text-to-Image Models via In-Context Experience Replay and Semantic-Preserving Prompt Rewriting
- Between a Rock and a Hard Place: The Tension Between Ethical Reasoning and Safety Alignment in LLMs
- Constitutional Classifiers++: Efficient Production-Grade Defenses against Universal Jailbreaks
- RainbowPlus: Enhancing Adversarial Prompt Generation via Evolutionary Quality-Diversity Search
- PersonaTeaming: Exploring How Introducing Personas Can Improve Automated AI Red-Teaming
Related papers
- AI-Mediated Communication Can Steer Collective Opinion
- Epistemic Infrastructures of Science in AI Era Should Rebalance Costs of Generation and Verification
- AI out-persuades expert humans in text-based dialogue
- Evaluating Ambient Clinical Scribes in India: The Need for Multilingual Real-World Clinical Conversation Data
- AI Value Alignment for Evolving Social Norms
- Voting Biases in Decentralized Autonomous Organization (DAO) Governance