BlueCodeAgent: A Blue Teaming Agent Powered by Automated Red Teaming for CodeGen AI
cs.SE
Submitted: 2025-10-20
Updated: 2026-10-29
Comments: Accepted at ICML 2026
Code: https://github.com/meta-llama/llama3
License: http://creativecommons.org/licenses/by/4.0/
The gist: Existing research on CodeGen AI security mainly focuses on red teaming, which aims to uncover vulnerabilities and risks in AI-generated code.
Terminology
Abstract
Existing research on CodeGen AI security mainly focuses on red teaming, which aims to uncover vulnerabilities and risks in AI-generated code. However, progress on the blue teaming side remains limited, as effective defenses require a deep security analysis of given tasks and edge cases. To fill in this gap, we propose BlueCodeAgent, an end-to-end blue teaming agent powered by automated red teaming. Our red teaming component generates diverse risky instances, providing effective edge cases and guidance for the subsequent blue teaming process. Our blue teaming agent then conducts multi-level defense, leveraging these red teaming examples to detect previously seen and unseen risk scenarios through constitution summarization and dynamic code analysis. Our evaluation across four representative code-related tasks (bias instruction detection, malicious instruction detection, vulnerable code detection, and prompt injection detection) shows that BlueCodeAgent achieves significant gains over diverse baselines. In particular, for vulnerability detection tasks, BlueCodeAgent integrates dynamic analysis to effectively reduce false positives, a challenging problem as base models tend to be over-conservative. Overall, with GPT-4o as the base model, BlueCodeAgent achieves an average F1 score improvement of 14.7% across four tasks compared to directly prompting the model, attributed to its ability to summarize actionable constitutions and perform dynamic analysis. Our code and data are available at https://github.com/1mocat/BlueCodeAgent.
Sources
- GPT-4 Technical Report
- Program Synthesis with Large Language Models
- Qwen Technical Report
- Constitutional AI: Harmlessness from AI Feedback
- CyberSecEval 2: A Wide-Ranging Cybersecurity Evaluation Suite for Large Language Models
- Vulnerability Detection with Code Language Models: How Far Are We?
- Vul-RAG: Enhancing LLM-based Vulnerability Detection via Knowledge-level RAG
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- Bias Testing and Mitigation in LLM-based Code Generation
- Black-Box Adversarial Attacks on LLM-Based Code Completion
- Poly-Guard: Massive Multi-Domain Safety Policy-Grounded Guardrail Dataset
- AmpleGCG: Learning a Universal and Transferable Generative Model of Adversarial Suffixes for Jailbreaking Both Open and Closed LLMs
- PurpCode: Reasoning for Safer Code Generation
- AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models
- HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal
- AdvPrompter: Fast Adaptive Adversarial Prompting for LLMs
- Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributions
- LLMs Cannot Reliably Identify and Reason About Security Vulnerabilities (Yet?): A Comprehensive Evaluation, Framework, and Benchmarks
Related papers
- ArkEval: Benchmarking and Evaluating Automated CodeRepair for ArkTS
- TestPrism: Rethinking Test Evaluation Beyond a Single Reference
- Mining Workflow Graphs for Black-Box Boundary Testing of Conversational LLM Agents
- Falsification-Based Verification of LLM-Generated Optimization Models: Sound Test Batteries and Their Detection Limits
- GitSkills: A Dataset of Agent Skills on GitHub
- For Coding Agents, the Repository Matters