RedCodeAgent: Automatic Red-teaming Agent against Diverse Code Agents
cs.SE
Submitted: 2025-10-02
Updated: 2026-10-29
Comments: Accepted at ICLR 2026
Code: https://github.com/meta-llama/llama3
License: http://creativecommons.org/licenses/by/4.0/
The gist: Code agents have gained widespread adoption due to their strong code generation capabilities and integration with code interpreters, enabling dynamic execution, debugging…
Terminology
Abstract
Code agents have gained widespread adoption due to their strong code generation capabilities and integration with code interpreters, enabling dynamic execution, debugging, and interactive programming capabilities. While these advancements have streamlined complex workflows, they have also introduced critical safety and security risks. Current static safety benchmarks and red-teaming tools are inadequate for identifying emerging real-world risky scenarios, as they fail to cover certain boundary conditions, such as the combined effects of different jailbreak tools. In this work, we propose RedCodeAgent, the first automated red-teaming agent designed to systematically uncover vulnerabilities in diverse code agents. With an adaptive memory module, RedCodeAgent can leverage existing jailbreak knowledge, dynamically select the most effective red-teaming tools and tool combinations in a tailored toolbox for a given input query, thus identifying vulnerabilities that might otherwise be overlooked. For reliable evaluation, we develop simulated sandbox environments to additionally evaluate the execution results of code agents, mitigating potential biases of LLM-based judges that only rely on static code. Through extensive evaluations across multiple state-of-the-art code agents, diverse risky scenarios, and various programming languages, RedCodeAgent consistently outperforms existing red-teaming methods, achieving higher attack success rates and lower rejection rates with high efficiency. We further validate RedCodeAgent on real-world code assistants, e.g., Cursor and Codeium, exposing previously unidentified security risks. By automating and optimizing red-teaming processes, RedCodeAgent enables scalable, adaptive, and effective safety assessments of code agents. Our code and data are available at https://github.com/1mocat/RedCodeAgent.
Sources
- GPT-4 Technical Report
- Qwen Technical Report
- Purple Llama CyberSecEval: A Secure Coding Benchmark for Language Models
- CyberSecEval 2: A Wide-Ranging Cybersecurity Evaluation Suite for Large Language Models
- AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Bases
- LLM Agents can Autonomously Exploit One-day Vulnerabilities
- Black-Box Adversarial Attacks on LLM-Based Code Completion
- JailbreakZoo: Survey, Landscapes, and Horizons in Jailbreaking Large Language and Vision-Language Models
- AmpleGCG: Learning a Universal and Transferable Generative Model of Adversarial Suffixes for Jailbreaking Both Open and Closed LLMs
- EIA: Environmental Injection Attack on Generalist Web Agents for Privacy Leakage
- RedTeamCUA: Realistic Adversarial Testing of Computer-Use Agents in Hybrid Web-OS Environments
- AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models
- Testing Language Model Agents Safely in the Wild
- AdvPrompter: Fast Adaptive Adversarial Prompting for LLMs
- CWEval: Outcome-driven Evaluation on Functionality and Security of LLM Code Generation
- Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
- The Rise and Potential of Large Language Model Based Agents: A Survey
- RedAgent: Red Teaming Large Language Models with Context-aware Autonomous Language Agent
- Jailbreak Attacks and Defenses Against Large Language Models: A Survey
- GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts
Related papers
- ArkEval: Benchmarking and Evaluating Automated CodeRepair for ArkTS
- TestPrism: Rethinking Test Evaluation Beyond a Single Reference
- Mining Workflow Graphs for Black-Box Boundary Testing of Conversational LLM Agents
- Falsification-Based Verification of LLM-Generated Optimization Models: Sound Test Batteries and Their Detection Limits
- GitSkills: A Dataset of Agent Skills on GitHub
- For Coding Agents, the Repository Matters