How Safe Are AI-Generated Patches? A Large-scale Study on Security Risks in LLM and Agentic Automated Program Repair on SWE-bench
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Security Radio. Generated commentary on the latest security and cryptography papers.
Nadia: I'm Nadia.
Elias: And I'm Elias, here with Priya.
Nadia: We're looking at the paper: "How Safe Are AI-Generated Patches? A Large-scale Study on Security Risks in LLM and Agentic Automated Program Repair on SWE-bench".
Elias: Here's the gist of the paper. How Safe Are AI-Generated Patches?
Nadia: A Large-scale Study on Security Risks in LLM and Agentic Automated Program Repair on SWE-bench is the first large-scale security evaluation of an LLM and three agent-generated patches for real-world issue resolution tasks.
Elias: Our findings reveal that Llama introduces many new vulnerabilities, exhibiting unique patterns not found in developers’ code.
Nadia: Pruning redundant reasoning tokens The study evaluates patches proposed by developers, a standalone LLM (Llama 3.3 Instruct-70B), and three top-performing agentic frameworks (OpenHands, AutoCodeRover, HoneyComb) using 20,000+ GitHub issues.
Paper summary: Nadia: So, this paper is called "How Safe Are AI-Generated Patches? A Large-scale Study on Security Risks in LLM and Agentic Automated Program Repair on SWE-bench". It’s basically the first big security check we’re doing for LLM patches and agent generated fixes for real world issues.
Elias: Right, it looks at developers' patches, a standalone Llama three point three Instruct model, and three agentic frameworks: OpenHands, AutoCodeRover, and HoneyComb <ref:2507.02976#pg1>. The main goal is to figure out when and where these automated program repair systems are most likely to generate vulnerable patches in the first place.
Priya: So, it’s not just about looking at the code itself; they’re checking the code, but also the issues and even how complex the whole project is. That context is important for real world risk.
Nadia: Exactly. And what they found is pretty significant: Llama introduces a lot of brand new vulnerabilities, showing patterns that aren't found in typical developer code at all.
Elias: They looked at over twenty thousand GitHub issues on the SWE-bench dataset to do this analysis <ref:2507.02976#pg1>. They used CodeQL, Semgrep, and Bandit with a majority voting strategy to catch things while still catching a lot of them.
Priya: That’s helpful for us because we need to know what the actual data shows before we trust any claimed result. What did the study actually find about where these risks show up?
Nadia: Well, they looked at three different areas: code characteristics, issue characteristics, and project characteristics. They found that vulnerable patches are most strongly linked to things like a higher number of files that are modified or unique file types being changed in a patch.
Elias: And when it comes to the issues themselves, vulnerabilities often show up in bugs where the issue descriptions lack crucial information like examples of the expected behavior or steps to reproduce. That’s a big signal they found about the context surrounding an issue.
Priya: So, it’s not just that the AI made a mistake in one line; it's that the task itself is poorly described or the resulting patch touches too much of a complex part of the system. That changes how we think about testing these systems.
Paper summary: Nadia: It suggests that assessing risk can’t just be about finding specific vulnerable lines of code anymore. You have to look at both what the AI generates and the broader context where it’s operating, which is something we need to develop complementary risk assessment techniques for.
Elias: They also pointed out some recurring vulnerability types they saw, like Eval Injection from Llama three point three-Generated Code and OS Command Injection from developer-written code <ref:2507.02976#pg1>. These often come from the model taking instructions too literally, like using an eval function or defaulting to insecure functions like MD5.
Priya: That makes sense because those are basic security mistakes that models can make when they don't fully grasp the intent of the prompt. But I wonder how much of that is just model misinterpretation versus actual malicious generation.
Nadia: The paper points out that agentic frameworks like OpenHands produced an order of magnitude more vulnerabilities than AutoCodeRover or HoneyComb did. This difference they attribute to the broad operational scope of the agent, meaning it often touches more files than necessary to fix a single issue.
Elias: That’s a cautionary tale about giving agents too much freedom on large codebases; unconstrained autonomy can significantly increase the risk of vulnerabilities, especially with those large cross file edits.
Priya: So for us, that means we should push for explicit limits on edit scope when we build these agentic workflows, like caps on the size of the diff or limits on how many files an agent can touch in one go.
Nadia: And they also found that higher risk patches in frameworks like AutoCodeRover exhibit abnormally high code complexity. That means when a patch touches really complex functions, there’s a higher chance of an issue showing up.
Elias: They also looked at project level factors like repository size and maintainability index, but those didn't show strong correlations with the total count of vulnerabilities found in the patches. The vulnerability counts seemed more tied to the code and issue level characteristics rather than how big or complex the whole project is.
Paper summary: Priya: That’s interesting because I always thought project size matters a lot for security. So, for this study, it seems like the immediate context around the specific bug report is more important than just looking at the whole repository's structure.
Nadia: Exactly. And they suggest that simple signals, like how widely a patch is spread across the codebase and what kind of information is in an issue description, can be combined into a prediction model that scores risk before and after the AI makes a patch.
Elias: That leads to the idea of integrating this risk assessment directly into CI/CD pipelines as an early warning system, which seems like a promising direction for how we handle these tools moving forward.
Priya: So, while they show us where the risks are most likely to appear, the paper’s main limitation is that it focuses on specific vulnerabilities found in the test sets from SWE-bench. It doesn't necessarily cover every single type of real world issue out there.
Nadia: That makes sense because their methodology relies on that specific dataset. They also flag that using these LLM fixes directly as a replacement for developer written patches without extra safeguards is likely to increase both the frequency and the diversity of vulnerabilities in codebases.
Elias: So, while this study gives us concrete signals about what Llama and agentic systems do well, it’s a warning that we need to build more robust guardrails around their deployment.
Priya: It means for anyone using these LLM based APR tools, integrating security scanning and some form of manual inspection into the workflow when you accept an AI patch is a necessary step.
Nadia: That’s the practical application here: we need to treat LLM generated patches as an additional source of risk and not just another quick fix. We need to figure out how to score that risk before it even hits the developer's screen, right?
Conclusion: Nadia: So we're wrapping up this look at how safe AI generated patches are in program repair tasks on SWE-bench, specifically focusing on Llama and three different agent frameworks like OpenHands, AutoCodeRover, and HoneyComb.
Elias: Yeah Nadia it’s really about testing the security risks when these systems actually try to fix real world code issues across thousands of problems.
Priya: The paper is looking at a lot of data here from twenty thousand plus GitHub issues paired with pull requests to see exactly where these vulnerabilities pop up in the process.
Nadia: They found that Llama can introduce up to eleven times more new vulnerabilities compared to when actual developers write the fixes themselves. That’s a pretty big jump.
Elias: What they really highlight is that these new kinds of risks aren't just random; they show unique patterns you wouldn't see in typical developer code.
Priya: And the study looked at three main areas for where these risks come from: the characteristics of the code itself, the details of the issue description, and things about the project as a whole.
Nadia: They concluded that vulnerable patches are most strongly tied to specific file changes and how much unique code is added or modified in a patch.
Elias: But they also found that issues where developers don't clearly explain what should happen or how to reproduce the bug are often where the AI messes up the most.
Priya: So, it means risk isn't just about finding one bad line of code; it’s about looking at the whole context of how much code is changing and how well the problem is described.
Nadia: Exactly. They suggest we need to build these prediction models that score risk based on those signals before an AI even proposes a fix.
Elias: It points toward integrating security checks right into those continuous delivery pipelines as an early warning system for these AI repairs.
Amirali Sajadi, Kostadin Damevski, Preetha Chatterjee
Drexel University · Virginia Commonwealth University
cs.CR, cs.LG, cs.SE
Submitted: 2025-06-30
Updated: 2026-11-01
Code: https://github.com/SWE-bench/experiments
Project page: https://spotbugs.github.io
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 83/100
The gist: How Safe Are AI-Generated Patches? A Large-scale Study on Security Risks in LLM and Agentic Automated Program Repair on SWE-bench is the first large-scale security evaluation of an LLM and three…
Key concepts
- SWE-bench
- This dataset consists of real-world GitHub issues paired with pull requests, providing a large collection of tasks for testing automated program repair systems. It simulates practical software maintenance problems that AI tools attempt to solve.
- Agentic Automated Program Repair (APR)
- This refers to systems where AI agents, like OpenHands or AutoCodeRover, autonomously propose and implement code changes to fix bugs in software. The study examines the security implications of these automated repair workflows.
- Vulnerability Characteristics
- The research identifies specific traits in code and issues that correlate with insecure patches. These include a high number of modified files, the presence of certain file types, and missing information within the bug report description as indicators of higher risk.
- Risk Assessment Integration
- The conclusion advocates for adding security checks directly into development workflows (CI/CD). This involves using simple signals—like patch spread across the codebase or issue detail—to predict and score the risk level of an AI-generated patch before it is accepted.
Terminology
Summary
How Safe Are AI-Generated Patches? p. 0 How Safe Are AI-Generated Patches? A Large-scale Study on Security Risks in
LLM and Agentic Automated Program Repair on SWE-bench
AMIRALI SAJADI, Drexel University, USA
KOSTADIN DAMEVSKI, Virginia Commonwealth University, USA
PREETHA CHATTERJEE, Drexel University, USA
Large language models (LLMs) and their agentic frameworks are increasingly adopted to perform development tasks such as automated
program repair (APR). A Large-scale Study on Security Risks in
LLM and Agentic Automated Program Repair on SWE-bench
AMIRALI SAJADI, Drexel University, USA
KOSTADIN DAMEVSKI, Virginia Commonwealth University, USA
PREETHA CHATTERJEE, Drexel University, USA
Large language models (LLMs) and their agentic frameworks are increasingly adopted to perform development tasks such as automated
program repair (APR). While prior work has identified security risks in LLM-generated code, most have focused on synthetic, simplified,
or isolated tasks that lack the complexity of real-world program repair. Finally, we analyze a wide
range of code, issue, and project-level factors to understand the conditions under which generating insecure patches is more likely. Our findings reveal that Llama introduces many new vulnerabilities, exhibiting unique patterns not found in developers’ code.Where the paper says it
Where the paper says it
Where the paper says it
Pruning redundant reasoning tokens
The study evaluates patches proposed by developers, a standalone LLM (Llama 3.3 Instruct-70B), and three top-performing agentic frameworks (OpenHands, AutoCodeRover, HoneyComb) using 20,000+ GitHub issues. p. 0 We evaluate patches proposed by developers, a standalone LLM (Llama 3. 3
Instruct-70B), and three top-performing agentic frameworks (OpenHands, AutoCodeRover, HoneyComb). We then examine how code-level, issue-level, and projectlevel factors influence the security of the proposed solutions, identifying conditions that result in vulnerability-inducing
AI-generated patches. Our overarching goal with this work is to identify when and where APR systems are most likely
to generate vulnerable patches, informing guidelines for when to trust their output and when to require additional
checks. We perform this analysis for all vulnerabilities introduced both by Llama and the
agentic workflows, allowing us to identify shared and distinct risk factors across different generation techniques. We group these factors into three categories: code-related, issue-related, and project-level characteristics:
Code-related Characteristics: We analyze various properties of the LLM-generated code, including:
# Files Added or Modified: A larger number of files can indicate broader changes and a higher overall risk [4, 25, 26, 63].Where the paper says it
Where the paper says it
Where the paper says it
How the training signal is built
The methodology leverages the SWE-bench dataset consisting of real-world GitHub issues paired with pull requests. For the standalone model, Llama 3. p. 2 SWE-bench includes a train set of approximately 19,000 issue–PR pairs collected from 37 popular Python
repositories and a test set of 2,294 issue–PR pairs from 12 repositories. For the standalone model, we use Llama 3. For the standalone model, we use Llama 3. 3 Instruct (70B) to generate patches for both the train and test sets of
SWE-bench, comprising approximately 21,000 instances in total. In contrast, for the agentic frameworks, we rely
on publicly released patches that are available only for the test set. Due to the substantial computational and financial
cost required to run these iterative frameworks across the train set, we restrict our framework-based analysis to the
test split.Where the paper says it
Where the paper says it
Where the paper says it
Where the gains come from
The analysis employs a multi-step process to detect vulnerabilities, combining CodeQL, Semgrep, and Bandit using a majority voting strategy to maximize recall and precision. The researchers also perform manual inspections to remove false positives and ensure validity. p. 5 This conservative
approach can provide a balanced and trustworthy basis for our large-scale comparative analysis across different patch
sources, including developer patches, standalone LLMs, and agentic frameworks. Step 3: To ensure validity of all instances and to guarantee that only valid security-relevant issues enter our
analysis, we performed a manual inspection in which the first author inspected the patched file, and judged whether
the tool-reported issue constituted a plausible security threat rather than a benign code smell. This multi-step process addresses known limitations of fully automated static analysis and ensures the quality of our
findings prior to our quantitative analyses. 5 Metrics for Empirical Analysis
We analyze the prevalence and properties of vulnerabilities (RQ1 & RQ2) using CWE categories.Where the paper says it
Where the paper says it
Code, Issue, and Project Characteristics of Vulnerable Instances (RQ3)
The analysis investigates code-level factors such as files added or modified, unique file types modified, LOC generated by LLM, and cyclomatic complexity of LLM-generated code. Issue-related characteristics include issue type classification and information completeness, such as the presence of expected behavior and steps to reproduce. p. 6 Information Completeness: Based on prior work emphasizing the importance of issue clarity [74], we manually annotate
whether each issue includes: (1) Expected Behavior: how the software is supposed to behave, (2) Observed Behavior: how
it actually behaves, and (3) Steps to Reproduce: the process needed to trigger the issue. The presence of this information
can aid in understanding the bug’s context and identifying the root cause of the problem. Further, size of a project has been shown to have direct associations with the number of potential vulnerabilities [24]. Cyclomatic Complexity and Maintainability Index of Project: We use Radon [1], a popular code analysis tool [29, 42], to
compute each project’s average maintainability index and cyclomatic complexity. We find that vulnerabilities in
LLM-generated patches are associated with distinctive code characteristics and are commonly observed in issues missing specific
types of information. These results suggest that contextual factors play a critical role in the security of the generated patches and
point toward the need for proactive risk assessment methods that account for both issue and code-level information.Where the paper says it
Where the paper says it
Where the paper says it
Conclusions
The study concludes that Llama can introduce up to 11x more new vulnerabilities than developers when addressing issues. Vulnerable outputs are associated with particular code and issue-level characteristics, such as a higher number of modified files and the absence of code snippets or certain information in the issue description. p. 2 We found that vulnerable outputs are more likely when LLM patches involve a higher number
of files and unique file type modifications. Further, most standalone LLM-generated vulnerabilities were observed in
instances involving bug-related issues that do not contain code examples, information about the expected behavior of
the code, or steps to reproduce the issue. We find that vulnerabilities in
LLM-generated patches are associated with distinctive code characteristics and are commonly observed in issues missing specific
types of information. These results suggest that contextual factors play a critical role in the security of the generated patches and
point toward the need for proactive risk assessment methods that account for both issue and code-level information.Where the paper says it
Where the paper says it
Llama 3.3-Generated Code CWE-95 (Eval Injection) 39, Developer-Written Code CWE-78 (OS Command Injection) 4, and Llama 3. p. 8 Vulnerability Types and Overlaps: Table 2 contains the most common vulnerabilities in Llama-generated and
developer-written code 1
. The most common vulnerabilities in Llama-generated patches include command injection
(CWE-78, 79 instances), eval injection (CWE-95, 39), insecure deserialization (CWE-502, 17), and path traversal (CWE-22,
14). 3-Generated Code
CWE-78 (OS Command Injection) 79
CWE-95 (Eval Injection) 39
CWE-502 (Insecure Deserialization) 17
CWE-22 (Path Traversal) 14
Developer-Written Code
CWE-78 (OS Command Injection) 4
CWE-377 (Insecure Temporary File) 3
CWE-732 (Permission Misconfiguration) 3
CWE-22 (Path Traversal) 3
Table 2. Most recurring CWE types found in Llama-generated and developer-written patches across the train & test sets. Similarly insecure deserialization (CWE-502) was repeated in 17 LLM patches
while being detected in only two developer patches. These issues likely stem from the model’s tendency to translate
task instructions into overly literal code, for example, using eval() to dynamically execute strings, or defaulting
to insecure algorithms like MD5.Where the paper says it
Where the paper says it
Where the paper says it
OH produced an order of magnitude more vulnerabilities than ACR or HC. This disparity is attributed to OH's broad operational scope, which means it often touches more files than necessary to resolve an issue. The results indicate that giving agents unconstrained autonomy over large repositories can substantially increase the risk of vulnerabilities, especially via large, cross-file edits.
Vulnerable patches tend to modify more files and file types, but involve fewer lines of generated code. Higher-risk patches in ACR exhibit abnormally high code complexity. p. 7 Maintainability index is derived from several factors, including cyclomatic complexity, lines of code, and Halstead
volume, and is designed to estimate how easy it is to maintain the codebase. A higher maintainability index suggests
code that is easier to understand and modify, which may reduce the risk of generating vulnerabilities. We found that vulnerable outputs are more likely when LLM patches involve a higher number
of files and unique file type modifications. Further, most standalone LLM-generated vulnerabilities were observed in
instances involving bug-related issues that do not contain code examples, information about the expected behavior of
the code, or steps to reproduce the issue. # Files: The number of files in a given project can serve as a proxy for its potential complexity and size. Prior work on
open-source software effort estimation has used file count as an indicator of project scale and structural complexity [61]. Generating more lines of
code may suggest that the model is undertaking more complex tasks, handling broader context, or replacing larger code
chunks, which can raise the likelihood of mistakes or security issues. Cyclomatic Complexity of LLM-Generated Code: This metric reflects the complexity of the functions introduced or
rewritten by the LLM.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Frameworks should incorporate explicit limits on edit scope e.g., caps on diff size, limits on the number of files that can be modified in a single attempt. p. 7 Projects with more contributors often have greater oversight, which can lower the risk of security
vulnerabilities. # Files: The number of files in a given project can serve as a proxy for its potential complexity and size. We find that vulnerabilities in
LLM-generated patches are associated with distinctive code characteristics and are commonly observed in issues missing specific
types of information. These results suggest that contextual factors play a critical role in the security of the generated patches and
point toward the need for proactive risk assessment methods that account for both issue and code-level information.Where the paper says it
Where the paper says it
The study presents the first large-scale security evaluation of an LLM and three agent-generated patches for real-world issue resolution tasks. The researchers find that Llama’s patches should be treated as an additional source of security risk. p. 0 We find that vulnerabilities in
LLM-generated patches are associated with distinctive code characteristics and are commonly observed in issues missing specific
types of information. These results suggest that contextual factors play a critical role in the security of the generated patches and
point toward the need for proactive risk assessment methods that account for both issue and code-level information. Hence, assessing
their security requires evaluation frameworks that capture the full end-to-end repair workflow rather than isolated
snippet-based analyses. Although recent benchmarks like SWE-bench [32] have enabled the assessment of LLMs on real-world software
engineering tasks, security-specific analyses in these contexts remain limited, and we still lack a systematic understanding of when APR-style LLM and agentic frameworks introduce new vulnerabilities, or how those vulnerabilities
relate to properties of the patches and the surrounding development context. Step 2: To increase confidence (precision), we adopt a conservative majority voting strategy: a generated or
developer-written patch is treated as a security risk only if at least two of the three tools independently identify new
vulnerabilities in the same file(s) it modifies or creates. Moving beyond aggregation, the majority vote dismisses findings
reported by only a single tool as likely false positives.Where the paper says it
Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Risk Assessment Integration: Integrate a risk model into CI/CD pipelines that scores APR tasks based on issue-level signals like information completeness and code-level features such as number of modified files, to route high-risk tasks to stricter security checks or human review.
-
Agentic Workflow Constraints: Implement explicit limits on edit scope for autonomous agentic frameworks, such as caps on diff size or limits on the number of files modified in a single attempt, to prevent excessive rewrites and large cross-file edits that lead to vulnerabilities.
-
Contextual Information Requirements: Require complete and precise problem descriptions from developers for APR systems, as vulnerability-prone patches are associated with issues where
expected behavior and reproduction steps are under-specified.
-
Code Complexity Safeguards: For agentic frameworks like AutoCodeRover, implement checks that flag or require human review when generated code exhibits abnormally high cyclomatic complexity, as this metric was found to be a significant indicator of security risk in those patches.
-
Distinguishing LLM Behavior: Develop methods to distinguish between LLM-generated vulnerabilities and developer errors by analyzing distinct vulnerability patterns, such as the prevalence of CWE-95 (eval injection) and CWE-502 (insecure deserialization) which were
far more prevalent in Llama-generated code.
Abstract
Large language models (LLMs) and agentic frameworks are increasingly used for development tasks such as automated program repair (APR). Prior work has identified security risks in LLM-generated code, but most studies focus on synthetic, simplified, or isolated tasks. We present a large-scale security analysis of LLM-generated patches using 20,000+ GitHub issues from SWE-bench. We evaluate developer-written patches, two standalone LLMs (Llama 3.3 Instruct-70B and GPT-5.4 nano), and four agentic frameworks (OpenHands, AutoCodeRover, HoneyComb, and SWE-agent). We also analyze code, issue, and project-level factors associated with insecure patches. Under our evaluated SWE-bench setup, Llama 3.3 introduced 135 manually confirmed vulnerabilities across the train and test sets, compared with 12 in developer-written patches for the same issues. On the SWE-bench test set, GPT-5.4 nano introduced 3 vulnerabilities compared with 2 in developer patches. All four evaluated agentic frameworks also introduced vulnerabilities. OpenHands had the highest retained count, with 44 vulnerabilities across 11 issue instances. Vulnerabilities are associated with broader code modifications and with issues missing specific information. These results suggest that APR systems require proactive risk assessment methods that consider both generated code and issue context.
Sources
- Red Teaming Program Repair Agents: When Correct Patches can Hide Vulnerabilities
- Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference
- Do Prompt Patterns Affect Code Quality? A First Empirical Assessment of ChatGPT-Generated Code
- Security Weaknesses of Copilot-Generated Code in GitHub Projects: An Empirical Study
- The Hitchhiker's Guide to Program Analysis, Part II: Deep Thoughts by LLMs
- Beyond Functional Correctness: Exploring Hallucinations in LLM-Generated Code
- Can We Trust Large Language Models Generated Code? A Framework for In-Context Learning, Security Patterns, and Code Evaluations Across Diverse LLMs
- Adversarial Bug Reports as a Security Risk in Language Model-Based Automated Program Repair
- The Art of Repair: Optimizing Iterative Program Repair with Instruction-Tuned Models
- LLM-CSEC: Empirical Evaluation of Security in C/C++ Code Generated by Large Language Models
- LLMs in Software Security: A Survey of Vulnerability Detection Techniques and Insights
- SALLM: Security Assessment of Generated Code
- Toward Secure Code Generation: Bridging Correctness and Security via Task-Adaptive Vulnerability Modeling and Execution-Based Benchmarking
- OpenHands: An Open Platform for AI Software Developers as Generalist Agents
- Are "Solved Issues" in SWE-bench Really Solved Correctly? An Empirical Study
- Enhancing repository-level software repair via repository-aware knowledge graphs
- SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering
- How Can ChatGPT Support Human Security Testers to Help Mitigate Supply Chain Attacks?
- Comparison of Static Application Security Testing Tools and Large Language Models for Repo-level Vulnerability Detection
Related papers
- From Strategy Search to Policy Learning: MCTS-Guided Adversarial Co-Evolution for LLM Safety Alignment
- Privacy Leakage from Recurrent State in LLM Inference
- Incrementally Verifiable Oracle Aided Computation
- Privilege and confidentiality in generative AI workflows
- BadMoE: Backdooring Mixture-of-Experts LLMs via Routing-Aware Triggers and Task-Underutilized Experts
- CoVault: Secure, Scalable Analytics of Personal Data