How Safe Are AI-Generated Patches? A Large-scale Study on Security Risks in LLM and Agentic Automated Program Repair on SWE-bench

summary

The gist

How Safe Are AI-Generated Patches? A Large-scale Study on Security Risks in LLM and Agentic Automated Program Repair on SWE-bench is the first large-scale security evaluation of an LLM and three…

In short

This study evaluated security risks from LLM and agent-generated code patches using 20,000+ GitHub issues on SWE-bench. Findings show Llama introduces up to 11x more new vulnerabilities than developers. Risk is linked to code characteristics like modified files and incomplete issue descriptions, suggesting integrated risk assessment in CI/CD pipelines.

Key concepts

SWE-bench
This dataset consists of real-world GitHub issues paired with pull requests, providing a large collection of tasks for testing automated program repair systems. It simulates practical software maintenance problems that AI tools attempt to solve.
Agentic Automated Program Repair (APR)
This refers to systems where AI agents, like OpenHands or AutoCodeRover, autonomously propose and implement code changes to fix bugs in software. The study examines the security implications of these automated repair workflows.
Vulnerability Characteristics
The research identifies specific traits in code and issues that correlate with insecure patches. These include a high number of modified files, the presence of certain file types, and missing information within the bug report description as indicators of higher risk.
Risk Assessment Integration
The conclusion advocates for adding security checks directly into development workflows (CI/CD). This involves using simple signals—like patch spread across the codebase or issue detail—to predict and score the risk level of an AI-generated patch before it is accepted.

Terminology used across episodes

This episode discusses

The paper

How Safe Are AI-Generated Patches? A Large-scale Study on Security Risks in LLM and Agentic Automated Program Repair on SWE-bench · Read on arXiv

Amirali Sajadi, Kostadin Damevski, Preetha Chatterjee

Drexel University · Virginia Commonwealth University

Large language models (LLMs) and agentic frameworks are increasingly used for development tasks such as automated program repair (APR). Prior work has identified security risks in LLM-generated code, but most studies focus on synthetic, simplified, or isolated tasks. We present a large-scale security analysis of LLM-generated patches using 20,000+ GitHub issues from SWE-bench. We evaluate developer-written patches, two standalone LLMs (Llama 3.3 Instruct-70B and GPT-5.4 nano), and four agentic frameworks (OpenHands, AutoCodeRover, HoneyComb, and SWE-agent). We also analyze code, issue, and project-level factors associated with insecure patches. Under our evaluated SWE-bench setup, Llama 3.3 introduced 135 manually confirmed vulnerabilities across the train and test sets, compared with 12 in developer-written patches for the same issues. On the SWE-bench test set, GPT-5.4 nano introduced 3 vulnerabilities compared with 2 in developer patches. All four evaluated agentic frameworks also introduced vulnerabilities. OpenHands had the highest retained count, with 44 vulnerabilities across 11 issue instances. Vulnerabilities are associated with broader code modifications and with issues missing specific information. These results suggest that APR systems require proactive risk assessment methods that consider both generated code and issue context.

Transcript

Introduction to the show: ident: Security Radio. Generated commentary on the latest security and cryptography papers.

Nadia: I'm Nadia.

Elias: And I'm Elias, here with Priya.

Nadia: We're looking at the paper: "How Safe Are AI-Generated Patches? A Large-scale Study on Security Risks in LLM and Agentic Automated Program Repair on SWE-bench".

Elias: Here's the gist of the paper. How Safe Are AI-Generated Patches?

Nadia: A Large-scale Study on Security Risks in LLM and Agentic Automated Program Repair on SWE-bench is the first large-scale security evaluation of an LLM and three agent-generated patches for real-world issue resolution tasks.

Elias: Our findings reveal that Llama introduces many new vulnerabilities, exhibiting unique patterns not found in developers’ code.

Nadia: Pruning redundant reasoning tokens The study evaluates patches proposed by developers, a standalone LLM (Llama 3.3 Instruct-70B), and three top-performing agentic frameworks (OpenHands, AutoCodeRover, HoneyComb) using 20,000+ GitHub issues.

Paper summary: Nadia: So, this paper is called "How Safe Are AI-Generated Patches? A Large-scale Study on Security Risks in LLM and Agentic Automated Program Repair on SWE-bench". It’s basically the first big security check we’re doing for LLM patches and agent generated fixes for real world issues.

Elias: Right, it looks at developers' patches, a standalone Llama three point three Instruct model, and three agentic frameworks: OpenHands, AutoCodeRover, and HoneyComb <ref:2507.02976#pg1>. The main goal is to figure out when and where these automated program repair systems are most likely to generate vulnerable patches in the first place.

Priya: So, it’s not just about looking at the code itself; they’re checking the code, but also the issues and even how complex the whole project is. That context is important for real world risk.

Nadia: Exactly. And what they found is pretty significant: Llama introduces a lot of brand new vulnerabilities, showing patterns that aren't found in typical developer code at all.

Elias: They looked at over twenty thousand GitHub issues on the SWE-bench dataset to do this analysis <ref:2507.02976#pg1>. They used CodeQL, Semgrep, and Bandit with a majority voting strategy to catch things while still catching a lot of them.

Priya: That’s helpful for us because we need to know what the actual data shows before we trust any claimed result. What did the study actually find about where these risks show up?

Nadia: Well, they looked at three different areas: code characteristics, issue characteristics, and project characteristics. They found that vulnerable patches are most strongly linked to things like a higher number of files that are modified or unique file types being changed in a patch.

Elias: And when it comes to the issues themselves, vulnerabilities often show up in bugs where the issue descriptions lack crucial information like examples of the expected behavior or steps to reproduce. That’s a big signal they found about the context surrounding an issue.

Priya: So, it’s not just that the AI made a mistake in one line; it's that the task itself is poorly described or the resulting patch touches too much of a complex part of the system. That changes how we think about testing these systems.

Paper summary: Nadia: It suggests that assessing risk can’t just be about finding specific vulnerable lines of code anymore. You have to look at both what the AI generates and the broader context where it’s operating, which is something we need to develop complementary risk assessment techniques for.

Elias: They also pointed out some recurring vulnerability types they saw, like Eval Injection from Llama three point three-Generated Code and OS Command Injection from developer-written code <ref:2507.02976#pg1>. These often come from the model taking instructions too literally, like using an eval function or defaulting to insecure functions like MD5.

Priya: That makes sense because those are basic security mistakes that models can make when they don't fully grasp the intent of the prompt. But I wonder how much of that is just model misinterpretation versus actual malicious generation.

Nadia: The paper points out that agentic frameworks like OpenHands produced an order of magnitude more vulnerabilities than AutoCodeRover or HoneyComb did. This difference they attribute to the broad operational scope of the agent, meaning it often touches more files than necessary to fix a single issue.

Elias: That’s a cautionary tale about giving agents too much freedom on large codebases; unconstrained autonomy can significantly increase the risk of vulnerabilities, especially with those large cross file edits.

Priya: So for us, that means we should push for explicit limits on edit scope when we build these agentic workflows, like caps on the size of the diff or limits on how many files an agent can touch in one go.

Nadia: And they also found that higher risk patches in frameworks like AutoCodeRover exhibit abnormally high code complexity. That means when a patch touches really complex functions, there’s a higher chance of an issue showing up.

Elias: They also looked at project level factors like repository size and maintainability index, but those didn't show strong correlations with the total count of vulnerabilities found in the patches. The vulnerability counts seemed more tied to the code and issue level characteristics rather than how big or complex the whole project is.

Paper summary: Priya: That’s interesting because I always thought project size matters a lot for security. So, for this study, it seems like the immediate context around the specific bug report is more important than just looking at the whole repository's structure.

Nadia: Exactly. And they suggest that simple signals, like how widely a patch is spread across the codebase and what kind of information is in an issue description, can be combined into a prediction model that scores risk before and after the AI makes a patch.

Elias: That leads to the idea of integrating this risk assessment directly into CI/CD pipelines as an early warning system, which seems like a promising direction for how we handle these tools moving forward.

Priya: So, while they show us where the risks are most likely to appear, the paper’s main limitation is that it focuses on specific vulnerabilities found in the test sets from SWE-bench. It doesn't necessarily cover every single type of real world issue out there.

Nadia: That makes sense because their methodology relies on that specific dataset. They also flag that using these LLM fixes directly as a replacement for developer written patches without extra safeguards is likely to increase both the frequency and the diversity of vulnerabilities in codebases.

Elias: So, while this study gives us concrete signals about what Llama and agentic systems do well, it’s a warning that we need to build more robust guardrails around their deployment.

Priya: It means for anyone using these LLM based APR tools, integrating security scanning and some form of manual inspection into the workflow when you accept an AI patch is a necessary step.

Nadia: That’s the practical application here: we need to treat LLM generated patches as an additional source of risk and not just another quick fix. We need to figure out how to score that risk before it even hits the developer's screen, right?

Conclusion: Nadia: So we're wrapping up this look at how safe AI generated patches are in program repair tasks on SWE-bench, specifically focusing on Llama and three different agent frameworks like OpenHands, AutoCodeRover, and HoneyComb.

Elias: Yeah Nadia it’s really about testing the security risks when these systems actually try to fix real world code issues across thousands of problems.

Priya: The paper is looking at a lot of data here from twenty thousand plus GitHub issues paired with pull requests to see exactly where these vulnerabilities pop up in the process.

Nadia: They found that Llama can introduce up to eleven times more new vulnerabilities compared to when actual developers write the fixes themselves. That’s a pretty big jump.

Elias: What they really highlight is that these new kinds of risks aren't just random; they show unique patterns you wouldn't see in typical developer code.

Priya: And the study looked at three main areas for where these risks come from: the characteristics of the code itself, the details of the issue description, and things about the project as a whole.

Nadia: They concluded that vulnerable patches are most strongly tied to specific file changes and how much unique code is added or modified in a patch.

Elias: But they also found that issues where developers don't clearly explain what should happen or how to reproduce the bug are often where the AI messes up the most.

Priya: So, it means risk isn't just about finding one bad line of code; it’s about looking at the whole context of how much code is changing and how well the problem is described.

Nadia: Exactly. They suggest we need to build these prediction models that score risk based on those signals before an AI even proposes a fix.

Elias: It points toward integrating security checks right into those continuous delivery pipelines as an early warning system for these AI repairs.

More episodes

← Home