Analysis of Local Open-Weight LLM Agents on Vulnerability-Reproduction Tasks

summary

The gist

Open-weight language models that fit on a single consumer GPU are becoming capable on a growing range of tasks…

In short

The study tested twelve local open-weight language models against ten CyberGym tasks designed to reproduce software vulnerabilities. Models consistently form correct hypotheses about defects, and success depends on whether that hypothesis generates a valid input that reaches the vulnerable code. The largest 27B model configuration performed best, showing improvement with newer generations.

Key concepts

Vulnerability-Reproduction
This task involves an agent reading existing code to guess a defect and then creating a specific input file that triggers that defect when run against the software. It tests the model's ability to understand code context and generate functional exploit inputs.
Hypothesis Formation
The core process where the AI agent reads real code, forms an educated guess about where a vulnerability exists, and constructs a specific test input based on that guess. The study found models are very good at forming the right initial hypothesis for these tasks.
Quantization
This refers to techniques used to reduce the size and computational requirements of large language models, such as using 4-bit versions instead of full precision. The research showed that quantization affects how long a run takes and how often it fails, depending on the specific model architecture.
Discriminating Tasks
These are the most difficult tasks in the evaluation set. They require more complex reasoning or larger code references to solve successfully. Success on these tasks indicates a model's higher capability for complex vulnerability reproduction.

Terminology used across episodes

This episode discusses

The paper

Analysis of Local Open-Weight LLM Agents on Vulnerability-Reproduction Tasks · Read on arXiv

Jorge García-Carrascoa, Javier Sanchisa, Álvaro Navarro, Alejandro Maté Matéa, Juan Trujillo

Lucentia Research, Department of Software and Computing Systems, University of Alicante

Open-weight language models that fit on a single consumer GPU are becoming capable on a growing range of tasks, and they matter because they can run locally without sending the code to a hosted service. Vulnerability reproduction, turning a vulnerability description into an input that crashes the vulnerable build, is a natural target for them, since it is an automated analysis task closer to test generation than to code comprehension and it runs against a build that must be executed rather than read. In order to assess whether current models can perform it, we evaluate twelve configurations of local models, spanning three generations, two architectures and three quantization levels, on ten CyberGym vulnerability-reproduction tasks, three runs each under one harness, and release the full artefact. We show that every configuration, down to the 4B model, solves most of the six easier tasks, and configurations separate only on the remaining four. For the harder tasks, the 27B model is required. Quantization did not change what the models solved in any matched pair, but it changed how fast they got there, and the mixture of experts reached the same scores in less wall clock. We also examine the agent trajectories. Almost every run we examined, including those that fail, names the vulnerable function and states the correct hypothesis, and what separates a solved task from an unsolved one is whether that hypothesis becomes a working input.

Transcript

Introduction to the show: ident: Security Radio. Generated commentary on the latest security and cryptography papers.

Nadia: We're looking at the paper: "Analysis of Local Open-Weight LLM Agents on Vulnerability-Reproduction Tasks".

Elias: Here's the gist of the paper. Open-weight language models that fit on a single consumer GPU are becoming capable on a growing range of tasks, and they matter because they can run locally without sending the code to a hosted service Reproducing vulnerabilities is closer to test generation than to code comprehension, requiring the agent to read a real code base, form a hypothesis about a defect, and construct an input that exercises it against a build it cannot compile itself The results show that the models almost always form the right hypothesis, and what decides whether a task is solved is whether that hypothesis becomes an input that reaches the vulnerable code The evaluation assessed twelve configurations of local models spanning three generations, two architectures and three quantization levels on ten CyberGym vulnerability-reproduction tasks Evaluation Setup The study evaluated twelve local open-weight model configurations spanning three generations, two architectures and three quantization levels on ten CyberGym vulnerability-reproduction tasks.

Nadia: Each configuration ran each of the ten tasks three times, giving 30 runs per configuration and 360 runs in total The agent received CyberGym’s own prompt verbatim, without modification.

Elias: CyberGym provides four difficulty levels that differ only in the information handed to the agent, ranging from Level 0 to Level 3.

Nadia: The evaluation loop involves the agent iterating by crafting candidate PoC inputs and submitting them to an execution server, which runs each candidate against the vulnerable and patched builds.

Paper summary: Nadia: So to summarize the paper "Analysis of Local Open-Weight LLM Agents on Vulnerability-Reproduction Tasks," they are checking if these local open-weight models can actually perform vulnerability reproduction on a set of ten CyberGym tasks. The authors are looking at twelve different setups—varying the model size, architecture, and how aggressively it’s quantized—to see what capabilities they have when running locally on consumer hardware.

Elias: The central claim is that these local models show competitive capabilities when performing tasks beyond just general-purpose coding. For instance, they found a 27B-parameter local model can get competitive results when writing code for embedded systems fifteen <ref:2611.06257#pg3,competitive results when writing code for embedded systems 15>.

Priya: So you’re pointing out that the evaluation isn't just on writing standard application code; it’s specifically on security tasks like vulnerability reproduction, which is a different kind of challenge three <ref:2611.06257#pg2>.

Nadia: Right. They found that the newest generation shows an improvement trend when looking at how much better the models are across those tasks. For example, Qwen three point six has hard-task success rates of seventy-three and thirty-three point three percent compared to Qwen three point five's similar rates one <ref:2611.06257#pg2>.

Elias: They also looked at how the model structure impacts things, finding that mixture of experts configurations are several times faster than the dense models of the same generation, which suggests architecture matters for speed too.

Priya: What about quantization? How did squeezing those models down to run on less memory affect their success rate on these security tasks?

Elias: Quantization changed how runs finish and how they fail more than what the model can actually solve, and this was mediated by the throughput of the setup one <ref:2611.06257#pg2>. They observed that two different four bit variants were on the same line, but one specific checkpoint was actually the best value for its generation when using dense twenty seven billion parameter models, though it was slower when using mixture of experts one <ref:2611.06257#pg2>.

Nadia: The study also looked at the actual paths taken by successful runs. They found three general patterns for how a run succeeds: first, a correct hypothesis translates into a correct file on the very first attempt; second, a correct hypothesis is carried by a malformed file that the agent later inspects and repairs; or third, it’s when the file itself gets rejected by the parser and then the agent resubmits different versions of it one <ref:2611.06257#pg2>.

Conclusion: Nadia: So looking at this paper, "Analysis of Local Open-Weight LLM Agents on Vulnerability-Reproduction Tasks," the main point is that open-weight models of this size are already useful for this kind of work and they are still improving. The strongest twenty seven billion parameter configuration reproduces two thirds of the discriminating tasks one <ref:2611.06257#pg2>.

Elias: That’s significant because that same strong configuration went from being one of those twelve runs under Qwen three point five to four under Qwen three point six one <ref:2611.06257#pg2>. It shows a real jump in capability just by moving to the newer generation.

Priya: So what does this mean for us, or for people who just listen to the show? It means that even without access to massive hosted services, we can get some decent security agents running locally on our own hardware today.

Nadia: Exactly. The paper suggests the next big improvement might come from the feedback that a larger model gives back to the agent, which is what they are testing next one <ref:2611.06257#pg2>.

Elias: It’s about making these local tools more reliable for complex security tasks, moving them from just interesting experiments to genuinely useful agents in practice.

Priya: The limitation they point out is that this study didn't systematically evaluate off-the-shelf open-weight models served locally on a single consumer grade GPU across all those sizes and architectures as general purpose agents one <ref:2611.06257#pg2>. They focused on this specific reproduction task.

Nadia: That’s the caveat. So while we see promise here, we still need more systematic testing to really map out what these local agents can do across every possible security scenario.

More episodes

← Home