Analysis of Local Open-Weight LLM Agents on Vulnerability-Reproduction Tasks

arXiv:2611.06257 · cs.CR, cs.LG, cs.SE · Submitted 2026-11-04 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Security Radio. Generated commentary on the latest security and cryptography papers.

Nadia: We're looking at the paper: "Analysis of Local Open-Weight LLM Agents on Vulnerability-Reproduction Tasks".

Elias: Here's the gist of the paper. Open-weight language models that fit on a single consumer GPU are becoming capable on a growing range of tasks, and they matter because they can run locally without sending the code to a hosted service Reproducing vulnerabilities is closer to test generation than to code comprehension, requiring the agent to read a real code base, form a hypothesis about a defect, and construct an input that exercises it against a build it cannot compile itself The results show that the models almost always form the right hypothesis, and what decides whether a task is solved is whether that hypothesis becomes an input that reaches the vulnerable code The evaluation assessed twelve configurations of local models spanning three generations, two architectures and three quantization levels on ten CyberGym vulnerability-reproduction tasks Evaluation Setup The study evaluated twelve local open-weight model configurations spanning three generations, two architectures and three quantization levels on ten CyberGym vulnerability-reproduction tasks.

Nadia: Each configuration ran each of the ten tasks three times, giving 30 runs per configuration and 360 runs in total The agent received CyberGym’s own prompt verbatim, without modification.

Elias: CyberGym provides four difficulty levels that differ only in the information handed to the agent, ranging from Level 0 to Level 3.

Nadia: The evaluation loop involves the agent iterating by crafting candidate PoC inputs and submitting them to an execution server, which runs each candidate against the vulnerable and patched builds.

Paper summary: Nadia: So to summarize the paper "Analysis of Local Open-Weight LLM Agents on Vulnerability-Reproduction Tasks," they are checking if these local open-weight models can actually perform vulnerability reproduction on a set of ten CyberGym tasks. The authors are looking at twelve different setups—varying the model size, architecture, and how aggressively it’s quantized—to see what capabilities they have when running locally on consumer hardware.

Elias: The central claim is that these local models show competitive capabilities when performing tasks beyond just general-purpose coding. For instance, they found a 27B-parameter local model can get competitive results when writing code for embedded systems fifteen <ref:2611.06257#pg3,competitive results when writing code for embedded systems 15>.

Priya: So you’re pointing out that the evaluation isn't just on writing standard application code; it’s specifically on security tasks like vulnerability reproduction, which is a different kind of challenge three <ref:2611.06257#pg2>.

Nadia: Right. They found that the newest generation shows an improvement trend when looking at how much better the models are across those tasks. For example, Qwen three point six has hard-task success rates of seventy-three and thirty-three point three percent compared to Qwen three point five's similar rates one <ref:2611.06257#pg2>.

Elias: They also looked at how the model structure impacts things, finding that mixture of experts configurations are several times faster than the dense models of the same generation, which suggests architecture matters for speed too.

Priya: What about quantization? How did squeezing those models down to run on less memory affect their success rate on these security tasks?

Elias: Quantization changed how runs finish and how they fail more than what the model can actually solve, and this was mediated by the throughput of the setup one <ref:2611.06257#pg2>. They observed that two different four bit variants were on the same line, but one specific checkpoint was actually the best value for its generation when using dense twenty seven billion parameter models, though it was slower when using mixture of experts one <ref:2611.06257#pg2>.

Nadia: The study also looked at the actual paths taken by successful runs. They found three general patterns for how a run succeeds: first, a correct hypothesis translates into a correct file on the very first attempt; second, a correct hypothesis is carried by a malformed file that the agent later inspects and repairs; or third, it’s when the file itself gets rejected by the parser and then the agent resubmits different versions of it one <ref:2611.06257#pg2>.

Conclusion: Nadia: So looking at this paper, "Analysis of Local Open-Weight LLM Agents on Vulnerability-Reproduction Tasks," the main point is that open-weight models of this size are already useful for this kind of work and they are still improving. The strongest twenty seven billion parameter configuration reproduces two thirds of the discriminating tasks one <ref:2611.06257#pg2>.

Elias: That’s significant because that same strong configuration went from being one of those twelve runs under Qwen three point five to four under Qwen three point six one <ref:2611.06257#pg2>. It shows a real jump in capability just by moving to the newer generation.

Priya: So what does this mean for us, or for people who just listen to the show? It means that even without access to massive hosted services, we can get some decent security agents running locally on our own hardware today.

Nadia: Exactly. The paper suggests the next big improvement might come from the feedback that a larger model gives back to the agent, which is what they are testing next one <ref:2611.06257#pg2>.

Elias: It’s about making these local tools more reliable for complex security tasks, moving them from just interesting experiments to genuinely useful agents in practice.

Priya: The limitation they point out is that this study didn't systematically evaluate off-the-shelf open-weight models served locally on a single consumer grade GPU across all those sizes and architectures as general purpose agents one <ref:2611.06257#pg2>. They focused on this specific reproduction task.

Nadia: That’s the caveat. So while we see promise here, we still need more systematic testing to really map out what these local agents can do across every possible security scenario.

Jorge García-Carrascoa, Javier Sanchisa, Álvaro Navarro, Alejandro Maté Matéa, Juan Trujillo

Lucentia Research, Department of Software and Computing Systems, University of Alicante

cs.CR, cs.LG, cs.SE

Submitted: 2026-11-04

Updated: 2026-11-04

Comments: 35 pages, 6 figures, 5 tables. Preprint of a manuscript under review at the Journal of Systems and Software (Elsevier). Artefact: https://doi.org/10.57967/hf/10888

Code: https://github.com/sgl-project/sglang

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 92/100

The gist: Open-weight language models that fit on a single consumer GPU are becoming capable on a growing range of tasks…

Key concepts

Vulnerability-Reproduction
This task involves an agent reading existing code to guess a defect and then creating a specific input file that triggers that defect when run against the software. It tests the model's ability to understand code context and generate functional exploit inputs.
Hypothesis Formation
The core process where the AI agent reads real code, forms an educated guess about where a vulnerability exists, and constructs a specific test input based on that guess. The study found models are very good at forming the right initial hypothesis for these tasks.
Quantization
This refers to techniques used to reduce the size and computational requirements of large language models, such as using 4-bit versions instead of full precision. The research showed that quantization affects how long a run takes and how often it fails, depending on the specific model architecture.
Discriminating Tasks
These are the most difficult tasks in the evaluation set. They require more complex reasoning or larger code references to solve successfully. Success on these tasks indicates a model's higher capability for complex vulnerability reproduction.

Summary

Open-weight language models that fit on a single consumer GPU are becoming capable on a growing range of tasks, and they matter because they can run locally without sending the code to a hosted service Reproducing vulnerabilities is closer to test generation than to code comprehension, requiring the agent to read a real code base, form a hypothesis about a defect, and construct an input that exercises it against a build it cannot compile itself The results show that the models almost always form the right hypothesis, and what decides whether a task is solved is whether that hypothesis becomes an input that reaches the vulnerable code The evaluation assessed twelve configurations of local models spanning three generations, two architectures and three quantization levels on ten CyberGym vulnerability-reproduction tasks

Evaluation Setup

The study evaluated twelve local open-weight model configurations spanning three generations, two architectures and three quantization levels on ten CyberGym vulnerability-reproduction tasks. Each configuration ran each of the ten tasks three times, giving 30 runs per configuration and 360 runs in total The agent received CyberGym’s own prompt verbatim, without modification. p. 9

Where the paper says it

Evaluation protocol Each configuration ran each of the ten tasks three times, giving 30 runs per configuration and 360 runs in total. The agent received CyberGym’s own prompt verbatim, without modification.

CyberGym provides four difficulty levels that differ only in the information handed to the agent, ranging from Level 0 to Level 3. p. 6
Where the paper says it

A non-final submission returns crash feedback to the agent so that it can refine its next candidate, whereas the outcome of the final submission determines the run’s score. CyberGym provides four difficulty levels that differ only in the information handed to the agent: • Level 0: the vulnerable source tree only.

The evaluation loop involves the agent iterating by crafting candidate PoC inputs and submitting them to an execution server, which runs each candidate against the vulnerable and patched builds. p. 6
Where the paper says it

Benchmark and task selection As introduced previously, each CyberGym task provides the agent with the source tree of a vulnerable program and a description of the bug, and is solved when the agent produces a single raw input that crashes the vulnerable build but not the patched build. Figure 1 illustrates the evaluation loop that every run follows: the agent iterates by crafting candidate PoC inputs and submitting them to the execution server, which runs each candidate against the vulnerable and patched builds.

Task Difficulty and Discrimination

The researchers differentiated between six easier tasks and four discriminating tasks based on success rates pooled over all twelve configurations. The relationship between task difficulty is quantified by a Spearman correlation of ρ = −0. p. 2

Where the paper says it

However, to the best of our knowledge, no work has systematically evaluated off-the-shelf openweight models served locally on a single consumer-grade GPU, across a range of sizes, architectures and quantization levels, as general-purpose agents on vulnerability reproduction. What makes this a software-engineering problem is where the difficulty sits.

81 (p = 0.004) across all ten tasks. p. 7
Where the paper says it

We evaluate on the ten tasks that the CyberGym authors designate as a representative subset of the suite, which spans over a thousand tasks and deliberately balances vulnerabilities that frontier agents solve most of the time with ones they rarely solve. We restrict the evaluation to this subset because our protocol is expensive to run: with twelve configurations and three trials, every task is executed thirty times, and each run comprises tens of tool calls across a series of candidate attempts.

Tasks with small references are solved by almost every configuration, while tasks whose witnesses run to tens or hundreds of kilobytes are solved only by the strongest. p. 0
Where the paper says it

We also examine the agent trajectories. Almost every run we examined, including those that fail, names the vulnerable function and states the correct hypothesis, and what separates a solved task from an unsolved one is whether that hypothesis becomes a working input.

The two hollow squares in Figure 3 represent the tasks whose reference PoC is an empty file, meaning almost any input satisfies them. p. 8
Where the paper says it

Table 1: The ten tasks from the CyberGym recommended subset, with the project and component each targets and the class of memory or logic defect its reference input triggers. Task ID Project Component Defect type arvo:10400 GraphicsMagick MNG image decoder heap buffer overflow arvo:1065 libmagic file-type matching use of uninitialised value arvo:24993 libheif HEIC colour conversion heap buffer overflow arvo:368 FreeType CFF font parser heap use-after-free arvo:3938 YARA rule compiler type confusion arvo:47101 binutils DWARF debug output heap buffer overflow oss-fuzz:370689421 Wt expression evaluator double free oss-fuzz:385167047 FFmpeg IP Movie demuxer use of uninitialised value oss-fuzz:42535201 Assimp MD3 mesh importer heap buffer overflow oss-fuzz:42535468 OpenSC PKCS#15 OpenPGP init heap buffer overflow 3.

Configuration Effects on Hard Tasks

The analysis focused on comparing twelve configurations across size, architecture, generation and quantization on the subset of discriminating tasks. Model size is a threshold rather than a gradient, as no configuration below 27B solved a hard task. p. 5

Where the paper says it

exploited any of the one-day vulnerabilities presented in an early evaluation, while GPT-4 succeeded on the majority. Each of the studies above departs from the setting we evaluate in at least one respect: the model is fine-tuned, or served through a hosted endpoint, or a single configuration is examined rather than a crossing of size, architecture and serving precision.

The newest generation shows an improvement trend with the generation axis, where Qwen 3. p. 9
Where the paper says it

Model configurations We evaluate twelve configurations drawn from three generations of the Qwen open-weight families [39]. They vary across four dimensions: generation (3.

6 has hard-task rates of 7. p. 0
Where the paper says it

de Sant Vicent del Raspeig, s/n, Sant Vicent del Raspeig, 03690, Spain Abstract Open-weight language models that fit on a single consumer GPU are becoming capable on a growing range of tasks, and they matter because they can run locally without sending the code to a hosted service. Vulnerability reproduction, turning a vulnerability description into an input that crashes the vulnerable build, is a natural target for them, since it is an automated analysis task closer to test generation than to code comprehension and it runs against a build that must be executed rather than read.

3 and 33.3 percent compared to Qwen 3. p. 9
Where the paper says it

CyberGym’s own harness uses a 3600 s cap; we doubled it because the shorter budget truncated the slowest dense models disproportionately, which would have biased the model-size comparison that is one of the objects of study. Model configurations We evaluate twelve configurations drawn from three generations of the Qwen open-weight families [39].

5's similar rates. Architecture changes cost and not capability, as the mixture-of-experts configurations are several times faster than the dense models of the same generation. p. 9
Where the paper says it

8); total parameter count (4B, 9B, 27B and a 35B mixture-of-experts (MoE) model with 3B active parameters); architecture (dense against mixture-of-experts); and serving precision (BF16, bfloat16; FP8, 8-bit floating point; INT4, 4-bit integer), where the 4-bit checkpoints are GPTQ, a post-training quantization method for generative pre-trained transformers, at group size 128 [40] and the FP8 checkpoints hold weights and activations at FP8 with dynamic scaling. All configurations use the officially released weights.

Role of Quantization and Trajectories

Quantization changed when runs finish and how they fail more than what the model can solve, mediated by throughput. The two 4-bit variants separate along the same line, with the 4-bit checkpoint being the best value of its generation on dense 27B but slower on mixture-of-experts. p. 9

Where the paper says it

They vary across four dimensions: generation (3. 8); total parameter count (4B, 9B, 27B and a 35B mixture-of-experts (MoE) model with 3B active parameters); architecture (dense against mixture-of-experts); and serving precision (BF16, bfloat16; FP8, 8-bit floating point; INT4, 4-bit integer), where the 4-bit checkpoints are GPTQ, a post-training quantization method for generative pre-trained transformers, at group size 128 [40] and the FP8 checkpoints hold weights and activations at FP8 with dynamic scaling.

Trajectory case studies revealed three general patterns for successful runs: a correct hypothesis translated into a correct file on the first attempt, a correct hypothesis carried by a malformed file that the agent later inspects and repairs, and a file the parser rejects while the agent resubmits variants of it. p. 0
Where the paper says it

We also examine the agent trajectories. Almost every run we examined, including those that fail, names the vulnerable function and states the correct hypothesis, and what separates a solved task from an unsolved one is whether that hypothesis becomes a working input.

Conclusions on Model Utility

The larger conclusion is that open-weight models of this size are already useful for this work and are still improving. The strongest 27B configuration reproduces two thirds of the discriminating tasks, and the same configuration went from one of those twelve runs under Qwen 3. p. 0

Where the paper says it

In order to assess whether current models can perform it, we evaluate twelve configurations of local models, spanning three generations, two architectures and three quantization levels, on ten CyberGym vulnerability-reproduction tasks, three runs each under one harness, and release the full artefact. We show that every configuration, down to the 4B model, solves most of the six easier tasks, and configurations separate only on the remaining four.

5 to four under 3. p. 0
Where the paper says it

In order to assess whether current models can perform it, we evaluate twelve configurations of local models, spanning three generations, two architectures and three quantization levels, on ten CyberGym vulnerability-reproduction tasks, three runs each under one harness, and release the full artefact. We show that every configuration, down to the 4B model, solves most of the six easier tasks, and configurations separate only on the remaining four.

6. The next improvement is likely to come from the feedback the agent receives as from a larger model. p. 6
Where the paper says it

The agent must decide whether the submission is final or not. A non-final submission returns crash feedback to the agent so that it can refine its next candidate, whereas the outcome of the final submission determines the run’s score.

Data Availability

The task trajectories, the submitted inputs and the per-submission results that support this study are openly available in Hugging Face at https://doi.org/10.57967/hf/10888. The deposit is released under CC BY 4. p. 3

Where the paper says it

A controlled evaluation of twelve local open-weight configurations (three generations, two architectures, three quantization levels) on ten CyberGym vulnerability-reproduction tasks, yielding 360 runs and 3,501 submitted candidates. The full artefact: agent trajectories, every submitted input and the per-submission outcomes on both builds, released as a public dataset [19].

0; the CyberGym material redistributed in it remains under CyberGym’s Apache-2. p. 6
Where the paper says it

5, and the difference appears only when the error budget is tightened. In the security domain, however, evaluation remains outcome-based.

0 licence. The funding bodies had no role in the study design, data collection, analysis, interpretation, manuscript preparation, or decision to submit. p. 6
Where the paper says it

4, at the level of individual runs and submissions rather than aggregate scores. Study design This section describes the design and setup of the experiments that were carried out to study how local open-weight agents perform on the task of Vulnerability reproduction.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Adaptive configuration selection based on task difficulty: The system should dynamically choose model size, architecture, generation, and quantization level based on which part of a vulnerability reproduction task it is currently facing to maximize success probability. This addresses RQ3 by allowing the agent to select the strongest configuration when wall-clock budget does not bind.

  2. Improved artifact verification: Implement a mechanism where the agent inspects its own output when feedback does not localize the error, as demonstrated in trajectory case studies (the runs that recover are the runs that inspect their own output). This helps distinguish between genuine solution finding and procedural looping or resampling failures.

  3. Quantization-aware attempt management: The system should adjust its iteration strategy based on quantization settings to avoid wasting attempts on inputs that have already been rejected by the execution server, particularly when using 4-bit weights where the budget does not bound those runs first.

Abstract

Open-weight language models that fit on a single consumer GPU are becoming capable on a growing range of tasks, and they matter because they can run locally without sending the code to a hosted service. Vulnerability reproduction, turning a vulnerability description into an input that crashes the vulnerable build, is a natural target for them, since it is an automated analysis task closer to test generation than to code comprehension and it runs against a build that must be executed rather than read. In order to assess whether current models can perform it, we evaluate twelve configurations of local models, spanning three generations, two architectures and three quantization levels, on ten CyberGym vulnerability-reproduction tasks, three runs each under one harness, and release the full artefact. We show that every configuration, down to the 4B model, solves most of the six easier tasks, and configurations separate only on the remaining four. For the harder tasks, the 27B model is required. Quantization did not change what the models solved in any matched pair, but it changed how fast they got there, and the mixture of experts reached the same scores in less wall clock. We also examine the agent trajectories. Almost every run we examined, including those that fail, names the vulnerable function and states the correct hypothesis, and what separates a solved task from an unsolved one is whether that hypothesis becomes a working input.

Sources

Related papers