Analysis of Local Open-Weight LLM Agents on Vulnerability-Reproduction Tasks
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Security Radio. Generated commentary on the latest security and cryptography papers.
Nadia: We're looking at the paper: "Analysis of Local Open-Weight LLM Agents on Vulnerability-Reproduction Tasks".
Elias: Here's the gist of the paper. Open-weight language models that fit on a single consumer GPU are becoming capable on a growing range of tasks, and they matter because they can run locally without sending the code to a hosted service Reproducing vulnerabilities is closer to test generation than to code comprehension, requiring the agent to read a real code base, form a hypothesis about a defect, and construct an input that exercises it against a build it cannot compile itself The results show that the models almost always form the right hypothesis, and what decides whether a task is solved is whether that hypothesis becomes an input that reaches the vulnerable code The evaluation assessed twelve configurations of local models spanning three generations, two architectures and three quantization levels on ten CyberGym vulnerability-reproduction tasks Evaluation Setup The study evaluated twelve local open-weight model configurations spanning three generations, two architectures and three quantization levels on ten CyberGym vulnerability-reproduction tasks.
Nadia: Each configuration ran each of the ten tasks three times, giving 30 runs per configuration and 360 runs in total The agent received CyberGym’s own prompt verbatim, without modification.
Elias: CyberGym provides four difficulty levels that differ only in the information handed to the agent, ranging from Level 0 to Level 3.
Nadia: The evaluation loop involves the agent iterating by crafting candidate PoC inputs and submitting them to an execution server, which runs each candidate against the vulnerable and patched builds.
Paper summary: Nadia: So to summarize the paper "Analysis of Local Open-Weight LLM Agents on Vulnerability-Reproduction Tasks," they are checking if these local open-weight models can actually perform vulnerability reproduction on a set of ten CyberGym tasks. The authors are looking at twelve different setups—varying the model size, architecture, and how aggressively it’s quantized—to see what capabilities they have when running locally on consumer hardware.
Elias: The central claim is that these local models show competitive capabilities when performing tasks beyond just general-purpose coding. For instance, they found a 27B-parameter local model can get competitive results when writing code for embedded systems fifteen <ref:2611.06257#pg3,competitive results when writing code for embedded systems 15>.
Priya: So you’re pointing out that the evaluation isn't just on writing standard application code; it’s specifically on security tasks like vulnerability reproduction, which is a different kind of challenge three <ref:2611.06257#pg2>.
Nadia: Right. They found that the newest generation shows an improvement trend when looking at how much better the models are across those tasks. For example, Qwen three point six has hard-task success rates of seventy-three and thirty-three point three percent compared to Qwen three point five's similar rates one <ref:2611.06257#pg2>.
Elias: They also looked at how the model structure impacts things, finding that mixture of experts configurations are several times faster than the dense models of the same generation, which suggests architecture matters for speed too.
Priya: What about quantization? How did squeezing those models down to run on less memory affect their success rate on these security tasks?
Elias: Quantization changed how runs finish and how they fail more than what the model can actually solve, and this was mediated by the throughput of the setup one <ref:2611.06257#pg2>. They observed that two different four bit variants were on the same line, but one specific checkpoint was actually the best value for its generation when using dense twenty seven billion parameter models, though it was slower when using mixture of experts one <ref:2611.06257#pg2>.
Nadia: The study also looked at the actual paths taken by successful runs. They found three general patterns for how a run succeeds: first, a correct hypothesis translates into a correct file on the very first attempt; second, a correct hypothesis is carried by a malformed file that the agent later inspects and repairs; or third, it’s when the file itself gets rejected by the parser and then the agent resubmits different versions of it one <ref:2611.06257#pg2>.
Conclusion: Nadia: So looking at this paper, "Analysis of Local Open-Weight LLM Agents on Vulnerability-Reproduction Tasks," the main point is that open-weight models of this size are already useful for this kind of work and they are still improving. The strongest twenty seven billion parameter configuration reproduces two thirds of the discriminating tasks one <ref:2611.06257#pg2>.
Elias: That’s significant because that same strong configuration went from being one of those twelve runs under Qwen three point five to four under Qwen three point six one <ref:2611.06257#pg2>. It shows a real jump in capability just by moving to the newer generation.
Priya: So what does this mean for us, or for people who just listen to the show? It means that even without access to massive hosted services, we can get some decent security agents running locally on our own hardware today.
Nadia: Exactly. The paper suggests the next big improvement might come from the feedback that a larger model gives back to the agent, which is what they are testing next one <ref:2611.06257#pg2>.
Elias: It’s about making these local tools more reliable for complex security tasks, moving them from just interesting experiments to genuinely useful agents in practice.
Priya: The limitation they point out is that this study didn't systematically evaluate off-the-shelf open-weight models served locally on a single consumer grade GPU across all those sizes and architectures as general purpose agents one <ref:2611.06257#pg2>. They focused on this specific reproduction task.
Nadia: That’s the caveat. So while we see promise here, we still need more systematic testing to really map out what these local agents can do across every possible security scenario.
Jorge García-Carrascoa, Javier Sanchisa, Álvaro Navarro, Alejandro Maté Matéa, Juan Trujillo
Lucentia Research, Department of Software and Computing Systems, University of Alicante
cs.CR, cs.LG, cs.SE
Submitted: 2026-11-04
Updated: 2026-11-04
Comments: 35 pages, 6 figures, 5 tables. Preprint of a manuscript under review at the Journal of Systems and Software (Elsevier). Artefact: https://doi.org/10.57967/hf/10888
Code: https://github.com/sgl-project/sglang
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 92/100
The gist: Open-weight language models that fit on a single consumer GPU are becoming capable on a growing range of tasks…
Key concepts
- Vulnerability-Reproduction
- This task involves an agent reading existing code to guess a defect and then creating a specific input file that triggers that defect when run against the software. It tests the model's ability to understand code context and generate functional exploit inputs.
- Hypothesis Formation
- The core process where the AI agent reads real code, forms an educated guess about where a vulnerability exists, and constructs a specific test input based on that guess. The study found models are very good at forming the right initial hypothesis for these tasks.
- Quantization
- This refers to techniques used to reduce the size and computational requirements of large language models, such as using 4-bit versions instead of full precision. The research showed that quantization affects how long a run takes and how often it fails, depending on the specific model architecture.
- Discriminating Tasks
- These are the most difficult tasks in the evaluation set. They require more complex reasoning or larger code references to solve successfully. Success on these tasks indicates a model's higher capability for complex vulnerability reproduction.
Summary
Open-weight language models that fit on a single consumer GPU are becoming capable on a growing range of tasks, and they matter because they can run locally without sending the code to a hosted service Reproducing vulnerabilities is closer to test generation than to code comprehension, requiring the agent to read a real code base, form a hypothesis about a defect, and construct an input that exercises it against a build it cannot compile itself The results show that the models almost always form the right hypothesis, and what decides whether a task is solved is whether that hypothesis becomes an input that reaches the vulnerable code The evaluation assessed twelve configurations of local models spanning three generations, two architectures and three quantization levels on ten CyberGym vulnerability-reproduction tasks
Evaluation Setup
The study evaluated twelve local open-weight model configurations spanning three generations, two architectures and three quantization levels on ten CyberGym vulnerability-reproduction tasks. Each configuration ran each of the ten tasks three times, giving 30 runs per configuration and 360 runs in total The agent received CyberGym’s own prompt verbatim, without modification. p. 9 Evaluation protocol
Each configuration ran each of the ten tasks three times, giving 30 runs
per configuration and 360 runs in total. The agent received CyberGym’s
own prompt verbatim, without modification. A non-final submission returns crash feedback
to the agent so that it can refine its next candidate, whereas the outcome of
the final submission determines the run’s score. CyberGym provides four difficulty levels that differ only in the information handed to the agent:
• Level 0: the vulnerable source tree only. Benchmark and task selection
As introduced previously, each CyberGym task provides the agent with
the source tree of a vulnerable program and a description of the bug, and is
solved when the agent produces a single raw input that crashes the vulnerable
build but not the patched build. Figure 1 illustrates the evaluation loop
that every run follows: the agent iterates by crafting candidate PoC inputs
and submitting them to the execution server, which runs each candidate
against the vulnerable and patched builds.Where the paper says it
Where the paper says it
Where the paper says it
Task Difficulty and Discrimination
The researchers differentiated between six easier tasks and four discriminating tasks based on success rates pooled over all twelve configurations. The relationship between task difficulty is quantified by a Spearman correlation of ρ = −0. p. 2 However, to the best
of our knowledge, no work has systematically evaluated off-the-shelf openweight models served locally on a single consumer-grade GPU, across a range
of sizes, architectures and quantization levels, as general-purpose agents on
vulnerability reproduction. What makes this a software-engineering problem is where the difficulty
sits. We evaluate on the ten tasks that the CyberGym authors designate as
a representative subset of the suite, which spans over a thousand tasks and
deliberately balances vulnerabilities that frontier agents solve most of the
time with ones they rarely solve. We restrict the evaluation to this subset
because our protocol is expensive to run: with twelve configurations and
three trials, every task is executed thirty times, and each run comprises
tens of tool calls across a series of candidate attempts. We also examine the agent trajectories. Almost every run we
examined, including those that fail, names the vulnerable function and states
the correct hypothesis, and what separates a solved task from an unsolved
one is whether that hypothesis becomes a working input. Table 1: The ten tasks from the CyberGym recommended subset, with the project and
component each targets and the class of memory or logic defect its reference input triggers. Task ID Project Component Defect type
arvo:10400 GraphicsMagick MNG image decoder heap buffer overflow
arvo:1065 libmagic file-type matching use of uninitialised value
arvo:24993 libheif HEIC colour conversion heap buffer overflow
arvo:368 FreeType CFF font parser heap use-after-free
arvo:3938 YARA rule compiler type confusion
arvo:47101 binutils DWARF debug output heap buffer overflow
oss-fuzz:370689421 Wt expression evaluator double free
oss-fuzz:385167047 FFmpeg IP Movie demuxer use of uninitialised value
oss-fuzz:42535201 Assimp MD3 mesh importer heap buffer overflow
oss-fuzz:42535468 OpenSC PKCS#15 OpenPGP init heap buffer overflow
3.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Configuration Effects on Hard Tasks
The analysis focused on comparing twelve configurations across size, architecture, generation and quantization on the subset of discriminating tasks. Model size is a threshold rather than a gradient, as no configuration below 27B solved a hard task. p. 5 exploited any of the one-day vulnerabilities presented in an early evaluation,
while GPT-4 succeeded on the majority. Each of the studies above departs
from the setting we evaluate in at least one respect: the model is fine-tuned,
or served through a hosted endpoint, or a single configuration is examined
rather than a crossing of size, architecture and serving precision. Model configurations
We evaluate twelve configurations drawn from three generations of the
Qwen open-weight families [39]. They vary across four dimensions: generation (3. de Sant Vicent del Raspeig, s/n, Sant Vicent del Raspeig, 03690, Spain
Abstract
Open-weight language models that fit on a single consumer GPU are becoming capable on a growing range of tasks, and they matter because they
can run locally without sending the code to a hosted service. Vulnerability
reproduction, turning a vulnerability description into an input that crashes
the vulnerable build, is a natural target for them, since it is an automated
analysis task closer to test generation than to code comprehension and it runs
against a build that must be executed rather than read. CyberGym’s own harness uses a 3600 s cap; we doubled it because
the shorter budget truncated the slowest dense models disproportionately,
which would have biased the model-size comparison that is one of the objects of study. Model configurations
We evaluate twelve configurations drawn from three generations of the
Qwen open-weight families [39]. 8); total parameter count (4B, 9B, 27B and a 35B
mixture-of-experts (MoE) model with 3B active parameters); architecture
(dense against mixture-of-experts); and serving precision (BF16, bfloat16;
FP8, 8-bit floating point; INT4, 4-bit integer), where the 4-bit checkpoints
are GPTQ, a post-training quantization method for generative pre-trained
transformers, at group size 128 [40] and the FP8 checkpoints hold weights and
activations at FP8 with dynamic scaling. All configurations use the officially
released weights.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Role of Quantization and Trajectories
Quantization changed when runs finish and how they fail more than what the model can solve, mediated by throughput. The two 4-bit variants separate along the same line, with the 4-bit checkpoint being the best value of its generation on dense 27B but slower on mixture-of-experts. p. 9 They vary across four dimensions: generation (3. 8); total parameter count (4B, 9B, 27B and a 35B
mixture-of-experts (MoE) model with 3B active parameters); architecture
(dense against mixture-of-experts); and serving precision (BF16, bfloat16;
FP8, 8-bit floating point; INT4, 4-bit integer), where the 4-bit checkpoints
are GPTQ, a post-training quantization method for generative pre-trained
transformers, at group size 128 [40] and the FP8 checkpoints hold weights and
activations at FP8 with dynamic scaling. We also examine the agent trajectories. Almost every run we
examined, including those that fail, names the vulnerable function and states
the correct hypothesis, and what separates a solved task from an unsolved
one is whether that hypothesis becomes a working input.Where the paper says it
Where the paper says it
Conclusions on Model Utility
The larger conclusion is that open-weight models of this size are already useful for this work and are still improving. The strongest 27B configuration reproduces two thirds of the discriminating tasks, and the same configuration went from one of those twelve runs under Qwen 3. p. 0 In order to assess
whether current models can perform it, we evaluate twelve configurations of
local models, spanning three generations, two architectures and three quantization levels, on ten CyberGym vulnerability-reproduction tasks, three runs
each under one harness, and release the full artefact. We show that every configuration, down to the 4B model, solves most
of the six easier tasks, and configurations separate only on the remaining
four. In order to assess
whether current models can perform it, we evaluate twelve configurations of
local models, spanning three generations, two architectures and three quantization levels, on ten CyberGym vulnerability-reproduction tasks, three runs
each under one harness, and release the full artefact. We show that every configuration, down to the 4B model, solves most
of the six easier tasks, and configurations separate only on the remaining
four. The agent must decide whether
the submission is final or not. A non-final submission returns crash feedback
to the agent so that it can refine its next candidate, whereas the outcome of
the final submission determines the run’s score.Where the paper says it
Where the paper says it
Where the paper says it
Data Availability
The task trajectories, the submitted inputs and the per-submission results that support this study are openly available in Hugging Face at https://doi.org/10.57967/hf/10888. The deposit is released under CC BY 4. p. 3 A controlled evaluation of twelve local open-weight configurations (three
generations, two architectures, three quantization levels) on ten CyberGym vulnerability-reproduction tasks, yielding 360 runs and 3,501
submitted candidates. The full artefact: agent trajectories, every submitted input and the
per-submission outcomes on both builds, released as a public dataset
[19]. 5, and the difference appears only when the error
budget is tightened. In the security domain, however, evaluation remains
outcome-based. 4, at the level of individual runs
and submissions rather than aggregate scores. Study design
This section describes the design and setup of the experiments that were
carried out to study how local open-weight agents perform on the task of
Vulnerability reproduction.Where the paper says it
Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Adaptive configuration selection based on task difficulty: The system should dynamically choose model size, architecture, generation, and quantization level based on which part of a vulnerability reproduction task it is currently facing to maximize success probability. This addresses RQ3 by allowing the agent to select the
strongest configuration
when wall-clock budget does not bind. -
Improved artifact verification: Implement a mechanism where the agent inspects its own output when feedback does not localize the error, as demonstrated in trajectory case studies (
the runs that recover are the runs that inspect their own output
). This helps distinguish between genuine solution finding and procedural looping or resampling failures. -
Quantization-aware attempt management: The system should adjust its iteration strategy based on quantization settings to avoid wasting attempts on inputs that have already been rejected by the execution server, particularly when using 4-bit weights where
the budget does not bound those runs first.
Abstract
Open-weight language models that fit on a single consumer GPU are becoming capable on a growing range of tasks, and they matter because they can run locally without sending the code to a hosted service. Vulnerability reproduction, turning a vulnerability description into an input that crashes the vulnerable build, is a natural target for them, since it is an automated analysis task closer to test generation than to code comprehension and it runs against a build that must be executed rather than read. In order to assess whether current models can perform it, we evaluate twelve configurations of local models, spanning three generations, two architectures and three quantization levels, on ten CyberGym vulnerability-reproduction tasks, three runs each under one harness, and release the full artefact. We show that every configuration, down to the 4B model, solves most of the six easier tasks, and configurations separate only on the remaining four. For the harder tasks, the 27B model is required. Quantization did not change what the models solved in any matched pair, but it changed how fast they got there, and the mixture of experts reached the same scores in less wall clock. We also examine the agent trajectories. Almost every run we examined, including those that fail, names the vulnerable function and states the correct hypothesis, and what separates a solved task from an unsolved one is whether that hypothesis becomes a working input.
Sources
- Scaling Laws for Neural Language Models
- Code Llama: Open Foundation Models for Code
- ReAct: Synergizing Reasoning and Acting in Language Models
- SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering
- VulnGym: Benchmarking Coding Agents for Repository-Level Vulnerability Detection
- SEC-bench: Automated Benchmarking of LLM Agents on Real-World Software Security Tasks
- From CVE Entries to Verifiable Exploits: An Automated Multi-Agent Framework for Reproducing CVEs
- CyberGym: Evaluating AI Agents' Real-World Cybersecurity Capabilities at Scale
- GLM-5: from Vibe Coding to Agentic Engineering
- Kimi K3: Open Frontier Intelligence
- DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
- Neuro-Symbolic Proof-of-Vulnerability Generation with Open-Weight Models
- ReproBench: Benchmarking LLM Agents on Reproducing Vulnerability From Scratch
- LLM Agents can Autonomously Exploit One-day Vulnerabilities
- ReposVul: A Repository-Level High-Quality Vulnerability Dataset
- VulEval: Towards Repository-Level Evaluation of Software Vulnerability Detection
- CVE-Bench: A Benchmark for AI Agents' Ability to Exploit Real-World Web Application Vulnerabilities
- All You Need Is A Fuzzing Brain: An LLM-Powered System for Automated Vulnerability Detection and Patching
- AutoCodeRover: Autonomous Program Improvement
- Agentless: Demystifying LLM-based Software Engineering Agents
Related papers
- From Strategy Search to Policy Learning: MCTS-Guided Adversarial Co-Evolution for LLM Safety Alignment
- Privacy Leakage from Recurrent State in LLM Inference
- Incrementally Verifiable Oracle Aided Computation
- Privilege and confidentiality in generative AI workflows
- BadMoE: Backdooring Mixture-of-Experts LLMs via Routing-Aware Triggers and Task-Underutilized Experts
- CoVault: Secure, Scalable Analytics of Personal Data