Analysis of Local Open-Weight LLM Agents on Vulnerability-Reproduction Tasks
summary
The gist
Open-weight language models that fit on a single consumer GPU are becoming capable on a growing range of tasks…
In short
The study tested twelve local open-weight language models against ten CyberGym tasks designed to reproduce software vulnerabilities. Models consistently form correct hypotheses about defects, and success depends on whether that hypothesis generates a valid input that reaches the vulnerable code. The largest 27B model configuration performed best, showing improvement with newer generations.
Key concepts
- Vulnerability-Reproduction
- This task involves an agent reading existing code to guess a defect and then creating a specific input file that triggers that defect when run against the software. It tests the model's ability to understand code context and generate functional exploit inputs.
- Hypothesis Formation
- The core process where the AI agent reads real code, forms an educated guess about where a vulnerability exists, and constructs a specific test input based on that guess. The study found models are very good at forming the right initial hypothesis for these tasks.
- Quantization
- This refers to techniques used to reduce the size and computational requirements of large language models, such as using 4-bit versions instead of full precision. The research showed that quantization affects how long a run takes and how often it fails, depending on the specific model architecture.
- Discriminating Tasks
- These are the most difficult tasks in the evaluation set. They require more complex reasoning or larger code references to solve successfully. Success on these tasks indicates a model's higher capability for complex vulnerability reproduction.
Terminology used across episodes
This episode discusses
- Analysis of Local Open-Weight LLM Agents on Vulnerability-Reproduction Tasks · Paper Radio
- Scaling Laws for Neural Language Models
- Code Llama: Open Foundation Models for Code
- ReAct: Synergizing Reasoning and Acting in Language Models
- SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering
- VulnGym: Benchmarking Coding Agents for Repository-Level Vulnerability Detection
- SEC-bench: Automated Benchmarking of LLM Agents on Real-World Software Security Tasks
- From CVE Entries to Verifiable Exploits: An Automated Multi-Agent Framework for Reproducing CVEs
- CyberGym: Evaluating AI Agents' Real-World Cybersecurity Capabilities at Scale
- GLM-5: from Vibe Coding to Agentic Engineering
- Kimi K3: Open Frontier Intelligence
- DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
- Neuro-Symbolic Proof-of-Vulnerability Generation with Open-Weight Models
- ReproBench: Benchmarking LLM Agents on Reproducing Vulnerability From Scratch · Paper Radio
- LLM Agents can Autonomously Exploit One-day Vulnerabilities
- ReposVul: A Repository-Level High-Quality Vulnerability Dataset
- VulEval: Towards Repository-Level Evaluation of Software Vulnerability Detection
- CVE-Bench: A Benchmark for AI Agents' Ability to Exploit Real-World Web Application Vulnerabilities
- All You Need Is A Fuzzing Brain: An LLM-Powered System for Automated Vulnerability Detection and Patching
- AutoCodeRover: Autonomous Program Improvement
- Agentless: Demystifying LLM-based Software Engineering Agents
The paper
Analysis of Local Open-Weight LLM Agents on Vulnerability-Reproduction Tasks · Read on arXiv
Jorge García-Carrascoa, Javier Sanchisa, Álvaro Navarro, Alejandro Maté Matéa, Juan Trujillo
Lucentia Research, Department of Software and Computing Systems, University of Alicante
Open-weight language models that fit on a single consumer GPU are becoming capable on a growing range of tasks, and they matter because they can run locally without sending the code to a hosted service. Vulnerability reproduction, turning a vulnerability description into an input that crashes the vulnerable build, is a natural target for them, since it is an automated analysis task closer to test generation than to code comprehension and it runs against a build that must be executed rather than read. In order to assess whether current models can perform it, we evaluate twelve configurations of local models, spanning three generations, two architectures and three quantization levels, on ten CyberGym vulnerability-reproduction tasks, three runs each under one harness, and release the full artefact. We show that every configuration, down to the 4B model, solves most of the six easier tasks, and configurations separate only on the remaining four. For the harder tasks, the 27B model is required. Quantization did not change what the models solved in any matched pair, but it changed how fast they got there, and the mixture of experts reached the same scores in less wall clock. We also examine the agent trajectories. Almost every run we examined, including those that fail, names the vulnerable function and states the correct hypothesis, and what separates a solved task from an unsolved one is whether that hypothesis becomes a working input.
Transcript
Introduction to the show: ident: Security Radio. Generated commentary on the latest security and cryptography papers.
Nadia: We're looking at the paper: "Analysis of Local Open-Weight LLM Agents on Vulnerability-Reproduction Tasks".
Elias: Here's the gist of the paper. Open-weight language models that fit on a single consumer GPU are becoming capable on a growing range of tasks, and they matter because they can run locally without sending the code to a hosted service Reproducing vulnerabilities is closer to test generation than to code comprehension, requiring the agent to read a real code base, form a hypothesis about a defect, and construct an input that exercises it against a build it cannot compile itself The results show that the models almost always form the right hypothesis, and what decides whether a task is solved is whether that hypothesis becomes an input that reaches the vulnerable code The evaluation assessed twelve configurations of local models spanning three generations, two architectures and three quantization levels on ten CyberGym vulnerability-reproduction tasks Evaluation Setup The study evaluated twelve local open-weight model configurations spanning three generations, two architectures and three quantization levels on ten CyberGym vulnerability-reproduction tasks.
Nadia: Each configuration ran each of the ten tasks three times, giving 30 runs per configuration and 360 runs in total The agent received CyberGym’s own prompt verbatim, without modification.
Elias: CyberGym provides four difficulty levels that differ only in the information handed to the agent, ranging from Level 0 to Level 3.
Nadia: The evaluation loop involves the agent iterating by crafting candidate PoC inputs and submitting them to an execution server, which runs each candidate against the vulnerable and patched builds.
Paper summary: Nadia: So to summarize the paper "Analysis of Local Open-Weight LLM Agents on Vulnerability-Reproduction Tasks," they are checking if these local open-weight models can actually perform vulnerability reproduction on a set of ten CyberGym tasks. The authors are looking at twelve different setups—varying the model size, architecture, and how aggressively it’s quantized—to see what capabilities they have when running locally on consumer hardware.
Elias: The central claim is that these local models show competitive capabilities when performing tasks beyond just general-purpose coding. For instance, they found a 27B-parameter local model can get competitive results when writing code for embedded systems fifteen <ref:2611.06257#pg3,competitive results when writing code for embedded systems 15>.
Priya: So you’re pointing out that the evaluation isn't just on writing standard application code; it’s specifically on security tasks like vulnerability reproduction, which is a different kind of challenge three <ref:2611.06257#pg2>.
Nadia: Right. They found that the newest generation shows an improvement trend when looking at how much better the models are across those tasks. For example, Qwen three point six has hard-task success rates of seventy-three and thirty-three point three percent compared to Qwen three point five's similar rates one <ref:2611.06257#pg2>.
Elias: They also looked at how the model structure impacts things, finding that mixture of experts configurations are several times faster than the dense models of the same generation, which suggests architecture matters for speed too.
Priya: What about quantization? How did squeezing those models down to run on less memory affect their success rate on these security tasks?
Elias: Quantization changed how runs finish and how they fail more than what the model can actually solve, and this was mediated by the throughput of the setup one <ref:2611.06257#pg2>. They observed that two different four bit variants were on the same line, but one specific checkpoint was actually the best value for its generation when using dense twenty seven billion parameter models, though it was slower when using mixture of experts one <ref:2611.06257#pg2>.
Nadia: The study also looked at the actual paths taken by successful runs. They found three general patterns for how a run succeeds: first, a correct hypothesis translates into a correct file on the very first attempt; second, a correct hypothesis is carried by a malformed file that the agent later inspects and repairs; or third, it’s when the file itself gets rejected by the parser and then the agent resubmits different versions of it one <ref:2611.06257#pg2>.
Conclusion: Nadia: So looking at this paper, "Analysis of Local Open-Weight LLM Agents on Vulnerability-Reproduction Tasks," the main point is that open-weight models of this size are already useful for this kind of work and they are still improving. The strongest twenty seven billion parameter configuration reproduces two thirds of the discriminating tasks one <ref:2611.06257#pg2>.
Elias: That’s significant because that same strong configuration went from being one of those twelve runs under Qwen three point five to four under Qwen three point six one <ref:2611.06257#pg2>. It shows a real jump in capability just by moving to the newer generation.
Priya: So what does this mean for us, or for people who just listen to the show? It means that even without access to massive hosted services, we can get some decent security agents running locally on our own hardware today.
Nadia: Exactly. The paper suggests the next big improvement might come from the feedback that a larger model gives back to the agent, which is what they are testing next one <ref:2611.06257#pg2>.
Elias: It’s about making these local tools more reliable for complex security tasks, moving them from just interesting experiments to genuinely useful agents in practice.
Priya: The limitation they point out is that this study didn't systematically evaluate off-the-shelf open-weight models served locally on a single consumer grade GPU across all those sizes and architectures as general purpose agents one <ref:2611.06257#pg2>. They focused on this specific reproduction task.
Nadia: That’s the caveat. So while we see promise here, we still need more systematic testing to really map out what these local agents can do across every possible security scenario.
More episodes
- 2608.27496-ROPE: Routed Origin Policy Enforcement against Indirect Prompt Injection
- Daily Summary for 2026-11-19
- 2607.07085-Is Randomness Necessary for Adaptive Data Analysis?
- The day in conversation for 2026-11-19
- Daily Summary for 2026-11-19
- 2609.10601-Threshold Choice, Not Sample Size, Bounds Trustless Verification of Nondeterministic Compound AI Workflows
- 2610.38786-Adversarial Process Disturbances at a Deployable Alarm Budget: Detector Evaluation on a Simulator and a Real Plant
- 2511.08660-Binary and Multiclass Cyberattack Classification on GeNIS Dataset
- 2604.27302-Static Attribution of Android Residential Proxy Malware Using Graph Kernels
- Daily Summary for 2026-11-18