CVE-Conditioned Exploit Generation with LLMs: A Data-Centric Study of Benchmarking and Fine-Tuning
summary
The gist
We study the task of CVE-conditioned exploit generation, where a model drafts proof-of-concept (PoC) exploits given software vulnerability context.
In short
Researchers tested how well various large language models can generate proof-of-concept exploits from software vulnerabilities (CVEs). They created a high-quality dataset using multi-stage refinement and found that fine-tuning a smaller model significantly improved its ability to generate complex Remote Code Execution (RCE) exploits, demonstrating that data quality is more important than just scaling up the model size.
Key concepts
- CVE-conditioned exploit generation
- This is the task where a language model takes information about a specific software vulnerability and writes a working code example (a proof-of-concept or PoC) that demonstrates how to exploit it. The goal is to turn a security flaw description into an actual attack script.
- Data Construction Pipeline
- A multi-step process used to create the training data. It involves standardizing messy vulnerability information, filtering out low-quality noise using rules related to CWE (Common Weakness Enumeration), and then refining the labels using a powerful LLM to ensure high accuracy.
- Supervised Fine-Tuning (SFT)
- A technique where a smaller, open-weight language model is trained further on a specific task. In this study, SFT was used on an 8B model; it proved highly effective at improving the generation of complex exploits like RCE but showed less benefit for simpler exploit types.
- LLM-as-a-Judge
- A method where another large language model is used to evaluate or score the output generated by a primary model. The paper notes limitations here, suggesting that while helpful, LLM judgments can introduce systematic biases and may not always be reliable for complex security tasks.
Terminology used across episodes
This episode discusses
- CVE-Conditioned Exploit Generation with LLMs: A Data-Centric Study of Benchmarking and Fine-Tuning · Paper Radio
- PoCo: Agentic Proof-of-Concept Exploit Generation for Smart Contracts
- CyberSecEval 2: A Wide-Ranging Cybersecurity Evaluation Suite for Large Language Models
- PentestGPT: An LLM-empowered Automatic Penetration Testing Tool
- Circular Reasoning: Understanding Self-Reinforcing Loops in Large Reasoning Models
- LLM Agents can Autonomously Exploit One-day Vulnerabilities
- LLM Agents can Autonomously Hack Websites
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- Critique-Guided Distillation for Robust Reasoning via Refinement · Paper Radio
- ARVO: Atlas of Reproducible Vulnerabilities for Open-Source Software
- Wait, Wait, Wait... Why Do Reasoning Models Loop? · Paper Radio
- Kimi K2.5: Visual Agentic Intelligence
- D-CIPHER: Dynamic Collaborative Intelligent Multi-Agent System with Planner and Heterogeneous Executors for Offensive Security
- CYBERSECEVAL 3: Advancing the Evaluation of Cybersecurity Risks and Capabilities in Large Language Models
- CyberGym: Evaluating AI Agents' Real-World Cybersecurity Capabilities at Scale
- AutoAttacker: A Large Language Model Guided System to Implement Automatic Cyber-attacks
- Qwen3 Technical Report
The paper
CVE-Conditioned Exploit Generation with LLMs: A Data-Centric Study of Benchmarking and Fine-Tuning · Read on arXiv
Cisco Systems, Inc. · Michigan State University
We study the task of CVE-conditioned exploit generation, where a model drafts proof-of-concept (PoC) exploits given software vulnerability context. We adopt a data-centric approach, constructing a high-quality dataset via multi-stage preprocessing and introducing a scalable evaluation framework with LLM-as-judge and fine-grained rubrics. Under this unified setup, we benchmark 17 large language models across 8 evaluation criteria, providing systematic insights into their zero-shot capabilities. We further show that a compact 8B open-weight model, when fine-tuned on curated data, achieves over 42.5% improvement in exploit quality and rivals some proprietary models when combined with simple test-time rejection strategies. Our results highlight the importance of data quality, structured supervision, and evaluation design for reliable exploit generation, suggesting that these factors can be as critical as model scale in adapting LLMs to cybersecurity tasks.
Transcript
Introduction to the show: ident: Security Radio. Generated commentary on the latest security and cryptography papers.
Nadia: We're looking at the paper: "CVE-Conditioned Exploit Generation with LLMs".
Elias: Here's the gist of the paper. We study the task of CVE-conditioned exploit generation, where a model drafts proof-of-concept (PoC) exploits given software vulnerability context.
Nadia: This research is significant because it systematically benchmarks 17 large language models across eight evaluation criteria to provide insights into their zero-shot capabilities.
Elias: The findings highlight the importance of data quality and structured supervision over model scaling in adapting LLMs to cybersecurity tasks.
Nadia: Problem Formulation and Data Construction The task is formalized as CVE-conditioned exploit generation where a model synthesizes structured exploit artifacts from a heterogeneous vulnerability context.
Paper summary: Nadia: So, we’re looking at this paper called "CVE-Conditioned Exploit Generation with LLMs: A Data-Centric Study of Benchmarking and Fine-Tuning." Basically, they're studying how models draft proof of concept exploits when you give them a software vulnerability context.
Elias: It seems the core idea is that it’s not just about scaling up these models; it’s really about how much you structure the data and supervise the process to get reliable results. The paper claims they benchmark seventeen large language models using eight different criteria to see how well they perform zero-shot, which is pretty important for seeing what these things can do out of the box.
Priya: From a privacy and measurement standpoint, I'm curious about that data construction part. If you’re building this dataset from fragmented sources like CVE program identifiers and NVD metadata, how much noise are they cleaning up before feeding it to the models?
Nadia: That’s the first big thing they tackle. They use a multi-stage preprocessing pipeline involving standardization, quality control with CWE-aware heuristics, and then semantic refinement using another strong reasoning LLM to rewrite labels. It sounds like they’re trying to fight the mess of raw data sources before anything gets trained on it.
Elias: Exactly. That refinement step is key because it combats that raw data fragmentation issue they mentioned in the summary of "CVE-Conditioned Exploit Generation with LLMs: A Data-Centric Study of Benchmarking and Fine-Tuning" <ref:2606.15123#pg2>, "Combines features into unified JSON (e.g., CVE Id, CWE, Affected Software)" The goal is to create a high-quality structured supervision signal.
Priya: But what does that refinement actually show us about the exploits themselves? Does it really lead to better exploit quality when you fine-tune a model? I mean, the paper says they used offline label denoising where a teacher model rewrites noisy labels for the student model.
Nadia: Well, they found some specific results on how that fine-tuning works. They trained this compact eight billion parameter open-weight model, Qwen3-8B <ref:2606.15123#pg1>. They observed that this fine-tuning boosts complex tasks like remote code execution or RCE significantly while simple tasks like path traversal don't show much improvement at all.
Elias: That’s a crucial distinction they make about the training signal. They found that the performance boost for the Qwen3-8B model on RCE vulnerabilities went from six point five eight to nine point three eight, which is a two point eighty increase over GPT-4o on those specific vulnerabilities, according to "CVE-Conditioned Exploit Generation with LLMs: A Data-Centric Study of Benchmarking and Fine-Tuning" <ref:2606.15123#pg2>.
Priya: So they’re showing that the quality of the supervision directly dictates what kind of exploit a model is good at generating. That makes sense for security research, because we can’t trust a model if it’s just memorized bad patterns from noisy data.
Nadia: Right, and it really highlights that data quality and structured supervision are more important than just scaling up the model size when you're trying to adapt LLMs for these specific cybersecurity tasks. They emphasize that this design allows them to disentangle the effects of the data, evaluation, and training signals in their unified framework according to "CVE-Conditioned Exploit Generation with LLMs: A Data-Centric Study of Benchmarking and Fine-Tuning" <ref:2606.15123#pg3>.
Elias: And that leads us nicely into what this study means for the future of these systems. The authors point out limitations regarding LLM-as-a-Judge mechanisms because they can introduce systematic bias, which is a concern for any evaluation setup.
Priya: So, what’s their suggestion for moving forward when using those judge mechanisms? Are they saying we should ditch them entirely or try something different?
Nadia: They suggest investigating hybrid evaluation pipelines where execution traces remain the primary oracle, meaning the actual running of the exploit is the main truth, while LLM judges are just providing a secondary assessment. That seems like a way to ground their evaluations in real behavior rather than just another layer of model judgment.
Elias: It’s about making sure that when we evaluate these systems, we prioritize what actually runs versus what the model *thinks* runs, especially when dealing with something as sensitive as exploit generation. That points toward a more robust way to measure capability in this area.
Priya: So, the implication for people who just listen to the show is that if you’re trying to use these tools for real security work, don't just look at the benchmark scores; you need to understand how that data was curated and supervised because those are what actually drive performance in complex scenarios.
Nadia: That’s the main point of this paper, "CVE-Conditioned Exploit Generation with LLMs: A Data-Centric Study of Benchmarking and Fine-Tuning." It gives us a clear roadmap for how to make these models reliable tools instead of just interesting experiments.
Elias: The authors are demonstrating that focusing on the data pipeline, the refinement process, and using structured supervision over massive scale is what actually yields tangible improvements in exploit generation quality for high-risk vulnerabilities like RCE.
Priya: So, to wrap up this segment, it’s a study showing that reliable exploit generation depends heavily on how you build the training signal and evaluate those results.
Nadia: Exactly. And that's where we head into discussing what the overall findings of "CVE-Conditioned Exploit Generation with LLMs: A Data-Centric Study of Benchmarking and Fine-Tuning" mean for how we approach building defense systems against these kinds of AI threats.
Conclusion: Nadia: So we’ve looked at how this paper on CVE-conditioned exploit generation works and what they found about these large language models.
Elias: It boils down to them systematically testing seventeen different large language models across eight criteria to see how well they handle zero-shot exploitation tasks.
Priya: And the main thing that stuck with me is their focus on the data construction method, which is really how much you can trust what these models are generating.
Nadia: Exactly. They spend a lot of time building this dataset from messy sources like MITRE CVE programs and NVD metadata, and they use a multi-stage process—standardization, quality control with CWE rules, and then semantic refinement using another strong language model—to clean up the noise.
Elias: That refinement step is what you need to hear about because it’s not just throwing raw data at the model; they are actively improving the labels before training begins.
Priya: And when they did that, they found a specific outcome with this eight billion parameter model, Qwen3-8B <ref:2606.15123#pg1>. They saw a big performance boost on complex exploits like remote code execution, RCE, jumping from six point five eight to nine point three eight score points.
Nadia: That jump is significant because it shows that for these complex tasks, the quality of the supervision really matters more than just making the model bigger or scaling it up blindly.
Elias: But there’s a caveat they mention—the fine-tuning strategy isn't perfect across all vulnerability types; they saw modest gains on path traversal vulnerabilities.
Priya: That tells us that what makes a model good at generating an exploit depends heavily on the specific type of exploit you’re looking for, not just how much training data it sees.
Nadia: So when we look at the title, "CVE-Conditioned Exploit Generation with LLMs," it really sums up the core idea: using these models to generate actual proof of concept code based on a known vulnerability context.
Elias: And the authors are basically arguing that for this kind of application, you need a rigorous way to structure the input and supervise the learning process to get dependable results.
Priya: What this means for us is that if we want to use AI to help us find or understand exploits in real-world software, we can’t just rely on a model’s general knowledge; we have to control the quality of the context it gets and how it learns from that context.
More episodes
- 2608.27496-ROPE: Routed Origin Policy Enforcement against Indirect Prompt Injection
- Daily Summary for 2026-11-19
- 2607.07085-Is Randomness Necessary for Adaptive Data Analysis?
- The day in conversation for 2026-11-19
- Daily Summary for 2026-11-19
- 2609.10601-Threshold Choice, Not Sample Size, Bounds Trustless Verification of Nondeterministic Compound AI Workflows
- 2610.38786-Adversarial Process Disturbances at a Deployable Alarm Budget: Detector Evaluation on a Simulator and a Real Plant
- 2511.08660-Binary and Multiclass Cyberattack Classification on GeNIS Dataset
- 2604.27302-Static Attribution of Android Residential Proxy Malware Using Graph Kernels
- Daily Summary for 2026-11-18