CVE-Conditioned Exploit Generation with LLMs: A Data-Centric Study of Benchmarking and Fine-Tuning
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Security Radio. Generated commentary on the latest security and cryptography papers.
Nadia: We're looking at the paper: "CVE-Conditioned Exploit Generation with LLMs".
Elias: Here's the gist of the paper. We study the task of CVE-conditioned exploit generation, where a model drafts proof-of-concept (PoC) exploits given software vulnerability context.
Nadia: This research is significant because it systematically benchmarks 17 large language models across eight evaluation criteria to provide insights into their zero-shot capabilities.
Elias: The findings highlight the importance of data quality and structured supervision over model scaling in adapting LLMs to cybersecurity tasks.
Nadia: Problem Formulation and Data Construction The task is formalized as CVE-conditioned exploit generation where a model synthesizes structured exploit artifacts from a heterogeneous vulnerability context.
Paper summary: Nadia: So, we’re looking at this paper called "CVE-Conditioned Exploit Generation with LLMs: A Data-Centric Study of Benchmarking and Fine-Tuning." Basically, they're studying how models draft proof of concept exploits when you give them a software vulnerability context.
Elias: It seems the core idea is that it’s not just about scaling up these models; it’s really about how much you structure the data and supervise the process to get reliable results. The paper claims they benchmark seventeen large language models using eight different criteria to see how well they perform zero-shot, which is pretty important for seeing what these things can do out of the box.
Priya: From a privacy and measurement standpoint, I'm curious about that data construction part. If you’re building this dataset from fragmented sources like CVE program identifiers and NVD metadata, how much noise are they cleaning up before feeding it to the models?
Nadia: That’s the first big thing they tackle. They use a multi-stage preprocessing pipeline involving standardization, quality control with CWE-aware heuristics, and then semantic refinement using another strong reasoning LLM to rewrite labels. It sounds like they’re trying to fight the mess of raw data sources before anything gets trained on it.
Elias: Exactly. That refinement step is key because it combats that raw data fragmentation issue they mentioned in the summary of "CVE-Conditioned Exploit Generation with LLMs: A Data-Centric Study of Benchmarking and Fine-Tuning" <ref:2606.15123#pg2>, "Combines features into unified JSON (e.g., CVE Id, CWE, Affected Software)" The goal is to create a high-quality structured supervision signal.
Priya: But what does that refinement actually show us about the exploits themselves? Does it really lead to better exploit quality when you fine-tune a model? I mean, the paper says they used offline label denoising where a teacher model rewrites noisy labels for the student model.
Nadia: Well, they found some specific results on how that fine-tuning works. They trained this compact eight billion parameter open-weight model, Qwen3-8B <ref:2606.15123#pg1>. They observed that this fine-tuning boosts complex tasks like remote code execution or RCE significantly while simple tasks like path traversal don't show much improvement at all.
Elias: That’s a crucial distinction they make about the training signal. They found that the performance boost for the Qwen3-8B model on RCE vulnerabilities went from six point five eight to nine point three eight, which is a two point eighty increase over GPT-4o on those specific vulnerabilities, according to "CVE-Conditioned Exploit Generation with LLMs: A Data-Centric Study of Benchmarking and Fine-Tuning" <ref:2606.15123#pg2>.
Priya: So they’re showing that the quality of the supervision directly dictates what kind of exploit a model is good at generating. That makes sense for security research, because we can’t trust a model if it’s just memorized bad patterns from noisy data.
Nadia: Right, and it really highlights that data quality and structured supervision are more important than just scaling up the model size when you're trying to adapt LLMs for these specific cybersecurity tasks. They emphasize that this design allows them to disentangle the effects of the data, evaluation, and training signals in their unified framework according to "CVE-Conditioned Exploit Generation with LLMs: A Data-Centric Study of Benchmarking and Fine-Tuning" <ref:2606.15123#pg3>.
Elias: And that leads us nicely into what this study means for the future of these systems. The authors point out limitations regarding LLM-as-a-Judge mechanisms because they can introduce systematic bias, which is a concern for any evaluation setup.
Priya: So, what’s their suggestion for moving forward when using those judge mechanisms? Are they saying we should ditch them entirely or try something different?
Nadia: They suggest investigating hybrid evaluation pipelines where execution traces remain the primary oracle, meaning the actual running of the exploit is the main truth, while LLM judges are just providing a secondary assessment. That seems like a way to ground their evaluations in real behavior rather than just another layer of model judgment.
Elias: It’s about making sure that when we evaluate these systems, we prioritize what actually runs versus what the model *thinks* runs, especially when dealing with something as sensitive as exploit generation. That points toward a more robust way to measure capability in this area.
Priya: So, the implication for people who just listen to the show is that if you’re trying to use these tools for real security work, don't just look at the benchmark scores; you need to understand how that data was curated and supervised because those are what actually drive performance in complex scenarios.
Nadia: That’s the main point of this paper, "CVE-Conditioned Exploit Generation with LLMs: A Data-Centric Study of Benchmarking and Fine-Tuning." It gives us a clear roadmap for how to make these models reliable tools instead of just interesting experiments.
Elias: The authors are demonstrating that focusing on the data pipeline, the refinement process, and using structured supervision over massive scale is what actually yields tangible improvements in exploit generation quality for high-risk vulnerabilities like RCE.
Priya: So, to wrap up this segment, it’s a study showing that reliable exploit generation depends heavily on how you build the training signal and evaluate those results.
Nadia: Exactly. And that's where we head into discussing what the overall findings of "CVE-Conditioned Exploit Generation with LLMs: A Data-Centric Study of Benchmarking and Fine-Tuning" mean for how we approach building defense systems against these kinds of AI threats.
Conclusion: Nadia: So we’ve looked at how this paper on CVE-conditioned exploit generation works and what they found about these large language models.
Elias: It boils down to them systematically testing seventeen different large language models across eight criteria to see how well they handle zero-shot exploitation tasks.
Priya: And the main thing that stuck with me is their focus on the data construction method, which is really how much you can trust what these models are generating.
Nadia: Exactly. They spend a lot of time building this dataset from messy sources like MITRE CVE programs and NVD metadata, and they use a multi-stage process—standardization, quality control with CWE rules, and then semantic refinement using another strong language model—to clean up the noise.
Elias: That refinement step is what you need to hear about because it’s not just throwing raw data at the model; they are actively improving the labels before training begins.
Priya: And when they did that, they found a specific outcome with this eight billion parameter model, Qwen3-8B <ref:2606.15123#pg1>. They saw a big performance boost on complex exploits like remote code execution, RCE, jumping from six point five eight to nine point three eight score points.
Nadia: That jump is significant because it shows that for these complex tasks, the quality of the supervision really matters more than just making the model bigger or scaling it up blindly.
Elias: But there’s a caveat they mention—the fine-tuning strategy isn't perfect across all vulnerability types; they saw modest gains on path traversal vulnerabilities.
Priya: That tells us that what makes a model good at generating an exploit depends heavily on the specific type of exploit you’re looking for, not just how much training data it sees.
Nadia: So when we look at the title, "CVE-Conditioned Exploit Generation with LLMs," it really sums up the core idea: using these models to generate actual proof of concept code based on a known vulnerability context.
Elias: And the authors are basically arguing that for this kind of application, you need a rigorous way to structure the input and supervise the learning process to get dependable results.
Priya: What this means for us is that if we want to use AI to help us find or understand exploits in real-world software, we can’t just rely on a model’s general knowledge; we have to control the quality of the context it gets and how it learns from that context.
Cisco Systems, Inc. · Michigan State University
cs.CR, cs.LG
Submitted: 2026-06-13
Updated: 2026-11-02
Comments: Technical Report
Code: https://github.com/meta-llama/llama-models
Project page: https://meta-llama.github.io/PurpleLlama/CyberSecEval
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 91/100
The gist: We study the task of CVE-conditioned exploit generation, where a model drafts proof-of-concept (PoC) exploits given software vulnerability context.
Key concepts
- CVE-conditioned exploit generation
- This is the task where a language model takes information about a specific software vulnerability and writes a working code example (a proof-of-concept or PoC) that demonstrates how to exploit it. The goal is to turn a security flaw description into an actual attack script.
- Data Construction Pipeline
- A multi-step process used to create the training data. It involves standardizing messy vulnerability information, filtering out low-quality noise using rules related to CWE (Common Weakness Enumeration), and then refining the labels using a powerful LLM to ensure high accuracy.
- Supervised Fine-Tuning (SFT)
- A technique where a smaller, open-weight language model is trained further on a specific task. In this study, SFT was used on an 8B model; it proved highly effective at improving the generation of complex exploits like RCE but showed less benefit for simpler exploit types.
- LLM-as-a-Judge
- A method where another large language model is used to evaluate or score the output generated by a primary model. The paper notes limitations here, suggesting that while helpful, LLM judgments can introduce systematic biases and may not always be reliable for complex security tasks.
Terminology
Summary
We study the task of CVE-conditioned exploit generation, where a model drafts proof-of-concept (PoC) exploits given software vulnerability context. p. 0 edu, {licli,kacheun2,vparla,gsundara}@cisco. com
Abstract
We study the task of CVE-conditioned exploit
generation, where a model drafts proof-ofconcept (PoC) exploits given software vulnerability context. 1 We adopt a data-centric approach, constructing a high-quality dataset via
multi-stage preprocessing and introducing a
scalable evaluation framework with LLM-asjudge and fine-grained rubrics. Under this unified setup, we benchmark 17 large language
models across 8 evaluation criteria, providing
systematic insights into their zero-shot capabilities. 5% improvement
in exploit quality and rivals some proprietary
models when combined with simple test-time
rejection strategies. Our results highlight the
importance of data quality, structured supervision, and evaluation design for reliable exploit
generation, suggesting that these factors can be
as critical as model scale in adapting LLMs to
cybersecurity tasks.Where the paper says it
Where the paper says it
Where the paper says it
Problem Formulation and Data Construction
The task is formalized as CVE-conditioned exploit generation where a model synthesizes structured exploit artifacts from a heterogeneous vulnerability context. This involves constructing a high-quality dataset via multi-stage preprocessing with normalization, filtering, and LLM-based refinement. p. 2 We summarize our main contributions as the
following four points:
① Problem formulation and dataset. We formalize
CVE-conditioned exploit generation and construct
a high-quality dataset via multi-stage preprocessing with normalization, filtering, and LLM-based
refinement. Vulnerability
identifiers and canonical descriptions are obtained
from the Common Vulnerabilities and Exposures
(CVE) program (MITRE Corporation), which provides standardized vulnerability indexing. For
structured technical metadata, we prioritize National Vulnerability Database (NVD) (National Institute of Standards and Technology), which offers comparable structured vulnerability information. Data-Centric Benchmarking of Exploit Generation in LLMs: Understanding the Impact of Fine-Tuning
Challenges
Fragmented sources
Inconsistent formats
Low-quality exploits
Datadiversity
Stage1: Standardization
Aggregation
Normalization
Multi-source-> UnifiedTemplates
Stage2: Quality Control
Filtering
Heuristic Selection
NoiseRemoval CWE+Regex
Stage3: Semantic Refinement
LLM Refinement
LLM Rewriting
CodeExtraction Quality Enhancement
Final Dataset
(High quality, Structured)
Figure 2: Overview of the preprocessing pipeline. Raw CVE data is standardized, filtered with CWE-aware
heuristics, and refined via LLM-based extraction and optional rewriting to improve data quality.Where the paper says it
Where the paper says it
Where the paper says it
How the Training Signal is Built
The training signal is built through a multi-stage refinement process designed to combat fragmented sources and low-quality exploits. This pipeline includes:
-
Standardization: Aggregation and normalization of heterogeneous features into unified templates. p. 5
Where the paper says it
2, we implement a structured multi-stage preprocessing pipeline that integrates normalization, deterministic filtering, heuristic selection, and LLM-based refinement. Specifically, heterogeneous features are first normalized into unified templates to mitigate cross-source inconsistencies.
-
Quality Control: Deterministic filtering using CWE-aware heuristics to remove noise. p. 5
Where the paper says it
Data-Centric Benchmarking of Exploit Generation in LLMs: Understanding the Impact of Fine-Tuning Challenges Fragmented sources Inconsistent formats Low-quality exploits Datadiversity Stage1: Standardization Aggregation Normalization Multi-source-> UnifiedTemplates Stage2: Quality Control Filtering Heuristic Selection NoiseRemoval CWE+Regex Stage3: Semantic Refinement LLM Refinement LLM Rewriting CodeExtraction Quality Enhancement Final Dataset (High quality, Structured) Figure 2: Overview of the preprocessing pipeline. Raw CVE data is standardized, filtered with CWE-aware heuristics, and refined via LLM-based extraction and optional rewriting to improve data quality.
-
Semantic Refinement: Using a strong reasoning-capable LLM to perform conditional rewriting of labels. p. 5
Where the paper says it
Subsequently, we leverage LLMs to perform semantic extraction and restructuring, including separating code from descriptive content and consolidating fragmented exploit implementations into executable scripts. Finally, for training data, we employ a strong reasoning-capable LLM to perform conditional rewriting, improving exploit completeness, clarity, and consistency for downstream supervision.
The final dataset is curated to ensure broad temporal coverage and diversity across vulnerability types and software ecosystems. p. 5 3, illustrating broad temporal
coverage and diversity across vulnerability types
and software ecosystems, ensuring a representative
evaluation setting. Notably, the distribution spans
both earlier and more recent vulnerabilities, capturing evolving exploit patterns and ensuring that the
dataset reflects realistic and dynamic security scenarios rather than being concentrated in a narrow
time window.Where the paper says it
Where the Gains Come From
Performance gains are primarily observed through supervised fine-tuning (SFT) on a compact 8B open-weight model. Specifically, SFT boosts complex tasks like Remote Code Execution (RCE) while simple tasks do not show obvious improvements. p. 1 , CVE
Id, CWE, Affected Software)
Standardization & Quality Check
CWE-AwareRegex Deterministic Filtering
Semantic Refinement (Teacher Model)
Label Denoising
CodeSeperation
Teacher LLM
Label Rewriting
. Compact Student Post-Train
Qwen3-8B
reasoning trace+ exploit
intro & snippets
Key Insight
PerformanceBoost &
Model Poratability
train on formatted exploits QLoRA
SFT
- SFT boosts
complex tasks like RCE
- Simple tasks like PT does not show
obvious
improvements
8 Criteria
Evaluation
Figure 1: Overview of our data-centric framework for CVE-conditioned exploit generation. We denote D =
{(xc,k, y˜c,k)}
K
k=1 to be the collected dataset after primitive preprocessing with noisy labels y˜c,k,
which may contain inconsistencies, incomplete exploit logic, or formatting noise due to heterogeneous data sources. A strong reasoning teacher
model f
T
is used offline to rewrite the labels:
yˆc,k,r = f
T
(xc,k, y˜c,k), (2)
producing a refined dataset Dˆ = {(xc,k, yˆc,k,r)},
where r denotes the reasoning chain-of-thought
(CoT) tokens demonstrate the thinking process. Compared to RCE, PT
exhibits more uniform performance across models, with smaller gaps in targeting accuracy and
reproducibility, suggesting that PT tasks impose
less stringent requirements on precise exploitation
logic and execution fidelity. Overall, o3 achieves the highest total score,
demonstrating strong and consistent performance
across all criteria, including targeting accuracy,
7Where the paper says it
Where the paper says it
Where the paper says it
Task Dependency and Limitations
The results reveal a clear task-dependent behavior of SFT, showing substantial improvements on complex RCE vulnerabilities but modest gains on Path Traversal (PT) vulnerabilities. This suggests that the current fine-tuning strategy is not uniformly optimal across different vulnerability types. p. 1 More broadly,
these challenges reveal a common trend: current approaches are largely model-centric or agent-centric,
emphasizing model scale or increasingly complex
tool-based pipelines, while under-exploring the role
of data quality, structured supervision, and systematic evaluation. As a result, it remains unclear how
much of the observed performance gains stem from
model scaling versus improvements in data curation and training signals, and there is a shortage
of unified benchmarks that enable controlled, reproducible, and fine-grained comparisons across
models. Second, safety alignment mechanisms in modern LLMs frequently lead
to refusals, reducing the effective rate of exploit
generation attempts, particularly for models such as
OpenAI o3 and GPT-5 (OpenAI, 2025c,a)
3
. Third,
many existing systems rely heavily on proprietary
APIs, raising concerns regarding data sovereignty
and limiting their applicability in sensitive, airgapped or production environments.Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Data-centric preprocessing pipeline for CVE-conditioned exploit generation improves data quality by combining features into unified JSON, applying CWE aware regex deterministic filtering, and using LLM based semantic refinement to enhance exploit summaries and code separation. This results in a high quality structured supervision signal for training.
-
Development of an 8 criterion rubric evaluation framework provides fine grained assessment across safety, correctness, and reproducibility for generated exploits. This systematic evaluation allows for controlled and reproducible comparison across different large language models.
-
Supervised fine tuning on compact student models like Qwen3-8B significantly improves performance on complex vulnerabilities such as remote code execution while revealing task dependent behavior. This demonstrates that targeted fine tuning can effectively compensate for model scale in complex exploit generation tasks.
Abstract
We study the task of CVE-conditioned exploit generation, where a model drafts proof-of-concept (PoC) exploits given software vulnerability context. We adopt a data-centric approach, constructing a high-quality dataset via multi-stage preprocessing and introducing a scalable evaluation framework with LLM-as-judge and fine-grained rubrics. Under this unified setup, we benchmark 17 large language models across 8 evaluation criteria, providing systematic insights into their zero-shot capabilities. We further show that a compact 8B open-weight model, when fine-tuned on curated data, achieves over 42.5% improvement in exploit quality and rivals some proprietary models when combined with simple test-time rejection strategies. Our results highlight the importance of data quality, structured supervision, and evaluation design for reliable exploit generation, suggesting that these factors can be as critical as model scale in adapting LLMs to cybersecurity tasks.
Sources
- PoCo: Agentic Proof-of-Concept Exploit Generation for Smart Contracts
- CyberSecEval 2: A Wide-Ranging Cybersecurity Evaluation Suite for Large Language Models
- PentestGPT: An LLM-empowered Automatic Penetration Testing Tool
- Circular Reasoning: Understanding Self-Reinforcing Loops in Large Reasoning Models
- LLM Agents can Autonomously Exploit One-day Vulnerabilities
- LLM Agents can Autonomously Hack Websites
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- Critique-Guided Distillation for Robust Reasoning via Refinement
- ARVO: Atlas of Reproducible Vulnerabilities for Open-Source Software
- Wait, Wait, Wait... Why Do Reasoning Models Loop?
- Kimi K2.5: Visual Agentic Intelligence
- D-CIPHER: Dynamic Collaborative Intelligent Multi-Agent System with Planner and Heterogeneous Executors for Offensive Security
- CYBERSECEVAL 3: Advancing the Evaluation of Cybersecurity Risks and Capabilities in Large Language Models
- CyberGym: Evaluating AI Agents' Real-World Cybersecurity Capabilities at Scale
- AutoAttacker: A Large Language Model Guided System to Implement Automatic Cyber-attacks
- Qwen3 Technical Report
Related papers
- From Strategy Search to Policy Learning: MCTS-Guided Adversarial Co-Evolution for LLM Safety Alignment
- Privacy Leakage from Recurrent State in LLM Inference
- Incrementally Verifiable Oracle Aided Computation
- Privilege and confidentiality in generative AI workflows
- BadMoE: Backdooring Mixture-of-Experts LLMs via Routing-Aware Triggers and Task-Underutilized Experts
- CoVault: Secure, Scalable Analytics of Personal Data