CVE-Conditioned Exploit Generation with LLMs: A Data-Centric Study of Benchmarking and Fine-Tuning

arXiv:2606.15123 · cs.CR, cs.LG · Submitted 2026-06-13 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Security Radio. Generated commentary on the latest security and cryptography papers.

Nadia: We're looking at the paper: "CVE-Conditioned Exploit Generation with LLMs".

Elias: Here's the gist of the paper. We study the task of CVE-conditioned exploit generation, where a model drafts proof-of-concept (PoC) exploits given software vulnerability context.

Nadia: This research is significant because it systematically benchmarks 17 large language models across eight evaluation criteria to provide insights into their zero-shot capabilities.

Elias: The findings highlight the importance of data quality and structured supervision over model scaling in adapting LLMs to cybersecurity tasks.

Nadia: Problem Formulation and Data Construction The task is formalized as CVE-conditioned exploit generation where a model synthesizes structured exploit artifacts from a heterogeneous vulnerability context.

Paper summary: Nadia: So, we’re looking at this paper called "CVE-Conditioned Exploit Generation with LLMs: A Data-Centric Study of Benchmarking and Fine-Tuning." Basically, they're studying how models draft proof of concept exploits when you give them a software vulnerability context.

Elias: It seems the core idea is that it’s not just about scaling up these models; it’s really about how much you structure the data and supervise the process to get reliable results. The paper claims they benchmark seventeen large language models using eight different criteria to see how well they perform zero-shot, which is pretty important for seeing what these things can do out of the box.

Priya: From a privacy and measurement standpoint, I'm curious about that data construction part. If you’re building this dataset from fragmented sources like CVE program identifiers and NVD metadata, how much noise are they cleaning up before feeding it to the models?

Nadia: That’s the first big thing they tackle. They use a multi-stage preprocessing pipeline involving standardization, quality control with CWE-aware heuristics, and then semantic refinement using another strong reasoning LLM to rewrite labels. It sounds like they’re trying to fight the mess of raw data sources before anything gets trained on it.

Elias: Exactly. That refinement step is key because it combats that raw data fragmentation issue they mentioned in the summary of "CVE-Conditioned Exploit Generation with LLMs: A Data-Centric Study of Benchmarking and Fine-Tuning" <ref:2606.15123#pg2>, "Combines features into unified JSON (e.g., CVE Id, CWE, Affected Software)" The goal is to create a high-quality structured supervision signal.

Priya: But what does that refinement actually show us about the exploits themselves? Does it really lead to better exploit quality when you fine-tune a model? I mean, the paper says they used offline label denoising where a teacher model rewrites noisy labels for the student model.

Nadia: Well, they found some specific results on how that fine-tuning works. They trained this compact eight billion parameter open-weight model, Qwen3-8B <ref:2606.15123#pg1>. They observed that this fine-tuning boosts complex tasks like remote code execution or RCE significantly while simple tasks like path traversal don't show much improvement at all.

Elias: That’s a crucial distinction they make about the training signal. They found that the performance boost for the Qwen3-8B model on RCE vulnerabilities went from six point five eight to nine point three eight, which is a two point eighty increase over GPT-4o on those specific vulnerabilities, according to "CVE-Conditioned Exploit Generation with LLMs: A Data-Centric Study of Benchmarking and Fine-Tuning" <ref:2606.15123#pg2>.

Priya: So they’re showing that the quality of the supervision directly dictates what kind of exploit a model is good at generating. That makes sense for security research, because we can’t trust a model if it’s just memorized bad patterns from noisy data.

Nadia: Right, and it really highlights that data quality and structured supervision are more important than just scaling up the model size when you're trying to adapt LLMs for these specific cybersecurity tasks. They emphasize that this design allows them to disentangle the effects of the data, evaluation, and training signals in their unified framework according to "CVE-Conditioned Exploit Generation with LLMs: A Data-Centric Study of Benchmarking and Fine-Tuning" <ref:2606.15123#pg3>.

Elias: And that leads us nicely into what this study means for the future of these systems. The authors point out limitations regarding LLM-as-a-Judge mechanisms because they can introduce systematic bias, which is a concern for any evaluation setup.

Priya: So, what’s their suggestion for moving forward when using those judge mechanisms? Are they saying we should ditch them entirely or try something different?

Nadia: They suggest investigating hybrid evaluation pipelines where execution traces remain the primary oracle, meaning the actual running of the exploit is the main truth, while LLM judges are just providing a secondary assessment. That seems like a way to ground their evaluations in real behavior rather than just another layer of model judgment.

Elias: It’s about making sure that when we evaluate these systems, we prioritize what actually runs versus what the model *thinks* runs, especially when dealing with something as sensitive as exploit generation. That points toward a more robust way to measure capability in this area.

Priya: So, the implication for people who just listen to the show is that if you’re trying to use these tools for real security work, don't just look at the benchmark scores; you need to understand how that data was curated and supervised because those are what actually drive performance in complex scenarios.

Nadia: That’s the main point of this paper, "CVE-Conditioned Exploit Generation with LLMs: A Data-Centric Study of Benchmarking and Fine-Tuning." It gives us a clear roadmap for how to make these models reliable tools instead of just interesting experiments.

Elias: The authors are demonstrating that focusing on the data pipeline, the refinement process, and using structured supervision over massive scale is what actually yields tangible improvements in exploit generation quality for high-risk vulnerabilities like RCE.

Priya: So, to wrap up this segment, it’s a study showing that reliable exploit generation depends heavily on how you build the training signal and evaluate those results.

Nadia: Exactly. And that's where we head into discussing what the overall findings of "CVE-Conditioned Exploit Generation with LLMs: A Data-Centric Study of Benchmarking and Fine-Tuning" mean for how we approach building defense systems against these kinds of AI threats.

Conclusion: Nadia: So we’ve looked at how this paper on CVE-conditioned exploit generation works and what they found about these large language models.

Elias: It boils down to them systematically testing seventeen different large language models across eight criteria to see how well they handle zero-shot exploitation tasks.

Priya: And the main thing that stuck with me is their focus on the data construction method, which is really how much you can trust what these models are generating.

Nadia: Exactly. They spend a lot of time building this dataset from messy sources like MITRE CVE programs and NVD metadata, and they use a multi-stage process—standardization, quality control with CWE rules, and then semantic refinement using another strong language model—to clean up the noise.

Elias: That refinement step is what you need to hear about because it’s not just throwing raw data at the model; they are actively improving the labels before training begins.

Priya: And when they did that, they found a specific outcome with this eight billion parameter model, Qwen3-8B <ref:2606.15123#pg1>. They saw a big performance boost on complex exploits like remote code execution, RCE, jumping from six point five eight to nine point three eight score points.

Nadia: That jump is significant because it shows that for these complex tasks, the quality of the supervision really matters more than just making the model bigger or scaling it up blindly.

Elias: But there’s a caveat they mention—the fine-tuning strategy isn't perfect across all vulnerability types; they saw modest gains on path traversal vulnerabilities.

Priya: That tells us that what makes a model good at generating an exploit depends heavily on the specific type of exploit you’re looking for, not just how much training data it sees.

Nadia: So when we look at the title, "CVE-Conditioned Exploit Generation with LLMs," it really sums up the core idea: using these models to generate actual proof of concept code based on a known vulnerability context.

Elias: And the authors are basically arguing that for this kind of application, you need a rigorous way to structure the input and supervise the learning process to get dependable results.

Priya: What this means for us is that if we want to use AI to help us find or understand exploits in real-world software, we can’t just rely on a model’s general knowledge; we have to control the quality of the context it gets and how it learns from that context.

Cisco Systems, Inc. · Michigan State University

cs.CR, cs.LG

Submitted: 2026-06-13

Updated: 2026-11-02

Comments: Technical Report

Code: https://github.com/meta-llama/llama-models

Project page: https://meta-llama.github.io/PurpleLlama/CyberSecEval

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 91/100

The gist: We study the task of CVE-conditioned exploit generation, where a model drafts proof-of-concept (PoC) exploits given software vulnerability context.

Key concepts

CVE-conditioned exploit generation
This is the task where a language model takes information about a specific software vulnerability and writes a working code example (a proof-of-concept or PoC) that demonstrates how to exploit it. The goal is to turn a security flaw description into an actual attack script.
Data Construction Pipeline
A multi-step process used to create the training data. It involves standardizing messy vulnerability information, filtering out low-quality noise using rules related to CWE (Common Weakness Enumeration), and then refining the labels using a powerful LLM to ensure high accuracy.
Supervised Fine-Tuning (SFT)
A technique where a smaller, open-weight language model is trained further on a specific task. In this study, SFT was used on an 8B model; it proved highly effective at improving the generation of complex exploits like RCE but showed less benefit for simpler exploit types.
LLM-as-a-Judge
A method where another large language model is used to evaluate or score the output generated by a primary model. The paper notes limitations here, suggesting that while helpful, LLM judgments can introduce systematic biases and may not always be reliable for complex security tasks.

Terminology

Summary

We study the task of CVE-conditioned exploit generation, where a model drafts proof-of-concept (PoC) exploits given software vulnerability context. p. 0

Where the paper says it

edu, {licli,kacheun2,vparla,gsundara}@cisco. com Abstract We study the task of CVE-conditioned exploit generation, where a model drafts proof-ofconcept (PoC) exploits given software vulnerability context.

This research is significant because it systematically benchmarks 17 large language models across eight evaluation criteria to provide insights into their zero-shot capabilities. p. 0
Where the paper says it

1 We adopt a data-centric approach, constructing a high-quality dataset via multi-stage preprocessing and introducing a scalable evaluation framework with LLM-asjudge and fine-grained rubrics. Under this unified setup, we benchmark 17 large language models across 8 evaluation criteria, providing systematic insights into their zero-shot capabilities.

The findings highlight the importance of data quality and structured supervision over model scaling in adapting LLMs to cybersecurity tasks. p. 0
Where the paper says it

5% improvement in exploit quality and rivals some proprietary models when combined with simple test-time rejection strategies. Our results highlight the importance of data quality, structured supervision, and evaluation design for reliable exploit generation, suggesting that these factors can be as critical as model scale in adapting LLMs to cybersecurity tasks.

Problem Formulation and Data Construction

The task is formalized as CVE-conditioned exploit generation where a model synthesizes structured exploit artifacts from a heterogeneous vulnerability context. This involves constructing a high-quality dataset via multi-stage preprocessing with normalization, filtering, and LLM-based refinement. p. 2

Where the paper says it

We summarize our main contributions as the following four points: ① Problem formulation and dataset. We formalize CVE-conditioned exploit generation and construct a high-quality dataset via multi-stage preprocessing with normalization, filtering, and LLM-based refinement.

The data source rationale prioritizes CVE program identifiers from MITRE Corporation and structured metadata from the National Vulnerability Database (NVD). p. 4
Where the paper says it

Vulnerability identifiers and canonical descriptions are obtained from the Common Vulnerabilities and Exposures (CVE) program (MITRE Corporation), which provides standardized vulnerability indexing. For structured technical metadata, we prioritize National Vulnerability Database (NVD) (National Institute of Standards and Technology), which offers comparable structured vulnerability information.

The preprocessing pipeline involves three stages: Standardization, Quality Control, and Semantic Refinement. p. 5
Where the paper says it

Data-Centric Benchmarking of Exploit Generation in LLMs: Understanding the Impact of Fine-Tuning Challenges Fragmented sources Inconsistent formats Low-quality exploits Datadiversity Stage1: Standardization Aggregation Normalization Multi-source-> UnifiedTemplates Stage2: Quality Control Filtering Heuristic Selection NoiseRemoval CWE+Regex Stage3: Semantic Refinement LLM Refinement LLM Rewriting CodeExtraction Quality Enhancement Final Dataset (High quality, Structured) Figure 2: Overview of the preprocessing pipeline. Raw CVE data is standardized, filtered with CWE-aware heuristics, and refined via LLM-based extraction and optional rewriting to improve data quality.

How the Training Signal is Built

The training signal is built through a multi-stage refinement process designed to combat fragmented sources and low-quality exploits. This pipeline includes:

  1. Standardization: Aggregation and normalization of heterogeneous features into unified templates. p. 5

    Where the paper says it

    2, we implement a structured multi-stage preprocessing pipeline that integrates normalization, deterministic filtering, heuristic selection, and LLM-based refinement. Specifically, heterogeneous features are first normalized into unified templates to mitigate cross-source inconsistencies.

  2. Quality Control: Deterministic filtering using CWE-aware heuristics to remove noise. p. 5

    Where the paper says it

    Data-Centric Benchmarking of Exploit Generation in LLMs: Understanding the Impact of Fine-Tuning Challenges Fragmented sources Inconsistent formats Low-quality exploits Datadiversity Stage1: Standardization Aggregation Normalization Multi-source-> UnifiedTemplates Stage2: Quality Control Filtering Heuristic Selection NoiseRemoval CWE+Regex Stage3: Semantic Refinement LLM Refinement LLM Rewriting CodeExtraction Quality Enhancement Final Dataset (High quality, Structured) Figure 2: Overview of the preprocessing pipeline. Raw CVE data is standardized, filtered with CWE-aware heuristics, and refined via LLM-based extraction and optional rewriting to improve data quality.

  3. Semantic Refinement: Using a strong reasoning-capable LLM to perform conditional rewriting of labels. p. 5

    Where the paper says it

    Subsequently, we leverage LLMs to perform semantic extraction and restructuring, including separating code from descriptive content and consolidating fragmented exploit implementations into executable scripts. Finally, for training data, we employ a strong reasoning-capable LLM to perform conditional rewriting, improving exploit completeness, clarity, and consistency for downstream supervision.

The final dataset is curated to ensure broad temporal coverage and diversity across vulnerability types and software ecosystems. p. 5

Where the paper says it

3, illustrating broad temporal coverage and diversity across vulnerability types and software ecosystems, ensuring a representative evaluation setting. Notably, the distribution spans both earlier and more recent vulnerabilities, capturing evolving exploit patterns and ensuring that the dataset reflects realistic and dynamic security scenarios rather than being concentrated in a narrow time window.

Where the Gains Come From

Performance gains are primarily observed through supervised fine-tuning (SFT) on a compact 8B open-weight model. Specifically, SFT boosts complex tasks like Remote Code Execution (RCE) while simple tasks do not show obvious improvements. p. 1

Where the paper says it

, CVE Id, CWE, Affected Software) Standardization & Quality Check CWE-AwareRegex Deterministic Filtering Semantic Refinement (Teacher Model) Label Denoising CodeSeperation Teacher LLM Label Rewriting . Compact Student Post-Train Qwen3-8B reasoning trace+ exploit intro & snippets Key Insight PerformanceBoost & Model Poratability train on formatted exploits QLoRA SFT - SFT boosts complex tasks like RCE - Simple tasks like PT does not show obvious improvements 8 Criteria Evaluation Figure 1: Overview of our data-centric framework for CVE-conditioned exploit generation.

The SFT framework uses offline label denoising where a teacher model rewrites noisy labels to produce a refined dataset. p. 8
Where the paper says it

We denote D = {(xc,k, y˜c,k)} K k=1 to be the collected dataset after primitive preprocessing with noisy labels y˜c,k, which may contain inconsistencies, incomplete exploit logic, or formatting noise due to heterogeneous data sources. A strong reasoning teacher model f T is used offline to rewrite the labels: yˆc,k,r = f T (xc,k, y˜c,k), (2) producing a refined dataset Dˆ = {(xc,k, yˆc,k,r)}, where r denotes the reasoning chain-of-thought (CoT) tokens demonstrate the thinking process.

The resulting SFT-enhanced Qwen3-8B significantly outperforms GPT-4o on RCE vulnerabilities by increasing the total score from 6. p. 6
Where the paper says it

Compared to RCE, PT exhibits more uniform performance across models, with smaller gaps in targeting accuracy and reproducibility, suggesting that PT tasks impose less stringent requirements on precise exploitation logic and execution fidelity. Overall, o3 achieves the highest total score, demonstrating strong and consistent performance across all criteria, including targeting accuracy, 7

58 to 9.38 (+2.80).

Task Dependency and Limitations

The results reveal a clear task-dependent behavior of SFT, showing substantial improvements on complex RCE vulnerabilities but modest gains on Path Traversal (PT) vulnerabilities. This suggests that the current fine-tuning strategy is not uniformly optimal across different vulnerability types. p. 1

Where the paper says it

More broadly, these challenges reveal a common trend: current approaches are largely model-centric or agent-centric, emphasizing model scale or increasingly complex tool-based pipelines, while under-exploring the role of data quality, structured supervision, and systematic evaluation. As a result, it remains unclear how much of the observed performance gains stem from model scaling versus improvements in data curation and training signals, and there is a shortage of unified benchmarks that enable controlled, reproducible, and fine-grained comparisons across models.

Furthermore, the paper identifies limitations in LLM-as-a-Judge mechanisms regarding systematic bias and generalizability. p. 1
Where the paper says it

Second, safety alignment mechanisms in modern LLMs frequently lead to refusals, reducing the effective rate of exploit generation attempts, particularly for models such as OpenAI o3 and GPT-5 (OpenAI, 2025c,a) 3 . Third, many existing systems rely heavily on proprietary APIs, raising concerns regarding data sovereignty and limiting their applicability in sensitive, airgapped or production environments.

Future work suggests investigating hybrid evaluation pipelines where execution traces remain the primary oracle while LLM judges provide secondary assessments.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Data-centric preprocessing pipeline for CVE-conditioned exploit generation improves data quality by combining features into unified JSON, applying CWE aware regex deterministic filtering, and using LLM based semantic refinement to enhance exploit summaries and code separation. This results in a high quality structured supervision signal for training.

  2. Development of an 8 criterion rubric evaluation framework provides fine grained assessment across safety, correctness, and reproducibility for generated exploits. This systematic evaluation allows for controlled and reproducible comparison across different large language models.

  3. Supervised fine tuning on compact student models like Qwen3-8B significantly improves performance on complex vulnerabilities such as remote code execution while revealing task dependent behavior. This demonstrates that targeted fine tuning can effectively compensate for model scale in complex exploit generation tasks.

Abstract

We study the task of CVE-conditioned exploit generation, where a model drafts proof-of-concept (PoC) exploits given software vulnerability context. We adopt a data-centric approach, constructing a high-quality dataset via multi-stage preprocessing and introducing a scalable evaluation framework with LLM-as-judge and fine-grained rubrics. Under this unified setup, we benchmark 17 large language models across 8 evaluation criteria, providing systematic insights into their zero-shot capabilities. We further show that a compact 8B open-weight model, when fine-tuned on curated data, achieves over 42.5% improvement in exploit quality and rivals some proprietary models when combined with simple test-time rejection strategies. Our results highlight the importance of data quality, structured supervision, and evaluation design for reliable exploit generation, suggesting that these factors can be as critical as model scale in adapting LLMs to cybersecurity tasks.

Sources

Related papers