Automated Exploit and Security Test Generation: A Systematic Literature Review
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Security Radio. Generated commentary on the latest security and cryptography papers.
Nadia: We're looking at the paper: "Automated Exploit and Security Test Generation".
Elias: Here's the gist of the paper. A Systematic Literature Review on Automated Exploit and Security Test Generation surveys existing techniques for automatically generating exploits and security tests for software vulnerabilities to aid researchers and practitioners in understanding their characteristics and usability.
Nadia: The exploit or the Proof of Concept of the vulnerability plays an important role in developing superior vulnerability repair techniques, as it can be used as an oracle to verify the correctness of the patches generated by the tools.
Elias: However, vulnerability exploits are often unavailable and require time and expert knowledge to craft.
Nadia: Obtaining them from exploit generation techniques can be another potential solution.
Title and authors: Nadia: So we're looking at this paper called "Automated Exploit and Security Test Generation: A Systematic Literature Review." It’s by Quan-Cuong Bui, Emanuele Iannone, Maria Camporese, Jorge Hinrichs, Catherine Tony, László Tóth, Fabio Palomba, Peter Hegedűs.
Elias: Yeah I’ve seen the title. It sounds like a big survey of how people are trying to automatically create exploits and security tests for software flaws. It’s basically mapping out what’s already been done in that area over time.
Nadia: Exactly, it’s surveying existing techniques to help researchers and practitioners figure out how these things work in practice and what they can actually be used for when you're trying to fix bugs or test defenses.
Priya: So, it’s a literature review aimed at understanding the characteristics and usability of automated exploit generation techniques. That sounds useful for knowing where the current efforts are strongest or weakest.
Elias: Right, it sets up this framework by grouping all these different methods into four main categories: automated exploit generation, security testing, fuzzing, and other techniques. It’s trying to organize a lot of scattered research into something coherent.
Nadia: And it focuses mainly on memory-based vulnerabilities in C/C++ programs and web-based injection issues in PHP and Java applications. That gives us a clear scope for what they are focusing on right now.
The paper's summary: Nadia: The main point of this paper is that the exploit or the proof of concept for a vulnerability is really important because it acts like an oracle, something you can use to check if your new patch actually fixed the problem.
Elias: But those exploits are often hard to find and take a lot of expert time to craft, so getting them from automated generation techniques is seen as another way forward.
Nadia: So the goal of this survey is to help researchers and practitioners understand these existing exploit generation techniques by analyzing their characteristics and how usable they are in real-world scenarios.
Priya: It seems like they’re trying to make sense of a lot of messy research into a usable understanding for the security community. That’s important because it helps us know what tools actually exist and what's practical versus just theoretical.
Elias: They found that most of the studies they reviewed were published between two thousand eleven and two thousand twenty-three which tells you the research landscape is quite recent <ref:2502.04953#pg1>. The techniques are grouped into AEG, security testing, fuzzing, and others.
Nadia: Within automated exploit generation they highlighted control-flow hijacking as the dominant technique in their selection of studies. That’s a big focus area for them right now.
The paper's improvements: Elias: They look at how to improve this field, and they suggest ways to make things more automated. For instance, they point out that automated exploit generation can be improved by creating a workflow that ties vulnerability identification directly with exploit synthesis to generate working exploits for known flaws in a more automatic way.
Nadia: That makes sense. They suggest using techniques like symbolic execution and genetic algorithms to synthesize what they call a confident "exploit" or proof of concept for vulnerabilities. That would automate the crafting process beyond just simple input generation.
Priya: I wonder if that means we move closer to systems that can generate actual working code snippets, not just test cases, which is a big step toward practical exploitation.
Elias: They also suggest enhancing security testing systems by incorporating a wider range of tests, perhaps combining model-based testing with learning-based testing to boost coverage. This would allow the system to create more robust security test cases by using methods listed in Table six <ref:2502.04953#pg1>.
Nadia: That sounds like they want to move beyond just one method and build a more comprehensive testing suite that covers different types of potential weaknesses.
Priya: And they also suggest developing standardized metrics for assessing automation levels, things like FA, SA, and Inter, so practitioners could better understand the degree of automation in these techniques. That would help with usability assessment too.
Elias: They also address the availability issue by suggesting a mechanism to automatically check if proposed tools are public or licensed. This directly tackles whether these techniques are actually usable right now without requiring human intervention for every single step.
Conclusion: Nadia: So, to wrap up this review of "Automated Exploit and Security Test Generation: A Systematic Literature Review," the authors found sixty-six relevant research papers introducing or improving techniques for automatically generating exploits or security tests for software vulnerabilities.
Elias: It’s a big collection of work, and it shows that automated methods are definitely gaining traction in this space over the last decade. But they also noted that more than eighty percent of the selected studies claimed their tools could be fully automated without any human intervention.
Priya: That eighty percent is interesting, but we have to remember what those claims actually mean in practice, right? We need to see if that automation holds up when you try to use it on a real target system.
Nadia: They also pointed out that most proposed techniques still lack available tools because of security concerns, which is a big hurdle for adoption. This survey helps researchers understand the current limitations of these practices and where the gaps are right now.
Elias: The authors plan to define a clear workflow for developing these automated exploit generation tools in their future work, which gives us a direction for what comes next in this research area.
Priya: It’s good to have that roadmap so we know what to look out for when the next wave of tools comes out. It’s about seeing if the theoretical improvements translate into something you can actually run on your machine.
Nadia: So, that's our take on this paper, "Automated Exploit and Security Test Generation: A Systematic Literature Review." It gives us a solid map of the automated exploit generation landscape as we move forward.
QUANG-CUONG BUI, EMANUELE IANNONE, MARIA CAMPORESE, TORGE HINRICHS, CATHERINE TONY, LÁSZLÓ TÓTH, FABIO PALOMBA, PÉTER HEGEDŰS, FABIO MASSACCI, RICCARDO SCANDARIATO
Hamburg University of Technology · University of Trento · University of Szeged
cs.CR, cs.SE
Submitted: 2025-02-07
Updated: 2026-11-04
Comments: This work was partially supported by EU-funded project Sec4AI4Sec (grant no. 101120393) and funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) -- 583980027 (SToCC)
Code: https://github.com/kaled-alshmrany/FuSeBMC
Project page: https://clarivate.com/products/web-of-science
License: http://creativecommons.org/licenses/by-sa/4.0/
Importance score: 81/100
The gist: A Systematic Literature Review on Automated Exploit and Security Test Generation surveys existing techniques for automatically generating exploits and security tests for software vulnerabilities to…
Key concepts
- Automated Exploit Generation (AEG)
- Techniques designed to automatically create working exploits or Proof of Concept code for software vulnerabilities. The goal is to reduce the time and expert knowledge needed to craft these malicious payloads.
- Security Testing
- A broad category of methods used to verify software security, including model-based, logic programming, and heuristic testing. These methods aim to find flaws in applications before they can be exploited by attackers.
- Fuzzing
- A technique where random or semi-random inputs are fed into a program to discover unexpected behavior or vulnerabilities. Coverage-based greybox fuzzing is highlighted as a prominent method within this category.
- Control-flow Hijacking
- A dominant automated exploit generation technique identified in the survey. It involves manipulating the program's execution flow by redirecting where the CPU goes next, often used to achieve arbitrary code execution in memory-based vulnerabilities.
Terminology
Summary
A Systematic Literature Review on Automated Exploit and Security Test Generation surveys existing techniques for automatically generating exploits and security tests for software vulnerabilities to aid researchers and practitioners in understanding their characteristics and usability. p. 0 The goal of this survey is to aid the researchers and practitioners in understanding the existing techniques for exploit
generation through the analysis of their characteristics and their usability in practice. We identify a list of exploit generation techniques
from literature and group them into four categories: automated exploit generation, security testing, fuzzing, and other techniques.Where the paper says it
Introduction to the Problem
The exploit or the Proof of Concept of the vulnerability plays an important role in developing superior vulnerability repair techniques, as it can be used as an oracle to verify the correctness of the patches generated by the tools. However, vulnerability exploits are often unavailable and require time and expert knowledge to craft. p. 0 , Hungary
FABIO MASSACCI, University of Trento, Italy
RICCARDO SCANDARIATO, Hamburg University of Technology, Germany
The exploit or the Proof of Concept of the vulnerability plays an important role in developing superior vulnerability repair techniques,
as it can be used as an oracle to verify the correctness of the patches generated by the tools. However, the vulnerability exploits are
often unavailable and require time and expert knowledge to craft. However, the vulnerability exploits are
often unavailable and require time and expert knowledge to craft. Obtaining them from the exploit generation techniques is another
potential solution. Obtaining them from the exploit generation techniques is another
potential solution. The goal of this survey is to aid the researchers and practitioners in understanding the existing techniques for exploit
generation through the analysis of their characteristics and their usability in practice.Where the paper says it
Where the paper says it
Where the paper says it
Scope and Categorization
The survey identifies a list of exploit generation techniques from literature and groups them into four categories: automated exploit generation, security testing, fuzzing, and other techniques. Most of the techniques focus on the memory-based vulnerabilities in C/C++ programs and web-based injection vulnerabilities in PHP and Java applications. p. 0 We identify a list of exploit generation techniques
from literature and group them into four categories: automated exploit generation, security testing, fuzzing, and other techniques. Most of the techniques focus on the memory-based vulnerabilities in C/C++ programs and web-based injection vulnerabilities in PHP
and Java applications. The scope of our survey. We aim to search and review the academic works (peer-reviewed) that introduce novel
techniques targeting generating exploits for security vulnerabilities. We do not focus on a specific family of techniques. For example, we may not cover all the primary fuzzing works in the literature.Where the paper says it
Where the paper says it
Where the paper says it
Methodology for Review
The paper follows guidelines for conducting systematic reviews in software engineering, involving three phases: planning the review, conducting the review, and reporting the results. The methodology involves identifying the need for the review and formulating research questions. p. 1 Section 3 describes
the methodology we adopted to conduct this literature review. We start from the goal of the survey and the research
questions we considered (3. To this end, we first identify the need for our review and
formulate the research questions. In the second phase, we define the search string and inclusion and exclusion criteria to
design our strategy for finding and selecting the primary studies of automated exploit generation. 2) and
the criteria that we derived and used to select the studies for the inclusion of our survey (Section 3. After all the
papers had been collected, they were organized, categorized, and reviewed (Section 3.Where the paper says it
Where the paper says it
Where the paper says it
Key Findings on Techniques
The survey reveals that the majority of the 66 selected studies were published between 2011 and 2023. The techniques are categorized into four main groups: AEG, Security Testing, Fuzzing, and Others. p. 5 This step identified four papers that could be
grouped together, in which the authors developed a security testing technique based on threat modeling [57, 80–82]. Based on our aimed scope of this survey study and the selection results of the paper list, we defined four main groups
of techniques of the relevant areas of automated exploit generation and security testing shown in Figure 1, that is, AEG,
Manuscript submitted to ACM , Control-flow hijacking AEG, Crash Analysis-based AEG, and Data-oriented AEG. Control-flow Hijacking is
the dominant technique with 10/12 AEG studies in our survey while we found only one for each of Crash Analysis-based
and Data-oriented AEG. Security testing has the majority of studies in our survey with a very wide range of testing
techniques. They expand in multiple classes of testing techniques: from model-based testing, logic programming-based
testing, heuristic-based testing, grammar-based testing, search-based testing to hybrid testing, learning-based testing,
Manuscript submitted to ACM Our survey
AEG Security Testing Fuzzing Others
Control-flow
Hijacking
Data-oriented
Threat Modelbased
Mutationbased
Grammarbased
Search-based
Heuristic-based
Whitebox
Coveragebased greybox
Stateful
Directed
Knowledgebased
Dynamic Symbolic Execution
Dynamic Taint
Analysis
String Analysis
Fig. Taxonomy of studies on Automated Exploit Generation.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Evaluation and Availability
The research questions aim to analyze the research on AEG made so far and their key characteristics, as well as evaluate the maturity of existing AEG techniques and their usability. The output of AEG techniques can be categorized into exploit, test cases, or input/payload. p. 6 AL
S01 [38] 2014 Binary Exploit Memory C/C++ General programs Implicit assertions FA
S02 [24] 2015 Source code Test cases Information flow Java General Java method Automated assertions FA
S03 [32] 2018 Binary Exploit Crash C/C++ Web browsers Implicit assertions Inter. S04 [51] 2018 Binary Exploit Memory C/C++ General programs Implicit assertions FA
S05 [37] 2019 Binary Exploit Memory (heap) C/C++ General programs Implicit assertions FA
S06 [76] 2019 Source code Exploit Memory (ROP) C/C++ General programs Implicit assertions FA
S07 [14] 2022 Binary Exploit Memory (JOP) C/C++ General programs Implicit assertions FA
S08 [83] 2022 Binary Exploit Memory (bof) C/C++ General programs Implicit assertions FA
S09 [75] 2023 Source code Input/Payload Memory C/C++ General programs Implicit assertions Inter. The last two columns provide
information on how the oracles generated from the tools were assessed and the automation level of the tools, which
can be Fully Automated (FA), Semi-Automated (SA), or Interactive (Inter. 4 REVIEW RESULTS
This section will answer all the formulated research questions based on the selected studies. Due to this fact, researchers and practitioners propose novel techniques and develop automated tools to deal with a
large volume of vulnerabilities. Security vulnerability research has recently gained ground not only in the academic
venues for security but also in software engineering and artificial intelligence.Where the paper says it
Where the paper says it
Where the paper says it
Conclusion
This work surveyed the literature on Automated Exploit Generation, finding 66 relevant research papers that introduce or improve techniques for automatically generating exploits or security tests for software security vulnerabilities. This survey can be used by security researchers to understand the limitations of the current state of AEG practices and develop new solutions to address them. p. 3 The purpose of this is to understand which techniques and approaches exist, their peculiarities,
and under which circumstances they can be used. This will provide security practitioners and researchers with a catalog
of usable tools and elements for possible advancements in AEG. A Systematic Literature Review on Automated Exploit
and Security Test Generation
QUANG-CUONG BUI, Hamburg University of Technology, Germany
EMANUELE IANNONE, Hamburg University of Technology, Germany
MARIA CAMPORESE, University of Trento, Italy
TORGE HINRICHS, Hamburg University of Technology, Germany
CATHERINE TONY, Hamburg University of Technology, Germany
LÁSZLÓ TÓTH, University of Szeged, Hungary
FABIO PALOMBA, University of Salerno, Italy
PÉTER HEGEDŰS, University of Szeged, FrontEndART Ltd. , Hungary
FABIO MASSACCI, University of Trento, Italy
RICCARDO SCANDARIATO, Hamburg University of Technology, Germany
The exploit or the Proof of Concept of the vulnerability plays an important role in developing superior vulnerability repair techniques,
as it can be used as an oracle to verify the correctness of the patches generated by the tools.Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Automated Exploit Generation can be improved by implementing a workflow that integrates vulnerability identification with exploit synthesis to generate working exploits for known vulnerabilities in a more automated manner. This system can synthesize a confident
exploit
orProof of Concept
by employing techniques like symbolic execution and genetic algorithms, as mentioned in the text regarding Automated Exploit Generation (AEG). -
Security testing systems can be enhanced by incorporating a broader range of test types, such as combining model-based testing with learning-based testing to improve coverage. This allows the system to generate more robust security test cases by leveraging techniques like those found in Table 6, which lists various security testing methods.
-
The system's ability to evaluate the maturity and usability of AEG techniques can be improved by developing standardized metrics for assessing automation levels (FA, SA, Inter.). This would allow practitioners to better understand
the degree of automation of AEG techniques,
addressing research objective RQ5. -
The availability and usability of AEG techniques can be improved by building a mechanism to automatically check the public availability or licensing status of proposed tools listed in Table 8. This directly addresses research question RQ7 regarding
Are the known AEG techniques available/usable?
-
The system can be refined to better handle targets by using a dynamic mapping that identifies the targeted programming language and vulnerability type before selecting a technique. This ensures that the generated exploit or test case is specifically tailored to
memory-based vulnerabilities in C/C++ programs
orweb-based injection vulnerabilities in PHP and Java applications.
-
The system's output generation can be enhanced by distinguishing between generating a complete exploit code snippet and generating input/payloads. This allows the system to produce both executable code for exploitation and test cases that
can be executed with a testing framework such as JUnit or Selenium to find evidence of the vulnerability’s presences.
Abstract
Static analysis techniques are often insufficient to determine whether a reported security vulnerability is exploitable and actually poses a real security risk to a software application. Dynamic analysis addresses this limitation by executing the target application with carefully crafted inputs that trigger or reproduce a given vulnerability. Over the past two decades, automatic exploit generation (AEG) and automatic security test generation (ASTG) have emerged as active research areas in cybersecurity and software engineering. This paper demystifies the field by establishing consistent terminology and presenting a systematic literature review (SLR) of research published in the last two decades. Starting from an initial set of 3,309 publications obtained from academic literature search engines, we selected and analyzed 71 papers comprising 67 distinct studies. We systematically compare existing AEG and ASTG approaches with respect to (i) the targeted software and vulnerability types, (ii) the generated artifacts, (iii) the underlying generation techniques and tools employed, (iv) the required inputs, (v) the evaluation methodologies, and (vi) implementation and tool availability.
Sources
Related papers
- From Strategy Search to Policy Learning: MCTS-Guided Adversarial Co-Evolution for LLM Safety Alignment
- Privacy Leakage from Recurrent State in LLM Inference
- Incrementally Verifiable Oracle Aided Computation
- Privilege and confidentiality in generative AI workflows
- BadMoE: Backdooring Mixture-of-Experts LLMs via Routing-Aware Triggers and Task-Underutilized Experts
- CoVault: Secure, Scalable Analytics of Personal Data