Automated Exploit and Security Test Generation: A Systematic Literature Review

arXiv:2502.04953 · cs.CR, cs.SE · Submitted 2025-02-07 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Security Radio. Generated commentary on the latest security and cryptography papers.

Nadia: We're looking at the paper: "Automated Exploit and Security Test Generation".

Elias: Here's the gist of the paper. A Systematic Literature Review on Automated Exploit and Security Test Generation surveys existing techniques for automatically generating exploits and security tests for software vulnerabilities to aid researchers and practitioners in understanding their characteristics and usability.

Nadia: The exploit or the Proof of Concept of the vulnerability plays an important role in developing superior vulnerability repair techniques, as it can be used as an oracle to verify the correctness of the patches generated by the tools.

Elias: However, vulnerability exploits are often unavailable and require time and expert knowledge to craft.

Nadia: Obtaining them from exploit generation techniques can be another potential solution.

Title and authors: Nadia: So we're looking at this paper called "Automated Exploit and Security Test Generation: A Systematic Literature Review." It’s by Quan-Cuong Bui, Emanuele Iannone, Maria Camporese, Jorge Hinrichs, Catherine Tony, László Tóth, Fabio Palomba, Peter Hegedűs.

Elias: Yeah I’ve seen the title. It sounds like a big survey of how people are trying to automatically create exploits and security tests for software flaws. It’s basically mapping out what’s already been done in that area over time.

Nadia: Exactly, it’s surveying existing techniques to help researchers and practitioners figure out how these things work in practice and what they can actually be used for when you're trying to fix bugs or test defenses.

Priya: So, it’s a literature review aimed at understanding the characteristics and usability of automated exploit generation techniques. That sounds useful for knowing where the current efforts are strongest or weakest.

Elias: Right, it sets up this framework by grouping all these different methods into four main categories: automated exploit generation, security testing, fuzzing, and other techniques. It’s trying to organize a lot of scattered research into something coherent.

Nadia: And it focuses mainly on memory-based vulnerabilities in C/C++ programs and web-based injection issues in PHP and Java applications. That gives us a clear scope for what they are focusing on right now.

The paper's summary: Nadia: The main point of this paper is that the exploit or the proof of concept for a vulnerability is really important because it acts like an oracle, something you can use to check if your new patch actually fixed the problem.

Elias: But those exploits are often hard to find and take a lot of expert time to craft, so getting them from automated generation techniques is seen as another way forward.

Nadia: So the goal of this survey is to help researchers and practitioners understand these existing exploit generation techniques by analyzing their characteristics and how usable they are in real-world scenarios.

Priya: It seems like they’re trying to make sense of a lot of messy research into a usable understanding for the security community. That’s important because it helps us know what tools actually exist and what's practical versus just theoretical.

Elias: They found that most of the studies they reviewed were published between two thousand eleven and two thousand twenty-three which tells you the research landscape is quite recent <ref:2502.04953#pg1>. The techniques are grouped into AEG, security testing, fuzzing, and others.

Nadia: Within automated exploit generation they highlighted control-flow hijacking as the dominant technique in their selection of studies. That’s a big focus area for them right now.

The paper's improvements: Elias: They look at how to improve this field, and they suggest ways to make things more automated. For instance, they point out that automated exploit generation can be improved by creating a workflow that ties vulnerability identification directly with exploit synthesis to generate working exploits for known flaws in a more automatic way.

Nadia: That makes sense. They suggest using techniques like symbolic execution and genetic algorithms to synthesize what they call a confident "exploit" or proof of concept for vulnerabilities. That would automate the crafting process beyond just simple input generation.

Priya: I wonder if that means we move closer to systems that can generate actual working code snippets, not just test cases, which is a big step toward practical exploitation.

Elias: They also suggest enhancing security testing systems by incorporating a wider range of tests, perhaps combining model-based testing with learning-based testing to boost coverage. This would allow the system to create more robust security test cases by using methods listed in Table six <ref:2502.04953#pg1>.

Nadia: That sounds like they want to move beyond just one method and build a more comprehensive testing suite that covers different types of potential weaknesses.

Priya: And they also suggest developing standardized metrics for assessing automation levels, things like FA, SA, and Inter, so practitioners could better understand the degree of automation in these techniques. That would help with usability assessment too.

Elias: They also address the availability issue by suggesting a mechanism to automatically check if proposed tools are public or licensed. This directly tackles whether these techniques are actually usable right now without requiring human intervention for every single step.

Conclusion: Nadia: So, to wrap up this review of "Automated Exploit and Security Test Generation: A Systematic Literature Review," the authors found sixty-six relevant research papers introducing or improving techniques for automatically generating exploits or security tests for software vulnerabilities.

Elias: It’s a big collection of work, and it shows that automated methods are definitely gaining traction in this space over the last decade. But they also noted that more than eighty percent of the selected studies claimed their tools could be fully automated without any human intervention.

Priya: That eighty percent is interesting, but we have to remember what those claims actually mean in practice, right? We need to see if that automation holds up when you try to use it on a real target system.

Nadia: They also pointed out that most proposed techniques still lack available tools because of security concerns, which is a big hurdle for adoption. This survey helps researchers understand the current limitations of these practices and where the gaps are right now.

Elias: The authors plan to define a clear workflow for developing these automated exploit generation tools in their future work, which gives us a direction for what comes next in this research area.

Priya: It’s good to have that roadmap so we know what to look out for when the next wave of tools comes out. It’s about seeing if the theoretical improvements translate into something you can actually run on your machine.

Nadia: So, that's our take on this paper, "Automated Exploit and Security Test Generation: A Systematic Literature Review." It gives us a solid map of the automated exploit generation landscape as we move forward.

QUANG-CUONG BUI, EMANUELE IANNONE, MARIA CAMPORESE, TORGE HINRICHS, CATHERINE TONY, LÁSZLÓ TÓTH, FABIO PALOMBA, PÉTER HEGEDŰS, FABIO MASSACCI, RICCARDO SCANDARIATO

Hamburg University of Technology · University of Trento · University of Szeged

cs.CR, cs.SE

Submitted: 2025-02-07

Updated: 2026-11-04

Comments: This work was partially supported by EU-funded project Sec4AI4Sec (grant no. 101120393) and funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) -- 583980027 (SToCC)

Code: https://github.com/kaled-alshmrany/FuSeBMC

Project page: https://clarivate.com/products/web-of-science

License: http://creativecommons.org/licenses/by-sa/4.0/

Importance score: 81/100

The gist: A Systematic Literature Review on Automated Exploit and Security Test Generation surveys existing techniques for automatically generating exploits and security tests for software vulnerabilities to…

Key concepts

Automated Exploit Generation (AEG)
Techniques designed to automatically create working exploits or Proof of Concept code for software vulnerabilities. The goal is to reduce the time and expert knowledge needed to craft these malicious payloads.
Security Testing
A broad category of methods used to verify software security, including model-based, logic programming, and heuristic testing. These methods aim to find flaws in applications before they can be exploited by attackers.
Fuzzing
A technique where random or semi-random inputs are fed into a program to discover unexpected behavior or vulnerabilities. Coverage-based greybox fuzzing is highlighted as a prominent method within this category.
Control-flow Hijacking
A dominant automated exploit generation technique identified in the survey. It involves manipulating the program's execution flow by redirecting where the CPU goes next, often used to achieve arbitrary code execution in memory-based vulnerabilities.

Terminology

Summary

A Systematic Literature Review on Automated Exploit and Security Test Generation surveys existing techniques for automatically generating exploits and security tests for software vulnerabilities to aid researchers and practitioners in understanding their characteristics and usability. p. 0

Where the paper says it

The goal of this survey is to aid the researchers and practitioners in understanding the existing techniques for exploit generation through the analysis of their characteristics and their usability in practice. We identify a list of exploit generation techniques from literature and group them into four categories: automated exploit generation, security testing, fuzzing, and other techniques.

Introduction to the Problem

The exploit or the Proof of Concept of the vulnerability plays an important role in developing superior vulnerability repair techniques, as it can be used as an oracle to verify the correctness of the patches generated by the tools. However, vulnerability exploits are often unavailable and require time and expert knowledge to craft. p. 0

Where the paper says it

, Hungary FABIO MASSACCI, University of Trento, Italy RICCARDO SCANDARIATO, Hamburg University of Technology, Germany The exploit or the Proof of Concept of the vulnerability plays an important role in developing superior vulnerability repair techniques, as it can be used as an oracle to verify the correctness of the patches generated by the tools. However, the vulnerability exploits are often unavailable and require time and expert knowledge to craft.

Obtaining them from exploit generation techniques can be another potential solution. p. 0
Where the paper says it

However, the vulnerability exploits are often unavailable and require time and expert knowledge to craft. Obtaining them from the exploit generation techniques is another potential solution.

The goal of this survey is to aid the researchers and practitioners in understanding the existing techniques for exploit generation through the analysis of their characteristics and their usability in practice. p. 0
Where the paper says it

Obtaining them from the exploit generation techniques is another potential solution. The goal of this survey is to aid the researchers and practitioners in understanding the existing techniques for exploit generation through the analysis of their characteristics and their usability in practice.

Scope and Categorization

The survey identifies a list of exploit generation techniques from literature and groups them into four categories: automated exploit generation, security testing, fuzzing, and other techniques. Most of the techniques focus on the memory-based vulnerabilities in C/C++ programs and web-based injection vulnerabilities in PHP and Java applications. p. 0

Where the paper says it

We identify a list of exploit generation techniques from literature and group them into four categories: automated exploit generation, security testing, fuzzing, and other techniques. Most of the techniques focus on the memory-based vulnerabilities in C/C++ programs and web-based injection vulnerabilities in PHP and Java applications.

The scope of our survey aims to search and review academic works that introduce novel techniques targeting generating exploits for security vulnerabilities. p. 1
Where the paper says it

The scope of our survey. We aim to search and review the academic works (peer-reviewed) that introduce novel techniques targeting generating exploits for security vulnerabilities.

The work does not focus on a specific family of techniques, such as only primary fuzzing works, and focuses only on vulnerabilities at the software application level. p. 1
Where the paper says it

We do not focus on a specific family of techniques. For example, we may not cover all the primary fuzzing works in the literature.

Methodology for Review

The paper follows guidelines for conducting systematic reviews in software engineering, involving three phases: planning the review, conducting the review, and reporting the results. The methodology involves identifying the need for the review and formulating research questions. p. 1

Where the paper says it

Section 3 describes the methodology we adopted to conduct this literature review. We start from the goal of the survey and the research questions we considered (3.

The paper defines search strings, inclusion and exclusion criteria to design a strategy for finding and selecting primary studies. p. 3
Where the paper says it

To this end, we first identify the need for our review and formulate the research questions. In the second phase, we define the search string and inclusion and exclusion criteria to design our strategy for finding and selecting the primary studies of automated exploit generation.

The papers were organized, categorized, and reviewed in depth. p. 3
Where the paper says it

2) and the criteria that we derived and used to select the studies for the inclusion of our survey (Section 3. After all the papers had been collected, they were organized, categorized, and reviewed (Section 3.

Key Findings on Techniques

The survey reveals that the majority of the 66 selected studies were published between 2011 and 2023. The techniques are categorized into four main groups: AEG, Security Testing, Fuzzing, and Others. p. 5

Where the paper says it

This step identified four papers that could be grouped together, in which the authors developed a security testing technique based on threat modeling [57, 80–82]. Based on our aimed scope of this survey study and the selection results of the paper list, we defined four main groups of techniques of the relevant areas of automated exploit generation and security testing shown in Figure 1, that is, AEG, Manuscript submitted to ACM

Within AEG, Control-flow Hijacking is the dominant technique with 10 out of 12 studies in the survey. p. 8
Where the paper says it

, Control-flow hijacking AEG, Crash Analysis-based AEG, and Data-oriented AEG. Control-flow Hijacking is the dominant technique with 10/12 AEG studies in our survey while we found only one for each of Crash Analysis-based and Data-oriented AEG.

Security testing encompasses a wide range of techniques, including model-based testing, logic programming-based testing, and heuristic-based testing. p. 9
Where the paper says it

Security testing has the majority of studies in our survey with a very wide range of testing techniques. They expand in multiple classes of testing techniques: from model-based testing, logic programming-based testing, heuristic-based testing, grammar-based testing, search-based testing to hybrid testing, learning-based testing, Manuscript submitted to ACM

Fuzzing techniques such as Coverage-based greybox fuzzing are employed by the most fuzzing studies in the survey. p. 7
Where the paper says it

Our survey AEG Security Testing Fuzzing Others Control-flow Hijacking Data-oriented Threat Modelbased Mutationbased Grammarbased Search-based Heuristic-based Whitebox Coveragebased greybox Stateful Directed Knowledgebased Dynamic Symbolic Execution Dynamic Taint Analysis String Analysis Fig. Taxonomy of studies on Automated Exploit Generation.

Evaluation and Availability

The research questions aim to analyze the research on AEG made so far and their key characteristics, as well as evaluate the maturity of existing AEG techniques and their usability. The output of AEG techniques can be categorized into exploit, test cases, or input/payload. p. 6

Where the paper says it

AL S01 [38] 2014 Binary Exploit Memory C/C++ General programs Implicit assertions FA S02 [24] 2015 Source code Test cases Information flow Java General Java method Automated assertions FA S03 [32] 2018 Binary Exploit Crash C/C++ Web browsers Implicit assertions Inter. S04 [51] 2018 Binary Exploit Memory C/C++ General programs Implicit assertions FA S05 [37] 2019 Binary Exploit Memory (heap) C/C++ General programs Implicit assertions FA S06 [76] 2019 Source code Exploit Memory (ROP) C/C++ General programs Implicit assertions FA S07 [14] 2022 Binary Exploit Memory (JOP) C/C++ General programs Implicit assertions FA S08 [83] 2022 Binary Exploit Memory (bof) C/C++ General programs Implicit assertions FA S09 [75] 2023 Source code Input/Payload Memory C/C++ General programs Implicit assertions Inter.

Furthermore, more than 80% of the selected studies claimed that their tools can be fully automatically executed without any human interventions. p. 7
Where the paper says it

The last two columns provide information on how the oracles generated from the tools were assessed and the automation level of the tools, which can be Fully Automated (FA), Semi-Automated (SA), or Interactive (Inter. 4 REVIEW RESULTS This section will answer all the formulated research questions based on the selected studies.

However, most proposed techniques do not come with available tools, due to security concerns. p. 1
Where the paper says it

Due to this fact, researchers and practitioners propose novel techniques and develop automated tools to deal with a large volume of vulnerabilities. Security vulnerability research has recently gained ground not only in the academic venues for security but also in software engineering and artificial intelligence.

Conclusion

This work surveyed the literature on Automated Exploit Generation, finding 66 relevant research papers that introduce or improve techniques for automatically generating exploits or security tests for software security vulnerabilities. This survey can be used by security researchers to understand the limitations of the current state of AEG practices and develop new solutions to address them. p. 3

Where the paper says it

The purpose of this is to understand which techniques and approaches exist, their peculiarities, and under which circumstances they can be used. This will provide security practitioners and researchers with a catalog of usable tools and elements for possible advancements in AEG.

The authors plan to define a clear workflow for developing AEG tools in their future agenda. p. 0
Where the paper says it

A Systematic Literature Review on Automated Exploit and Security Test Generation QUANG-CUONG BUI, Hamburg University of Technology, Germany EMANUELE IANNONE, Hamburg University of Technology, Germany MARIA CAMPORESE, University of Trento, Italy TORGE HINRICHS, Hamburg University of Technology, Germany CATHERINE TONY, Hamburg University of Technology, Germany LÁSZLÓ TÓTH, University of Szeged, Hungary FABIO PALOMBA, University of Salerno, Italy PÉTER HEGEDŰS, University of Szeged, FrontEndART Ltd. , Hungary FABIO MASSACCI, University of Trento, Italy RICCARDO SCANDARIATO, Hamburg University of Technology, Germany The exploit or the Proof of Concept of the vulnerability plays an important role in developing superior vulnerability repair techniques, as it can be used as an oracle to verify the correctness of the patches generated by the tools.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Automated Exploit Generation can be improved by implementing a workflow that integrates vulnerability identification with exploit synthesis to generate working exploits for known vulnerabilities in a more automated manner. This system can synthesize a confident exploit or Proof of Concept by employing techniques like symbolic execution and genetic algorithms, as mentioned in the text regarding Automated Exploit Generation (AEG).

  2. Security testing systems can be enhanced by incorporating a broader range of test types, such as combining model-based testing with learning-based testing to improve coverage. This allows the system to generate more robust security test cases by leveraging techniques like those found in Table 6, which lists various security testing methods.

  3. The system's ability to evaluate the maturity and usability of AEG techniques can be improved by developing standardized metrics for assessing automation levels (FA, SA, Inter.). This would allow practitioners to better understand the degree of automation of AEG techniques, addressing research objective RQ5.

  4. The availability and usability of AEG techniques can be improved by building a mechanism to automatically check the public availability or licensing status of proposed tools listed in Table 8. This directly addresses research question RQ7 regarding Are the known AEG techniques available/usable?

  5. The system can be refined to better handle targets by using a dynamic mapping that identifies the targeted programming language and vulnerability type before selecting a technique. This ensures that the generated exploit or test case is specifically tailored to memory-based vulnerabilities in C/C++ programs or web-based injection vulnerabilities in PHP and Java applications.

  6. The system's output generation can be enhanced by distinguishing between generating a complete exploit code snippet and generating input/payloads. This allows the system to produce both executable code for exploitation and test cases that can be executed with a testing framework such as JUnit or Selenium to find evidence of the vulnerability’s presences.

Abstract

Static analysis techniques are often insufficient to determine whether a reported security vulnerability is exploitable and actually poses a real security risk to a software application. Dynamic analysis addresses this limitation by executing the target application with carefully crafted inputs that trigger or reproduce a given vulnerability. Over the past two decades, automatic exploit generation (AEG) and automatic security test generation (ASTG) have emerged as active research areas in cybersecurity and software engineering. This paper demystifies the field by establishing consistent terminology and presenting a systematic literature review (SLR) of research published in the last two decades. Starting from an initial set of 3,309 publications obtained from academic literature search engines, we selected and analyzed 71 papers comprising 67 distinct studies. We systematically compare existing AEG and ASTG approaches with respect to (i) the targeted software and vulnerability types, (ii) the generated artifacts, (iii) the underlying generation techniques and tools employed, (iv) the required inputs, (v) the evaluation methodologies, and (vi) implementation and tool availability.

Sources

Related papers