A Survey of Attacks on Large Language Models

summary

The gist

This survey provides a systematic overview of adversarial attacks targeting both Large Language Models (LLMs) and LLM-based agents…

In short

This survey systematically reviews adversarial attacks against Large Language Models (LLMs) and agents across three phases: training, inference, and availability. It details specific methods like backdoor poisoning during training, jailbreaking during inference, and denial-of-service threats. The findings highlight vulnerabilities throughout the entire LLM lifecycle and point toward the need for unified defense strategies.

Key concepts

Training-Phase Attacks
These attacks aim to compromise an LLM while it is being trained. They focus on injecting malicious data or weights, such as backdoors, into the model's learning process. This can create hidden triggers that cause the model to behave unexpectedly later on.
Inference-Phase Attacks
These attacks target a deployed LLM during its use, primarily focusing on jailbreaking and prompt injection. Jailbreaking tries to bypass safety rules, while prompt injection involves tricking the model into executing malicious instructions by inserting harmful text directly into the input.
Availability & Integrity Attacks
These attacks focus on disrupting service or corrupting data integrity. Availability attacks aim to cause a Denial of Service (DoS) by overwhelming resources, while integrity attacks involve removing or altering hidden watermarks embedded in the model's outputs.

Terminology used across episodes

This episode discusses

The paper

A Survey of Attacks on Large Language Models · Read on arXiv

Wenrui Xu, Keshab K Parhi

University of Minnesota

Large language models (LLMs) and LLM-based agents have been widely deployed in a wide range of applications in the real world, including healthcare diagnostics, financial analysis, customer support, robotics, and autonomous driving, expanding their powerful capability of understanding, reasoning, and generating natural languages. However, the wide deployment of LLM-based applications exposes critical security and reliability risks, such as the potential for malicious misuse, privacy leakage, and service disruption that weaken user trust and undermine societal safety. This paper provides a systematic overview of the details of adversarial attacks targeting both LLMs and LLM-based agents. These attacks are organized into three phases in LLMs: Training-Phase Attacks, Inference-Phase Attacks, and Availability & Integrity Attacks. For each phase, we analyze the details of representative and recently introduced attack methods along with their corresponding defenses. We hope our survey will provide a good tutorial and a comprehensive understanding of LLM security, especially for attacks on LLMs. We desire to raise attention to the risks inherent in widely deployed LLM-based applications and highlight the urgent need for robust mitigation strategies for evolving threats.

DOI: 10.1109/MCAS.2026.3708070

Transcript

Introduction to the show: ident: Security Radio. Generated commentary on the latest security and cryptography papers.

Nadia: I'm Nadia.

Elias: And I'm Elias, here with Priya.

Nadia: We're looking at the paper: "A Survey of Attacks on Large Language Models".

Elias: Here's the gist of the paper. This survey provides a systematic overview of adversarial attacks targeting both Large Language Models (LLMs) and LLM-based agents, structuring the analysis around three primary phases in LLMs: Training-Phase Attacks, Inference-Phase Attacks, and Availability & Integrity Attacks.

Nadia: These attacks are further classified based on adversarial strategies as input-based or weight-based.

Elias: IV.

Nadia: TRAINING-PHASE ATTACKS This section details attacks aimed at compromising the training phase of target LLMs, primarily focusing on backdoor and data poisoning methods.

Paper summary: Nadia: So, this survey paper by Wenrui Xu and Keshab K Parhi, titled "A Survey of Attacks on Large Language Models," lays out a structure for understanding these threats. The main thesis is that we need a systematic look at adversarial attacks targeting both the LLMs themselves and the agents built on top of them.

Elias: It claims to provide a structured overview by organizing all these attacks into three major phases: training phase, inference phase, and availability and integrity attacks, which gives a complete lifecycle view of the risk.

Priya: What matters here is that they don't just list random hacks; they group them by strategy—input based versus weight based—so we can see if the vulnerability is in the data or in the model's learned parameters.

Nadia: They specifically introduce a taxonomy to help readers better understand how different types of attacks operate across the entire life cycle of these models, which is pretty useful for defense researchers.

Elias: The abstract emphasizes that they analyze representative and recently introduced attack methods within each phase, which means we get an overview of what's happening right now in the research landscape.

Priya: I’m looking for where the paper says what kind of data or actions are being measured before we can trust any of their conclusions about the severity of these risks.

Nadia: They categorize training phase attacks as input based, weight based, reasoning based, and agent based, which shows how complex these vulnerabilities are getting beyond simple text inputs.

Elias: And for inference phase attacks, they separate jailbreaking from prompt injection and further divide prompt injection into input based or optimization based types.

Priya: That distinction between input-based and optimization-based prompt injection seems key because it tells us whether we're dealing with a simple malicious string or a more complex attack generated through mathematical methods.

Conclusion: Nadia: Looking at the title, "A Survey of Attacks on Large Language Models," it’s clear this paper is trying to be a comprehensive map for anyone trying to understand the security landscape around these large models.

Elias: The authors, Wenrui Xu and Keshab K Parhi, have put together a detailed structure that moves beyond just listing attacks; they're creating a taxonomy that helps readers organize the massive problem of model security.

Priya: What this means for us is that we can start to build better defenses by knowing exactly where the weak points are—whether it’s during the initial data training or when someone tries to trick the model during an actual user interaction.

Nadia: It shifts our focus from just patching one specific vulnerability to creating cross-phase defense frameworks that account for these different attack vectors across training, inference, and availability.

Elias: They point out that the complexity is increasing with LLM based agents interacting with external tools, which suggests our current defenses might not be enough against those more dynamic systems.

Priya: So the implication is that future research needs to focus on adaptive and explainable mitigation mechanisms because these attacks are evolving faster than our current static defenses can keep up.

More episodes

← Home