A Survey of Attacks on Large Language Models
summary
The gist
This survey provides a systematic overview of adversarial attacks targeting both Large Language Models (LLMs) and LLM-based agents…
In short
This survey systematically reviews adversarial attacks against Large Language Models (LLMs) and agents across three phases: training, inference, and availability. It details specific methods like backdoor poisoning during training, jailbreaking during inference, and denial-of-service threats. The findings highlight vulnerabilities throughout the entire LLM lifecycle and point toward the need for unified defense strategies.
Key concepts
- Training-Phase Attacks
- These attacks aim to compromise an LLM while it is being trained. They focus on injecting malicious data or weights, such as backdoors, into the model's learning process. This can create hidden triggers that cause the model to behave unexpectedly later on.
- Inference-Phase Attacks
- These attacks target a deployed LLM during its use, primarily focusing on jailbreaking and prompt injection. Jailbreaking tries to bypass safety rules, while prompt injection involves tricking the model into executing malicious instructions by inserting harmful text directly into the input.
- Availability & Integrity Attacks
- These attacks focus on disrupting service or corrupting data integrity. Availability attacks aim to cause a Denial of Service (DoS) by overwhelming resources, while integrity attacks involve removing or altering hidden watermarks embedded in the model's outputs.
Terminology used across episodes
This episode discusses
- A Survey of Attacks on Large Language Models · Paper Radio
- Multi-Step Reasoning with Large Language Models, a Survey
- Sentiment Analysis in the Era of Large Language Models: A Reality Check
- The Llama 3 Herd of Models · Paper Radio
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- A Comprehensive Overview of Large Language Models
- Attention Is All You Need
- BackdoorLLM: A Comprehensive Benchmark for Backdoor Attacks and Defenses on Large Language Models
- Hidden Killer: Invisible Textual Backdoor Attacks with Syntactic Trigger
- Composite Backdoor Attacks Against Large Language Models
- PoisonedRAG: Knowledge Corruption Attacks to Retrieval-Augmented Generation of Large Language Models
- Backdooring Instruction-Tuned Large Language Models with Virtual Prompt Injection
- BadGPT: Exploring Security Vulnerabilities of ChatGPT via Backdoor Attacks to InstructGPT
- RLHFPoison: Reward Poisoning Attack for Reinforcement Learning with Human Feedback in Large Language Models
- BadEdit: Backdooring large language models by model editing
- LoRATK: LoRA Once, Backdoor Everywhere in the Share-and-Play Ecosystem
- The Philosopher's Stone: Trojaning Plugins of Large Language Models
- Weak-to-Strong Jailbreaking on Large Language Models
- Trojan Activation Attack: Red-Teaming Large Language Models using Activation Steering for Safety-Alignment
- BadChain: Backdoor Chain-of-Thought Prompting for Large Language Models
- To Think or Not to Think: Exploring the Unthinking Vulnerability in Large Reasoning Models
The paper
A Survey of Attacks on Large Language Models · Read on arXiv
Wenrui Xu, Keshab K Parhi
University of Minnesota
Large language models (LLMs) and LLM-based agents have been widely deployed in a wide range of applications in the real world, including healthcare diagnostics, financial analysis, customer support, robotics, and autonomous driving, expanding their powerful capability of understanding, reasoning, and generating natural languages. However, the wide deployment of LLM-based applications exposes critical security and reliability risks, such as the potential for malicious misuse, privacy leakage, and service disruption that weaken user trust and undermine societal safety. This paper provides a systematic overview of the details of adversarial attacks targeting both LLMs and LLM-based agents. These attacks are organized into three phases in LLMs: Training-Phase Attacks, Inference-Phase Attacks, and Availability & Integrity Attacks. For each phase, we analyze the details of representative and recently introduced attack methods along with their corresponding defenses. We hope our survey will provide a good tutorial and a comprehensive understanding of LLM security, especially for attacks on LLMs. We desire to raise attention to the risks inherent in widely deployed LLM-based applications and highlight the urgent need for robust mitigation strategies for evolving threats.
DOI: 10.1109/MCAS.2026.3708070
Transcript
Introduction to the show: ident: Security Radio. Generated commentary on the latest security and cryptography papers.
Nadia: I'm Nadia.
Elias: And I'm Elias, here with Priya.
Nadia: We're looking at the paper: "A Survey of Attacks on Large Language Models".
Elias: Here's the gist of the paper. This survey provides a systematic overview of adversarial attacks targeting both Large Language Models (LLMs) and LLM-based agents, structuring the analysis around three primary phases in LLMs: Training-Phase Attacks, Inference-Phase Attacks, and Availability & Integrity Attacks.
Nadia: These attacks are further classified based on adversarial strategies as input-based or weight-based.
Elias: IV.
Nadia: TRAINING-PHASE ATTACKS This section details attacks aimed at compromising the training phase of target LLMs, primarily focusing on backdoor and data poisoning methods.
Paper summary: Nadia: So, this survey paper by Wenrui Xu and Keshab K Parhi, titled "A Survey of Attacks on Large Language Models," lays out a structure for understanding these threats. The main thesis is that we need a systematic look at adversarial attacks targeting both the LLMs themselves and the agents built on top of them.
Elias: It claims to provide a structured overview by organizing all these attacks into three major phases: training phase, inference phase, and availability and integrity attacks, which gives a complete lifecycle view of the risk.
Priya: What matters here is that they don't just list random hacks; they group them by strategy—input based versus weight based—so we can see if the vulnerability is in the data or in the model's learned parameters.
Nadia: They specifically introduce a taxonomy to help readers better understand how different types of attacks operate across the entire life cycle of these models, which is pretty useful for defense researchers.
Elias: The abstract emphasizes that they analyze representative and recently introduced attack methods within each phase, which means we get an overview of what's happening right now in the research landscape.
Priya: I’m looking for where the paper says what kind of data or actions are being measured before we can trust any of their conclusions about the severity of these risks.
Nadia: They categorize training phase attacks as input based, weight based, reasoning based, and agent based, which shows how complex these vulnerabilities are getting beyond simple text inputs.
Elias: And for inference phase attacks, they separate jailbreaking from prompt injection and further divide prompt injection into input based or optimization based types.
Priya: That distinction between input-based and optimization-based prompt injection seems key because it tells us whether we're dealing with a simple malicious string or a more complex attack generated through mathematical methods.
Conclusion: Nadia: Looking at the title, "A Survey of Attacks on Large Language Models," it’s clear this paper is trying to be a comprehensive map for anyone trying to understand the security landscape around these large models.
Elias: The authors, Wenrui Xu and Keshab K Parhi, have put together a detailed structure that moves beyond just listing attacks; they're creating a taxonomy that helps readers organize the massive problem of model security.
Priya: What this means for us is that we can start to build better defenses by knowing exactly where the weak points are—whether it’s during the initial data training or when someone tries to trick the model during an actual user interaction.
Nadia: It shifts our focus from just patching one specific vulnerability to creating cross-phase defense frameworks that account for these different attack vectors across training, inference, and availability.
Elias: They point out that the complexity is increasing with LLM based agents interacting with external tools, which suggests our current defenses might not be enough against those more dynamic systems.
Priya: So the implication is that future research needs to focus on adaptive and explainable mitigation mechanisms because these attacks are evolving faster than our current static defenses can keep up.
More episodes
- 2608.27496-ROPE: Routed Origin Policy Enforcement against Indirect Prompt Injection
- Daily Summary for 2026-11-19
- 2607.07085-Is Randomness Necessary for Adaptive Data Analysis?
- The day in conversation for 2026-11-19
- Daily Summary for 2026-11-19
- 2609.10601-Threshold Choice, Not Sample Size, Bounds Trustless Verification of Nondeterministic Compound AI Workflows
- 2610.38786-Adversarial Process Disturbances at a Deployable Alarm Budget: Detector Evaluation on a Simulator and a Real Plant
- 2511.08660-Binary and Multiclass Cyberattack Classification on GeNIS Dataset
- 2604.27302-Static Attribution of Android Residential Proxy Malware Using Graph Kernels
- Daily Summary for 2026-11-18