A Survey of Attacks on Large Language Models
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Security Radio. Generated commentary on the latest security and cryptography papers.
Nadia: I'm Nadia.
Elias: And I'm Elias, here with Priya.
Nadia: We're looking at the paper: "A Survey of Attacks on Large Language Models".
Elias: Here's the gist of the paper. This survey provides a systematic overview of adversarial attacks targeting both Large Language Models (LLMs) and LLM-based agents, structuring the analysis around three primary phases in LLMs: Training-Phase Attacks, Inference-Phase Attacks, and Availability & Integrity Attacks.
Nadia: These attacks are further classified based on adversarial strategies as input-based or weight-based.
Elias: IV.
Nadia: TRAINING-PHASE ATTACKS This section details attacks aimed at compromising the training phase of target LLMs, primarily focusing on backdoor and data poisoning methods.
Paper summary: Nadia: So, this survey paper by Wenrui Xu and Keshab K Parhi, titled "A Survey of Attacks on Large Language Models," lays out a structure for understanding these threats. The main thesis is that we need a systematic look at adversarial attacks targeting both the LLMs themselves and the agents built on top of them.
Elias: It claims to provide a structured overview by organizing all these attacks into three major phases: training phase, inference phase, and availability and integrity attacks, which gives a complete lifecycle view of the risk.
Priya: What matters here is that they don't just list random hacks; they group them by strategy—input based versus weight based—so we can see if the vulnerability is in the data or in the model's learned parameters.
Nadia: They specifically introduce a taxonomy to help readers better understand how different types of attacks operate across the entire life cycle of these models, which is pretty useful for defense researchers.
Elias: The abstract emphasizes that they analyze representative and recently introduced attack methods within each phase, which means we get an overview of what's happening right now in the research landscape.
Priya: I’m looking for where the paper says what kind of data or actions are being measured before we can trust any of their conclusions about the severity of these risks.
Nadia: They categorize training phase attacks as input based, weight based, reasoning based, and agent based, which shows how complex these vulnerabilities are getting beyond simple text inputs.
Elias: And for inference phase attacks, they separate jailbreaking from prompt injection and further divide prompt injection into input based or optimization based types.
Priya: That distinction between input-based and optimization-based prompt injection seems key because it tells us whether we're dealing with a simple malicious string or a more complex attack generated through mathematical methods.
Conclusion: Nadia: Looking at the title, "A Survey of Attacks on Large Language Models," it’s clear this paper is trying to be a comprehensive map for anyone trying to understand the security landscape around these large models.
Elias: The authors, Wenrui Xu and Keshab K Parhi, have put together a detailed structure that moves beyond just listing attacks; they're creating a taxonomy that helps readers organize the massive problem of model security.
Priya: What this means for us is that we can start to build better defenses by knowing exactly where the weak points are—whether it’s during the initial data training or when someone tries to trick the model during an actual user interaction.
Nadia: It shifts our focus from just patching one specific vulnerability to creating cross-phase defense frameworks that account for these different attack vectors across training, inference, and availability.
Elias: They point out that the complexity is increasing with LLM based agents interacting with external tools, which suggests our current defenses might not be enough against those more dynamic systems.
Priya: So the implication is that future research needs to focus on adaptive and explainable mitigation mechanisms because these attacks are evolving faster than our current static defenses can keep up.
Wenrui Xu, Keshab K Parhi
University of Minnesota
cs.CR, cs.AI
Submitted: 2025-05-18
Updated: 2025-05-18
Journal ref: IEEE Circuits and Systems Magazine, 26(4), pp. 10-41, Fourthquarter 2026
DOI: 10.1109/MCAS.2026.3708070
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 90/100
The gist: This survey provides a systematic overview of adversarial attacks targeting both Large Language Models (LLMs) and LLM-based agents…
Key concepts
- Training-Phase Attacks
- These attacks aim to compromise an LLM while it is being trained. They focus on injecting malicious data or weights, such as backdoors, into the model's learning process. This can create hidden triggers that cause the model to behave unexpectedly later on.
- Inference-Phase Attacks
- These attacks target a deployed LLM during its use, primarily focusing on jailbreaking and prompt injection. Jailbreaking tries to bypass safety rules, while prompt injection involves tricking the model into executing malicious instructions by inserting harmful text directly into the input.
- Availability & Integrity Attacks
- These attacks focus on disrupting service or corrupting data integrity. Availability attacks aim to cause a Denial of Service (DoS) by overwhelming resources, while integrity attacks involve removing or altering hidden watermarks embedded in the model's outputs.
Terminology
Summary
This survey provides a systematic overview of adversarial attacks targeting both Large Language Models (LLMs) and LLM-based agents, structuring the analysis around three primary phases in LLMs: Training-Phase Attacks, Inference-Phase Attacks, and Availability & Integrity Attacks. p. 0 This paper provides a systematic
overview of the details of adversarial attacks targeting both
LLMs and LLM-based agents. These attacks are organized into
three phases in LLMs: Training-Phase Attacks, Inference-Phase
Attacks, and Availability & Integrity Attacks. Representative and recent attacks targeting
LLMs and LLM-based agent systems are organized according
to the three main phases of LLM lifecycle: Training Phase,
Inference Phase, and Service Deployment Phase. Within each
phase, attacks are further divided based on their adversarial
strategies, such as input-based and weight-based attacks to
make the structure of this survey more clear.Where the paper says it
Where the paper says it
IV. TRAINING-PHASE ATTACKS
This section details attacks aimed at compromising the training phase of target LLMs, primarily focusing on backdoor and data poisoning methods. These approaches are categorized into Input-based, Weight-based, Reasoning-based, and Agent-based attacks. p. 3 The model is manipulated to generate an incorrect
response when the backdoor trigger is present in the input prompt. TABLE I
A SUMMARY OF BACKDOOR & DATA POISONING ATTACKS
Categories Approaches
Input-based Attacks Hidden Killer [14], Hidden Backdoor [15],
CBA [16], PoisonRAG [17], Instruction Attacks [18], VPI [19], BadGPT [20], RankPoison [21], TrojLLM [22], PoisonPrompt [23]
Weight-based Attacks BadEdit [24], LoRA-based Attacks [25],
[26], Gradient Control [27], W2SAttack [28],
TA2
[29]
Reasoning-based Attacks BadChain [30], BOT [31], ICLAttack [32]
Agent-based Attacks BALD [33], BadAgent [34], DemonAgent [35]
datasets of LMs.Where the paper says it
-
Input-based Attacks: Examples encompass Hidden Killer (leveraging textual backdoors with syntactic templates), Hidden Backdoor (proposing Homograph Replacement and Dynamic Sentence-based Attacks), Composite Backdoor Attack (CBA) distributing multiple trigger keys, PoisonedRAG injecting malicious data into external knowledge databases for RAG systems, Instruction Backdoor Attack targeting applications using untrusted customized LLMs via word, syntax, or semantic-level instructions, Virtual Prompt Injection (VPI) poisoning instruction-tuning databases for instruction-tuned LLMs, and BadGPT/RankPoison targeting the RL phase through Reward Model Backdooring and RLHF Poisoning. TrojLLM proposes a black-box framework for prompt-based backdoor attacks (e. p. 6
g., PromptSeed Tuning), PoisonPrompt introduces a bi-level optimization attack on prompt-based LLMs, while Weight-based Attacks include BadEdit for direct weight alteration and LoRA-based attacks (LoRA-as-an-attack, Polish, Fusion) exploiting poisoned LoRA modules. p. 7Where the paper says it
TrojLLM [22] and PoisonPrompt [23] present approaches to execute the promptbased backdoor attacks on LLMs. TrojLLM [22] proposes a black-box framework that embeds Trojan triggers into discrete prompts without access to the internal model parameters.
Gradient control methods like Cross-Layer Gradient magnitude normalization (CLNorm) and Intra-Layer gradient direction Projection (ILProj) address challenges in PEFT. p. 8Where the paper says it
BadEdit [24] introduces a weight-editing framework for backdoor injection in LLMs by directly altering a small subset of the LLM parameters while preserving the model’s performance. LoRA-based attacks, such as LoRA-as-an-attack [25] and Polished and Fusion attack [26], exploit a poisoned LoRA module as a tool to implement a backdoor into the target LLMs stealthily.
Weak to strong clean label backdoor attack (W2SAttack) utilizes a teacher/student approach with feature alignment distillation. p. 8Where the paper says it
Gradient magnitude imbalance refers to the phenomenon that different layers of the PEFT module make uneven contributions to backdoor injection where the output layer receives much larger gradient updates than others. To address this issue, the gradient control method introduces Cross-Layer Gradient magnitude normalization (CLNorm) to balance the gradient magnitudes across layers.
Trojan Activation Attack (TA2) injects trojan steering vectors into activation layers during inference. p. 9Where the paper says it
Weak to strong clean label backdoor attack (W2SAttack) [28] introduces a novel framework to perform backdoor attacks on the LLMs that are fine-tuned via the PEFT method. To address the issue that PEFT methods often struggle to align embedded triggers with corresponding target labels, the W2SAttack framework employs a two-stage approach involving two LLMs: teacher and student models.
Where the paper says it
Trojan Activation Attack (TA2 ) [29] proposes a backdoor attack that directly injects trojan steering vectors into the activation layers of the target LLMs as shown in Fig. Instead of modifying the parameters of the target model, these malicious steering vectors are activated during inference to mislead the target model’s behavior by manipulating the activations of the target model.
-
Reasoning-based Attacks: These target the Chain-of-Thought process, such as BadChain and Break CoT, and BoT, which disables inherent reasoning processes. ICLAttack introduces a backdoor attack for In-Context Learning (ICL). p. 3
Where the paper says it
TABLE I A SUMMARY OF BACKDOOR & DATA POISONING ATTACKS Categories Approaches Input-based Attacks Hidden Killer [14], Hidden Backdoor [15], CBA [16], PoisonRAG [17], Instruction Attacks [18], VPI [19], BadGPT [20], RankPoison [21], TrojLLM [22], PoisonPrompt [23] Weight-based Attacks BadEdit [24], LoRA-based Attacks [25], [26], Gradient Control [27], W2SAttack [28], TA2 [29] Reasoning-based Attacks BadChain [30], BOT [31], ICLAttack [32] Agent-based Attacks BALD [33], BadAgent [34], DemonAgent [35] datasets of LMs. Hidden Killer [14] introduces a textual backdoor attack that leverages chosen syntactic templates as triggers on early LMs.
-
Agent-based Attacks: Frameworks like BALD target decision-making systems (Word Injection Attack BALDword, Scenario Manipulation Attack BALDscene, Knowledge Injection Attack BALDRAG), BadAgent targets LLM agents across multiple tasks, and DemonAgent introduces the Dynamically Encrypted Multi-Backdoor Implantation Attack.
V. INFERENCE-PHASE ATTACKS
This section concentrates on Jailbreaking and Prompt Injection attacks.
-
Jailbreaking Attacks: These involve crafting inputs to bypass safety restrictions, categorized into Direct Attacks (Rule-based, Translation-based, Self-learning) and Indirect Attacks (Implicit and Cognitive Manipulation). Examples include GPTfuzzer for rule-based attacks, LRL jailbreaking exploiting cross-lingual vulnerabilities, and J2 for self-learning attacks. p. 0
Indirect methods involve Puzzler exploiting implicit clues, Persona Modulation guiding LLMs to specific personas, Persuasive Adversarial Prompt (PAP), and Reasoning-Augmented Conversation (RACE) leveraging reasoning capability. Cognitive manipulation includes Dual Intention Escape (DIE), integrating psychological principles into jailbreaking.Where the paper says it
In this paper, these attacks are systematically summarized into three different categories: Training-Phase attacks, Inference-Phase attacks, and Availability & Integrity attacks. Additionally, this paper extends the discussion to the attacks specific to LLM-based agents and highlights the vulnerabilities introduced by the architecture of the agent systems and their interactions with external tools and environments.
-
Prompt Injection Attacks: These involve directly inserting malicious instructions or data into the input to generate harmful outputs, categorized as Input-based, Optimization-based, and Other attacks. Input-based attacks include Obvious Malicious Injection (OMI) and Goal Hijacking Injection (GHI), Vocabulary Attack exploiting adversarial vocabulary, and Prompt Injection Frameworks formalizing compromised data construction (Naive Attack, Escape Characters, Context Ignoring). p. 0
Optimization-based attacks utilize gradient methods like Automatic and Universal Attacks to generate effective injection data. p. 3Where the paper says it
In this paper, we primarily focus on the development and recent advancements in various attack strategies targeting LLMs and LLM-based agents, including the methodology, implications, and challenges posed to LLM security. This paper provides a comprehensive summary of the development and recent advancements in adversarial attacks on LLMs, including threat strategies such as jailbreaking, backdoor and data poisoning, prompt injection, DoS, and watermarking attacks.
JudgeDeveiver targets LLM-as-a-Judge by optimizing input sequences. p. 4Where the paper says it
The model is manipulated to generate an incorrect response when the backdoor trigger is present in the input prompt. TABLE I A SUMMARY OF BACKDOOR & DATA POISONING ATTACKS Categories Approaches Input-based Attacks Hidden Killer [14], Hidden Backdoor [15], CBA [16], PoisonRAG [17], Instruction Attacks [18], VPI [19], BadGPT [20], RankPoison [21], TrojLLM [22], PoisonPrompt [23] Weight-based Attacks BadEdit [24], LoRA-based Attacks [25], [26], Gradient Control [27], W2SAttack [28], TA2 [29] Reasoning-based Attacks BadChain [30], BOT [31], ICLAttack [32] Agent-based Attacks BALD [33], BadAgent [34], DemonAgent [35] datasets of LMs.
Goal-guided Generative Prompt Injection Attack (G2PIA) focuses on query-free black box attacks maximizing KL divergence between clean and injected inputs. Prompt Infection introduces self-replicating attacks across multiagent systems, such as Prompt Infection: LLM-to-LLM Prompt Injection within Multi-Agent Systems. p. 2Where the paper says it
Instruction Backdoor Attack [18] proposes an approach that targets applications using untrusted customized LLMs, such as text classification systems, by embedding malicious instructions into their prompts. These embedded instructions manipulate the target LLM to generate the attacker-desired responses when input prompts contain pre-trained triggers in instructions.
Where the paper says it
Representative and recent attacks targeting LLMs and LLM-based agent systems are organized according to the three main phases of LLM lifecycle: Training Phase, Inference Phase, and Service Deployment Phase. Within each phase, attacks are further divided based on their adversarial strategies, such as input-based and weight-based attacks to make the structure of this survey more clear.
VI. AVAILABILITY & INTEGRITY ATTACKS
This section addresses Denial of Service (DoS) and Watermarking attacks.
-
Denial of Service (DoS) Attacks: These aim to overwhelm service resources, classified as Repetition, Recursion, Count, Long Article, and Source Code instructions. Prominent DoS attacks include regular expression DoS (ReDoS), poisoning-based DoS (P-DoS), and safeguard-based DoS. p. 0
Where the paper says it
In this paper, we primarily focus on the development and recent advancements in various attack strategies targeting LLMs and LLM-based agents, including the methodology, implications, and challenges posed to LLM security. This paper provides a comprehensive summary of the development and recent advancements in adversarial attacks on LLMs, including threat strategies such as jailbreaking, backdoor and data poisoning, prompt injection, DoS, and watermarking attacks.
-
Watermarking Attacks: These embed subtle patterns for detection and are categorized into Paraphrasing and Prompting attacks. Self Color Testing-based Substitution (SCTS) Attack focuses on watermark removal via color identification, while Black-Box scrubbing Attack (B4) reformulates removal as a constrained optimization problem.
VII. CONCLUSION
The survey concludes with three key insights: the vulnerability of LLMs across the entire pipeline; the increasing complexity of threats evolving from LLMs to LLM-based multi-agents augmented with external tools; and the limitations of current defenses against adaptive attacks. Future research directions emphasize developing a unified threat classification, designing cross-phase defense frameworks, and advancing adaptive and explainable mitigation mechanisms. The survey underscores the urgent need for robust strategies against evolving threats, particularly concerning LLM-based agents interacting with external environments. p. 0 We hope
our survey will provide a good tutorial and a comprehensive
understanding of LLM security, especially for attacks on LLMs. We desire to raise attention to the risks inherent in widely
deployed LLM-based applications and highlight the urgent need
for robust mitigation strategies for evolving threats.Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Unified Threat Classification and Benchmarking: Develop a unified classification of threats and benchmarks to enable consistent evaluation across different LLMs and scenarios for defense strategies.
-
Cross-Phase Defense Framework: Design a cross-phase defense framework that offers comprehensive protection across the full model lifecycle, covering Training Phase attacks, Inference Phase attacks, and Availability & Integrity attacks.
-
Adaptive Prompt Injection Defenses: Implement novel defense mechanisms that are both adaptive and robust to defend against evolving prompt injection attacks through methods like gradient-based optimization and JudgeDeveiver.
-
Agent-Specific Backdoor Mitigation: Develop specific defenses for LLM-based agents by addressing the risks introduced by their architecture, such as those related to profiling, memory, planning, and action modules.
-
Watermark Evasion Resilience: Enhance watermarking techniques to resist adversarial strategies like Self Color Testing (SCTS) and Black-Box scrubbing (B4), ensuring that seemingly clean text cannot be generated from watermarked AI-generated texts.
Abstract
Large language models (LLMs) and LLM-based agents have been widely deployed in a wide range of applications in the real world, including healthcare diagnostics, financial analysis, customer support, robotics, and autonomous driving, expanding their powerful capability of understanding, reasoning, and generating natural languages. However, the wide deployment of LLM-based applications exposes critical security and reliability risks, such as the potential for malicious misuse, privacy leakage, and service disruption that weaken user trust and undermine societal safety. This paper provides a systematic overview of the details of adversarial attacks targeting both LLMs and LLM-based agents. These attacks are organized into three phases in LLMs: Training-Phase Attacks, Inference-Phase Attacks, and Availability & Integrity Attacks. For each phase, we analyze the details of representative and recently introduced attack methods along with their corresponding defenses. We hope our survey will provide a good tutorial and a comprehensive understanding of LLM security, especially for attacks on LLMs. We desire to raise attention to the risks inherent in widely deployed LLM-based applications and highlight the urgent need for robust mitigation strategies for evolving threats.
Sources
- Multi-Step Reasoning with Large Language Models, a Survey
- Sentiment Analysis in the Era of Large Language Models: A Reality Check
- The Llama 3 Herd of Models
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- A Comprehensive Overview of Large Language Models
- Attention Is All You Need
- BackdoorLLM: A Comprehensive Benchmark for Backdoor Attacks and Defenses on Large Language Models
- Hidden Killer: Invisible Textual Backdoor Attacks with Syntactic Trigger
- Composite Backdoor Attacks Against Large Language Models
- PoisonedRAG: Knowledge Corruption Attacks to Retrieval-Augmented Generation of Large Language Models
- Backdooring Instruction-Tuned Large Language Models with Virtual Prompt Injection
- BadGPT: Exploring Security Vulnerabilities of ChatGPT via Backdoor Attacks to InstructGPT
- RLHFPoison: Reward Poisoning Attack for Reinforcement Learning with Human Feedback in Large Language Models
- BadEdit: Backdooring large language models by model editing
- LoRATK: LoRA Once, Backdoor Everywhere in the Share-and-Play Ecosystem
- The Philosopher's Stone: Trojaning Plugins of Large Language Models
- Weak-to-Strong Jailbreaking on Large Language Models
- Trojan Activation Attack: Red-Teaming Large Language Models using Activation Steering for Safety-Alignment
- BadChain: Backdoor Chain-of-Thought Prompting for Large Language Models
- To Think or Not to Think: Exploring the Unthinking Vulnerability in Large Reasoning Models
Related papers
- From Strategy Search to Policy Learning: MCTS-Guided Adversarial Co-Evolution for LLM Safety Alignment
- Privacy Leakage from Recurrent State in LLM Inference
- Incrementally Verifiable Oracle Aided Computation
- Privilege and confidentiality in generative AI workflows
- BadMoE: Backdooring Mixture-of-Experts LLMs via Routing-Aware Triggers and Task-Underutilized Experts
- CoVault: Secure, Scalable Analytics of Personal Data