A Survey of Attacks on Large Language Models

arXiv:2505.12567 · cs.CR, cs.AI · Submitted 2025-05-18 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Security Radio. Generated commentary on the latest security and cryptography papers.

Nadia: I'm Nadia.

Elias: And I'm Elias, here with Priya.

Nadia: We're looking at the paper: "A Survey of Attacks on Large Language Models".

Elias: Here's the gist of the paper. This survey provides a systematic overview of adversarial attacks targeting both Large Language Models (LLMs) and LLM-based agents, structuring the analysis around three primary phases in LLMs: Training-Phase Attacks, Inference-Phase Attacks, and Availability & Integrity Attacks.

Nadia: These attacks are further classified based on adversarial strategies as input-based or weight-based.

Elias: IV.

Nadia: TRAINING-PHASE ATTACKS This section details attacks aimed at compromising the training phase of target LLMs, primarily focusing on backdoor and data poisoning methods.

Paper summary: Nadia: So, this survey paper by Wenrui Xu and Keshab K Parhi, titled "A Survey of Attacks on Large Language Models," lays out a structure for understanding these threats. The main thesis is that we need a systematic look at adversarial attacks targeting both the LLMs themselves and the agents built on top of them.

Elias: It claims to provide a structured overview by organizing all these attacks into three major phases: training phase, inference phase, and availability and integrity attacks, which gives a complete lifecycle view of the risk.

Priya: What matters here is that they don't just list random hacks; they group them by strategy—input based versus weight based—so we can see if the vulnerability is in the data or in the model's learned parameters.

Nadia: They specifically introduce a taxonomy to help readers better understand how different types of attacks operate across the entire life cycle of these models, which is pretty useful for defense researchers.

Elias: The abstract emphasizes that they analyze representative and recently introduced attack methods within each phase, which means we get an overview of what's happening right now in the research landscape.

Priya: I’m looking for where the paper says what kind of data or actions are being measured before we can trust any of their conclusions about the severity of these risks.

Nadia: They categorize training phase attacks as input based, weight based, reasoning based, and agent based, which shows how complex these vulnerabilities are getting beyond simple text inputs.

Elias: And for inference phase attacks, they separate jailbreaking from prompt injection and further divide prompt injection into input based or optimization based types.

Priya: That distinction between input-based and optimization-based prompt injection seems key because it tells us whether we're dealing with a simple malicious string or a more complex attack generated through mathematical methods.

Conclusion: Nadia: Looking at the title, "A Survey of Attacks on Large Language Models," it’s clear this paper is trying to be a comprehensive map for anyone trying to understand the security landscape around these large models.

Elias: The authors, Wenrui Xu and Keshab K Parhi, have put together a detailed structure that moves beyond just listing attacks; they're creating a taxonomy that helps readers organize the massive problem of model security.

Priya: What this means for us is that we can start to build better defenses by knowing exactly where the weak points are—whether it’s during the initial data training or when someone tries to trick the model during an actual user interaction.

Nadia: It shifts our focus from just patching one specific vulnerability to creating cross-phase defense frameworks that account for these different attack vectors across training, inference, and availability.

Elias: They point out that the complexity is increasing with LLM based agents interacting with external tools, which suggests our current defenses might not be enough against those more dynamic systems.

Priya: So the implication is that future research needs to focus on adaptive and explainable mitigation mechanisms because these attacks are evolving faster than our current static defenses can keep up.

Wenrui Xu, Keshab K Parhi

University of Minnesota

cs.CR, cs.AI

Submitted: 2025-05-18

Updated: 2025-05-18

Journal ref: IEEE Circuits and Systems Magazine, 26(4), pp. 10-41, Fourthquarter 2026

DOI: 10.1109/MCAS.2026.3708070

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 90/100

The gist: This survey provides a systematic overview of adversarial attacks targeting both Large Language Models (LLMs) and LLM-based agents…

Key concepts

Training-Phase Attacks
These attacks aim to compromise an LLM while it is being trained. They focus on injecting malicious data or weights, such as backdoors, into the model's learning process. This can create hidden triggers that cause the model to behave unexpectedly later on.
Inference-Phase Attacks
These attacks target a deployed LLM during its use, primarily focusing on jailbreaking and prompt injection. Jailbreaking tries to bypass safety rules, while prompt injection involves tricking the model into executing malicious instructions by inserting harmful text directly into the input.
Availability & Integrity Attacks
These attacks focus on disrupting service or corrupting data integrity. Availability attacks aim to cause a Denial of Service (DoS) by overwhelming resources, while integrity attacks involve removing or altering hidden watermarks embedded in the model's outputs.

Terminology

Summary

This survey provides a systematic overview of adversarial attacks targeting both Large Language Models (LLMs) and LLM-based agents, structuring the analysis around three primary phases in LLMs: Training-Phase Attacks, Inference-Phase Attacks, and Availability & Integrity Attacks. p. 0

Where the paper says it

This paper provides a systematic overview of the details of adversarial attacks targeting both LLMs and LLM-based agents. These attacks are organized into three phases in LLMs: Training-Phase Attacks, Inference-Phase Attacks, and Availability & Integrity Attacks.

These attacks are further classified based on adversarial strategies as input-based or weight-based. p. 2
Where the paper says it

Representative and recent attacks targeting LLMs and LLM-based agent systems are organized according to the three main phases of LLM lifecycle: Training Phase, Inference Phase, and Service Deployment Phase. Within each phase, attacks are further divided based on their adversarial strategies, such as input-based and weight-based attacks to make the structure of this survey more clear.

IV. TRAINING-PHASE ATTACKS

This section details attacks aimed at compromising the training phase of target LLMs, primarily focusing on backdoor and data poisoning methods. These approaches are categorized into Input-based, Weight-based, Reasoning-based, and Agent-based attacks. p. 3

Where the paper says it

The model is manipulated to generate an incorrect response when the backdoor trigger is present in the input prompt. TABLE I A SUMMARY OF BACKDOOR & DATA POISONING ATTACKS Categories Approaches Input-based Attacks Hidden Killer [14], Hidden Backdoor [15], CBA [16], PoisonRAG [17], Instruction Attacks [18], VPI [19], BadGPT [20], RankPoison [21], TrojLLM [22], PoisonPrompt [23] Weight-based Attacks BadEdit [24], LoRA-based Attacks [25], [26], Gradient Control [27], W2SAttack [28], TA2 [29] Reasoning-based Attacks BadChain [30], BOT [31], ICLAttack [32] Agent-based Attacks BALD [33], BadAgent [34], DemonAgent [35] datasets of LMs.

Representative attack methods include:

  • Input-based Attacks: Examples encompass Hidden Killer (leveraging textual backdoors with syntactic templates), Hidden Backdoor (proposing Homograph Replacement and Dynamic Sentence-based Attacks), Composite Backdoor Attack (CBA) distributing multiple trigger keys, PoisonedRAG injecting malicious data into external knowledge databases for RAG systems, Instruction Backdoor Attack targeting applications using untrusted customized LLMs via word, syntax, or semantic-level instructions, Virtual Prompt Injection (VPI) poisoning instruction-tuning databases for instruction-tuned LLMs, and BadGPT/RankPoison targeting the RL phase through Reward Model Backdooring and RLHF Poisoning. TrojLLM proposes a black-box framework for prompt-based backdoor attacks (e. p. 6

    Where the paper says it

    TrojLLM [22] and PoisonPrompt [23] present approaches to execute the promptbased backdoor attacks on LLMs. TrojLLM [22] proposes a black-box framework that embeds Trojan triggers into discrete prompts without access to the internal model parameters.

    g., PromptSeed Tuning), PoisonPrompt introduces a bi-level optimization attack on prompt-based LLMs, while Weight-based Attacks include BadEdit for direct weight alteration and LoRA-based attacks (LoRA-as-an-attack, Polish, Fusion) exploiting poisoned LoRA modules. p. 7
    Where the paper says it

    BadEdit [24] introduces a weight-editing framework for backdoor injection in LLMs by directly altering a small subset of the LLM parameters while preserving the model’s performance. LoRA-based attacks, such as LoRA-as-an-attack [25] and Polished and Fusion attack [26], exploit a poisoned LoRA module as a tool to implement a backdoor into the target LLMs stealthily.

    Gradient control methods like Cross-Layer Gradient magnitude normalization (CLNorm) and Intra-Layer gradient direction Projection (ILProj) address challenges in PEFT. p. 8
    Where the paper says it

    Gradient magnitude imbalance refers to the phenomenon that different layers of the PEFT module make uneven contributions to backdoor injection where the output layer receives much larger gradient updates than others. To address this issue, the gradient control method introduces Cross-Layer Gradient magnitude normalization (CLNorm) to balance the gradient magnitudes across layers.

    Weak to strong clean label backdoor attack (W2SAttack) utilizes a teacher/student approach with feature alignment distillation. p. 8
    Where the paper says it

    Weak to strong clean label backdoor attack (W2SAttack) [28] introduces a novel framework to perform backdoor attacks on the LLMs that are fine-tuned via the PEFT method. To address the issue that PEFT methods often struggle to align embedded triggers with corresponding target labels, the W2SAttack framework employs a two-stage approach involving two LLMs: teacher and student models.

    Trojan Activation Attack (TA2) injects trojan steering vectors into activation layers during inference. p. 9
    Where the paper says it

    Trojan Activation Attack (TA2 ) [29] proposes a backdoor attack that directly injects trojan steering vectors into the activation layers of the target LLMs as shown in Fig. Instead of modifying the parameters of the target model, these malicious steering vectors are activated during inference to mislead the target model’s behavior by manipulating the activations of the target model.

  • Reasoning-based Attacks: These target the Chain-of-Thought process, such as BadChain and Break CoT, and BoT, which disables inherent reasoning processes. ICLAttack introduces a backdoor attack for In-Context Learning (ICL). p. 3

    Where the paper says it

    TABLE I A SUMMARY OF BACKDOOR & DATA POISONING ATTACKS Categories Approaches Input-based Attacks Hidden Killer [14], Hidden Backdoor [15], CBA [16], PoisonRAG [17], Instruction Attacks [18], VPI [19], BadGPT [20], RankPoison [21], TrojLLM [22], PoisonPrompt [23] Weight-based Attacks BadEdit [24], LoRA-based Attacks [25], [26], Gradient Control [27], W2SAttack [28], TA2 [29] Reasoning-based Attacks BadChain [30], BOT [31], ICLAttack [32] Agent-based Attacks BALD [33], BadAgent [34], DemonAgent [35] datasets of LMs. Hidden Killer [14] introduces a textual backdoor attack that leverages chosen syntactic templates as triggers on early LMs.

  • Agent-based Attacks: Frameworks like BALD target decision-making systems (Word Injection Attack BALDword, Scenario Manipulation Attack BALDscene, Knowledge Injection Attack BALDRAG), BadAgent targets LLM agents across multiple tasks, and DemonAgent introduces the Dynamically Encrypted Multi-Backdoor Implantation Attack.

V. INFERENCE-PHASE ATTACKS

This section concentrates on Jailbreaking and Prompt Injection attacks.

  • Jailbreaking Attacks: These involve crafting inputs to bypass safety restrictions, categorized into Direct Attacks (Rule-based, Translation-based, Self-learning) and Indirect Attacks (Implicit and Cognitive Manipulation). Examples include GPTfuzzer for rule-based attacks, LRL jailbreaking exploiting cross-lingual vulnerabilities, and J2 for self-learning attacks. p. 0

    Where the paper says it

    In this paper, these attacks are systematically summarized into three different categories: Training-Phase attacks, Inference-Phase attacks, and Availability & Integrity attacks. Additionally, this paper extends the discussion to the attacks specific to LLM-based agents and highlights the vulnerabilities introduced by the architecture of the agent systems and their interactions with external tools and environments.

    Indirect methods involve Puzzler exploiting implicit clues, Persona Modulation guiding LLMs to specific personas, Persuasive Adversarial Prompt (PAP), and Reasoning-Augmented Conversation (RACE) leveraging reasoning capability. Cognitive manipulation includes Dual Intention Escape (DIE), integrating psychological principles into jailbreaking.

  • Prompt Injection Attacks: These involve directly inserting malicious instructions or data into the input to generate harmful outputs, categorized as Input-based, Optimization-based, and Other attacks. Input-based attacks include Obvious Malicious Injection (OMI) and Goal Hijacking Injection (GHI), Vocabulary Attack exploiting adversarial vocabulary, and Prompt Injection Frameworks formalizing compromised data construction (Naive Attack, Escape Characters, Context Ignoring). p. 0

    Where the paper says it

    In this paper, we primarily focus on the development and recent advancements in various attack strategies targeting LLMs and LLM-based agents, including the methodology, implications, and challenges posed to LLM security. This paper provides a comprehensive summary of the development and recent advancements in adversarial attacks on LLMs, including threat strategies such as jailbreaking, backdoor and data poisoning, prompt injection, DoS, and watermarking attacks.

    Optimization-based attacks utilize gradient methods like Automatic and Universal Attacks to generate effective injection data. p. 3
    Where the paper says it

    The model is manipulated to generate an incorrect response when the backdoor trigger is present in the input prompt. TABLE I A SUMMARY OF BACKDOOR & DATA POISONING ATTACKS Categories Approaches Input-based Attacks Hidden Killer [14], Hidden Backdoor [15], CBA [16], PoisonRAG [17], Instruction Attacks [18], VPI [19], BadGPT [20], RankPoison [21], TrojLLM [22], PoisonPrompt [23] Weight-based Attacks BadEdit [24], LoRA-based Attacks [25], [26], Gradient Control [27], W2SAttack [28], TA2 [29] Reasoning-based Attacks BadChain [30], BOT [31], ICLAttack [32] Agent-based Attacks BALD [33], BadAgent [34], DemonAgent [35] datasets of LMs.

    JudgeDeveiver targets LLM-as-a-Judge by optimizing input sequences. p. 4
    Where the paper says it

    Instruction Backdoor Attack [18] proposes an approach that targets applications using untrusted customized LLMs, such as text classification systems, by embedding malicious instructions into their prompts. These embedded instructions manipulate the target LLM to generate the attacker-desired responses when input prompts contain pre-trained triggers in instructions.

    Goal-guided Generative Prompt Injection Attack (G2PIA) focuses on query-free black box attacks maximizing KL divergence between clean and injected inputs. Prompt Infection introduces self-replicating attacks across multiagent systems, such as Prompt Infection: LLM-to-LLM Prompt Injection within Multi-Agent Systems. p. 2
    Where the paper says it

    Representative and recent attacks targeting LLMs and LLM-based agent systems are organized according to the three main phases of LLM lifecycle: Training Phase, Inference Phase, and Service Deployment Phase. Within each phase, attacks are further divided based on their adversarial strategies, such as input-based and weight-based attacks to make the structure of this survey more clear.

VI. AVAILABILITY & INTEGRITY ATTACKS

This section addresses Denial of Service (DoS) and Watermarking attacks.

  • Denial of Service (DoS) Attacks: These aim to overwhelm service resources, classified as Repetition, Recursion, Count, Long Article, and Source Code instructions. Prominent DoS attacks include regular expression DoS (ReDoS), poisoning-based DoS (P-DoS), and safeguard-based DoS. p. 0

    Where the paper says it

    In this paper, we primarily focus on the development and recent advancements in various attack strategies targeting LLMs and LLM-based agents, including the methodology, implications, and challenges posed to LLM security. This paper provides a comprehensive summary of the development and recent advancements in adversarial attacks on LLMs, including threat strategies such as jailbreaking, backdoor and data poisoning, prompt injection, DoS, and watermarking attacks.

  • Watermarking Attacks: These embed subtle patterns for detection and are categorized into Paraphrasing and Prompting attacks. Self Color Testing-based Substitution (SCTS) Attack focuses on watermark removal via color identification, while Black-Box scrubbing Attack (B4) reformulates removal as a constrained optimization problem.

VII. CONCLUSION

The survey concludes with three key insights: the vulnerability of LLMs across the entire pipeline; the increasing complexity of threats evolving from LLMs to LLM-based multi-agents augmented with external tools; and the limitations of current defenses against adaptive attacks. Future research directions emphasize developing a unified threat classification, designing cross-phase defense frameworks, and advancing adaptive and explainable mitigation mechanisms. The survey underscores the urgent need for robust strategies against evolving threats, particularly concerning LLM-based agents interacting with external environments. p. 0

Where the paper says it

We hope our survey will provide a good tutorial and a comprehensive understanding of LLM security, especially for attacks on LLMs. We desire to raise attention to the risks inherent in widely deployed LLM-based applications and highlight the urgent need for robust mitigation strategies for evolving threats.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Unified Threat Classification and Benchmarking: Develop a unified classification of threats and benchmarks to enable consistent evaluation across different LLMs and scenarios for defense strategies.

  2. Cross-Phase Defense Framework: Design a cross-phase defense framework that offers comprehensive protection across the full model lifecycle, covering Training Phase attacks, Inference Phase attacks, and Availability & Integrity attacks.

  3. Adaptive Prompt Injection Defenses: Implement novel defense mechanisms that are both adaptive and robust to defend against evolving prompt injection attacks through methods like gradient-based optimization and JudgeDeveiver.

  4. Agent-Specific Backdoor Mitigation: Develop specific defenses for LLM-based agents by addressing the risks introduced by their architecture, such as those related to profiling, memory, planning, and action modules.

  5. Watermark Evasion Resilience: Enhance watermarking techniques to resist adversarial strategies like Self Color Testing (SCTS) and Black-Box scrubbing (B4), ensuring that seemingly clean text cannot be generated from watermarked AI-generated texts.

Abstract

Large language models (LLMs) and LLM-based agents have been widely deployed in a wide range of applications in the real world, including healthcare diagnostics, financial analysis, customer support, robotics, and autonomous driving, expanding their powerful capability of understanding, reasoning, and generating natural languages. However, the wide deployment of LLM-based applications exposes critical security and reliability risks, such as the potential for malicious misuse, privacy leakage, and service disruption that weaken user trust and undermine societal safety. This paper provides a systematic overview of the details of adversarial attacks targeting both LLMs and LLM-based agents. These attacks are organized into three phases in LLMs: Training-Phase Attacks, Inference-Phase Attacks, and Availability & Integrity Attacks. For each phase, we analyze the details of representative and recently introduced attack methods along with their corresponding defenses. We hope our survey will provide a good tutorial and a comprehensive understanding of LLM security, especially for attacks on LLMs. We desire to raise attention to the risks inherent in widely deployed LLM-based applications and highlight the urgent need for robust mitigation strategies for evolving threats.

Sources

Related papers