The Vulnerability of LLM Rankers to Prompt Injection Attacks
summary
The gist
Large Language Models (LLMs) can be compromised through simple adversarial prompt injection attacks that significantly alter their ranking decisions The extent to which this vulnerability persists…
In short
This study tested how easily Large Language Model (LLM) rankers can be tricked by adversarial prompt injections designed to change their ranking decisions. The research found that encoder-decoder architectures are naturally more resistant to these attacks than other models. It also showed how specific injection methods and model scaling affect the ranking quality.
Key concepts
- Preference Vulnerability Assessment (ASR)
- This measures how susceptible a model is to jailbreak attacks by counting how often an injected prompt successfully causes the model to change its decision in favor of the attacker's target. A high ASR indicates a strong vulnerability.
- Ranking Vulnerability Assessment (nDCG@10)
- This quantifies the real-world impact of an attack on search quality by measuring changes in ranking performance using nDCG@10. When attacks succeed, this metric drops significantly, showing the operational damage to the system's usefulness.
- Decision Objective Hijacking
- This is a type of prompt injection where the attacker manipulates the model's goal or objective function. The injected prompt forces the LLM to prioritize a specific outcome dictated by the attacker rather than its original, intended ranking criteria.
- Encoder-Decoder Architectures Resilience
- The paper found that models built with an encoder-decoder structure, such as Flan-T5, show strong inherent resilience against jailbreak attacks. These architectures maintain high ranking quality even when subjected to objective hijacking or criteria hijacking compared to other model types.
Terminology used across episodes
This episode discusses
- The Vulnerability of LLM Rankers to Prompt Injection Attacks · Paper Radio
- GPT-4 Technical Report
- The Llama 3 Herd of Models · Paper Radio
- Gemma 3 Technical Report
- Manipulating Large Language Models to Increase Product Visibility
- A Survey of Generative Search and Recommendation in the Era of Large Language Models
- Holistic Evaluation of Language Models
- Zero-Shot Listwise Document Reranking with a Large Language Model
- Adversarial Search Engine Optimization for Large Language Models
- Ranking Manipulation for Conversational Search Engines
- RankVicuna: Zero-Shot Listwise Document Reranking with Open-Source Large Language Models
- Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agents
- StealthRank: LLM Ranking Manipulation via Stealthy Prompt Optimization
- BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models
- Fact or Fiction: Verifying Scientific Claims
- Are LLMs Reliable Rankers? Rank Manipulation via Two-Stage Token Optimization
- Qwen3 Technical Report
The paper
The Vulnerability of LLM Rankers to Prompt Injection Attacks · Read on arXiv
University of Queensland · CSIRO
Large Language Models (LLMs) have emerged as powerful re-rankers. Recent research has however showed that simple prompt injections embedded within a candidate document (i.e., jailbreak prompt attacks) can significantly alter an LLM's ranking decisions. While this poses serious security risks to LLM-based ranking pipelines, the extent to which this vulnerability persists across diverse LLM families, architectures, and settings remains largely under-explored. In this paper, we present a comprehensive empirical study of jailbreak prompt attacks against LLM rankers. We focus our evaluation on two complementary tasks: (1) Preference Vulnerability Assessment, measuring intrinsic susceptibility via attack success rate (ASR); and (2) Ranking Vulnerability Assessment, quantifying the operational impact on the ranking's quality (nDCG@10). We systematically examine three prevalent ranking paradigms (pairwise, listwise, setwise) under two injection variants: decision objective hijacking and decision criteria hijacking. Beyond reproducing prior findings, we expand the analysis to cover vulnerability scaling across model families, position sensitivity, backbone architectures, and cross-domain robustness. Our results characterize the boundary conditions of these vulnerabilities, revealing critical insights such as that encoder-decoder architectures exhibit strong inherent resilience to jailbreak attacks. We publicly release our code and additional experimental results at https://github.com/ielab/LLM-Ranker-Attack.
Transcript
Introduction to the show: ident: Security Radio. Generated commentary on the latest security and cryptography papers.
Nadia: We're looking at the paper: "The Vulnerability of LLM Rankers to Prompt Injection Attacks".
Elias: Here's the gist of the paper. Large Language Models (LLMs) can be compromised through simple adversarial prompt injection attacks that significantly alter their ranking decisions The extent to which this vulnerability persists across diverse LLM families, architectures, and settings remains largely under-explored This paper presents a comprehensive empirical study of jailbreak prompt attacks against LLM rankers The results characterize the boundary conditions of these vulnerabilities, revealing critical insights such as that encoder-decoder architectures exhibit strong inherent resilience to jailbreak attacks Core Research Objectives The study focuses on two complementary tasks: (1) Preference Vulnerability Assessment, measuring intrinsic susceptibility via attack success rate (ASR); and (2) Ranking Vulnerability Assessment, quantifying the operational impact on the ranking’s quality (nDCG@10) The researchers systematically examine three prevalent ranking paradigms (pairwise, listwise, setwise) under two injection variants: decision objective hijacking and decision criteria hijacking Beyond reproducing prior findings, they expand the analysis to cover vulnerability scaling across model families, position sensitivity, backbone architectures, and cross-domain robustness Vulnerability Assessment Metrics The paper defines two primary metrics for evaluating the attacks 1.
Nadia: Preference Vulnerability Assessment (Task 1) measures intrinsic susceptibility via attack success rate (ASR) An attack is counted as successful if the model changes its decision in favor of the injected target 2.
Title and authors: Nadia: So, to recap, this paper by Yu Yin et al., "The Vulnerability of LLM Rankers to Prompt Injection Attacks," is a deep dive into how prompt injections affect an LLM's ability to rank documents across different systems. They confirm that these attacks can significantly alter the ranking decisions.
Elias: They focus on two main tasks: measuring the intrinsic susceptibility through attack success rate, and then measuring the operational impact on quality using nDCG@ten across various paradigms like pairwise, listwise, and setwise ranking <ref:2602.16752#pg1>.
Priya: It sounds like they are trying to establish a concrete boundary condition for where these vulnerabilities lie. What do you think that means for the systems we use every day?
Nadia: It means they’re trying to figure out exactly how hard it is to break the ranking mechanism and whether that breaking point changes depending on the model family or the specific way we set up our ranking system.
Elias: They systematically examine three different ways ranking can be done—pairwise, listwise, and setwise—and they test two types of injections: decision objective hijacking and decision criteria hijacking.
Priya: So they are testing whether the attack changes what the model is trying to optimize for, or if it changes the rules it uses to compare things? That sounds like a very technical way to look at susceptibility.
The paper's summary: Nadia: The authors summarize their work by confirming that larger LLMs are generally more vulnerable and capable models show greater susceptibility, contrary to what you might expect from size alone. They also found position sensitivity matters a lot when testing the attack success rate across different document positions.
Elias: That’s a key finding because it challenges the idea that just having a bigger model automatically makes it safer against these types of manipulation attempts. The paper also discovered that encoder-decoder architectures show great robustness compared to decoder-only models when dealing with decision objective hijacking.
Priya: So, if I'm thinking about building a ranking system, this suggests that the architecture itself—whether it's the structure of the LLM—might be a bigger factor in defense than just how big the model is.
Nadia: Exactly. And they also found that vulnerability extends across different domains, meaning susceptibility isn't just stuck in one area like search or general text and it can generalize to other areas too, which is something we need to consider for broader security.
Elias: They also found that high attack success rate does not always translate into severe ranking quality degradation; they saw cases where the model maintains fidelity to the instructions even when manipulated.
Priya: That’s an important caveat because if we only look at the success rate, we might over-warn about a system's fragility without seeing how much that actually hurts our end user.
The paper's improvements: Nadia: The paper suggests several ways to improve defenses. First, they point to using encoder-decoder architectures like Flan-T5 because those models show great inherent resilience against these jailbreak attacks. It means they should be a priority for systems where we anticipate manipulation.
Elias: They also suggest dynamic prompt placement strategies based on the ranking paradigm, specifically using back-of-passage injections in setwise ranking because they noted that back-placed prompts typically exceed front-placed variants by roughly twenty points for models like Qwen3 under decision criteria hijacking.
Priya: That’s practical—so if we're using a setwise method, we can strategically place the prompt further down to see if that position sensitivity actually helps us keep the quality up when an attack tries to hit us.
Nadia: They also suggest training or fine-tuning LLM rankers on diverse datasets because they saw that vulnerability largely generalizes across domains, implying that training on varied data should stabilize performance in new areas.
Elias: And finally, for operational quality preservation, they propose a two-stage reranking pipeline with a safety layer to actively monitor the nDCG@ten metric and trigger fallback mechanisms if the degradation goes past a certain threshold <ref:2602.16752#pg1>.
Conclusion: Nadia: So, to wrap up on "The Vulnerability of LLM Rankers to Prompt Injection Attacks," the authors show that while larger models are often more vulnerable, architectural choices like encoder-decoder structures offer better inherent resilience against objective hijacking.
Elias: They also highlighted position sensitivity as a confirmed factor and showed that vulnerability isn't strictly limited to one type of ranking method or domain.
Priya: From my side, what this means for the actual deployment is that we need layered defenses: checking the architecture first, then being aware of where we place our prompts, and finally having an operational safety net watching that nDCG@ten score <ref:2602.16752#pg1>.
Nadia: It suggests that defense isn't just about patching one specific vulnerability but understanding how these attacks scale across model capabilities and ranking paradigms.
Elias: They lay out a clear picture of where the gaps are, confirming that even in complex setups, those prompt injections can cause significant operational impact on the ranking quality.
Priya: It’s a good reminder that we need to keep testing across those diverse domains and architectures to truly understand the full scope of this susceptibility.
More episodes
- 2608.27496-ROPE: Routed Origin Policy Enforcement against Indirect Prompt Injection
- Daily Summary for 2026-11-19
- 2607.07085-Is Randomness Necessary for Adaptive Data Analysis?
- The day in conversation for 2026-11-19
- Daily Summary for 2026-11-19
- 2609.10601-Threshold Choice, Not Sample Size, Bounds Trustless Verification of Nondeterministic Compound AI Workflows
- 2610.38786-Adversarial Process Disturbances at a Deployable Alarm Budget: Detector Evaluation on a Simulator and a Real Plant
- 2511.08660-Binary and Multiclass Cyberattack Classification on GeNIS Dataset
- 2604.27302-Static Attribution of Android Residential Proxy Malware Using Graph Kernels
- Daily Summary for 2026-11-18