The Vulnerability of LLM Rankers to Prompt Injection Attacks

Abstract

Large Language Models (LLMs) have emerged as powerful re-rankers. Recent research has shown that simple prompt injections embedded within a candidate document (jailbreak prompt attacks) can significantly alter an LLM’s ranking decisions. While this poses serious security risks to LLM-based ranking pipelines, the extent to which this vulnerability persists across diverse LLM families, architectures, and settings remains largely under-explored. This paper presents a comprehensive empirical study of jailbreak prompt attacks against LLM rankers, focusing on two complementary tasks: (1) Preference Vulnerability Assessment, measuring intrinsic susceptibility via attack success rate (ASR); and (2) Ranking Vulnerability Assessment, quantifying the operational impact on ranking quality (nDCG@10). Three prevalent ranking paradigms are examined — pairwise, listwise, and setwise — under two injection variants: decision objective hijacking and decision-related variants. Key findings include that larger and more capable LLMs show greater susceptibility to attacks, the position of injected content makes a statistically significant difference, encoder-decoder models show greater robustness, and high ASR corresponds to severe nDCG@10 degradation across domains.