Evaluating the Effects of Prompt Perturbation on Bias and Hallucination in Large Language Models

arXiv:2609.35804 · cs.CL, cs.AI, cs.HC · Submitted 2026-09-18 · Read on arXiv

cs.CL, cs.AI, cs.HC

Submitted: 2026-09-18

Updated: 2026-09-18

Comments: 14 pages. Published in ICONIP 2024 (Neural Information Processing), LNCS 15290, Springer Nature, 2025

Journal ref: Neural Information Processing (ICONIP 2024), Lecture Notes in Computer Science (LNCS), vol. 15290, pp. 361-374, Springer, 2025

DOI: 10.1007/978-981-96-6588-4_25

Code: https://github.com/fiddler-labs/fiddler-auditor

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Terminology

• Hallucination — 23× in this paper · explained in 3 episodes
• Accuracy — 21× in this paper · explained in 1 episode
• Robustness — 20× in this paper · explained in 5 episodes
• F1 Score — 10× in this paper · explained in 3 episodes
• Benchmark — 7× in this paper · explained in 2 episodes
• Consistency — 4× in this paper · explained in 3 episodes
• Decision-Support Tool — 3× in this paper · explained in 2 episodes
• Embeddings — 2× in this paper · explained in 2 episodes
• Large Language Models (LLMs) — 2× in this paper · explained in 6 episodes
• Prompt Engineering — 2× in this paper · explained in 4 episodes
• Adversarial Attacks — 1× in this paper · explained in 1 episode
• Adversarial Examples — 1× in this paper · explained in 2 episodes
• Artificial Intelligence (AI) — 1× in this paper · explained in 2 episodes
• BIG-Bench Hard — 1× in this paper
• Failure Modes — 1× in this paper · explained in 3 episodes
• False Discovery Rate — 1× in this paper
• False Negative Rate — 1× in this paper
• False Omission Rate — 1× in this paper
• False Positive Rate — 1× in this paper
• Generative AI — 1× in this paper · explained in 2 episodes
• Perplexity — 1× in this paper · explained in 3 episodes
• Research in Human-Computer Interaction — 1× in this paper

Sources

• GPT-4 Technical Report
• PaLM 2 Technical Report
• Quantifying and Reducing Stereotypes in Word Embeddings
• Uncovering Biases with Reflective Large Language Models
• Measuring Massive Multitask Language Understanding
• Language Models are Few-Shot Learners
• On Faithfulness and Factuality in Abstractive Summarization
• Large Language Models: A Survey
• Beyond Accuracy: Behavioral Testing of NLP models with CheckList
• Aequitas: A Bias and Fairness Audit Toolkit
• Societal Biases in Language Generation: Progress and Challenges
• Moral Mimicry: Large Language Models Produce Moral Rationalizations Tailored to Political Identity
• Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
• Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them
• Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Models
• Large Language Models are not Fair Evaluators
• DataFrame QA: A Universal LLM Framework on DataFrame Question Answering Without Data Exposure
• CHBias: Bias Evaluation and Mitigation of Chinese Conversational Language Models

Related papers

• Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
• Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
• Subliminal Steering: Stronger Encoding of Hidden Signals
• MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
• The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
• Untangling the Mechanisms of Misleading Context in Medical Question Answering