Are Fake Citations a Form of Reward Hacking? A Pre-Registered Study of Post-Training and Citation Cues
cs.AI, cs.CL, cs.LG
Submitted: 2026-10-31
Updated: 2026-10-31
Comments: 49 pages, 3 figures, 20 tables. Code and data: https://github.com/zainnab-sparq/citation-reward-hacking (archived at doi:10.5281/zenodo.23845528)
Code: https://github.com/zainnab-sparq/citation-reward-hacking
License: http://creativecommons.org/licenses/by/4.0/
The gist: Post-trained language models fabricate citations at high rates.
Terminology
Abstract
Post-trained language models fabricate citations at high rates. We ask whether that is reward hacking: scoring well on a proxy for scholarship while doing worse on what it stands for. We ran four pre-registered experiments on 900 frozen questions (US federal case law, computationally adjacent scientific literature, and medical literature) across twelve open-weights checkpoints covering four post-training stages of three model families, then refit on fifteen. The clearest result holds the weights fixed and changes only the prompt. Asking for citations raises fabrication sharply in post-trained checkpoints and not in base ones: a 47.8-point interaction on twelve checkpoints (Holm-adjusted p = 2e-34) and 41.5 points on fifteen, in all three domains. A length-matched control separates the citation cue from a demand for a longer answer, so the effect is specific to citations, not verbosity. Post-training also raises fabrication overall, from 4.8% to 56.7%, but our registered prediction that it would rise most where fabrication is most tempting did not hold. A preference model trained only on surface citation features prefers post-trained output 86% of the time, and holding citation count fixed shows it rewards quantity. Our answer: possibly, but not proven. Post-trained models produce fabricated citations at far higher rates when asked for sources, and a naive evaluator that counts sources without checking them would prefer their output for citing more. But the targeted pattern we registered did not appear, and plain instruction-following could explain the same results. Proving that the reward causes fabrication needs an experiment that changes the reward during training, which this study does not do. Pre-registration: doi:10.17605/OSF.IO/S53HY. Code and data: https://github.com/zainnab-sparq/citation-reward-hacking
Sources
- Risks from Learned Optimization in Advanced Machine Learning Systems
- Why Language Models Hallucinate
- Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools
- AI Deception: A Survey of Examples, Risks, and Potential Solutions
- Right Answer, Wrong Method: Shortcut Hacking Misleads the Evaluation of LLM Reasoning on Frontier Science Benchmarks
- Language Models Learn to Mislead Humans via RLHF
Related papers
- Multi-Objective Structured Pruning of LLMs for Latency and Model Size Optimization
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Embeddings for Preferences, Not Semantics
- Can We Trust Item Response Theory for AI Evaluation?
- Diagnosing Task Insensitivity in Language Agents