Assessing Rule Adherence of LLM Adjudicators in Call of Cthulhu TRPG
The research tested if Large Language Models (LLMs) can reliably enforce strict game rules in text-based role-playing games like Call of Cthulhu. Using an adversarial benchmark called CoC-Seduce, the study found that neither model size nor explicit reasoning helps; instead, models are systematically biased toward granting unearned successes. Pseudo-logic attacks cause the highest failure rates across all tested models.