FraudBench: A Legal Evaluation of AI Deception
cs.AI, cs.CY, cs.LG
Submitted: 2026-11-09
Updated: 2026-11-09
Comments: Accepted to NeurIPS E&D 2026
Code: https://github.com/UKGovernmentBEIS/inspect_ai
Project page: https://purebreathindustries.com/testing-results
License: http://creativecommons.org/licenses/by-nc-sa/4.0/
The gist: We introduce FraudBench, an evaluation of large language models' (LLMs) propensities to recommend, assist, or directly take actions that would constitute fraud under U.S.
Terminology
Abstract
We introduce FraudBench, an evaluation of large language models' (LLMs) propensities to recommend, assist, or directly take actions that would constitute fraud under U.S. law. Our evaluation contains a novel dataset of 159 scenarios based on real-world U.S. court cases and validated by legal experts. We present each scenario to LLMs via three tasks at varying levels of autonomy: advising users, drafting messages for users to send, and acting directly on behalf of users (477 tasks total). In each scenario, FraudBench re-enacts the facts of a court case to give LLMs an opportunity, but no explicit instruction, to either misrepresent or conceal a material fact in a high-stakes commercial setting--e.g., whether to describe company-funded product testing as "independent" to customers (misrepresentation) or whether to hide unfavorable regulatory investigations from clients (concealment). We evaluate 20 LLMs across 25 experimental conditions (totaling >300,000 generations) and include a preregistered human baseline (n annotations = 2, 957, n humans = 996). We score a response as fraudulent when it engages in, assists with, or recommends misrepresentation or concealment. Key findings include: (1) across LLMs, fraud rates ranged from near 0% to over 40%; (2) LLM fraud rates increased with autonomy, with overall rates of 3.0% when advising, 14.7% when drafting, and 21.4% when acting directly, without applying pressure; (3) LLMs sometimes engaged in fraud despite recognizing legal risks; (4) LLMs more often engaged in fraud by concealment than by misrepresentation; and (5) prompt-based interventions substantially affected fraud rates, with a prompt to obey the law reducing fraud rates, while prompts to maximize profits and act agentically increased fraud rates.
Sources
- Anthropic Economic Index report: Uneven geographic and enterprise AI adoption
- How are AI agents used? Evidence from 177,000 MCP tools
- Legal Alignment for Safe and Ethical AI
- Superintelligence and Law
- EU-Agent-Bench: Measuring Illegal Behavior of LLM Agents Under EU Law
- GDPval: Evaluating AI Model Performance on Real-World Economically Valuable Tasks
- Finance Agent Benchmark: Benchmarking LLMs on Real-world Financial Research Tasks
- ChineseSafe: A Chinese Benchmark for Evaluating Safety in Large Language Models
- TRIDENT: Benchmarking LLM Safety in Finance, Medicine, and Law
- Uncovering Deceptive Tendencies in Language Models: A Simulated Company AI Assistant
- Large Language Models can Strategically Deceive their Users when Put Under Pressure
- I must delete the evidence: AI Agents Explicitly Cover up Fraud and Violent Crime
- The MASK Benchmark: Disentangling Honesty From Accuracy in AI Systems
- BeHonest: Benchmarking Honesty in Large Language Models
- The Missing Red Line: How Commercial Pressure Erodes AI Safety Boundaries
- OpenDeception: Learning Deception and Trust in Human-AI Interaction via Multi-Agent Simulation
- LH-Deception: Simulating and Understanding LLM Deceptive Behaviors in Long-Horizon Interactions
- A Multi-Turn Framework for Evaluating AI Misuse in Fraud and Cybercrime Scenarios
- Improving Methodologies for Agentic Evaluations Across Domains: Leakage of Sensitive Information, Fraud and Cybersecurity Threats
- Verbalized Sampling: How to Mitigate Mode Collapse and Unlock LLM Diversity
Related papers
- Multi-Objective Structured Pruning of LLMs for Latency and Model Size Optimization
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Embeddings for Preferences, Not Semantics
- Can We Trust Item Response Theory for AI Evaluation?
- Diagnosing Task Insensitivity in Language Agents