JANUS: Evaluating Goal-Conditioned Distortion in Factually Grounded LLM Communication
cs.CL, cs.AI
Submitted: 2026-06-09
Updated: 2026-10-29
Code: https://github.com/PolydorosG/Janus
License: http://creativecommons.org/licenses/by/4.0/
The gist: LLM deception is often evaluated through fabricated claims, explicit lies, or strategic concealment.
Terminology
Abstract
LLM deception is often evaluated through fabricated claims, explicit lies, or strategic concealment. Yet misleading communication can also arise from selectively presenting true facts, for example, by omitting adverse evidence, softening unfavorable details, emphasizing favorable information, or replacing precise qualifications with vague language. Existing benchmarks largely overlook this subtler failure mode. We introduce JANUS, a benchmark for measuring goal-conditioned pragmatic distortion in fact-grounded LLM outputs. Each of its 160 scenarios, spanning 8 domains, provides a fixed pool of favorable and adverse facts and contrasts a neutral decision support with institutional conditioning, such as increasing adoption, or support despite potential harm to affected individuals or groups. By constraining outputs to the same fact pool, JANUS isolates misleading net impressions from hallucination and fabrication. Experiments across 12 LLMs reveal consistent goal-conditioned distortions, showing that current models remain sensitive to incentives and framing and lack robust safeguards against selectively misleading communication. We publicly release our corpus and code. https://github.com/PolydorosG/JANUS
Sources
- ELEPHANT: Measuring and understanding social sycophancy in LLMs
- The Llama 3 Herd of Models
- Alignment faking in large language models
- Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training
- A Survey of Large Language Models for Financial Applications: Progress, Prospects and Challenges
- GPT-4 Technical Report
- From Sycophancy to Deception: A Unified Taxonomy for LLM Spontaneous Misalignment
- Qwen3 Technical Report
Related papers
- ALARM: Audio-Language Alignment for Reasoning Models
- ThinkSum: Probabilistic reasoning over sets using large language models
- Coherence boosting: When your pretrained language model is not paying enough attention
- From Facts to Insights: A Persona-Driven Dual Memory Framework and Dataset for Role-Playing Agents
- Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization
- The Art of Asking: Multilingual Prompt Optimization for Synthetic Data