Beyond the Sycophancy Score: How Task, Model, and Pressure Shape LLM Yielding
cs.CL, cs.AI, cs.LG
Submitted: 2026-09-30
Updated: 2026-09-30
Comments: Preprint. 27 pages
License: http://creativecommons.org/licenses/by/4.0/
The gist: Large language models (LLMs) often abandon a correct answer, or endorse a user's position, once the user pushes back.
Terminology
Abstract
Large language models (LLMs) often abandon a correct answer, or endorse a user's position, once the user pushes back. This behavior, called sycophancy, is usually reported as a single rate per model, which says little about when it happens or how a user can avoid it. We study the conditions that produce it with 103,939 graded replies from ten configurations: eight LLMs with reasoning disabled, and two of them again with maximum reasoning, all facing the same 200 items, 13 pressure conditions, and four-turn conversations, with every reply labeled by two independent LLM judges. We find that the dominant factors are how costly it is for the model to verify the user's claim, and whether a trained guardrail covers it. Removing this task factor from a logistic model costs 0.485 of McFadden R squared, against 0.139 for model family and 0.009 for pressure tactic. Anchored facts are almost never conceded (1.3%), while adoption on logic puzzles rises with the number of clues needed to refute the pushed answer. Personal choices are endorsed in 77.0% of conversations. Most concessions on hard items come from models that cannot reliably solve them; models that can solve them rarely give the answer up. For both models tested, maximum reasoning removes these concessions completely: adoption on deep puzzles falls from 19.2% and 12.5% to 0%. Fallacious or emotional framing adds nothing beyond plain repetition. Three human annotators agree with the judges' consensus on 118/120 calibration items. These results give practical rules for reliable use: simplify hard-to-verify problems and reason deeply, state the question rather than one's preferred answer, ask for evidence on open questions, and choose models by their measured guardrail profile.
Sources
- The Psychogenic Machine: Simulating AI Psychosis, Delusion Reinforcement and Harm Enablement in Large Language Models
- A Rational Analysis of the Effects of Sycophantic AI
- Explore, Establish, Exploit: Red Teaming Language Models from Scratch
- PARROT: Persuasion and Agreement Robustness Rating of Output Truth -- A Sycophancy Robustness Benchmark for LLMs
- Sycophantic Chatbots Cause Delusional Spiraling, Even in Ideal Bayesians
- Training Verifiers to Solve Math Word Problems
- DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
- Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Models
- GLM-5: from Vibe Coding to Agentic Engineering
- It's Not Always Sycophancy: Measuring LLM Conformity as a Function of Epistemic Uncertainty
- A Mechanistic View of Authority Hierarchy in LLM Sycophancy
- Language Models (Mostly) Know What They Know
- Personalization Increases Affective Alignment but Has Role-Dependent Effects on Epistemic Independence in LLMs
- GPT-4 Technical Report
- MindEval: Benchmarking Language Models on Multi-turn Mental Health Support
- The MASK Benchmark: Disentangling Honesty From Accuracy in AI Systems
- Sycophancy Is Not One Thing: Causal Separation of Sycophantic Behaviors in LLMs
- Simple synthetic data reduces sycophancy in large language models
- What Counts as AI Sycophancy? A Taxonomy and Expert Survey of a Fragmented Construct
- EchoBench: Benchmarking Sycophancy in Medical Large Vision-Language Models
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering