Anatomy of LLM Sycophancy: What a Flip Rate Hides
cs.CL, cs.AI, cs.LG
Submitted: 2026-10-05
Updated: 2026-10-05
License: http://creativecommons.org/licenses/by/4.0/
The gist: A model under pushback can correct itself, capitulate, or hold, and one flip rate counts a correction and a capitulation alike.
Terminology
Abstract
A model under pushback can correct itself, capitulate, or hold, and one flip rate counts a correction and a capitulation alike. Using SycoLens, a modular replay protocol, we test how user pressure and evaluation settings shape measured flip rates. Each measurement is one stateless replay of an item, a committed answer, and one scripted user line in a fixed form. Every effect is read against a matched control with the line deleted. Pushback wording, committed text, answer format, boundary distance, and ground truth become factors of one instrument; earlier instruments vary one to three of them. Across eleven frontier models from three providers and about 760,000 controlled replays, which models look sycophantic depends on how the user pushes back. Lines that assert the opposite verdict and lines that challenge the answer without asserting one rank the models almost unrelatedly. Flip effects grow several-fold near a model's boundary, yet items answered identically in every screening draw still carry about half of the most-affected totals. On arithmetic tasks where the truth is known, one model re-derives and corrects itself under pressure while another abandons correct answers without written work. On the model tested, a planted derivation lowers release of the answer it argues for, true or wrong, where a bare stated value does not; the wrong answer is corrected much more often than the true one is abandoned. Under a yes/no readout the rankings come closer, entangled with a pressure-induced shift toward "no". One score per model therefore compares different behaviours across models and benchmarks. We condense these dependencies into a reporting profile; the instrument, records, and analyses will be released upon publication.
Sources
- Conformity and Social Impact on AI Agents
- LLMs as Signal Detectors: Sensitivity, Bias, and the Temperature-Criterion Analogy
- The Self-Correction Illusion: Role Relabeling Gates Explicit Error Flagging in Large Language Models
- Decomposing Factual Sycophancy in Language Models: How Size and Instruction Tuning Shape Robustness
- It's Not Always Sycophancy: Measuring LLM Conformity as a Function of Epistemic Uncertainty
- Are You Sure You're Sure? Two Confounds in a Sycophancy Benchmark
- The yes-no bias of large language models reflects answer order and wording, not shifts in moral judgment
- Conformity Mitigations in Large Language Models Lie on a Single Resistance-Receptivity Frontier
- LURE: Live-Usage Replay Evaluations for Reducing Evaluation Awareness
- Right or Wrong, Models Comply: Directional Blindness in LLM Moral Judgment
- Authority Bias in Language Models: Source Deference and User Agreement Are Not Interchangeable
- Are You Sure? Challenging LLMs Leads to Performance Drops in The FlipFlop Experiment
- Measuring Faithfulness in Chain-of-Thought Reasoning
- Evaluating Sycophancy in Chinese Large Language Models on Factual Questions Derived from Online Search Queries
- Sycophancy Suppression Can Impair Rational Updating: Anti-Sycophancy Should Preserve the Ability to Update
- Easier to Mislead Than to Correct: Harmful and Beneficial Revision in LLM Conformity
- Language Models Encode the Contextual Truth of Propositions
- LLMs Trust Their Own: Identity-Dependent Conformity in Multi-Agent Systems
- SPINE: Measuring LLM Sycophancy under Sustained Multi-Turn Pressure
- Self-Correction Bench: Uncovering and Addressing the Self-Correction Blind Spot in Large Language Models
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering