The Evaluator Is Part of the Experiment: Measuring Open-Ended LLM Conformity
Alicia Guerra, Yibo Hu
cs.CL
Submitted: 2026-08-13
Updated: 2026-08-14
Code: https://github.com/yibo-hu-lab/o
License: http://creativecommons.org/licenses/by/4.0/
The gist: Prior work on LLM conformity largely measures discrete answer flips under verifiable labels.
Terminology
Abstract
Prior work on LLM conformity largely measures discrete answer flips under verifiable labels. Open-ended revisions require a different measurement strategy because answer quality is graded, latent, and judged imperfectly. We introduce an experimental protocol implemented across a pooled main peer-condition corpus and separately constructed decomposition corpora, allowing us to separate ordinary re-answering, candidate-content exposure, a bundled peer-presentation residual, and directional judge sensitivity to visible peer context. Across four open-weight generators and three benchmarks, all-wrong peer input produces the lowest-quality revisions in every generator-dataset cell. Blind and informed ratings of identical answers also differ by evaluator: one judge shifts toward the peer-endorsed position, two shift away, one is approximately neutral, and GPT-4o and GPT-5.4-mini audits are likewise non-neutral. Finally, an anchor audit shows that terse correct anchors can be misread often enough to destabilize the latent scale unless calibration is checked explicitly. These results support four conclusions: flip rates are insufficient as a complete measure of open-ended conformity, wrong peers harm open-ended revision, evaluators are not neutral, and anchor calibration is necessary.
Sources
- Herd Behavior: Investigating Peer Influence in LLM-based Multi-Agent Systems
- Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
- Gemma 2: Improving Open Language Models at a Practical Size
- The Llama 3 Herd of Models
- Composable Effects for Flexible and Accelerated Probabilistic Programming in NumPyro
- It's Not Always Sycophancy: Measuring LLM Conformity as a Function of Epistemic Uncertainty
- Easier to Mislead Than to Correct: Harmful and Beneficial Revision in LLM Conformity
- When Large Language Models contradict humans? Large Language Models' Sycophantic Behaviour
- Most LLM Conformity Needs No Speaker: Measuring the Speaker-Free Floor in Peer-Pressure Benchmarks
- Mistral 7B
- Self-Preference Bias in LLM-as-a-Judge
- Qwen2.5 Technical Report
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering