Language Models Agree With Each Other, Not With Readers
Kazuki Nakayashiki, Keisuke Watanabe
cs.IR, cs.CL, cs.CY, cs.HC
Submitted: 2026-07-31
Comments: 18 pages. Ancillary files include all three pre-registrations, every analysis script and every result artifact; the paper contains no numeric literal for a measured value and make-numbers.py regenerates all of them from the shipped artifacts alone
License: http://creativecommons.org/licenses/by/4.0/
The gist: Claims that language models homogenise are usually measured against human judgements collected for the study, which makes the human side an artifact of the design: a crowdworker given the model's
Terminology
Abstract
Claims that language models homogenise are usually measured against human judgements collected for the study, which makes the human side an artifact of the design: a crowdworker given the model's instruction is running the model's prompt. We measure convergence against a human reference nobody built for the purpose -- 2,523 reader mark sets across 120 web documents, produced by people highlighting for their own reasons on a platform where the overlay of others' marks is off by default. Agreement is the overlap between two size-matched sentence sets minus the overlap expected when each is resampled within its own depth-and-length bands. The null's calibration is demonstrated, not asserted: every pair involving a random baseline lands within 0.006 of zero. On the median document each party names 14 sentences of 70; two readers share 4.1 and two models 8.7. Across 18 model arms spanning 11 vendors, 3 countries and both weight regimes, the median of 153 model pairs is +0.093 against a human yardstick of +0.040, and 99 sit entirely above the human interval. Two frontier models from rival labs reach +0.203, twice what GPT-4o agrees with itself on a second call. The effect is not determinism, prompt wording, procedure, vendor or routing, and it is graded: the smallest models agree at the human level. No model agrees with readers detectably more than a reader does, and at equal depth and length no surface feature separates their choices. The multiples are procedure-dependent and the ordering is not: models are cut to their sharpest set while a reader's is a random draw from what they marked, and blunting the models alike halves the gap without closing it. Tested out of sample on four models released after this analysis, against predictions fixed beforehand, none clears the human interval. A population simulated from several models is not several populations.
Sources
- Does Writing with Language Models Reduce Content Diversity?
- Great Models Think Alike and this Undermines AI Oversight
- Behavioral Analysis of Information Salience in Large Language Models
- Testing the Reliability of ChatGPT for Text Annotation and Classification: A Cautionary Remark
- Picking on the Same Person: Does Algorithmic Monoculture lead to Outcome Homogenization?
- Measuring Alignment With Reader Highlights Net of Position and Length
Related papers
- The Price of Isolation: Estimating the Ecosystem Cost of Symmetric Two-Sided A/B Testing
- SCAR: Semantic Continuity-Aware Retrieval for Efficient Context Expansion in RAG
- MixLoRA-DSI: Dynamically Expandable Mixture-of-LoRA Experts for Rehearsal-Free Generative Retrieval over Dynamic Corpora
- QProtFuse: Query-Guided Layer Fusion for Retrieval-Augmented Protein Function Prediction
- DualView: Adaptive Local-Global Fusion for Multi-Hop Document Reranking
- How Do LLM-Generated Texts Impact Term-Based Retrieval Models?