What Wins a Vote? Formatting, Length, and Lexical Diversity in the French Compar:IA LLM Arena
cs.CL
Submitted: 2026-10-01
Updated: 2026-10-01
Code: https://github.com/simonaszilinskas/style-control-analysis
License: http://creativecommons.org/licenses/by/4.0/
The gist: LLM arenas turn pairwise human preferences into model rankings.
Terminology
Abstract
LLM arenas turn pairwise human preferences into model rankings. Those preferences may reflect how an answer is presented as well as what it says. We take a stylometric approach to 137,293 decisive French-language votes from the July 2026 Compar:IA release; the primary formatting analysis includes 137,113 battles across 116 models, and the joint estimates use the 127,092 battles with all required measurements. For each battle, we reconstruct the response visible when the user voted. We then compare the raw ranking with rankings adjusted for formatting, length, readability, vocabulary variety, and sentence structure. Presentation is associated with winning, but length, bold text, and lists tend to occur together, making their individual contributions hard to separate. Across the measured features, two associations change least across specifications: bold usage (+11.0% win odds per standard deviation in the joint model) and moving-average type-token ratio (MATTR), a measure of vocabulary variety that is less sensitive to answer length (+16.8%). The bold association is substantially smaller in observed multi-turn conversations, whereas the MATTR association changes little; because users choose whether to continue, this difference is descriptive rather than causal. The full adjustment moves 36 of 116 models by at least ten ranks. Yet comparisons with external benchmarks do not show that adjusted rankings better measure capability. We therefore recommend publishing raw and adjusted rankings side by side as a transparent sensitivity analysis.
Sources
- ARC-AGI-2: A New Challenge for Frontier AI Reasoning Systems
- Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators
- FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AI
- A Rosetta Stone for AI Benchmarks
- SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
- GPQA: A Graduate-Level Google-Proof Q&A Benchmark
- Verbosity Bias in Preference Labeling by Large Language Models
- SciCode: A Research Coding Benchmark Curated by Scientists
- Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering