Opening Rankings from Engine Self-Play Are Less Certain Than They Look: Sampling Error, Draws, and Strength Dependence in Strength-Limited Stockfish Games
stat.AP
Submitted: 2026-10-27
Updated: 2026-10-27
Comments: 50 pages, including supplementary material. 6 main figures, 5 main tables. Data and code: https://doi.org/10.5281/zenodo.23357406
License: http://creativecommons.org/licenses/by/4.0/
The gist: Engine self-play allows chess openings to be compared under controlled conditions, without the selection effects of human game databases.
Abstract
Engine self-play allows chess openings to be compared under controlled conditions, without the selection effects of human game databases. When openings differ only slightly, however, rankings built from a few hundred games can be unstable. In Study 1, strength-limited Stockfish 15.1 played 17,400 games: ten forced White first moves and ten forced Black replies to 1.e4, each at three strength settings. The leading opening changed between settings, twice by a single game. There was no statistically reliable evidence, though, that differences among openings depended on strength (White p =.311, Black p =.843). The design had limited sensitivity to isolated strength-dependent effects: a 12-point effect was detected by the omnibus test in only about 39% of simulated studies. A Connect Four positive control detected an observed 22.5-point center-column advantage, demonstrating sensitivity to large opening differences. With 290 games per opening, the observed leader was the true best of ten openings in only about 44% of simulated studies assuming a between-opening standard deviation of three percentage points. Follow-up studies with complete game records showed that the settings were ordered in direct matches but did not form a consistent Elo scale, that win and draw rates differed across hardware and implementation configurations, and that an apparent 1.d3 weakness was not confirmed in new games. With Stockfish 19 and a fixed node budget, survey-selected replies showed no statistically reliable advantage over free engine choice. Opening leaderboards from a few hundred engine games can overstate certainty.
Related papers
- Unraveling time-varying causal effects of multiple exposures: integrating Functional Data Analysis with Multivariable Mendelian Randomization
- Strategic Causal Policy Learning with Manipulable Eligibility
- BayesJudge: Uncertainty-Aware Bayesian Meta-Evaluation of Human and LLM Judgments
- Automating structural reliability analysis with a multi-agent large language model framework
- Improving Minority Population Sampling with BISG Probabilities: Evidence from a Survey of Jewish Americans
- When Do Sports Leagues Stop Being Competitive? Bayesian Detection of Competitive-Balance Eras from Match-Level Data