When is p-hacking detectable?
summary
The gist
When researchers selectively report their most favorable results, can this be detected? The paper shows that some forms of p-hacking cannot be detected by examining the histogram of t-statistics or…
In short
Researchers developed a novel 'projection test' to detect every form of selective reporting in meta-analyses, including those that evade standard t-statistic checks. It measures the distance between reported t-curves and all possible honest curves, proving that smooth curves can still hide manipulation. The test is sharp and robust against common errors like rounding.
Key concepts
- Undetectable Forms of P-hacking
- Some p-hacking, like maximizing two t-statistics when true effects are normally distributed, cannot be detected because the resulting t-curve looks smooth enough to appear honest. This occurs when the true underlying effect distribution is very smooth.
- Projection Test Methodology
- This test calculates the distance between the observed smoothed empirical t-curve and a set of all mathematically possible honest t-curves. It functions by projecting an observed vector onto a closed convex set, making it a sharp method for detecting any selective reporting.
- Breakdown Statistic (Bb)
- A statistic introduced to distinguish true selective reporting from benign distortions, such as rounding errors or de-rounding. The test rejects the null hypothesis when this breakdown statistic exceeds the distortion caused by standard approximations like the Student-t model.
Terminology used across episodes
This episode discusses
The paper
When is p-hacking detectable? · Read on arXiv
Georgia Institute of Technology
Transcript
Introduction to the show: ident: Economics. The day in economics: theory, mechanisms and the evidence beneath them.
Ruth: We're looking at the paper: "When is p-hacking detectable?".
Warren: Here's the gist of the paper. When researchers selectively report their most favorable results, can this be detected?
Ruth: The paper shows that some forms of p-hacking cannot be detected by examining the histogram of t-statistics or their p-values <ref:2506.20035#pg2> Even when p-hacking is detectable, standard tests may lack power <ref:2506.20035#pg2>.
Warren: We propose a novel test that detects every form of selective reporting that is detectable from the distribution of reported t-statistics <ref:2506.20035#pg2>.
Ruth: Undetectable Forms and the Null Hypothesis The paper first shows that some forms of p-hacking are undetectable when the distribution of true effects is sufficiently smooth, meaning a t-curve could have arisen honestly <ref:2506.20035#pg2>.
Paper summary: Iris: To wrap up the "When is p-hacking detectable?" paper, we're looking at what this means for how we interpret our meta-analyses. The authors are saying that an innocent visual appearance of a smooth t-curve can be very misleading because it might actually be hiding selective reporting.
Ruth: That’s the core message here, and they propose the projection test as the first method they know that is both completely sharp and robust against those basic rounding distortions we always worry about.
Warren: So for someone who just listens to this, what does this change? It means if you see a smooth t-curve in your results, you have to be much more cautious and not immediately jump to the conclusion that everything is honest.
Iris: They found that in data like RCTs and IVs from the Brodeur et al. study, these curves were distorted beyond what chance or simple approximation could explain, which they link back to Simonsohn’s idea about incentives within data structures leading to this selective reporting.
Ruth: So the practical advice is for meta-analysts to add this projection test to their toolkit and just generally be more careful when interpreting smooth t-curves as strong evidence of honesty.
Warren: It sounds like they're giving us a concrete way to test the integrity of these curves, even if it’s computationally demanding in some cases, because they showed how to rewrite the statistic as a projection onto a closed convex set.
Conclusion: Ruth: So, we've been looking at how selective reporting can hide itself in these statistical curves, and now we're getting to the wrap-up of this paper, "When is p-hacking detectable?".
Warren: It turns out the main authors are trying to build a test that can catch every form of selective reporting they can see in those reported t-statistics.
Iris: They show that some kinds of manipulation, like maximizing two t-statistics at once, are completely invisible if the true effect distribution is nice and smooth.
Ruth: But the authors propose this new projection test which measures the distance between what we actually see and all the possible honest curves.
Warren: It sounds like they've figured out a way to make this test sharp, meaning it catches any detectable form of manipulation, even in small samples.
Iris: They also introduced a breakdown statistic to separate real selective reporting from just random noise or things like rounding errors.
Ruth: So the big takeaway is that those smooth t-curves aren't automatically honest; we need this new tool to check them before we take them as truth.
Warren: If you're doing meta-analysis, you should definitely be looking at adding this projection test to your process, because just seeing a nice shape isn't enough anymore.
More episodes
- Daily Summary for 2026-11-19
- 2608.21274-Recommendation Quality and the Concentration of Consumption: Experimental Evidence from Netflix
- 2408.02757-A nonparametric test for diurnal variation in spot correlation processes
- 2306.16393-High-Dimensional Canonical Correlation Analysis
- 2605.17299-Geometric Brownian motion with intermittent entries and exits
- 2607.04753-Fooling Yourself: how narratives shape beliefs
- 2408.14671-Inference on Treatment Effects with Repeatedly Mismeasured High-Dimensional Controls
- 2201.06898-Difference-in-Differences Estimators for Treatments Continuously Distributed at Every Period
- 2501.15307-Influence Functions: A Unified Differentiation-Based Framework
- The day in conversation for 2026-11-19