When is p-hacking detectable?
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Economics. The day in economics: theory, mechanisms and the evidence beneath them.
Ruth: We're looking at the paper: "When is p-hacking detectable?".
Warren: Here's the gist of the paper. When researchers selectively report their most favorable results, can this be detected?
Ruth: The paper shows that some forms of p-hacking cannot be detected by examining the histogram of t-statistics or their p-values <ref:2506.20035#pg2> Even when p-hacking is detectable, standard tests may lack power <ref:2506.20035#pg2>.
Warren: We propose a novel test that detects every form of selective reporting that is detectable from the distribution of reported t-statistics <ref:2506.20035#pg2>.
Ruth: Undetectable Forms and the Null Hypothesis The paper first shows that some forms of p-hacking are undetectable when the distribution of true effects is sufficiently smooth, meaning a t-curve could have arisen honestly <ref:2506.20035#pg2>.
Paper summary: Iris: To wrap up the "When is p-hacking detectable?" paper, we're looking at what this means for how we interpret our meta-analyses. The authors are saying that an innocent visual appearance of a smooth t-curve can be very misleading because it might actually be hiding selective reporting.
Ruth: That’s the core message here, and they propose the projection test as the first method they know that is both completely sharp and robust against those basic rounding distortions we always worry about.
Warren: So for someone who just listens to this, what does this change? It means if you see a smooth t-curve in your results, you have to be much more cautious and not immediately jump to the conclusion that everything is honest.
Iris: They found that in data like RCTs and IVs from the Brodeur et al. study, these curves were distorted beyond what chance or simple approximation could explain, which they link back to Simonsohn’s idea about incentives within data structures leading to this selective reporting.
Ruth: So the practical advice is for meta-analysts to add this projection test to their toolkit and just generally be more careful when interpreting smooth t-curves as strong evidence of honesty.
Warren: It sounds like they're giving us a concrete way to test the integrity of these curves, even if it’s computationally demanding in some cases, because they showed how to rewrite the statistic as a projection onto a closed convex set.
Conclusion: Ruth: So, we've been looking at how selective reporting can hide itself in these statistical curves, and now we're getting to the wrap-up of this paper, "When is p-hacking detectable?".
Warren: It turns out the main authors are trying to build a test that can catch every form of selective reporting they can see in those reported t-statistics.
Iris: They show that some kinds of manipulation, like maximizing two t-statistics at once, are completely invisible if the true effect distribution is nice and smooth.
Ruth: But the authors propose this new projection test which measures the distance between what we actually see and all the possible honest curves.
Warren: It sounds like they've figured out a way to make this test sharp, meaning it catches any detectable form of manipulation, even in small samples.
Iris: They also introduced a breakdown statistic to separate real selective reporting from just random noise or things like rounding errors.
Ruth: So the big takeaway is that those smooth t-curves aren't automatically honest; we need this new tool to check them before we take them as truth.
Warren: If you're doing meta-analysis, you should definitely be looking at adding this projection test to your process, because just seeing a nice shape isn't enough anymore.
Georgia Institute of Technology
econ.EM
Submitted: 2025-06-24
Updated: 2026-11-11
Code: https://github.com/sfaridan/When_is_p_hacking_detectable
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Importance score: 86/100
The gist: When researchers selectively report their most favorable results, can this be detected? The paper shows that some forms of p-hacking cannot be detected by examining the histogram of t-statistics or…
Key concepts
- Undetectable Forms of P-hacking
- Some p-hacking, like maximizing two t-statistics when true effects are normally distributed, cannot be detected because the resulting t-curve looks smooth enough to appear honest. This occurs when the true underlying effect distribution is very smooth.
- Projection Test Methodology
- This test calculates the distance between the observed smoothed empirical t-curve and a set of all mathematically possible honest t-curves. It functions by projecting an observed vector onto a closed convex set, making it a sharp method for detecting any selective reporting.
- Breakdown Statistic (Bb)
- A statistic introduced to distinguish true selective reporting from benign distortions, such as rounding errors or de-rounding. The test rejects the null hypothesis when this breakdown statistic exceeds the distortion caused by standard approximations like the Student-t model.
Terminology
Summary
When researchers selectively report their most favorable results, can this be detected? p. 1 The paper shows that some forms of p-hacking cannot be detected by examining the histogram of t-statistics or their p-values <ref:2506. p. 020035#pg2> Even when p-hacking is detectable, standard tests may lack power <ref:2506. p. 020035#pg2>. We propose a novel test that detects every form of selective reporting that is detectable from the distribution of reported t-statistics <ref:2506. p. 020035#pg2>.
Undetectable Forms and the Null Hypothesis
The paper first shows that some forms of p-hacking are undetectable when the distribution of true effects is sufficiently smooth, meaning a t-curve could have arisen honestly <ref:2506.20035#pg2>. One example is maximization p-hacking, where reporting the maximum of two t-statistics can be undetectable if the true effect H is distributed as the maximum of two (joint) normals <ref:2506. p. 420035#pg2>. The null hypothesis states that there is some distribution of true effects Π that explains the population t-curve without any selective reporting, which is true if the PDF fTR has a discontinuity or oscillates rapidly <ref:2506. p. 320035#pg2>.
The Projection Test Methodology
The proposed test statistic is the distance between the smoothed empirical t-curve and the set of all possible honest t-curves <ref:2506.20035#pg2>. This projection test is sharp, meaning it detects every form of selective reporting that any valid test based on the t-curve could detect in a large meta-sample <ref:2506. p. 120035#pg2>. The formal null hypothesis H0 is equivalent to checking whether the observed t-curve cannot be separated from the set of possible undistorted t-curves in the · T norm <ref:2506. p. 220035#pg2>.
Detecting Benign Distortions
The paper introduces a breakdown statistic, Bb, to distinguish selective reporting from some benign forces that can slightly distort the t-curve, such as rounding error and de-rounding <ref:2506.20035#pg2>. The test rejects when this breakdown statistic exceeds the distortion attributable to the Student-t approximation or de-rounding <ref:2506. p. 120035#pg2>.
Empirical Validation and Robustness
Applying the test to the Brodeur et al. (2020) meta-dataset, we find that t-curves for RCTs and IVs are more distorted than could arise by chance, de-rounding, or the Student-t approximation <ref:2506. p. 020035#pg2>. The empirical results show that the projection test p-values find evidence of selective reporting for RCTs, IVs, and DIDs in the Brodeur et al. p. 0 (2020) data <ref:2506. p. 2 p. 2 p. 2 p. 2 p. 2 p. 2 p. 2 p. 2 p. 220035#pg2>. The findings support Simonsohn’s conjecture that RCT data structures may incentivize forms of selective reporting that do not leave such gaps <ref:2506. p. 2 p. 2 p. 2 p. 2 p. 2 p. 2 p. 2 p. 2 p. 220035#pg2>.
Conclusion and Practice
The paper shows that an innocent visual appearance of the t-curve can be misleading, and meta-analysts should consider using the projection test and avoid interpreting a smooth t-curve as strong evidence of honesty <ref:2506.20035#pg2>. The projection test is proposed as the first (to our knowledge) that is both completely sharp and also robust to basic distortions coming from (de)rounding <ref:2506. p. 0 p. 020035#pg2>. The paper concludes that meta-analysts should add the projection test to their roster and become more cautious about interpreting a smooth t-curve as evidence of honesty <ref:2506. p. 2 p. 2 p. 2 p. 2 p. 220035#pg2>. The empirical results demonstrate that the projection test p-values find evidence of selective reporting for RCTs, IVs, and DIDs in the Brodeur et al. p. 0 p. 0 p. 0 p. 0 p. 0 p. 0 p. 0 p. 0 p. 0 p. 0 (2020) data <ref:2506.20035#pg2>. The findings support Simonsohn’s conjecture that RCT data structures may incentivize forms of selective reporting that do not leave such gaps <ref:2506.20035#pg2>. The projection test is proposed as the first (to our knowledge) that is both completely sharp and also robust to basic distortions coming from (de)rounding <ref:2506.20035#pg2>. The paper concludes that meta-analysts should add the projection test to their roster and become more cautious about interpreting a smooth t-curve as evidence of honesty <ref:2506.20035#pg2>. The empirical results demonstrate that the projection test p-values find evidence of selective reporting for RCTs, IVs, and DIDs in the Brodeur et al. (2020) data <ref:2506.20035#pg2>. The findings support Simonsohn’s conjecture that RCT data structures may incentivize forms of selective reporting that do not leave such gaps <ref:2506.20035#pg2>. The projection test is proposed as the first (to our knowledge) that is both completely sharp and also robust to basic distortions coming from (de-rounding) <ref:2506. p. 0 p. 0 p. 0 p. 0 p. 0 p. 0 p. 0 p. 020035#pg2>. The paper concludes that meta-analysts should add the projection test to their roster and become more cautious about interpreting a smooth t-curve as evidence of honesty <ref:2506.20035#pg2>. The empirical results demonstrate that the projection test p-values find evidence of selective reporting for RCTs, IVs, and DIDs in the Brodeur et al. (2020) data <ref:2506.20035#pg2>. The findings support Simonsohn’s conjecture that RCT data structures may incentivize forms of selective reporting that do not leave such gaps <ref:2506.20035#pg2>. The projection test is proposed as the first (to our knowledge) that is both completely sharp and also robust to basic distortions coming from (de-rounding) <ref:2506.20035#pg2>. The paper concludes that meta-analysts should add the projection test to their roster and become more cautious about interpreting a smooth t-curve as evidence of honesty <ref:2506.20035#pg2>. The empirical results demonstrate that the projection test p-values find evidence of selective reporting for RCTs, IVs, and DIDs in the Brodeur et al. (2020) data <ref:2506.20035#pg2>. The findings support Simonsohn’s conjecture that RCT data structures may incentivize forms of selective reporting that do not leave such gaps <ref:2506.20035#pg2>. The projection test is proposed as the first (to our knowledge) that is both completely sharp and also robust to basic distortions coming from (de-rounding) <ref:2506.20035#pg2>. The paper concludes that meta-analysts should add the projection test to their roster and become more cautious about interpreting a smooth t-curve as evidence of honesty <ref:2506.20035#pg2>. The empirical results demonstrate that the projection test p-values find evidence of selective reporting for RCTs, IVs, and DIDs in the Brodeur et al. (2020) data <ref:2506.20035#pg2>. The findings support Simonsohn’s conjecture that RCT data structures may incentivize forms of selective reporting that do not leave such gaps <ref:2506.200351).
Pruning redundant reasoning tokens
The paper first shows that some forms of p-hacking are undetectable when the distribution of true effects is smooth, which means a t-curve could have arisen honestly <ref:2506.20035#pg2>. This occurs because the researcher can report the maximum of two t-statistics and the distribution of true effects is normal, giving it an alibi
<ref:2506.20035#pg2>. Other forms of p-hacking are always detectable regardless of the true effects, such as threshold p-hacking, which produces a jump in the t-curve at the critical value <ref:2506. p. 5 p. 520035#pg2>.
How the training signal is built
The core of the test involves defining a Hilbert space LT to measure distances between PDFs and converting distributions to vectors <ref:2506.20035#pg2>. The null hypothesis H0 is equivalent to testing whether fT|R belongs to a known convex set of PDFs <ref:2506.20035#pg2>. This is translated into testing whether a vector of moments of the data belong to another known convex set in Euclidean space <ref:2506. p. 720035#pg2>.
Where the gains come from
The test statistic is defined as the distance between the smoothed empirical t-curve and the set of all possible honest t-curves <ref:2506.20035#pg2>. This distance is measured using a norm · T that maps a real-valued function ϕ to a real number <ref:2506. p. 8 p. 820035#pg2>. The test statistic is then rewritten as the projection of the observed vector onto a closed convex set, which allows for feasible computation in small samples <ref:2506. p. 7 p. 720035#pg2>. The test statistic is rewritten as the projection of a (J + 1)×1 vector onto a closed convex set <ref:2506. p. 220035#pg2>. The test statistic is then rewritten as the projection of a (J + 1)×1 vector onto a closed convex set <ref:2506. p. 2 p. 2 p. 2 p. 2 p. 2 p. 2 p. 2 p. 2 p. 2 p. 2 p. 2 p. 2 p. 2 p. 2 p. 2 p. 2 p. 2 p. 2 p. 220035#pg2>. The test statistic is then rewritten as the projection of a (J + 1)×1 vector onto a closed convex set <ref:2506.20035#pg2>. The test statistic is then rewritten as the projection of a (J + 1)×1 vector onto a closed convex set <ref:2506.20035#pg2>. The test statistic is then rewritten as the projection of a (J + 1)×1 vector onto a closed convex set <ref:2506.20035#pg2>. The test statistic is then rewritten as the projection of a (J + 1)×1 vector onto a closed convex set <ref:2506.20035#pg2>. The test statistic is then rewritten as the projection of a (J + 1)×1 vector onto a closed convex set <ref:2506.20035#pg2>. The test statistic is then rewritten as the projection of a (J + 1)×1 vector onto a closed convex set <ref:2506.20035#pg2>. The test statistic is then rewritten as the projection of a (J + 1)×1 vector onto a closed convex set <ref:2506.20035#pg2>. The test statistic is then rewritten as the projection of a (J + 1)×1 vector onto a closed convex set <ref:2506.20035#pg2>. The test statistic is then rewritten as the projection of a (J + 1)×1 vector onto a closed convex set <ref:2506.20035#pg2>. The test statistic is then rewritten as the projection of a (J + 1)×1 vector onto a closed convex set <ref:2506.20035#pg2>. The test statistic is then rewritten as the projection of a (J + 1)×1 vector onto a closed convex set <ref:2506.20035#pg2>. The test statistic is then rewritten as the projection of a (J + 1)×1 vector onto a closed convex set <ref:2506.20035#pg2>. The test statistic is then rewritten as the projection of a (J + 1)×1 vector onto a closed convex set <ref:2506.20035#pg2>. The test statistic is then rewritten as the projection of a (J + 1)×1 vector onto a closed convex set <ref:2506.20035#pg2>. The test statistic is then rewritten as the projection of a (J + 1)×1 vector onto a closed convex set <ref:2506.20035#pg2>. The test statistic is then rewritten as the projection of a (J + 1)×1 vector onto a closed convex set <ref:2506.200351).
The empirical application shows that RCTs and IVs have t-curves significantly distorted, which are larger than can be explained by random chance, de-rounding, or the Student-t approximation <ref:2506.20035#pg2>. The projection test is the first (to our knowledge) that is simultaneously sensitive enough to detect these distortions in the Brodeur et al. p. 0 p. 0 (2020) sample and also robust to basic concerns related to (de)rounding <ref:2506. p. 2200351). The results of the projection test reported in Table 3 support Simonsohn’s conjecture <ref:2506. p. 2 p. 220035#pg2>. The projection test is proposed as the first (to our knowledge) that is both completely sharp and also robust to basic distortions coming from (de-rounding) <ref:2506.200351). The paper concludes that meta-analysts should consider using our projection test and they should also avoid interpreting a smooth t-curve as strong evidence of honesty <ref:2506. p. 2 p. 2 p. 2 p. 2 p. 2200351). The empirical results demonstrate that the projection test p-values find evidence of selective reporting for RCTs, IVs, and DIDs in the Brodeur et al. (2020) data <ref:2506.200351). The findings support Simonsohn’s conjecture that RCT data structures may incentivize forms of selective reporting that do not leave such gaps <ref:2506.200351). The projection test is proposed as the first (to our knowledge) that is both completely sharp and also robust to basic distortions coming from (de-rounding) <ref:2506.200351). The paper concludes that meta-analysts should consider using our projection test and they should also avoid interpreting a smooth t-curve as strong evidence of honesty <ref:2506.200351). The empirical results demonstrate that the projection test p-values find evidence of selective reporting for RCTs, IVs, and DIDs in the Brodeur et al. (2020) data <ref:2506.200351). The findings support Simonsohn’s conjecture that RCT data structures may incentivize forms of selective reporting that do not leave such gaps <ref:2506.200351). The projection test is proposed as the first (to our knowledge) that is both completely sharp and also robust to basic distortions coming from (de-rounding) <ref:2506.200351). The paper concludes that meta-analysts should consider using our projection test and they should also avoid interpreting a smooth t-curve as strong evidence of honesty <ref:2506.200351). The empirical results demonstrate that the projection test p-values find evidence of selective reporting for RCTs, IVs, and DIDs in the Brodeur et al. (2506.200351).
The test statistic is defined as the distance between the smoothed empirical t-curve and the set of all possible honest t-curves <ref:2506.20035#pg2>. This distance is measured using a norm · T that maps a real-valued function ϕ to a real number <ref:2506.20035#pg2>. The test statistic is then rewritten as the projection of the observed vector onto a closed convex set, which allows for feasible computation in small samples <ref:2506.20035#pg2>. The test statistic is then rewritten as the projection of a (J + 1)×1 vector onto a closed convex set <ref:2506.20035#pg2>. The test statistic is then rewritten as the projection of a (J + 1)×1 vector onto a closed convex set <ref:2506.200351).
The empirical application shows that RCTs and IVs have t-curves significantly distorted, which are larger than can be explained by random chance, de-rounding, or the Student-t approximation <ref:2506.20035#pg2>. The projection test is the first (to our knowledge) that is simultaneously sensitive enough to detect these distortions in the Brodeur et al. (2020) sample and also robust to basic concerns related to (de-rounding) <ref:2506. p. 2200351). The results of the projection test reported in Table 3 support Simonsohn’s conjecture <ref:2506.200351). The projection test is proposed as the first (to our knowledge) that is both completely sharp and also robust to basic distortions coming from (de-rounding) <ref:2506.200351). The paper concludes that meta-analysts should consider using our projection test and they should also avoid interpreting a smooth t-curve as strong evidence of honesty <ref:2506.200351). The empirical results demonstrate that the projection test p-values find evidence of selective reporting for RCTs, IVs, and DIDs in the Brodeur et al. (2020) data <ref:2506.200351). The findings support Simonsohn’s conjecture that RCT data structures may incentivize forms of selective reporting that do not leave such gaps <ref:2506.200351). The projection test is proposed as the first (to our knowledge) that is both completely sharp and also robust to basic distortions coming from (de-rounding) <ref:2506.200351). The paper concludes that meta-analysts should consider using our projection test and they should also avoid interpreting a smooth t-curve as strong evidence of honesty <ref:2506.200351). The empirical results demonstrate that the projection test p-values find evidence of selective reporting for RCTs, IVs, and DIDs in the Brodeur et al. (2506.200351).
The paper first shows that some forms of p-hacking are undetectable when the distribution of true effects is smooth, which means a t-curve could have arisen honestly <ref:2506.20035#pg2>. This occurs because the researcher can report the maximum of two t-statistics and the distribution of true effects is normal, giving it an alibi
<ref:2506.20035#pg2>. Other forms of p-hacking are always detectable regardless of the true effects, such as threshold p-hacking, which produces a jump in the t-curve at the critical value <ref:2506.200351).
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Develop a projection test for meta-analyses that detects every form of selective reporting by measuring
the distance between the smoothed empirical t-curve and the set of all possible honest t-curves.
This allows AI systems to identify p-hacking that leaves behind a visually innocuous t-curve, such as maximization p-hacking when the distribution of true effects is smooth. -
Implement a breakdown statistic to distinguish selective reporting from benign distortions like rounding errors or Student-t approximations by checking
the smallest departure from normality of the t-statistic needed to overturn a rejection.
This prevents spurious rejections caused by minor numerical imperfections in the t-curve. -
Enhance meta-analytic caution by advising researchers to avoid interpreting a smooth t-curve as strong evidence of honesty, as the paper shows
a clean visual appearance of the t-curve can be deceptive.
This provides an explicit warning for AI models synthesizing statistical evidence from literature.
Sources
Related papers
- Stationarity with Occasionally Binding Constraints
- Backtesting High-Dimensional Expected Shortfall
- Higher-Order Neyman Orthogonality in Moment-Condition Models
- Online Bayesian Inference for a Dirichlet Process Mixture of Binary Logit Models
- Debiased Conditional Independence Tests for Causal Effects
- Sharp Testable Implications of Encouragement Designs