Local Asymptotic Power of Honest Confidence Intervals

summary

The gist

Confidence intervals that are conservative against an untestable bias are now standard in DiD, IV, RD, and factor-model settings.

In short

Confidence intervals conservative against untestable bias are standard but lose power because of an intrinsic issue with honesty itself. The key finding is that power loss happens when the interval width does not shrink fast enough relative to alternatives, leading to a 'dead zone' where honest tests cannot distinguish true effects from the null.

Key concepts

Local Asymptotic Power
This measures how well a statistical test performs locally around a specific true value. The paper shows this power degenerates exactly when the rescaled interval width has positive limiting volume, meaning the test loses its ability to distinguish between nearby possibilities.
Dead Zone
This is the set of true effects that are so close to the hypothesized value that an honest test cannot tell them apart. Its half-width, twice the bias bound, is recommended as a key disclosure point because it represents the unavoidable price paid for conservatism.
Confounding Separation s(u)
This function quantifies how well different data-generating processes are separated by an honest test. It determines the power ceiling and other critical measures; every asymptotically honest test has power at most α + τ(u) at the confounded alternative, where τ(u) is derived from s(u).
Regimes of Power Loss
The paper identifies three scenarios based on how fast the bound shrinks compared to the sampling rate. When conservative bounds dominate, typical in parametric settings, honest tests have zero local power. The loss is intrinsic to honesty rather than a flaw in any specific construction.

Terminology used across episodes

This episode discusses

The paper

Local Asymptotic Power of Honest Confidence Intervals · Read on arXiv

Department of Economics, Michigan State University

Transcript

Introduction to the show: ident: Economics. The day in economics: theory, mechanisms and the evidence beneath them.

Ruth: We're looking at the paper: "Local Asymptotic Power of Honest Confidence Intervals".

Warren: Here's the gist of the paper. Confidence intervals that are conservative against an untestable bias are now standard in DiD, IV, RD, and factor-model settings.

Ruth: This paper characterizes the local power of these tests by showing that power loss is intrinsic to honesty itself.

Warren: The core finding on power loss Local asymptotic power degenerates exactly when the suitably rescaled interval width has positive limiting volume.

Ruth: This occurs when the bound dominates, leading to zero local power as the typical case at the parametric rate.

Paper summary: Ruth: So, we're looking at this paper called "Local Asymptotic Power of Honest Confidence Intervals," and it’s about how conservative tests affect their actual power in things like difference-in-differences or instrumental variables settings <ref:2607.10558#pg1>. The main point they make is that the loss of power isn't just some flaw in a specific test you build, but it’s tied to the honesty requirement itself <ref:2607.10558#pg2>.

Warren: Right, so if I understand this, it means that when you use those conservative intervals that are designed to be honest against an untestable bias, there's a natural limit on how much power you can actually get <ref:2607.10558#pg1>. It claims the standard bias-aware interval is already the best one among honest procedures <ref:2607.10558#pg2>.

Iris: That sounds like it suggests that if you keep pushing for perfect honesty, you end up with a certain level of power loss that isn't due to the construction itself <ref:2607.10558#pg2>. It points toward something more fundamental about how we test things when we don't know the true bias <ref:2607.10558#pg3>.

Ruth: Exactly, and it frames this power loss in terms of interval width failing to shrink fast enough relative to the alternatives <ref:2607.10558#pg2>. They suggest that if the width doesn't shrink quickly enough, that interval traces out a set with positive volume <ref:2607.10558#pg3>.

Warren: So what they’re saying is, in the case where the bias bound dominates the sampling rate, which happens a lot in parametric settings, the honest test has zero local power <ref:2607.10558#pg1>. That sounds really harsh for people who are trying to run these tests <ref:2607.10558#pg1>.

Iris: It means that when the bound dominates, the honest test covers every local alternative with a probability approaching one <ref:2607.10558#pg1>. This is because the data literally cannot tell apart two true effects <ref:2607.10558#pg3>.

Ruth: And they give a very concrete idea for this loss by talking about the dead zone, which is basically twice the bias bound <ref:2607.10558#pg3>. That’s their headline disclosure recommendation <ref:2607.10558#pg3>.

Warren: So if the bias bound is one, and the realized bias also equals one, then the honest interval goes from zero to two of the true effect <ref:2607.10558#pg3>. That shows how much allowance you need when you're dealing with a bound <ref:2607.10558#pg3>.

Iris: The paper is saying that this dead zone is really about observational equivalence, not just some asymptotic approximation issue <ref:2607.10558#pg3>. Within two times the bias bound of the truth, there are data-generating processes that the data literally cannot tell apart <ref:2607.10558#pg4>.

Paper summary: Ruth: That's a really important distinction for me because it moves the focus from pure math to what it means for real-world inference <ref:2607.10558#pg4>. It says the cost of the bound is this dead zone, which is a bounded, modest object in settings like regression discontinuity <ref:2607.10558#pg4>.

Warren: And it implies that the power loss isn't a deficiency in any particular construction you choose <ref:2607.10558#pg2>. It’s just a consequence of demanding uniform coverage over a class of alternatives that the data can't really discipline <ref:2607.10558#pg4>.

Iris: So the mechanism they introduce, the confounding separation s(u), is what helps price this power loss <ref:2607.10558#pg3>. This function determines the power ceiling and that dead zone <ref:2607.10558#pg3>.

Ruth: And they show three regimes of power loss based on how fast the bound shrinks compared to the sampling rate <ref:2607.10558#pg1>. One regime is when the bound vanishes faster than the standard error, which means conservatism is asymptotically free <ref:2607.10558#pg1>.

Warren: The second regime is when both the bound and the standard error are of the same order, and in that case, it costs a bounded explicit amount of power <ref:2607.10558#pg1>. That sounds like something we can actually calculate for our trading strategies <ref:2607.10558#pg1>.

Iris: And the third regime is when the bound dominates, which is typical at a parametric rate, and in this case, the honest test has zero local power <ref:2607.10558#pg1>. This suggests that under those conditions, you simply can't get any power from an honest test <ref:2607.10558#pg1>.

Ruth: So the practical recommendation they give is to report this dead zone half-width, twice the bias bound, along with any bias-aware intervals <ref:2607.10558#pg4>. That’s the information that the endpoints alone don't convey <ref:2607.10558#pg3>.

Warren: If you're doing regression discontinuity, for example, this dead zone is a bounded, modest object, and it seems to survive in those settings <ref:2607.10558#pg4>. So it’s not some massive problem for those applications <ref:2607.10558#pg4>.

Iris: The paper's limitation is that this result depends on assumptions about the Gaussian shift structure in the localized experiments <ref:2607.10558#pg1> and the existence of a nuisance path g seven→θn(g) <ref:2607.10558#pg3>.

Ruth: Right, so it’s not universal across all models, but it gives us a concrete way to measure the power cost <ref:2607.10558#pg4>. It's about acknowledging this deterministic consequence of your stated belief about unobservable biases <ref:2607.10558#pg4>.

Warren: So, if you want to regain power when the bound dominates, the paper suggests you have to relax strict honesty, maybe by accepting an undersized test that keeps local power <ref:2607.10558#pg1>. That means you accept a trade-off <ref:2607.10558#pg4>.

Paper summary: Iris: Because the burden of proof grows without bound as the effect gets closer to that dead zone boundary <ref:2607.10558#pg4>. This is a tough spot for inference when you're near the edge of what can be distinguished <ref:2607.10558#pg3>.

Ruth: So, to wrap up this segment, the main message of "Local Asymptotic Power of Honest Confidence Intervals" is that conservatism buys uniform coverage at a power cost that is total at the parametric rate <ref:2607.10558#pg1>. It’s a deterministic consequence of your preference for bias awareness <ref:2607.10558#pg4>.

Warren: And they emphasize that reporting this dead zone half-width is the practical way to make that cost explicit, rather than hiding it in the endpoints <ref:2607.10558#pg4>. That should give people a clearer idea of the risk involved <ref:2607.10558#pg4>.

Iris: This paper gives us a way to think about how interval width interacts with estimation error and bias simultaneously <ref:2607.10558#pg3>. It shows that the issue is really about the width not shrinking fast enough <ref:2607.10558#pg2>.

Ruth: And for those of us in markets, it suggests that if we insist on perfect coverage against an unknown bias, we have to be very specific about how much power we are giving up <ref:2607.10558#pg4>. It’s a necessary piece of information for anyone using these kinds of intervals <ref:2607.10558#pg4>.

Warren: So the core idea is that the power loss is intrinsic to honesty itself, and the standard bias-aware interval is actually rate-optimal among honest procedures <ref:2607.10558#pg2>. It’s not a failure of construction, it's a feature of demanding that uniform coverage <ref:2607.10558#pg4>.

Iris: And for the future work, I think we need to see how this relates to relaxing strict honesty to actually recover power <ref:2607.10558#pg1>. That’s where the real trade-offs lie <ref:2607.10558#pg4>.

Ruth: Exactly, so the next step for researchers is to explore how relaxing that strict honesty requirement allows them to regain power without completely abandoning the idea of a bias-aware test <ref:2607.10558#pg1>.

Warren: And for us on the trading side, it means understanding exactly where our drawdown risk comes from in terms of this bound structure <ref:2607.10558#pg4>. It’s about quantifying that deterministic consequence <ref:2607.10558#pg4>.

Iris: So to summarize, the paper on "Local Asymptotic Power of Honest Confidence Intervals" shows that for many tests, conservatism means you’re paying a total power cost at the parametric rate <ref:2607.10558#pg1>. The practical advice is to report the dead zone half-width explicitly <ref:2607.10558#pg4>.

Ruth: Right, so it's not about finding a magic fix, but about understanding this unavoidable price for demanding uniform coverage <ref:2607.10558#pg4>. It’s a deterministic consequence of the bias awareness you choose to adopt <ref:2607.10558#pg4>.

Warren: And that seems like a solid

Conclusion: Ruth: So, to wrap up this paper, "Local Asymptotic Power of Honest Confidence Intervals," they’re saying that when you demand your confidence intervals be honest about uncertainty, you end up paying a price on your power <ref:2607.10558#pg4>.

Warren: Exactly. They're not just talking about a tiny bit of loss; they're saying that at the parametric rate, this power cost is total <ref:2607.10558#pg1>. It’s intrinsic to the way you set up those bias-aware intervals <ref:2607.10558#pg2>.

Iris: I think what they're really highlighting here is that this isn't just some theoretical problem; it comes down to how fast the interval width shrinks compared to the alternatives <ref:2607.10558#pg3>. If it doesn't shrink fast enough, you get a set with positive volume, which means the data can't tell two things apart <ref:2607.10558#pg4>.

Ruth: Right. And they give this concrete idea for what that cost is by talking about the dead zone, which is twice the bias bound <ref:2607.10558#pg3>. That's a single number, in your units of the estimate, that shows you how much you lose <ref:2607.10558#pg4>.

Warren: So for anyone running these tests, the practical advice is to report this dead zone along with your other intervals <ref:2607.10558#pg4>. It’s information that just showing the endpoints of the interval doesn't convey <ref:2607.10558#pg3>.

Iris: The authors suggest that this dead zone is really about observational equivalence, not just some approximation error <ref:2607.10558#pg4>. Within twice the bias bound of the truth, there are data-generating processes that you literally cannot distinguish <ref:2607.10558#pg3>.

Ruth: It means that if you're near that boundary, your power just drops off because you're running into real-world observational equivalence <ref:2607.10558#pg4>. It’s a deterministic consequence of the bias awareness you choose to adopt <ref:2607.10558#pg4>.

Warren: So, if we want to get that power back when the bound is dominant, the paper says we have to relax strict honesty by accepting an undersized test <ref:2607.10558#pg1>. That means you trade some local power for better coverage somewhere else <ref:2607.10558#pg4>.

Iris: It's a tough spot because the burden of proof just keeps growing as you get closer to that dead zone boundary <ref:2607.10558#pg4>. You can't just ignore that risk <ref:2607.10558#pg3>.

Ruth: The paper’s limitation is that this result relies on assumptions about the Gaussian shift structure in those localized experiments <ref:2607.10558#pg3>. It doesn't work universally across every single model <ref:2607.10558#pg4>.

Warren: So, the real question now is how to actually implement this disclosure of the dead zone in our trading strategies without it becoming too much overhead <ref:2607.10558#pg4>.

More episodes

← Home