Post-Selection Inference for Partially Identified Parameters

summary

The gist

Interval identification of parameters such as average treatment effects, average partial effects and welfare is particularly common when using observational data and experimental data with imperfect…

In short

The paper develops new inference tools for parameters like average treatment effects when data is selected based on estimated sets or data-dependent rules. It introduces three types of confidence intervals—conditional, projection, and hybrid CIs—to provide valid statistical estimates for these complex scenarios where standard methods fail.

Key concepts

Conditional CIs
These are confidence intervals that are valid assuming the treatment or policy of interest was chosen from a pre-estimated set. They offer correct coverage under this specific assumption, which is useful when selection is based on known best performers.
Projection CIs
These intervals do not require knowing the exact data-dependent selection rule beforehand. They provide a baseline for inference that does not depend on having a perfect understanding of how the object was chosen from the data.
Hybrid CIs
These are designed to perform best overall by balancing the length properties of conditional and projection CIs. They aim to limit the worst-case length performance, offering a more robust confidence interval when dealing with complex data-dependent selections.

Terminology used across episodes

This episode discusses

The paper

Post-Selection Inference for Partially Identified Parameters · Read on arXiv

Sukjin Han, Adam McCloskey

University of Bristol · University of Colorado

Transcript

Introduction to the show: ident: Economics. The day in economics: theory, mechanisms and the evidence beneath them.

Ruth: We're looking at the paper: "Post-Selection Inference for Partially Identified Parameters".

Warren: Here's the gist of the paper. Interval identification of parameters such as average treatment effects, average partial effects and welfare is particularly common when using observational data and experimental data with imperfect compliance due to the endogeneity of individuals’ treatment uptake.

Ruth: In this paper, we develop new inference tools for interval-identified parameters chosen via these forms of selection.

Warren: Problem Statement The researcher is typically interested in a treatment or policy that is either selected from the estimated set of best-performers or arises from a data-dependent selection rule.

Ruth: It is now well-known that selecting an object of interest from data invalidates standard inference tools.

Title and authors: Ruth: So, we’re looking at this paper called "Post-Selection Inference for Partially Identified Parameters". The authors, Sukjin Han and Adam McCloskey, they are tackling a tricky part of econometrics where you have selection issues.

Warren: Right. It’s about how to make sense of things when you pick a treatment or policy based on the data itself, which messes up the standard ways we calculate things like average treatment effects or welfare.

Iris: Exactly. The core problem here is that standard tools don't work properly once you select something from a set of best performers or follow some kind of data-dependent rule, especially when those parameters are only partially identified to begin with.

Ruth: So what they’re proposing is new inference tools for these kinds of parameters, specifically three types of confidence intervals: conditional CIs, projection CIs, and hybrid CIs.

Warren: Three types sounds like a lot to digest, but the goal is to give us a way to gauge how well those selected treatments or policies are actually expected to perform in the real world.

Iris: That's right. The paper shows how these new tools build on older ideas of conditional and hybrid inference used in other point-identified contexts, like work by Lee et al., Fithian et al., and Tibshirani et al.

Ruth: So, how does this translate to what’s happening in the data? They develop these CIs for parameters chosen by either selecting from an estimated set or following a data-dependent rule.

Warren: The conditional CIs are asymptotically valid when you're interested in a treatment or policy picked from an estimated set of best performers, which is one way this selection happens.

Iris: But the other two types, the projection and hybrid CIs, are specifically designed for those parameters chosen by data-dependent selection rules where the object of interest is uniquely determined by that rule.

Ruth: That’s a key distinction. So we have these three types of confidence intervals, each tailored to a specific way something gets chosen from the data.

Warren: And they analyze how these CIs perform in finite samples, which is important because we don't always have infinite data to rely on asymptotic results alone.

Title and authors: Iris: They do look at the coverage and length properties of these proposed confidence intervals in finite samples, and they found that the hybrid CI tends to perform best overall by limiting the worst-case length performance of the conditional CI relative to its own projection CI.

Ruth: That means even though some CIs might not have uniformly correct coverage, this hybrid one gives us a better worst-case bound for how long the interval is likely to be.

Warren: It sounds like they’ve found a way to get a more robust estimate of the uncertainty when we’re dealing with these tricky selection mechanisms.

Iris: The technical appendix suggests that the conditional CI has correct coverage for all general data-dependent policies and even for a modest sample size of one hundred though the projection CI tends to be conservative with true coverage above its nominal level of ninety-five percent.

Ruth: That's a bit reassuring, but they also point out that the conventional CI can substantially under-cover in some cases.

Warren: So it’s not just about finding a new tool; it’s about understanding exactly where the existing tools fall short when selection rules are involved.

Iris: Exactly. And the paper shows how their general inference framework can incorporate several problems of interest for data-dependent selection, like Manski bounds or bounds on dynamic treatment effects, which is pretty broad.

Ruth: That generality is what makes this work interesting because it lets you apply the framework to a wide class of problems beyond just schooling and training examples.

Warren: It sounds like they’re trying to create a toolkit that can handle these kinds of identification issues across many different economic setups.

Iris: The paper shows that their general framework incorporates bounds for average potential outcomes, bounds derived from linear programming, and even welfare bounds for treatment allocation rules that are partially identified by observational data.

Ruth: So it’s not just a single technique; it’s a structure that can apply to problems involving Manski bounds or Balke and Pearl type parameters.

Warren: And they also show how to incorporate inference on parameters chosen via asymptotically optimal treatment choice rules into this framework. That's pretty comprehensive.

Title and authors: Iris: The main point is the development of these three specific confidence interval types—conditional, projection, and hybrid—to handle these data-dependent selections in partially identified contexts.

Ruth: So to wrap up, the paper provides a new set of tools for inferring parameters when you select them based on data selection rules or from an estimated set.

Warren: The hybrid CI seems like the best practical choice because it controls the worst-case length performance compared to other options.

Iris: It gives us better coverage guarantees in finite samples, even if we acknowledge that some of these methods, like the projection CI, might be a bit conservative with their nominal coverage.

Ruth: And they show how this general framework can handle a huge variety of problems nesting these examples—from dynamic treatment effects to welfare bounds.

Warren: It’s about making sure that when we use these tools to estimate things like average welfare or treatment effects, the uncertainty we report is as accurate as possible given the identification constraints.

Iris: The paper shows how this framework can be applied to many different examples of interest, which gives us confidence in its general applicability beyond just the specific cases they study.

Ruth: It’s a solid development for anyone working with observational data and experimental data where compliance isn't perfect, because those settings are really common.

Warren: It gives us more reliable ways to make decisions about treatments or policies that we actually select from our observed data instead of just relying on theoretical assumptions that might not hold up.

Iris: The main implication is that we now have a structured way to move beyond the limitations of standard tools when selection rules are involved and parameters are only partially identified.

Ruth: It’s a lot of heavy lifting for the econometricians dealing with these identification issues, but it provides a clearer path forward for inference in these complex scenarios.

Warren: Yeah, it sounds like they’ve given us something concrete to work with when we have those kinds of selection rules in our models.

Iris: So that’s the gist of "Post-Selection Inference for Partially Identified Parameters," moving from standard tools to a more robust set of interval identification methods.

The paper's summary: Ruth: So basically, the paper is building new ways to get confidence intervals for these tricky parameters. It’s not just about the basic tools we use when we don't have perfect data compliance anymore.

Warren: Exactly. They introduce three types of confidence intervals: conditional, projection, and hybrid CIs. These are designed specifically for when you select something from an estimated set or follow a data-dependent selection rule.

Iris: The point is that standard tools just break down after you pick something based on the data itself, especially when those parameters aren't fully identified to begin with. This paper fixes that by building on ideas from other inference frameworks.

Ruth: Right. The conditional CIs work for when you’re picking from a set of best performers, and the projection and hybrid CIs are for those more complex rules where the object you care about is totally determined by that selection rule.

Warren: And they show how these new methods perform in finite samples, which is important because we can't always rely on just asymptotic results when we’re actually running a model.

Iris: They found that the hybrid CI ends up being the best overall because it limits the worst-case length of that conditional CI compared to its own projection CI. It offers better worst-case performance in practice.

Ruth: That means even if some of these confidence intervals aren't perfectly correct on coverage, this hybrid one gives us a tighter bound on how long the interval is likely to be. It’s a practical way to manage uncertainty.

Warren: So what this changes for someone running a strategy or analyzing data? It means we have a more robust way to estimate things like average welfare or treatment effects when our selection process isn't perfectly clean.

Iris: It gives us valid inference on those parameters even in these tough identification settings. The technical appendix suggests the conditional CI has correct coverage for most general data-dependent policies and even for a sample size of one hundred people.

Ruth: But the projection CI tends to be very conservative, meaning its actual coverage is often above ninety-five percent, while the conventional CI can substantially under-cover.

Warren: So we have a choice here. You can have a tool that's asymptotically valid but might be too wide or too narrow in small samples, versus something that’s designed to give you the best possible worst-case bound.

Iris: And they also show this general framework can handle many different problems, including things like bounds on dynamic treatment effects and welfare bounds derived from linear programming. It’s pretty generalizable.

Ruth: That generality is key. It suggests we can use this structure for a whole class of problems, not just the examples they use, which opens up a lot of possibilities for applying these inference methods in different economic settings.

Warren: So it sounds like this framework is designed to be quite versatile, allowing us to handle selection rules across many different scenarios. But what are the real-world costs of running this kind of complex inference?

Iris: The main implication is that we now have a structured way to move beyond the limitations of standard tools when selection rules are involved and parameters are only partially identified. We can make more reliable decisions about treatments or policies based on our observed data.

The paper's improvements: Tom: So we’re looking at what these authors suggest for future work on this post-selection inference paper. It seems they have some ideas on how to make their general framework even more powerful for different kinds of problems.

Ruth: Right, they are pushing to integrate things like Manski bounds and linear programming bounds directly into the main inference framework. They want this structure to handle a wider variety of selection rules, not just the ones in those examples.

Warren: That makes sense from a modeling standpoint. If you can plug in tools that give you bounds on dynamic treatment effects or welfare directly, then your choice of interval CIs becomes much more flexible and comprehensive.

Iris: I agree with the idea of making it more general, but they also focus on designing specific probabilistic lower and upper bounds for those identified intervals. They want concrete math for the conditional CIs and projection CIs based on sufficient statistics.

Ruth: So they are moving from just providing three types of intervals to actually giving us formulas that define the exact lower and upper bounds for those types, which is crucial for practical application.

Warren: That’s useful because it gives us a clearer picture of the precision we can expect when we use these tools. It moves beyond just saying "here's a hybrid CI" to showing you exactly how wide that interval should be based on the data structure.

Iris: And they also propose unconditional confidence intervals using projection and hybrid methods to cover the entire identified interval, aiming for uniformly correct asymptotic coverage regardless of how the object is selected from the data. That’s a big step toward making it truly universal.

Ruth: So they are aiming for this kind of comprehensive tool that works well whether you're picking something from an estimated set or following some complex rule, and they want it to work consistently across different types of problems.

Warren: It sounds like the goal is to create a truly universal inference toolkit for these kinds of tricky identification issues in observational data. But we still have to remember the practical side—how much computational power does this require?

Iris: The paper doesn't really focus on computational complexity, but it does emphasize that the framework is designed to be general enough to handle a wide class of problems nesting these examples. It’s about structural capability more than raw speed for now.

Ruth: So the focus is on making it theoretically sound and broadly applicable first, with the idea that this structure can solve many different identification puzzles in econometrics. But what does this mean for us when we look at other related papers, like those on dynamic coordination or model refutation?

Warren: It means if you are dealing with dynamic effects or complex modeling where selection is a factor, you might be able to use this overarching framework instead of trying to force a standard tool that just doesn't work.

Iris: Exactly. It suggests that the challenge isn't inventing one new trick, but building a general inference structure that can adapt its bounds and construction based on the specific selection mechanism present in any given problem.

Conclusion: Ruth: So we’re wrapping up this look at "Post-Selection Inference for Partially Identified Parameters." Essentially, the paper shows how to create three new types of confidence intervals—conditional, projection, and hybrid—to handle parameters that are selected based on data rules or estimates.

Warren: Right. The big takeaway is that even when you can't perfectly identify something because of selection issues or partial identification, you still have a structured way to report uncertainty.

Iris: I think the crucial part is the hybrid CI; it’s what they show performs best in finite samples by limiting the worst-case length compared to other methods. It gives us a better idea of how wide our interval will be in reality.

Ruth: It’s about moving past the standard tools that just break down once you introduce data-dependent selection rules. This work provides a new way for people to make sense of average treatment effects or welfare when compliance isn't perfect.

Warren: And the general framework they build, incorporating things like Manski bounds and dynamic treatment effect bounds, seems to be pretty versatile for tackling many different identification problems in one place.

Iris: It’s a broad toolkit that can apply to various economic scenarios, which is what makes it useful beyond just these specific schooling examples. We're not just fixing one problem; we're building a general inference structure.

Ruth: So the paper gives us concrete tools—conditional, projection, and hybrid CIs—that help gauge how well selected treatments or policies are expected to perform in practice.

Warren: It’s about giving us more reliable estimates of uncertainty when the selection process isn't perfectly clean. The hybrid CI seems like a very solid practical choice for managing that uncertainty across different contexts.

Iris: I just want to emphasize that while the conditional CI has good coverage for most general data-dependent policies, we still have to be careful with the projection CI because its coverage can stay above ninety-five percent even when it's supposed to be nominal.

Ruth: That’s a fair caveat. And they point out that this method doesn't solve every single identification headache; it stops where the underlying assumptions about what variation is doing the work are too restrictive for that specific problem.

Warren: So, while this paper offers a strong new structure for inference after selection, there are still specific limitations on what kind of parameters or rules it can perfectly handle. But it opens up a lot of doors for future research in this area.

Iris: It certainly sets the stage well for looking at how these ideas might combine with other areas we’ve been talking about, like uniform inference for clustered data or even model refutation after data rejects our initial assumptions.

Ruth: Exactly. This paper is a solid step forward in building that more robust toolbox for dealing with selection issues in complex data settings.

More episodes

← Home