Prediction Sets and Conformal Inference with Interval Outcomes
summary
The gist
Given data on a random variable Y, a prediction set with miscoverage level alpha ∈ (0, 1) is a set that contains a new draw of Y with probability 1-alpha.
In short
The paper develops methods to estimate prediction sets for censored or interval-valued outcomes when covariates are known. It introduces an oracle prediction set and proposes estimation procedures using conformal inference, which provides finite-sample valid sets that are consistent with the ideal oracle set under partial identification. This approach is validated on real economic data, showing robust performance and capturing spatial variation better than simpler methods.
Key concepts
- Prediction Set C
- A measurable subset of the outcome variable Y that has a probability of containing a new draw at least equal to 1-alpha. The oracle prediction set is the smallest possible such set with minimal volume that meets this coverage requirement.
- Partial Identification
- A condition arising from interval censoring where the exact value of Y is unknown but constrained within an interval (YL, YU). The paper focuses on characterizing optimal prediction sets under this constraint, particularly when using interval censoring.
- Conformal Inference Set ($ ilde{C}_n^I$)
- A prediction set constructed using conformal inference that maintains finite-sample validity. It uses a set-based score function to penalize both over- and under-coverage, ensuring the resulting set is valid for a given miscoverage level alpha.
- Oracle Prediction Set
- The theoretically optimal prediction set defined as the subset of Y with the smallest possible volume that guarantees at least 1-alpha coverage. The goal of the estimation procedures is to construct an estimator that converges to this minimal volume set.
Terminology used across episodes
This episode discusses
The paper
Prediction Sets and Conformal Inference with Interval Outcomes · Read on arXiv
University College London · Harvard University
Given data on a random variable Y, a prediction set with miscoverage level α in (0,1) is a set that contains a new draw of Y with probability at least 1-α. Among all prediction sets satisfying this coverage property, the oracle prediction set is the one with minimal volume. The oracle prediction set offers a complementary view of the distribution of Y, beyond point estimators such as the mean and quantiles, and has attracted considerable interest recently. This paper develops methods for estimating such prediction sets conditional on observed covariates when Y censored or interval-valued. We characterise the oracle prediction set under partial identification induced by interval censoring and propose consistent estimators for both oracle prediction intervals and more general oracle prediction sets consisting of multiple disjoint intervals. In addition, we apply conformal inference to construct finite-sample valid prediction sets for interval outcomes that remain consistent as the sample size grows, using a conformity score tailored to interval data. The proposed procedure accounts for irreducible prediction uncertainty due to the stochastic nature of outcomes, modelling uncertainty arising from partial identification, and sampling uncertainty that vanishes as sample size increases. We conduct Monte Carlo simulations and two empirical applications using UK job postings data and the US Current Population Survey. The results demonstrate the robustness and efficiency of the proposed methods.
Transcript
Introduction to the show: ident: Economics. The day in economics: theory, mechanisms and the evidence beneath them.
Ruth: I'm Ruth.
Warren: And I'm Warren, here with Iris.
Ruth: We're looking at the paper: "Prediction Sets and Conformal Inference with Interval Outcomes".
Warren: Here's the gist of the paper. Given data on a random variable Y, a prediction set with miscoverage level alpha ∈ (0, 1) is a set that contains a new draw of Y with probability 1-alpha.
Ruth: This paper develops methods for estimating such prediction sets conditional on observed covariates when Y is censored or interval-valued.
Warren: Problem Definition and Oracle Sets The paper introduces the concept of a prediction set C, which is a measurable subset of Y such that P Y (Y ∈ C) ≥ 1-alpha.
Ruth: The oracle prediction set is defined as the set with the minimal volume that satisfies this coverage property.
Paper summary: Ruth: So we're looking at this paper, "Prediction Sets and Conformal Inference with Interval Outcomes." It tackles the idea of finding a set that covers a new observation with a certain probability, say one minus alpha. The main thing they claim is that they can estimate these sets even when the outcome variable Y is either censored or interval-valued.
Warren: So, if we've got data like income brackets instead of exact numbers, the traditional methods get really messy because you don't have a single point to anchor on. This paper focuses on characterizing this optimal prediction set and then proposing consistent estimators for both those sets and more general sets made of multiple disjoint intervals.
Iris: I wonder what partial identification means here, Ruth? Because when we have interval censoring, the whole structure of what we know about the underlying distribution changes because we only observe some part of the true value.
Ruth: Exactly. They characterize this oracle prediction set under partial identification that arises from interval censoring by defining a marginally valid condition where you get at least one minus alpha probability of seeing a new draw in the next sample, and they define the oracle set as the minimal volume one satisfying those validity conditions.
Warren: And then they move into proposing an actual estimation procedure for this oracle prediction set under interval censoring, which relates it to the conditional distribution of those observed upper and lower brackets. They also handle cases where the oracle prediction set might be a union of several disjoint intervals with the required coverage.
Iris: That sounds like they are building something that goes beyond just a single interval forecast, which is interesting because it lets you capture more complex patterns in censored data, like those seen in real surveys.
Ruth: Right. And to make this useful for finite samples, they use conformal inference to construct a prediction set C tilde n using exchangeable data from the sample. This set has finite-sample validity while still having the asymptotic property of being consistent for the oracle prediction set under partial identification as the sample size gets bigger.
Warren: So they propose a split conformal inference procedure specifically to build this finite-sample valid conformal prediction set, which we can call C tilde I. This means it’s something you can actually use right now on a smaller dataset without it being completely unreliable.
Iris: But what are the practical implications for someone trying to use this in a real-world application? Does this just give us a better number, or does it fundamentally change how we think about prediction when the data is incomplete?
Ruth: It changes how we think about prediction sets because they aren't just point estimates like the mean or quantiles. They offer a complementary view of the distribution, and for people who only look at point estimators like that, this approach gives them a richer picture of uncertainty.
Paper summary: Warren: And from a quantitative standpoint, they show that their proposed estimators have smaller volume compared to those prediction sets constructed using quantile regression methods. That suggests that if you use this method with interval-valued data, you might get a more precise forecast region for the same level of coverage.
Iris: That comparison against quantile regression is important because quantile methods are pretty standard, but this work shows that accounting for the interval structure gives us a tighter volume estimate. What about limitations? What’s the catch here?
Ruth: The authors do note that they focus specifically on partial identification induced by interval censoring, and future research could look at prediction sets under partial identification beyond just interval censoring. They also point out that designing conformity scores in the context of partial identification is an important topic for further investigation.
Warren: So, while they get us consistent estimators for the oracle prediction set and a conformal prediction set with finite sample miscoverage, they aren't claiming this works perfectly everywhere or for every kind of data structure yet. They're showing how to get close, but there are still areas needing more exploration.
Iris: It seems like the real power here is combining the theory of characterization with the practical tool of conformal inference to get something that behaves well both theoretically and empirically when applied to real datasets like job adverts or surveys.
Ruth: Right. The empirical validation on datasets like the UK job advert data and US Current Population Survey data showed that these proposed prediction sets are robust in their finite sample coverage properties and have smaller volume than alternative methods. They even found substantial cross-city heterogeneity in predicted salary distributions for the UK job market.
Warren: And for the US CPS data, they noted that incorporating interval-valued income data shifts the right tail of the distribution significantly compared to using imputed income when constructing prediction intervals there. That suggests that ignoring the interval structure can lead to really different and potentially misleading predictions about high earners in those samples.
Iris: So, it seems like for someone only listening to this, this paper is showing that when you have messy data like brackets or censoring, you need a specific theoretical framework—like partial identification—and then a specific tool, conformal inference—to get prediction sets that are both valid in the short term and consistent as the data grows.
Ruth: That's the gist of it. It’s about moving beyond simple point estimates to get a more complete picture of uncertainty when you're dealing with censored or interval-valued outcomes. This paper, "Prediction Sets and Conformal Inference with Interval Outcomes," provides the tools to do that consistently under certain conditions.
Warren: And looking ahead, the authors suggest we should look at prediction sets under partial identification in ways other than just interval censoring. They also highlight that figuring out how to design those conformity scores properly when you have partial identification is a key area for future research in this field.
Conclusion: Ruth: So, we've seen how they build these prediction sets using interval data and conformal inference to get coverage probabilities that work even when the outcome isn't a single number but a range of possibilities.
Warren: It’s about getting those sets, the oracle ones in particular, which are the tightest possible regions that still guarantee you see the new draw with one minus alpha probability.
Iris: And they show how to estimate them using observed data, specifically by looking at the upper and lower brackets you actually see from your samples.
Ruth: The authors are focused on making sure these estimators aren't just theoretical constructs, but something that actually works in practice for real-world economic data like salary surveys.
Warren: The main result is that their proposed methods can have a smaller volume than prediction sets built using quantile regression, which suggests they get a tighter fit around the true distribution.
Iris: That tightness comes from how they handle the partial identification issue, which is what makes this estimation possible when you only see parts of the true value.
Ruth: So it means that for economists and data scientists working with censored or interval-valued outcomes, there's a way to get more informative prediction regions than just using standard point estimates.
Warren: It’s a practical tool that helps build those regions with finite samples while still aiming for that perfect fit as the sample grows larger.
Iris: The next thing we need to look at is how these ideas translate when we move beyond interval censoring and into other types of partial identification problems in modeling.
More episodes
- Daily Summary for 2026-11-19
- 2608.21274-Recommendation Quality and the Concentration of Consumption: Experimental Evidence from Netflix
- 2408.02757-A nonparametric test for diurnal variation in spot correlation processes
- 2306.16393-High-Dimensional Canonical Correlation Analysis
- 2605.17299-Geometric Brownian motion with intermittent entries and exits
- 2607.04753-Fooling Yourself: how narratives shape beliefs
- 2408.14671-Inference on Treatment Effects with Repeatedly Mismeasured High-Dimensional Controls
- 2201.06898-Difference-in-Differences Estimators for Treatments Continuously Distributed at Every Period
- 2501.15307-Influence Functions: A Unified Differentiation-Based Framework
- The day in conversation for 2026-11-19