Prediction Sets and Conformal Inference with Interval Outcomes
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Economics. The day in economics: theory, mechanisms and the evidence beneath them.
Ruth: I'm Ruth.
Warren: And I'm Warren, here with Iris.
Ruth: We're looking at the paper: "Prediction Sets and Conformal Inference with Interval Outcomes".
Warren: Here's the gist of the paper. Given data on a random variable Y, a prediction set with miscoverage level alpha ∈ (0, 1) is a set that contains a new draw of Y with probability 1-alpha.
Ruth: This paper develops methods for estimating such prediction sets conditional on observed covariates when Y is censored or interval-valued.
Warren: Problem Definition and Oracle Sets The paper introduces the concept of a prediction set C, which is a measurable subset of Y such that P Y (Y ∈ C) ≥ 1-alpha.
Ruth: The oracle prediction set is defined as the set with the minimal volume that satisfies this coverage property.
Paper summary: Ruth: So we're looking at this paper, "Prediction Sets and Conformal Inference with Interval Outcomes." It tackles the idea of finding a set that covers a new observation with a certain probability, say one minus alpha. The main thing they claim is that they can estimate these sets even when the outcome variable Y is either censored or interval-valued.
Warren: So, if we've got data like income brackets instead of exact numbers, the traditional methods get really messy because you don't have a single point to anchor on. This paper focuses on characterizing this optimal prediction set and then proposing consistent estimators for both those sets and more general sets made of multiple disjoint intervals.
Iris: I wonder what partial identification means here, Ruth? Because when we have interval censoring, the whole structure of what we know about the underlying distribution changes because we only observe some part of the true value.
Ruth: Exactly. They characterize this oracle prediction set under partial identification that arises from interval censoring by defining a marginally valid condition where you get at least one minus alpha probability of seeing a new draw in the next sample, and they define the oracle set as the minimal volume one satisfying those validity conditions.
Warren: And then they move into proposing an actual estimation procedure for this oracle prediction set under interval censoring, which relates it to the conditional distribution of those observed upper and lower brackets. They also handle cases where the oracle prediction set might be a union of several disjoint intervals with the required coverage.
Iris: That sounds like they are building something that goes beyond just a single interval forecast, which is interesting because it lets you capture more complex patterns in censored data, like those seen in real surveys.
Ruth: Right. And to make this useful for finite samples, they use conformal inference to construct a prediction set C tilde n using exchangeable data from the sample. This set has finite-sample validity while still having the asymptotic property of being consistent for the oracle prediction set under partial identification as the sample size gets bigger.
Warren: So they propose a split conformal inference procedure specifically to build this finite-sample valid conformal prediction set, which we can call C tilde I. This means it’s something you can actually use right now on a smaller dataset without it being completely unreliable.
Iris: But what are the practical implications for someone trying to use this in a real-world application? Does this just give us a better number, or does it fundamentally change how we think about prediction when the data is incomplete?
Ruth: It changes how we think about prediction sets because they aren't just point estimates like the mean or quantiles. They offer a complementary view of the distribution, and for people who only look at point estimators like that, this approach gives them a richer picture of uncertainty.
Paper summary: Warren: And from a quantitative standpoint, they show that their proposed estimators have smaller volume compared to those prediction sets constructed using quantile regression methods. That suggests that if you use this method with interval-valued data, you might get a more precise forecast region for the same level of coverage.
Iris: That comparison against quantile regression is important because quantile methods are pretty standard, but this work shows that accounting for the interval structure gives us a tighter volume estimate. What about limitations? What’s the catch here?
Ruth: The authors do note that they focus specifically on partial identification induced by interval censoring, and future research could look at prediction sets under partial identification beyond just interval censoring. They also point out that designing conformity scores in the context of partial identification is an important topic for further investigation.
Warren: So, while they get us consistent estimators for the oracle prediction set and a conformal prediction set with finite sample miscoverage, they aren't claiming this works perfectly everywhere or for every kind of data structure yet. They're showing how to get close, but there are still areas needing more exploration.
Iris: It seems like the real power here is combining the theory of characterization with the practical tool of conformal inference to get something that behaves well both theoretically and empirically when applied to real datasets like job adverts or surveys.
Ruth: Right. The empirical validation on datasets like the UK job advert data and US Current Population Survey data showed that these proposed prediction sets are robust in their finite sample coverage properties and have smaller volume than alternative methods. They even found substantial cross-city heterogeneity in predicted salary distributions for the UK job market.
Warren: And for the US CPS data, they noted that incorporating interval-valued income data shifts the right tail of the distribution significantly compared to using imputed income when constructing prediction intervals there. That suggests that ignoring the interval structure can lead to really different and potentially misleading predictions about high earners in those samples.
Iris: So, it seems like for someone only listening to this, this paper is showing that when you have messy data like brackets or censoring, you need a specific theoretical framework—like partial identification—and then a specific tool, conformal inference—to get prediction sets that are both valid in the short term and consistent as the data grows.
Ruth: That's the gist of it. It’s about moving beyond simple point estimates to get a more complete picture of uncertainty when you're dealing with censored or interval-valued outcomes. This paper, "Prediction Sets and Conformal Inference with Interval Outcomes," provides the tools to do that consistently under certain conditions.
Warren: And looking ahead, the authors suggest we should look at prediction sets under partial identification in ways other than just interval censoring. They also highlight that figuring out how to design those conformity scores properly when you have partial identification is a key area for future research in this field.
Conclusion: Ruth: So, we've seen how they build these prediction sets using interval data and conformal inference to get coverage probabilities that work even when the outcome isn't a single number but a range of possibilities.
Warren: It’s about getting those sets, the oracle ones in particular, which are the tightest possible regions that still guarantee you see the new draw with one minus alpha probability.
Iris: And they show how to estimate them using observed data, specifically by looking at the upper and lower brackets you actually see from your samples.
Ruth: The authors are focused on making sure these estimators aren't just theoretical constructs, but something that actually works in practice for real-world economic data like salary surveys.
Warren: The main result is that their proposed methods can have a smaller volume than prediction sets built using quantile regression, which suggests they get a tighter fit around the true distribution.
Iris: That tightness comes from how they handle the partial identification issue, which is what makes this estimation possible when you only see parts of the true value.
Ruth: So it means that for economists and data scientists working with censored or interval-valued outcomes, there's a way to get more informative prediction regions than just using standard point estimates.
Warren: It’s a practical tool that helps build those regions with finite samples while still aiming for that perfect fit as the sample grows larger.
Iris: The next thing we need to look at is how these ideas translate when we move beyond interval censoring and into other types of partial identification problems in modeling.
University College London · Harvard University
econ.EM, stat.ME
Submitted: 2025-01-17
Updated: 2026-11-11
Code: https://github.com/lwg342/prediction
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 85/100
The gist: Given data on a random variable Y, a prediction set with miscoverage level alpha ∈ (0, 1) is a set that contains a new draw of Y with probability 1-alpha.
Key concepts
- Prediction Set C
- A measurable subset of the outcome variable Y that has a probability of containing a new draw at least equal to 1-alpha. The oracle prediction set is the smallest possible such set with minimal volume that meets this coverage requirement.
- Partial Identification
- A condition arising from interval censoring where the exact value of Y is unknown but constrained within an interval (YL, YU). The paper focuses on characterizing optimal prediction sets under this constraint, particularly when using interval censoring.
- Conformal Inference Set ($ ilde{C}_n^I$)
- A prediction set constructed using conformal inference that maintains finite-sample validity. It uses a set-based score function to penalize both over- and under-coverage, ensuring the resulting set is valid for a given miscoverage level alpha.
- Oracle Prediction Set
- The theoretically optimal prediction set defined as the subset of Y with the smallest possible volume that guarantees at least 1-alpha coverage. The goal of the estimation procedures is to construct an estimator that converges to this minimal volume set.
Terminology
Summary
Given data on a random variable Y, a prediction set with miscoverage level alpha ∈ (0, 1) is a set that contains a new draw of Y with probability 1-alpha. p. 0 Prediction Sets and Conformal Inference with Interval
Outcomes ∗
Weiguang Liu
UCL
Aureo de Paula ´
UCL, IFS and CeMMAP
Elie Tamer
Harvard University
February 24, 2026
Abstract
Given data on a random variable 𝑌, a prediction set with miscoverage level 𝛼 ∈ (0, 1)
is a set that contains a new draw of 𝑌 with probability 1−𝛼. Among all prediction sets satisfying this coverage property, the oracle prediction set is the one with minimal volume. The oracle prediction set offers a complementary view of the distribution of 𝑌, beyond
point estimators such as the mean and quantiles, and has attracted considerable interest
recently. This paper develops methods for estimating such prediction sets conditional
on observed covariates when 𝑌 is censored or interval-valued.Where the paper says it
Where the paper says it
Problem Definition and Oracle Sets
The paper introduces the concept of a prediction set C, which is a measurable subset of Y such that P Y (Y ∈ C) ≥ 1-alpha. The oracle prediction set is defined as the set with the minimal volume that satisfies this coverage property. p. 7 For example, the prediction set 𝐶 such that 𝐶(𝑥) = (−∞, ∞), for all
𝑥 ∈ X , is technically valid at any miscoverage level, but it is also uninformative. Therefore,
our target parameter of interest is an oracle prediction set, denoted by 𝐶
∗
P𝐼
, which is defined
as the minimal-volume prediction set that satisfies the validity conditions in Definition 2. We use these data to construct
prediction sets 𝐶𝑛 (𝑥) ≡ 𝐶(𝑥;𝑋1, 𝑌1, . , 𝑋𝑛, 𝑌𝑛) ⊂ Y for a given miscoverage level 𝛼 at 𝑥 ∈ X .Where the paper says it
Where the paper says it
Characterization under Partial Identification
The paper characterizes the oracle prediction set under partial identification induced by interval censoring. A prediction set C is defined as marginally valid if inf P ∈ P I P Yn+1 ∈ C(Xn+1) ≥ 1-alpha. p. 7 1 (Validity under Partial Identification). We say a prediction set 𝐶 is marginally
valid under partial identification with miscoverage level 𝛼, if,
inf
𝑃∈P𝐼
𝑃
𝑌𝑛+1 ∈ 𝐶(𝑋𝑛+1)
≥ 1 − 𝛼, (2)
and 𝐶 is conditionally valid under partial identification with miscoverage level 𝛼 if for almost
everywhere 𝑥 ∈ X ,
inf
𝑃∈P𝐼
𝑃
𝑌𝑛+1 ∈ 𝐶(𝑥) | 𝑋𝑛+1 = 𝑥
≥ 1 − 𝛼. For example, the prediction set 𝐶 such that 𝐶(𝑥) = (−∞, ∞), for all
𝑥 ∈ X , is technically valid at any miscoverage level, but it is also uninformative. Therefore,
our target parameter of interest is an oracle prediction set, denoted by 𝐶
∗
P𝐼
, which is defined
as the minimal-volume prediction set that satisfies the validity conditions in Definition 2.Where the paper says it
Where the paper says it
Estimation Procedures and Consistency
The paper proposes a feasible estimation procedure for the oracle prediction set under interval censoring, which relates it to the conditional distribution of the observed upper and lower brackets. The oracle prediction set may be the union of disjoint intervals, which is addressed by providing an approach that allows for a union of disjoint intervals with the prescribed coverage. p. 3 The
oracle prediction set may be the union of disjoint intervals, for instance, when the prediction
density is multimodal. We provide an approach that allows for the prediction set estimator
to be a union of disjoint intervals with the prescribed coverage. Section 3 provides a
study of the consistency of such sets using kernel estimators of the conditional distribution. Section 4 provides results on finite sample coverage using a conformal inference procedure
under interval censoring.Where the paper says it
Where the paper says it
Finite-Sample Validity via Conformal Inference
The paper uses recent developments on conformal inference to propose a conformal prediction set C tilde n using a sample of exchangeable data, which has finite-sample validity while also enjoying the asymptotic property of being consistent for the oracle prediction set under partial identification. A split conformal inference procedure is described to construct a finite-sample valid conformal prediction set C tilde I. p. 0 In addition, we apply conformal inference to
construct finite-sample valid prediction sets for interval outcomes that remain consistent
as the sample size grows, using a conformity score tailored to interval data. The proposed
procedure accounts for irreducible prediction uncertainty due to the stochastic nature of
outcomes, modelling uncertainty arising from partial identification, and sampling uncertainty that vanishes as sample size increases.Where the paper says it
Empirical Validation
The methods are applied to two real-world datasets: the UK job advert dataset and the US Current Population Survey (CPS) data. Monte Carlo simulations demonstrate that the proposed prediction set is robust in its finite sample coverage properties and has a smaller volume compared to alternative prediction sets constructed with quantile regression methods. p. 3 We illustrate our construction using a set of Monte Carlo experiments, where we consider
both fixed and random censoring. Our proposed prediction set is shown to have robust finite sample coverage properties and to have a smaller volume compared with the alternative
prediction sets constructed with quantile regression methods. We conduct Monte Carlo simulations and
two empirical applications using UK job postings data and the US Current Population
Survey. The results demonstrate the robustness and efficiency of the proposed methods. As a competing alternative prediction set estimator, we show in the
appendix that a conformal quantile regression method provides a valid prediction interval for
interval-valued outcomes, which can be easier to implement in practice, but in general will not
be efficient in terms of the volume of the prediction set. In addition, quantile regression-based
methods are not suitable when the prediction set is not a single interval. For example, respondents are often asked whether their money in savings accounts, in investments, or their income falls into a sequence of brackets such as [$10𝐾, $20𝐾],
[$20𝐾, $50𝐾], etc, while also being given the option to provide a point answer to such questions (see Heeringa and Suzman (1995) and Moore and Loomis (2001)). Interval-valued data
also accommodate common patterns such as right censoring, where an outcome variable is
greater than a specified value (i.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Conclusion and Future Directions
The paper concludes by providing consistent estimators for the oracle prediction set and a conformal prediction set that maintains a finite sample miscoverage property. The authors suggest that future research could explore prediction sets under partial identification beyond interval censoring. p. 3 Our method addresses these limitations and offers a complementary
perspective on the distribution of outcomes beyond existing quantile-based methods. We first define the oracle prediction sets under partial identification. We use recent developments on conformal inference to propose a conformal prediction set
𝐶˜
𝑛 using a sample of exchangeable data, which has finite-sample validity while also enjoying
the asymptotic property of being consistent for the oracle prediction set under partial identification. Our conformal procedure is designed for interval-valued outcomes, which requires the
design of a particular conformity score function to handle interval data. We conduct Monte Carlo simulations and
two empirical applications using UK job postings data and the US Current Population
Survey. The results demonstrate the robustness and efficiency of the proposed methods. Here, 𝜗1−𝛼 is the (1−𝛼) (1+1/𝑛2)-th empirical
quantile of the set of conformity scores
max{𝜏ˆ0 (𝑋𝑖) − 𝑌𝑖
, 𝑌𝑖 − 𝜏ˆ1 (𝑋𝑖)} : 𝑖 ∈ I2
estimated in a
calibration sub-sample I2 with sample size 𝑛2. This corrects for under- or over-coverage and
allows for the construction of prediction intervals satisfying desired coverage levels in finite
samples using quantile estimates. We characterise the oracle
prediction set under partial identification induced by interval censoring and propose consistent estimators for both oracle prediction intervals and more general oracle prediction
sets consisting of multiple disjoint intervals. In addition, we apply conformal inference to
construct finite-sample valid prediction sets for interval outcomes that remain consistent
as the sample size grows, using a conformity score tailored to interval data. There are several related areas connected to our theoretical development. The classical
way to construct estimators for the oracle prediction set relies on the level set estimation
(see, for example, Wilks (1941), Polonik (1995) and Samworth and Wand (2010)). We illustrate our construction using a set of Monte Carlo experiments, where we consider
both fixed and random censoring. Our proposed prediction set is shown to have robust finite sample coverage properties and to have a smaller volume compared with the alternative
prediction sets constructed with quantile regression methods. We conduct Monte Carlo simulations and
two empirical applications using UK job postings data and the US Current Population
Survey. The results demonstrate the robustness and efficiency of the proposed methods. For each 𝑖 ∈ I, the predictor 𝑋𝑖
is a 𝑑-dimensional random vector with support
X ⊂ ℝ
𝑑
. The latent outcome 𝑌𝑖 and the observed lower and upper bounds 𝑌
𝐿
𝑖
and 𝑌
𝑈
𝑖
are
one-dimensional. For each 𝑖 ∈ I, the predictor 𝑋𝑖
is a 𝑑-dimensional random vector with support
X ⊂ ℝ
𝑑
. The latent outcome 𝑌𝑖 and the observed lower and upper bounds 𝑌
𝐿
𝑖
and 𝑌
𝑈
𝑖
are
one-dimensional. Here, 𝜗1−𝛼 is the (1−𝛼) (1+1/𝑛2)-th empirical
quantile of the set of conformity scores
max{𝜏ˆ0 (𝑋𝑖) − 𝑌𝑖
, 𝑌𝑖 − 𝜏ˆ1 (𝑋𝑖)} : 𝑖 ∈ I2
estimated in a
calibration sub-sample I2 with sample size 𝑛2. This corrects for under- or over-coverage and
allows for the construction of prediction intervals satisfying desired coverage levels in finite
samples using quantile estimates. This paper develops methods for estimating such prediction sets conditional
on observed covariates when 𝑌 is censored or interval-valued. We characterise the oracle
prediction set under partial identification induced by interval censoring and propose consistent estimators for both oracle prediction intervals and more general oracle prediction
sets consisting of multiple disjoint intervals. We conduct Monte Carlo simulations and
two empirical applications using UK job postings data and the US Current Population
Survey. The results demonstrate the robustness and efficiency of the proposed methods. The oracle prediction set offers a complementary view of the distribution of 𝑌, beyond
point estimators such as the mean and quantiles, and has attracted considerable interest
recently. This paper develops methods for estimating such prediction sets conditional
on observed covariates when 𝑌 is censored or interval-valued.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Pruning redundant reasoning tokens
The paper is organized into sections detailing the setup, consistency proofs, and empirical applications of prediction sets under censoring. The structure begins with defining optimal sets and approaches to obtaining estimators with censoring. p. 5 Outline
This paper is organised as follows. In Section 2, we provide the setup and definitions of optimal
sets and approaches to obtaining estimators of such sets with censoring.Where the paper says it
How the training signal is built
The setup involves a random sample D bar = (X i, Y i, YL i, YU i) drawn independently from a joint distribution P. The observed subvector Z i = (X i, YL i, YU i) is used for estimation. p. 6 2 Setup and definitions
Suppose there is a random sample D¯ = {𝑍¯
𝑖 = (𝑋𝑖
, 𝑌𝑖
, 𝑌𝐿
𝑖
, 𝑌𝑈
𝑖
) ∈ ℝ
𝑑+3
: 𝑖 ∈ I} with sample
size |I| = 𝑛 drawn independently from a joint distribution 𝑃. We only observe the subvector 𝑍𝑖 = (𝑋𝑖
, 𝑌𝐿
𝑖
, 𝑌𝑈
𝑖
), and 𝑌𝑖
is not directly observed but assumed to satisfy the condition
𝑌
𝐿
𝑖 ≤ 𝑌𝑖 ≤ 𝑌
𝑈
𝑖
.Where the paper says it
Where the gains come from
The gains come from characterizing the oracle prediction set under partial identification and proposing a feasible estimation strategy based on observed data. The conformal inference procedure achieves finite-sample validity using a set-based score function, penalizing both over- and under-coverage. p. 3 We use recent developments on conformal inference to propose a conformal prediction set
𝐶˜
𝑛 using a sample of exchangeable data, which has finite-sample validity while also enjoying
the asymptotic property of being consistent for the oracle prediction set under partial identification. Our conformal procedure is designed for interval-valued outcomes, which requires the
design of a particular conformity score function to handle interval data. We illustrate our construction using a set of Monte Carlo experiments, where we consider
both fixed and random censoring. Our proposed prediction set is shown to have robust finite sample coverage properties and to have a smaller volume compared with the alternative
prediction sets constructed with quantile regression methods.Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Improved prediction set estimation for interval outcomes: The system can construct a finite-sample valid prediction set using split conformal inference, which is distribution-free, maintaining coverage probability of 1 minus alpha for all distributions P.
-
Consistent oracle prediction set estimation under partial identification: The system can estimate the oracle prediction set C star by solving a feasible optimization problem that replaces the constraint based on the conditional distribution of Y with one based on the joint distribution of (Y L, Y U).
-
Robustness to censoring mechanisms: The system can construct a prediction set estimator that does not require a specification of the censoring mechanism, making it robust to both random and fixed censoring schemes.
-
Handling multimodal distributions: The system can estimate oracle prediction sets that consist of multiple disjoint intervals, which is necessary when the conditional distribution of Y given X is multimodal.
-
Spatial variation analysis in economic data: When applied to datasets like UK job postings, the system can estimate local conformal prediction intervals that capture spatial variation in predicted salary distributions across geographic locations without imposing parametric assumptions or relying on imputation.
Abstract
Given data on a random variable Y, a prediction set with miscoverage level α in (0,1) is a set that contains a new draw of Y with probability at least 1-α. Among all prediction sets satisfying this coverage property, the oracle prediction set is the one with minimal volume. The oracle prediction set offers a complementary view of the distribution of Y, beyond point estimators such as the mean and quantiles, and has attracted considerable interest recently. This paper develops methods for estimating such prediction sets conditional on observed covariates when Y censored or interval-valued. We characterise the oracle prediction set under partial identification induced by interval censoring and propose consistent estimators for both oracle prediction intervals and more general oracle prediction sets consisting of multiple disjoint intervals. In addition, we apply conformal inference to construct finite-sample valid prediction sets for interval outcomes that remain consistent as the sample size grows, using a conformity score tailored to interval data. The proposed procedure accounts for irreducible prediction uncertainty due to the stochastic nature of outcomes, modelling uncertainty arising from partial identification, and sampling uncertainty that vanishes as sample size increases. We conduct Monte Carlo simulations and two empirical applications using UK job postings data and the US Current Population Survey. The results demonstrate the robustness and efficiency of the proposed methods.
Related papers
- Stationarity with Occasionally Binding Constraints
- Backtesting High-Dimensional Expected Shortfall
- Higher-Order Neyman Orthogonality in Moment-Condition Models
- Online Bayesian Inference for a Dirichlet Process Mixture of Binary Logit Models
- Debiased Conditional Independence Tests for Causal Effects
- Sharp Testable Implications of Encouragement Designs