Changes-in-Changes for Ordered Choice Models with Underreporting
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Economics. The day in economics: theory, mechanisms and the evidence beneath them.
Ruth: We're looking at the paper: "Changes-in-Changes for Ordered Choice Models with Underreporting".
Warren: Here's the gist of the paper. The paper develops a Difference-in-Differences (CiC) framework specifically designed for discrete, ordered outcomes that are subject to underreporting, which commonly occurs in self-reported surveys concerning socially undesirable or stigmatized behaviors.
Ruth: The framework derives nonparametric bounds for both the counterfactual outcome distribution after treatment and the factual outcome distribution, as well as associated quantile treatment effects when outcomes are underreported.
Warren: These bounds are shown to be sharp uniformly across outcome levels under additional support conditions, and suitable estimation and bootstrap inference procedures are proposed.
Ruth: Furthermore, an extension considers a semiparametric underreporting model that allows for the point identification and estimation of distributional treatment effects.
Title and authors: Ruth: So we're looking at this new paper today called "Changes-in-Changes for Ordered Choice Models with Underreporting". It tackles how to do a difference in differences when you're dealing with outcomes that people might not be reporting accurately.
Warren: Right. It deals with those discrete, ordered outcomes, like survey responses where someone might hide what they actually did because it’s something socially undesirable.
Iris: So the authors are developing a whole framework called Changes-in-Changes for this problem, and they aim to get beyond just looking at the average effect.
Ruth: Exactly. They derive nonparametric bounds for both the counterfactual outcome distribution and the factual outcome distribution, as well as bounds for those quantile treatment effects when things are underreported.
Warren: That sounds like a big deal because it gives you something concrete about what the true effect might be, not just one number.
Iris: The paper claims these bounds are sharp uniformly across all outcome levels if you have some additional support conditions in place, and they propose specific ways to estimate these bounds and do bootstrap inference.
Ruth: That’s important because it means we can get really tight confidence sets around our predictions for how much things change.
Warren: And they also extend this by looking at a semiparametric underreporting model, which lets them point identify and estimate distributional treatment effects directly.
Iris: That extension is interesting because it allows them to use observed covariates in the underreporting model itself to see how reporting behavior might be affected by the treatment status.
Ruth: It means they can recover those Distributional Treatment Effects on the Treated, or DTTs, even with that extra layer of complexity introduced by covariates.
Warren: So, what we're looking at is moving from just mean effects to understanding how different parts of the outcome distribution are affected by a policy change.
Iris: And this paper does that by focusing on Quantile Treatment Effects on the Treated, which they define as the difference between the first quantile of the outcome under treatment and the first quantile under no treatment.
Ruth: It formalizes underreporting by defining reported consumption, which is less than or equal to true consumption with probability one, and it sets an upper bound on how likely someone is to report a value that's too low.
Warren: So the core idea here is that the reported data isn't just noisy; it follows a specific pattern of misreporting, and they use that structure to find more robust results.
Iris: They derive bounds for those Quantile Treatment Effects using these distribution bounds, showing that the quantile treatment effects fall between some lower bound and an upper bound depending on the underreporting probability.
Ruth: That’s a major result because it links the uncertainty about misreporting directly to the uncertainty about how much a specific quantile moves.
Warren: And they show these bounds are sharp, meaning you can't get much tighter than what they provide, which is good for setting expectations on those effects.
Iris: They also developed estimation and inference procedures for all those bound functions and the resulting quantile treatment effects, using repeated cross-sectional sampling and bootstrap methods.
Ruth: So we’re not just getting a theoretical range; we’re getting practical tools to actually estimate those bounds with confidence in them across different outcome levels.
Warren: And they apply this whole structure empirically to look at the impact of recreational marijuana legalization on the consumption behavior of eighth-grade students in several U.S. states, which is a concrete application.
Iris: In their empirical analysis, they define that reported consumption and investigate how legalization affects short-term consumption frequency while accounting for potential misreporting with an alpha value.
Ruth: Their initial analysis without underreporting suggested little evidence of effects outside the extreme upper tail, but when they allow for underreporting, the pattern gets more nuanced.
Warren: What they found is that only positive quantile treatment effects are ruled out at moderate upper-tail quantiles, while at the very top end, negative quantile treatment effects can be rejected.
Iris: That’s interesting because it aligns with results from a semiparametric model that allows for misreporting, suggesting that allowing for misreporting reveals a significant negative difference in treatment effect at the zero consumption level and significant increases at both positive consumption levels.
Ruth: So what this means for us is that without accounting for misreporting, you might miss important distributional shifts across usage levels entirely.
Warren: It suggests that policy impacts aren't just about whether something goes up or down overall, but where in the distribution those changes happen.
Iris: And they conclude by showing that for all values of the outcome level within a certain range, the bounds converge to a specific probability based on their model parameters.
Ruth: So to wrap up, this paper "Changes-in-Changes for Ordered Choice Models with Underreporting" gives us a rigorous way to handle uncertainty when we know our reported data might be hiding the truth.
Warren: It moves beyond just the mean effect and gives us bounds on quantile treatment effects that are more informative about different usage groups.
Iris: The key contribution is developing these sharp nonparametric bounds and then providing estimation and inference methods to actually use them across all outcome levels, especially when you extend it with a semiparametric model for richer identification.
Ruth: It's a powerful tool for economists interested in policy impacts because it tells us not just what the average effect is, but how that effect might vary depending on the level of behavior we are looking at.
Warren: The application to marijuana legalization shows that even when data is imperfect, you can still extract nuanced insights into how different consumption levels respond to a change.
Iris: But they do flag a limitation: their model relies on assumptions like parallel trends for the mean of the untreated potential outcome, which is standard in DiD but still an assumption.
Ruth: That’s fair. And it also notes that their semiparametric extension depends on having some structure in how covariates affect the reporting probability itself.
Warren: So while this paper gives us better tools for dealing with underreporting, the real world application still depends on those underlying structural assumptions about the data generation process.
Iris: Exactly. It's a solid framework, but it's not a magic solution to every identification problem in causal inference.
The paper's summary: Ruth: So, to recap, this paper is about building a framework for difference in differences when outcomes are things people don't report accurately, like survey responses about behaviors that might be stigmatized or socially undesirable.
Warren: Right. They’re not just looking at the average change; they’re trying to get bounds—limits—on what the true effect could actually be on different parts of the outcome distribution.
Iris: Exactly. It tries to give you those bounds for both what happened after treatment and what would have happened if treatment hadn't occurred, and it does this even when we know people are underreporting their consumption or behavior.
Ruth: What’s really interesting is that they derive these bounds using the difference in differences structure, which is a specific way of looking at policy changes over time.
Warren: And they show these bounds are sharp uniformly across all levels of the outcome variable if you assume some basic support conditions are met. That means the range they give you isn't just a loose guess; it’s tight.
Iris: They also propose estimation and inference methods for these bound functions, using repeated cross-sectional sampling and bootstrap methods to get confidence sets that are valid across all outcome levels.
Ruth: So, what this changes for us is moving from just guessing the average effect to getting a distribution of possible effects.
Warren: It lets you see how different usage levels might actually react differently to a policy change, instead of just seeing one single number.
Iris: And they extend it further by adding a semiparametric model that lets them use observed data about covariates—like demographics or state-level characteristics—to help point identify and estimate those distributional effects more precisely.
Ruth: That extension is key because it means you can see how people’s reporting behavior itself might change based on some other factors while still recovering the treatment effect.
Warren: So, the application they use is looking at recreational marijuana legalization and how it affects consumption frequency among high school students across different states.
Iris: In their study, they found that without accounting for underreporting, there was very little evidence of any effect outside of the extreme upper tail of consumption levels.
Ruth: But when they allow for underreporting, the picture gets much more complex. They found that positive effects are ruled out at moderate upper-tail quantiles, but they can actually reject negative effects at the very top end.
Warren: That’s a pretty nuanced finding because it suggests that policy impacts aren't uniform across all usage levels, which is what we want to know when you're looking at real-world behavior.
Iris: It also points toward results from a semiparametric model that allows for misreporting, suggesting there might be a significant negative difference at zero consumption and increases at both positive consumption levels.
Ruth: So the bottom line is that if you don't account for this reporting behavior, you miss these important distributional shifts across usage levels entirely.
Warren: They show that allowing misreporting reveals a real pattern of effect across the whole spectrum of usage, not just one average figure.
Iris: It's a solid framework for getting those bounds and doing the math right with confidence in your results.
The paper's improvements: Tom: So, to recap this part is about where they suggest making things better than what they just presented in that first paper about underreporting in ordered choice models.
Ruth: Right. They're looking at how to take those nonparametric bounds and estimation procedures we talked about and make them more robust or flexible for real-world use.
Warren: They focus on the estimation part, specifically suggesting they use Hadamard directional derivatives when constructing the estimators for those bounds. That’s a specific way to calculate a derivative that should help with getting better estimates when you only have cross-sectional data.
Iris: I see how that fits with what we need for inference. If you can get a cleaner estimator of the bound, it makes constructing those confidence sets much more reliable, especially across all those different outcome levels we discussed earlier.
Ruth: Exactly. It’s about moving from just a theoretical range to having actual numbers you can trust when you're bootstrapping or running your analysis on real data.
Warren: And they also mention improving the semiparametric extension by incorporating observed covariates into the threshold-crossing representation itself. That makes it more flexible for identifying those distributional effects we were interested in earlier.
Iris: That’s a big deal because it means we can see how people's reporting choices might be influenced by those covariates while still recovering the treatment effect on the outcome, which is what we wanted.
Ruth: So they're essentially saying, okay, our initial framework is good for theory and setting bounds, but these suggested improvements help us build a practical tool that handles the messy reality of data collection better.
Warren: It’s about reducing the reliance on those strong assumptions—like parallel trends or perfect structure in reporting—and making the estimation less sensitive to those things.
Iris: The limitation they flag is that while these methods are better, they still rely on having some structure in how covariates affect the reporting probability itself, so it's not a magic fix for every possible data problem.
Ruth: So while these improvements make the tools stronger and more practical for estimation, we still need to be careful about those underlying assumptions about the data generation process.
Warren: That makes sense. So they’re not claiming perfection, just better ways to run the strategy we have so far. They’re suggesting an iterative improvement path.
Iris: And this leads us into the next topic, which is how this framework connects with other estimation techniques for spillover effects and under interference in comparison group methods.
Conclusion: Tom: So, to wrap up this whole thing, we're looking at "Changes-in-Changes for Ordered Choice Models with Underreporting." The main point is that this framework lets us get rigorous bounds on how much an outcome might change when people are underreporting their choices.
Ruth: It’s about giving economists a way to get distributional insights instead of just a single average effect, which is important because policy impacts aren't always uniform across the whole population.
Warren: Yeah, it moves us past just calculating one number and starts showing us the range of possibilities for that change based on how much people might be hiding their true behavior.
Iris: From my side, the identification works because it uses a difference in differences structure, which is solid if you assume parallel trends hold for the untreated potential outcome.
Ruth: That's a necessary assumption we have to live with in this setup, but the paper does a good job showing what happens when we relax those conditions slightly.
Warren: The numbers they use show that even with underreporting present, you can still reject negative effects at certain high levels of consumption where people are likely being honest.
Iris: That’s the nuance; it shows that honesty matters depending on how much behavior is involved, which is a critical check for any causal claim.
Ruth: So, we've seen that even with imperfect data, we can extract these bounds and use estimation procedures to get confidence in our predictions for those usage levels.
Warren: It’s a lot of work to build all those estimators and inference procedures, but if they are valid across all outcome levels, it’s a powerful tool for quantitative analysis.
Iris: I just think the extension to the semiparametric model is where the real power lies for future work, because it lets us incorporate more observed data about reporting itself.
Ruth: Agreed. It opens up ways to study how external factors might influence people's reporting decisions, which is a huge area for understanding behavior in markets and incentives.
Warren: So the implication here is that when we use this "Changes-in-Changes for Ordered Choice Models with Underreporting" framework, we get a much richer picture of policy effects than just the mean difference.
Iris: It gives us tools to be more precise about uncertainty in those bounds and where those effects are actually happening within the distribution.
Ruth: That’s what it is; better tools for making inferences about complex, real-world behavior when we know data isn't perfect.
Goethe University Frankfurt
econ.EM, stat.ME
Submitted: 2024-01-01
Updated: 2026-09-10
Journal ref: Journal of Business & Economic Statistics (2026)
DOI: 10.1080/07350015.2026.2732285
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Importance score: 92/100
The gist: The paper develops a Difference-in-Differences (CiC) framework specifically designed for discrete, ordered outcomes that are subject to underreporting…
Key concepts
- Quantile Treatment Effects on the Treated (QTTs)
- These measure the effect of a treatment on specific quantiles (e.g., the 10th or 90th percentile) of an outcome, rather than just the average effect. They help determine how a policy change impacts different groups within the treated population, providing nuanced distributional insights.
- Underreporting Probability ($\alpha$)
- This parameter formalizes how likely someone is to underreport their true consumption level. It quantifies the chance that an observed reported value is lower than the actual value, allowing researchers to model and account for this common issue in self-reported data.
- Nonparametric Bounds
- These are mathematical limits placed on the possible true distribution of outcomes, even when reporting is imperfect. By deriving sharp bounds, the authors establish guaranteed ranges for what the counterfactual outcome distribution could look like under different conditions.
- Semiparametric Underreporting Model
- This extension allows researchers to incorporate observed characteristics (covariates) into the model of how people misreport. This enables precise identification and estimation of distributional treatment effects by accounting for how covariates influence reporting behavior.
Terminology
Summary
The paper develops a Difference-in-Differences (CiC) framework specifically designed for discrete, ordered outcomes that are subject to underreporting, which commonly occurs in self-reported surveys concerning socially undesirable or stigmatized behaviors. p. 0 The framework derives nonparametric bounds for both the counterfactual outcome distribution after treatment and the factual outcome distribution, as well as associated quantile treatment effects when outcomes are underreported. p. 0 These bounds are shown to be sharp uniformly across outcome levels under additional support conditions, and suitable estimation and bootstrap inference procedures are proposed. p. 0 Furthermore, an extension considers a semiparametric underreporting model that allows for the point identification and estimation of distributional treatment effects. p. 3
The primary objects of interest include Quantile Treatment Effects on the Treated (QTTs), defined as delta QTT(tau):= QY(1)D=1(tau) - QY(0)D=1(tau). p. 8 The analysis formalizes underreporting by defining potential reported
consumption, C(d), such that C(d) at most Y(d) with probability one, and formalizing the underreporting probability as (C(d) at most j Y(d) at least j + 1, G = g, T = t) at most alpha.
Key contributions include:
-
Deriving sharp nonparametric bounds for the counterfactual distribution F Y11(0) under Assumptions 1 and 3, incorporating the CiC structure of Assumption 2. p. 1 These bounds are presented in Proposition 2 as L(0) 11, alpha(y) at most F Y11(0)(y) at most U(0) 11, alpha(y), where y in R. p. 7
-
Deriving bounds for the QTTs using these distribution bounds in Corollary 3, establishing that LQTT, alpha(tau) at most delta QTT(tau) at most UQTT, alpha(tau) for tau in (0, 1). p. 2
-
Developing estimation and inference theory for all bound functions and resulting QTTs. p. 3
-
Applying the framework empirically to investigate the impact of recreational marijuana legalization on the consumption behavior of 8th-grade students across several U. p. 0S. states.
The empirical analysis involves defining reported consumption C(d) and investigating how legalization affects short-term consumption frequency, accounting for potential misreporting (alpha). p. 3 Initial analysis without underreporting suggests little evidence of non-zero effects outside the extreme upper tail. p. 4 When underreporting is allowed, the pattern becomes more nuanced: only positive QTTs are ruled out at moderate upper-tail quantiles, while at the very top, negative QTTs can be rejected. p. 4 This aligns with results from a semiparametric model that allows for misreporting, suggesting that allowing for misreporting leads to a significant negative Difference-in-Treatment (DTT) at the zero consumption level and significant increases at both positive consumption levels. p. 4
The paper establishes estimation and inference procedures using repeated cross-sectional sampling and bootstrap methods, showing that confidence bands for the counterfactual bounds are asymptotically valid and uniform over outcome levels. p. 3 The final result confirms that for all y in J = 1 - gamma, N to infinity (0) 11, alpha(y), U(0) 11, alpha(y) in CS(0) alpha (y; 1-gamma) = 1 - gamma. The analysis concludes that while without misreporting there is little evidence of impact, accounting for misreporting reveals nuanced distributional effects across usage levels. p. 4
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Develop a Difference-in-Differences framework for discrete, ordered outcomes subject to underreporting that derives sharp nonparametric bounds for both the counterfactual and factual outcome distributions as well as for associated quantile treatment effects when outcomes are underreported. This allows the AI system to provide rigorous, distribution-level causal inference rather than just mean effects.
-
Implement estimation and inference procedures based on repeated cross-sectional sampling by constructing estimators of the bounds using Hadamard directional derivatives and bootstrap methods, leading to uniformly valid Confidence Sets for these bounds across all outcome levels. This enables the AI to quantify uncertainty around its distributional predictions with high precision.
-
Extend the analysis to a semiparametric underreporting model that allows for point identification and estimation of distributional treatment effects by incorporating observed covariates into the threshold-crossing representation. This capability permits the AI to estimate how reporting behavior may be affected by treatment status through covariates while still recovering Distributional Treatment Effects on the Treated (DTTs).
-
Integrate a semiparametric model based on a real analytic extrapolation assumption to characterize discrete Changes-in-Changes models as threshold-crossing representations for latent outcome variables with location and scale restrictions. This allows the system to construct flexible parametric models that jointly capture consumption and reporting decisions while maintaining identification of treatment parameters.
-
Apply the methodology to investigate heterogeneous effects of policy changes by analyzing Quantile Treatment Effects (QTTs) across the entire consumption distribution, rather than just the mean, providing nuanced insights into how legalization impacts different usage levels. This capability moves beyond summary statistics to reveal complex distributional shifts caused by policy interventions.
Sources
- Misclassification in Difference-in-differences Models
- Inference on Directionally Differentiable Functions
- Evaluating the Impact of Regulatory Policies on Social Welfare in Difference-in-Difference Settings
- Partial Identification of Binary Choice Models with Misreported Outcomes
Related papers
- Stationarity with Occasionally Binding Constraints
- Backtesting High-Dimensional Expected Shortfall
- Higher-Order Neyman Orthogonality in Moment-Condition Models
- Online Bayesian Inference for a Dirichlet Process Mixture of Binary Logit Models
- Debiased Conditional Independence Tests for Causal Effects
- Sharp Testable Implications of Encouragement Designs