(Empirical) Bayes Approaches to Parallel Trends
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Economics. The day in economics: theory, mechanisms and the evidence beneath them.
Ruth: We're looking at the paper: "(Empirical) Bayes Approaches to Parallel Trends".
Warren: Here's the gist of the paper. (Empirical) Bayes Approaches to Parallel Trends Researchers employing a difference-indifferences (DiD) design are often unsure about the validity of the parallel trends assumption It is common to test for “pretrends”, yet such tests may be underpowered, and relying on them leads to statistical issues from pre-testing Recent work by Manski and Pepper (2018) and Rambachan and Roth (2023, RR) has made progress on obtaining more credible inference when parallel trends may be violated by adopting a partial identification approach We instead consider a Bayesian approach where the researcher imposes a prior on the violations of parallel trends δ This allows them to form posterior means and credible sets (CSs) for the treatment effect τpost For settings with many pre-treatment periods, we also consider empirical Bayes (EB) approaches, where the “prior” for the violations of parallel trends is calibrated using the pre-trends Fully Bayesian Approach The fully Bayesian approach involves imposing a prior πτ,δ(·) over τ, δ From Bayes’ rule, we have that p(τ, δ βˆ) ∝ l(βˆ δ + τ) · πτ,δ(τ, δ) Consequently, given a prior πτδ it is straightforward to compute the posterior p(τ βˆ) The following result characterizes the posterior mean for τpost when the prior is uninformative PROPOSITION 1: If the prior for τ is uninformative (i.e.
Ruth: πτδ ∝ 1), then E
τpost βˆ: = E
βpost βˆ: − E
E[δpost δpre = βpre: βˆ] Proposition 1 shows that the posterior mean for τpost is simply the difference between the posterior for βpost and the posterior for δpost It shows further that the posterior for δpost can be written as an iterated expectation, where the inner expectation is based on the conditional prior of δpost given δpre, and the outer expectation is over the posterior for βpre βˆ] Empirical Illustration The example using a Gaussian Prior demonstrates how to calculate posterior means Suppose we have a Gaussian prior for δ, δ ∼ N (µδ, Vδ) and impose the uninformative prior for τ To derive the posterior mean, note that the formula for the conditional mean of a Gaussian vector implies that E
δpost δpre: = µδpost + Γ′ V (δpre − µδpre),for ΓV = V−1 δpre Vδpre,δpost Applying Proposition 1, E
τpost βˆ: = β∗ post−µδpost−Γ′ V (β∗ pre−µδpre),where β∗ = E
β βˆ: is the posterior mean for β Empirical Bayes Approaches In Empirical Bayes approaches, the parameters of the time series process for violations of parallel trends are learned from the pre-trends As a simple illustration, suppose that violations of parallel trends across consecutive periods are governed by wt:= δt − δt−1 iid∼ N (µ, σ2) The parameters µ and σ2 can thus be estimated via maximum likelihood, which will be consistent (under mild regularity conditions on Σw) as the number of pre-treatment periods grows large, ¯T → ∞ One caveat to this approach is that the consistency of the estimates for the prior depends on the number of pre-treatment periods ¯T being large An alternative to the EB approach when the number of periods is moderate is to consider a hierarchical Bayes model Where the gains come from The gains in these approaches stem from imposing ex ante information about what violations of parallel trends may look like This allows one to form posterior means and credible sets (CSs) for the treatment effect τpost The Bayesian approach allows the researcher to impose ex ante information about what violations of parallel trends may look like, and thus to potentially obtain more informative results than the partial identification approach using worst-case bounds In settings with many pre-treatment periods, we also consider empirical Bayes (EB) approaches, where the “prior” for the violations of parallel trends is calibrated using the pre-trends The posterior CSs are wider than the OLS CIs, since the OLS CIs assume that parallel trends holds exactly, whereas our prior puts positive weight on violations of parallel trends Nevertheless, the CSs are informative, excluding zero in 3 out of 4 post-treatment periods The posterior means are also somewhat closer to zero than the OLS estimates This is because δpre and δpost are correlated under the imposed prior; thus, the primarily positive estimates for βˆ pre lead to a posterior that the posttreatment bias δpost is positive The posterior means for the treatment effects are substantially closer to zero than the OLS estimates The CSs are also substantially wider than the OLS CIs, especially in later post-treatment periods, owing to uncertainty about the violations of parallel trends REFERENCES Benzarti, Youssef, and Dorian Carloni.
Ruth: “Who Really Benefits from Consumption Tax Cuts?
Title and authors: Ruth: So we're looking at this paper titled "Empirical Bayes Approaches to Parallel Trends." It sounds like they are tackling a really tricky problem in difference-in-differences designs where the parallel trends assumption is often shaky.
Warren: Yeah, it's about how to get more solid results when you can't be totally sure that the groups are tracking each other over time. The title itself points to using empirical Bayes methods for this issue.
Iris: It suggests moving beyond just testing for pretrends, which the authors say can be underpowered and lead to statistical problems from doing that testing first. They propose a Bayesian approach instead of relying solely on pretests (Roth, two thousand twenty-two) <ref:2404.11839#pg1>.
Ruth: Exactly. Instead of just checking if trends exist before the treatment, they suggest imposing a prior on how big those parallel trend violations might be to get more credible inferences about the post-treatment effect.
Warren: So it’s about letting the data inform what we think those violations look like, instead of just assuming they are zero.
Iris: Right. And that leads into their main approach, which involves using a prior on the violations of parallel trends, called delta, to update the posterior for the treatment effect tau post (Roth, two thousand twenty-two) <ref:2404.11839#pg1,a prior on the violations of parallel trends>.
The paper's summary: Ruth: What they’re actually doing is setting up a fully Bayesian framework where they have a prior over both the violation of parallel trends and the treatment effect itself. It uses Bayes rule to get this posterior distribution for the treatment effect tau post given what we observe about the data, like our estimated values for beta hat.
Warren: The paper lays out this formula, which shows how you calculate that posterior distribution by combining a likelihood function with a prior on both parameters (Roth, two thousand twenty-two) <ref:2404.11839#pg1>. It’s all about updating beliefs.
Iris: If we use an uninformative prior for the treatment effect tau post given the violations of parallel trends delta, they derive Proposition one which shows the posterior mean for tau post is just the difference between the posterior for beta post and some expectation involving delta (Roth, two thousand twenty-two) <ref:2404.11839#pg3,the violations of parallel trends>.
Ruth: That's a key result because it simplifies how we think about that bias; it basically says you calculate the bias in two separate steps, one for each part.
Warren: So they show that the posterior mean for tau post is found by taking the difference between the posterior of beta post and an iterated expectation involving delta pre (Roth, two thousand twenty-two) <ref:2404.11839#pg1>.
Iris: They even give an empirical illustration using a Gaussian prior for delta to show how you calculate that posterior mean in practice, relating it back to the pre-treatment parameters like mu delta post minus Gamma prime V times the difference between pre-treatment and post-treatment means (Roth, two thousand twenty-two) <ref:2404.11839#pg1>.
The paper's improvements: Ruth: The real improvement they highlight is that by using this Bayesian framework, you get posterior means and credible sets for the treatment effect tau post, not just a single point estimate. That’s much more informative.
Warren: And those credible sets are better than the confidence intervals from standard OLS because OLS assumes parallel trends hold exactly, whereas this Bayesian approach actually gives some weight to violations of parallel trends (Roth, two thousand twenty-two) <ref:2404.11839#pg1>.
Iris: They also note that in settings with many pre-treatment periods, they suggest empirical Bayes approaches where they learn the parameters for those parallel trend violations directly from the pretrends using maximum likelihood.
Ruth: And that learning process is consistent as you get more and more pre-treatment periods, which is a nice practical thing to know about it. The caveat there is just that the consistency of those estimates depends on having enough pre-treatment periods, say t bar going to infinity (Roth, two thousand twenty-two) <ref:2404.11839#pg1>.
Warren: So you can use empirical Bayes if you have a lot of data points before the treatment starts to calibrate your prior on those violations. But if you don't have that many periods, it gets messy again.
Iris: The main takeaway is that this gives us a way to quantify our uncertainty about the parallel trends assumption in a structured way, rather than just ignoring it or using worst-case bounds (Roth, two thousand twenty-two) <ref:2404.11839#pg1>.
Conclusion: Ruth: To wrap up, the paper on "Empirical Bayes Approaches to Parallel Trends" shows that we can use Bayesian methods to incorporate prior information about how parallel trends might actually violate our assumptions.
Warren: It leads to posterior credible sets that are wider than OLS intervals, but they are still informative because they exclude zero in three out of four post-treatment periods (Roth, two thousand twenty-two) <ref:2404.11839#pg1>.
Iris: And the posterior means for the treatment effects end up being closer to zero than the OLS estimates because of how pre-treatment and post-treatment violations are correlated under their imposed prior (Roth, two thousand twenty-two) <ref:2404.11839#pg1>.
Ruth: So essentially, this approach gives us a richer picture of what’s happening when we estimate that post-treatment bias. It’s about moving from a fixed assumption to an updated belief structure about the data.
Warren: It’s about using that structure to make our conclusions more robust, even when the assumptions are tricky to pin down exactly.
Iris: We should remember that it relies on setting up a prior for those parallel trend violations, which is where your initial choice of prior really matters for the final results (Roth, two thousand twenty-two) <ref:2404.11839#pg1>.
Brown University
econ.EM, stat.ME
Submitted: 2024-04-18
Updated: 2024-04-18
Journal ref: AEA Papers and Proceedings 114 (2024), 606-609
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Importance score: 79/100
The gist: (Empirical) Bayes Approaches to Parallel Trends Researchers employing a difference-indifferences (DiD) design are often unsure about the validity of the parallel trends assumption It is common to…
Key concepts
- Fully Bayesian Approach
- This method involves imposing a prior distribution over both the treatment effect (tau) and the parallel trends violations (delta). By using Bayes' rule, researchers can calculate posterior distributions for these parameters, leading to more robust inferences about the treatment effect.
- Empirical Bayes Approaches
- These approaches learn the time-series structure of parallel trend violations directly from observed pre-treatment data. For instance, if violations follow a simple random walk pattern (wt = delta_t - delta_{t-1}), the parameters governing this process can be estimated using maximum likelihood.
- Proposition 1
- This proposition provides a formula for calculating the posterior mean of the post-treatment effect (taupost) when assuming an uninformative prior for tau. It shows that this mean is simply the difference between the posterior means of both the post-treatment and parallel trends violation parameters.
- Credible Sets (CSs)
- Credible sets are intervals derived from Bayesian analysis that represent the range of plausible values for a parameter, given the data and prior information. These sets are often wider than standard confidence intervals because they account for uncertainty about potential violations of parallel trends.
Terminology
Summary
(Empirical) Bayes Approaches to Parallel Trends
Researchers employing a difference-indifferences (DiD) design are often unsure about the validity of the parallel trends assumption It is common to test for “pretrends”, yet such tests may be underpowered, and relying on them leads to statistical issues from pre-testing Recent work by Manski and Pepper (2018) and Rambachan and Roth (2023, RR) has made progress on obtaining more credible inference when parallel trends may be violated by adopting a partial identification approach We instead consider a Bayesian approach where the researcher imposes a prior on the violations of parallel trends δ This allows them to form posterior means and credible sets (CSs) for the treatment effect τpost For settings with many pre-treatment periods, we also consider empirical Bayes (EB) approaches, where the “prior” for the violations of parallel trends is calibrated using the pre-trends
Fully Bayesian Approach
The fully Bayesian approach involves imposing a prior πτ,δ(·) over τ, δ From Bayes’ rule, we have that p(τ, δ βˆ) ∝ l(βˆ δ + τ) · πτ,δ(τ, δ) Consequently, given a prior πτδ it is straightforward to compute the posterior p(τ βˆ) The following result characterizes the posterior mean for τpost when the prior is uninformative PROPOSITION 1: If the prior for τ is uninformative (i.e. πτδ ∝ 1), then E[τpost βˆ] = E[βpost βˆ] − E[E[δpost δpre = βpre] βˆ] Proposition 1 shows that the posterior mean for τpost is simply the difference between the posterior for βpost and the posterior for δpost It shows further that the posterior for δpost can be written as an iterated expectation, where the inner expectation is based on the conditional prior of δpost given δpre, and the outer expectation is over the posterior for βpre βˆ]
Empirical Illustration
The example using a Gaussian Prior demonstrates how to calculate posterior means Suppose we have a Gaussian prior for δ, δ ∼ N (µδ, Vδ) and impose the uninformative prior for τ To derive the posterior mean, note that the formula for the conditional mean of a Gaussian vector implies that E[δpost δpre] = µδpost + Γ′ V (δpre − µδpre),for ΓV = V−1 δpre Vδpre,δpost Applying Proposition 1, E[τpost βˆ] = β∗ post−µδpost−Γ′ V (β∗ pre−µδpre),where β∗ = E[β βˆ] is the posterior mean for β
Empirical Bayes Approaches
In Empirical Bayes approaches, the parameters of the time series process for violations of parallel trends are learned from the pre-trends As a simple illustration, suppose that violations of parallel trends across consecutive periods are governed by wt:= δt − δt−1 iid∼ N (µ, σ2) The parameters µ and σ2 can thus be estimated via maximum likelihood, which will be consistent (under mild regularity conditions on Σw) as the number of pre-treatment periods grows large, ¯T → ∞ One caveat to this approach is that the consistency of the estimates for the prior depends on the number of pre-treatment periods ¯T being large An alternative to the EB approach when the number of periods is moderate is to consider a hierarchical Bayes model
Where the gains come from
The gains in these approaches stem from imposing ex ante information about what violations of parallel trends may look like This allows one to form posterior means and credible sets (CSs) for the treatment effect τpost The Bayesian approach allows the researcher to impose ex ante information about what violations of parallel trends may look like, and thus to potentially obtain more informative results than the partial identification approach using worst-case bounds In settings with many pre-treatment periods, we also consider empirical Bayes (EB) approaches, where the “prior” for the violations of parallel trends is calibrated using the pre-trends The posterior CSs are wider than the OLS CIs, since the OLS CIs assume that parallel trends holds exactly, whereas our prior puts positive weight on violations of parallel trends Nevertheless, the CSs are informative, excluding zero in 3 out of 4 post-treatment periods The posterior means are also somewhat closer to zero than the OLS estimates This is because δpre and δpost are correlated under the imposed prior; thus, the primarily positive estimates for βˆ pre lead to a posterior that the posttreatment bias δpost is positive The posterior means for the treatment effects are substantially closer to zero than the OLS estimates The CSs are also substantially wider than the OLS CIs, especially in later post-treatment periods, owing to uncertainty about the violations of parallel trends
REFERENCES
Benzarti, Youssef, and Dorian Carloni. 2019. “Who Really Benefits from Consumption Tax Cuts? p. 3 Evidence from a Large VAT Reform in France. p. 3” Amer3A recent literature surveyed in Roth et al. p. 3 (2023) has shown that such specifications may be difficult to interpret under treatment effect heterogeneity; one could re-do the analysis here with the event-study from one of the estimators developed for this case Giacomini, Raffaella, and Toru Kitagawa. p. 3 2021. “Robust Bayesian Inference for Set-Identified Models. p. 3” Econometrica, 89(4): 1519–1556 Leavitt, Thomas. p. 3 2020. “Beyond Parallel Trends: Improvements on Estimation and Inference in the Difference-in-Differences Design. p. 3” Working paper Lovenheim, Michael F. p. 3, and Alexander Will´en. p. 3 2019. “The Long-Run Effects of Teacher Collective Bargaining. p. 3” American Economic Journal: Economic Policy, 11(3): 292–324 Manski, Charles F. p. 3, and John V. p. 3 Pepper. 2018. “How Do Right-to-Carry Laws Affect Crime Rates? p. 3 Coping with Ambiguity Using Bounded-Variation Assumptions. p. 3” The Review of Economics and Statistics, 100(2): 232–244 Moon, Hyungsik Roger, and Frank Schorfheide. p. 3 2012. “Bayesian and Frequentist Inference in Partially Identified Models. p. 3” Econometrica, 80(2): 755–782 Rambachan, Ashesh, and Jonathan Roth. p. 3 2023. “A More Credible Approach to Parallel Trends. p. 0” The Review of Economic Studies, 90(5): 2555–2591 Roth, Jonathan. p. 3 2022. “Pretest with Caution: Event-Study Estimates after Testing for Parallel Trends. p. 3” American Economic Review: Insights, 4(3): 305–322 Roth, Jonathan, Pedro H. p. 3 C. Sant’Anna, Alyssa Bilinski, and John Poe. p. 3 2023. “What’s trending in difference-in-differences? p. 3 A synthesis of the recent econometrics literature. p. 3
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Bayesian inference allows for more informative results than partial identification approaches by imposing a prior on violations of parallel trends, potentially obtaining
more informative results than the partial identification approach using worst-case bounds.
-
The fully Bayesian approach enables researchers to
impose ex ante information about what violations of parallel trends may look like,
allowing for the formation of posterior means and credible sets (CSs) for treatment effects rather than just point estimates. -
Empirical Bayes approaches can learn the time-series dependence of parallel trend violations from pre-trends, allowing researchers to
learn the time-series dependence of violations of parallel trends from the pre-trends
when assuming violations come from a stationary process. -
The improved AI system can calculate posterior means for treatment effects by using
Proposition 1: If the prior for tau is uninformative (i.e. pi taudelta proportional to 1), then E[taupost beta hat] = E[beta post beta hat] − E[E[delta post deltapre = betapre] beta hat].
-
The AI system can utilize specific prior structures, such as a Gaussian prior for violations of parallel trends, to derive posterior means for treatment effects by applying the formula:
E[taupost beta hat] = beta post − mu deltapost−Gamma′V(beta pre−mu pre).
-
The system can adjust point estimates by shrinking them towards the prior mean, as shown in the equation for posterior mean for beta pre:
beta∗ pre = (Σ-1βˆpre + V-1pre)−1(Σ-1βˆpreβˆpre + V-1premu pre).
Related papers
- Stationarity with Occasionally Binding Constraints
- Backtesting High-Dimensional Expected Shortfall
- Higher-Order Neyman Orthogonality in Moment-Condition Models
- Online Bayesian Inference for a Dirichlet Process Mixture of Binary Logit Models
- Debiased Conditional Independence Tests for Causal Effects
- Sharp Testable Implications of Encouragement Designs