(Empirical) Bayes Approaches to Parallel Trends

summary

The gist

(Empirical) Bayes Approaches to Parallel Trends Researchers employing a difference-indifferences (DiD) design are often unsure about the validity of the parallel trends assumption It is common to…

In short

(Empirical) Bayes approaches address uncertainty in difference-in-differences by using priors to model parallel trends violations. Instead of relying solely on pre-testing, this method allows researchers to form posterior means and credible sets for treatment effects, providing more informative results than standard methods.

Key concepts

Fully Bayesian Approach
This method involves imposing a prior distribution over both the treatment effect (tau) and the parallel trends violations (delta). By using Bayes' rule, researchers can calculate posterior distributions for these parameters, leading to more robust inferences about the treatment effect.
Empirical Bayes Approaches
These approaches learn the time-series structure of parallel trend violations directly from observed pre-treatment data. For instance, if violations follow a simple random walk pattern (wt = delta_t - delta_{t-1}), the parameters governing this process can be estimated using maximum likelihood.
Proposition 1
This proposition provides a formula for calculating the posterior mean of the post-treatment effect (taupost) when assuming an uninformative prior for tau. It shows that this mean is simply the difference between the posterior means of both the post-treatment and parallel trends violation parameters.
Credible Sets (CSs)
Credible sets are intervals derived from Bayesian analysis that represent the range of plausible values for a parameter, given the data and prior information. These sets are often wider than standard confidence intervals because they account for uncertainty about potential violations of parallel trends.

Terminology used across episodes

This episode discusses

The paper

(Empirical) Bayes Approaches to Parallel Trends · Read on arXiv

Brown University

DOI: 10.1257/pandp.20241048

Transcript

Introduction to the show: ident: Economics. The day in economics: theory, mechanisms and the evidence beneath them.

Ruth: We're looking at the paper: "(Empirical) Bayes Approaches to Parallel Trends".

Warren: Here's the gist of the paper. (Empirical) Bayes Approaches to Parallel Trends Researchers employing a difference-indifferences (DiD) design are often unsure about the validity of the parallel trends assumption It is common to test for “pretrends”, yet such tests may be underpowered, and relying on them leads to statistical issues from pre-testing Recent work by Manski and Pepper (2018) and Rambachan and Roth (2023, RR) has made progress on obtaining more credible inference when parallel trends may be violated by adopting a partial identification approach We instead consider a Bayesian approach where the researcher imposes a prior on the violations of parallel trends δ This allows them to form posterior means and credible sets (CSs) for the treatment effect τpost For settings with many pre-treatment periods, we also consider empirical Bayes (EB) approaches, where the “prior” for the violations of parallel trends is calibrated using the pre-trends Fully Bayesian Approach The fully Bayesian approach involves imposing a prior πτ,δ(·) over τ, δ From Bayes’ rule, we have that p(τ, δ βˆ) ∝ l(βˆ δ + τ) · πτ,δ(τ, δ) Consequently, given a prior πτδ it is straightforward to compute the posterior p(τ βˆ) The following result characterizes the posterior mean for τpost when the prior is uninformative PROPOSITION 1: If the prior for τ is uninformative (i.e.

Ruth: πτδ ∝ 1), then E

τpost βˆ: = E

βpost βˆ: − E

E[δpost δpre = βpre: βˆ] Proposition 1 shows that the posterior mean for τpost is simply the difference between the posterior for βpost and the posterior for δpost It shows further that the posterior for δpost can be written as an iterated expectation, where the inner expectation is based on the conditional prior of δpost given δpre, and the outer expectation is over the posterior for βpre βˆ] Empirical Illustration The example using a Gaussian Prior demonstrates how to calculate posterior means Suppose we have a Gaussian prior for δ, δ ∼ N (µδ, Vδ) and impose the uninformative prior for τ To derive the posterior mean, note that the formula for the conditional mean of a Gaussian vector implies that E

δpost δpre: = µδpost + Γ′ V (δpre − µδpre),for ΓV = V−1 δpre Vδpre,δpost Applying Proposition 1, E

τpost βˆ: = β∗ post−µδpost−Γ′ V (β∗ pre−µδpre),where β∗ = E

β βˆ: is the posterior mean for β Empirical Bayes Approaches In Empirical Bayes approaches, the parameters of the time series process for violations of parallel trends are learned from the pre-trends As a simple illustration, suppose that violations of parallel trends across consecutive periods are governed by wt:= δt − δt−1 iid∼ N (µ, σ2) The parameters µ and σ2 can thus be estimated via maximum likelihood, which will be consistent (under mild regularity conditions on Σw) as the number of pre-treatment periods grows large, ¯T → ∞ One caveat to this approach is that the consistency of the estimates for the prior depends on the number of pre-treatment periods ¯T being large An alternative to the EB approach when the number of periods is moderate is to consider a hierarchical Bayes model Where the gains come from The gains in these approaches stem from imposing ex ante information about what violations of parallel trends may look like This allows one to form posterior means and credible sets (CSs) for the treatment effect τpost The Bayesian approach allows the researcher to impose ex ante information about what violations of parallel trends may look like, and thus to potentially obtain more informative results than the partial identification approach using worst-case bounds In settings with many pre-treatment periods, we also consider empirical Bayes (EB) approaches, where the “prior” for the violations of parallel trends is calibrated using the pre-trends The posterior CSs are wider than the OLS CIs, since the OLS CIs assume that parallel trends holds exactly, whereas our prior puts positive weight on violations of parallel trends Nevertheless, the CSs are informative, excluding zero in 3 out of 4 post-treatment periods The posterior means are also somewhat closer to zero than the OLS estimates This is because δpre and δpost are correlated under the imposed prior; thus, the primarily positive estimates for βˆ pre lead to a posterior that the posttreatment bias δpost is positive The posterior means for the treatment effects are substantially closer to zero than the OLS estimates The CSs are also substantially wider than the OLS CIs, especially in later post-treatment periods, owing to uncertainty about the violations of parallel trends REFERENCES Benzarti, Youssef, and Dorian Carloni.

Ruth: “Who Really Benefits from Consumption Tax Cuts?

Title and authors: Ruth: So we're looking at this paper titled "Empirical Bayes Approaches to Parallel Trends." It sounds like they are tackling a really tricky problem in difference-in-differences designs where the parallel trends assumption is often shaky.

Warren: Yeah, it's about how to get more solid results when you can't be totally sure that the groups are tracking each other over time. The title itself points to using empirical Bayes methods for this issue.

Iris: It suggests moving beyond just testing for pretrends, which the authors say can be underpowered and lead to statistical problems from doing that testing first. They propose a Bayesian approach instead of relying solely on pretests (Roth, two thousand twenty-two) <ref:2404.11839#pg1>.

Ruth: Exactly. Instead of just checking if trends exist before the treatment, they suggest imposing a prior on how big those parallel trend violations might be to get more credible inferences about the post-treatment effect.

Warren: So it’s about letting the data inform what we think those violations look like, instead of just assuming they are zero.

Iris: Right. And that leads into their main approach, which involves using a prior on the violations of parallel trends, called delta, to update the posterior for the treatment effect tau post (Roth, two thousand twenty-two) <ref:2404.11839#pg1,a prior on the violations of parallel trends>.

The paper's summary: Ruth: What they’re actually doing is setting up a fully Bayesian framework where they have a prior over both the violation of parallel trends and the treatment effect itself. It uses Bayes rule to get this posterior distribution for the treatment effect tau post given what we observe about the data, like our estimated values for beta hat.

Warren: The paper lays out this formula, which shows how you calculate that posterior distribution by combining a likelihood function with a prior on both parameters (Roth, two thousand twenty-two) <ref:2404.11839#pg1>. It’s all about updating beliefs.

Iris: If we use an uninformative prior for the treatment effect tau post given the violations of parallel trends delta, they derive Proposition one which shows the posterior mean for tau post is just the difference between the posterior for beta post and some expectation involving delta (Roth, two thousand twenty-two) <ref:2404.11839#pg3,the violations of parallel trends>.

Ruth: That's a key result because it simplifies how we think about that bias; it basically says you calculate the bias in two separate steps, one for each part.

Warren: So they show that the posterior mean for tau post is found by taking the difference between the posterior of beta post and an iterated expectation involving delta pre (Roth, two thousand twenty-two) <ref:2404.11839#pg1>.

Iris: They even give an empirical illustration using a Gaussian prior for delta to show how you calculate that posterior mean in practice, relating it back to the pre-treatment parameters like mu delta post minus Gamma prime V times the difference between pre-treatment and post-treatment means (Roth, two thousand twenty-two) <ref:2404.11839#pg1>.

The paper's improvements: Ruth: The real improvement they highlight is that by using this Bayesian framework, you get posterior means and credible sets for the treatment effect tau post, not just a single point estimate. That’s much more informative.

Warren: And those credible sets are better than the confidence intervals from standard OLS because OLS assumes parallel trends hold exactly, whereas this Bayesian approach actually gives some weight to violations of parallel trends (Roth, two thousand twenty-two) <ref:2404.11839#pg1>.

Iris: They also note that in settings with many pre-treatment periods, they suggest empirical Bayes approaches where they learn the parameters for those parallel trend violations directly from the pretrends using maximum likelihood.

Ruth: And that learning process is consistent as you get more and more pre-treatment periods, which is a nice practical thing to know about it. The caveat there is just that the consistency of those estimates depends on having enough pre-treatment periods, say t bar going to infinity (Roth, two thousand twenty-two) <ref:2404.11839#pg1>.

Warren: So you can use empirical Bayes if you have a lot of data points before the treatment starts to calibrate your prior on those violations. But if you don't have that many periods, it gets messy again.

Iris: The main takeaway is that this gives us a way to quantify our uncertainty about the parallel trends assumption in a structured way, rather than just ignoring it or using worst-case bounds (Roth, two thousand twenty-two) <ref:2404.11839#pg1>.

Conclusion: Ruth: To wrap up, the paper on "Empirical Bayes Approaches to Parallel Trends" shows that we can use Bayesian methods to incorporate prior information about how parallel trends might actually violate our assumptions.

Warren: It leads to posterior credible sets that are wider than OLS intervals, but they are still informative because they exclude zero in three out of four post-treatment periods (Roth, two thousand twenty-two) <ref:2404.11839#pg1>.

Iris: And the posterior means for the treatment effects end up being closer to zero than the OLS estimates because of how pre-treatment and post-treatment violations are correlated under their imposed prior (Roth, two thousand twenty-two) <ref:2404.11839#pg1>.

Ruth: So essentially, this approach gives us a richer picture of what’s happening when we estimate that post-treatment bias. It’s about moving from a fixed assumption to an updated belief structure about the data.

Warren: It’s about using that structure to make our conclusions more robust, even when the assumptions are tricky to pin down exactly.

Iris: We should remember that it relies on setting up a prior for those parallel trend violations, which is where your initial choice of prior really matters for the final results (Roth, two thousand twenty-two) <ref:2404.11839#pg1>.

More episodes

← Home