The Illusion of Learning from Observational Data: An Empirical Bayes Perspective

summary

The gist

Randomized experiments have long been the gold standard for scientists seeking to learn about cause and effect.

In short

The paper shows that observational studies can create an 'illusion of learning' because they don't account for how biases are distributed across studies. By using calibration studies to learn this bias distribution, researchers can apply empirical Bayes shrinkage to observational data, allowing them to estimate causal effects with vanishing risk as more data is gathered.

Key concepts

Illusion of Learning
When assuming flat priors on both the true effect and observational biases, the posterior mean remains fixed at the experimental estimate. This means observational studies provide no new information about the causal quantity of interest beyond what is learned from a single experiment alone, creating a misleading sense of improvement.
Empirical Bayes Shrinkage
This technique uses prior knowledge to adjust estimates from observational data. By learning the distribution of biases through calibration studies, shrinkage reduces the posterior variance and shifts the estimate towards the true causal effect, making observational studies informative.
Calibration Studies
These are experiments specifically designed to target a known zero causal effect. They serve as a tool to reveal how observational biases are distributed across different research approaches. This knowledge of bias is crucial for using empirical Bayes effectively in observational settings.

Terminology used across episodes

This episode discusses

The paper

The Illusion of Learning from Observational Data: An Empirical Bayes Perspective · Read on arXiv

Columbia University

Randomized experiments have long been the gold standard for scientists seeking to learn about cause and effect. When randomized experiments are infeasible, scientists often resort to observational studies, which are widely available and often large but rely on untestable assumptions that, when violated, may result in biased estimates. Uncertainty about bias leads to a phenomenon known as the illusion of learning from observational research (Gerber, Green and Kaplan, 2004a): absent prior information about bias, observational results cannot meaningfully contribute to the estimation of a causal parameter. To shatter the illusion, we take an empirical Bayes perspective. We show that the distribution of observational biases can be learned from calibration studies-experiments that target a causal effect that is known a priori to be zero. Calibration identifies the distribution of observational bias and allows observational studies to inform the estimation of causal parameters via empirical Bayes shrinkage. We formalize the illusion phenomenon in an empirical Bayes setting and show that, with an increasing number of calibration and observation studies, both the bias distribution and the causal effect can be consistently recovered. We illustrate our method through a simulation study and a semi-synthetic application based on Ferraro and Miranda (2013)'s water-usage experiment.

Transcript

Introduction to the show: ident: Statistics. The day in statistics: methods, theory and the inference under them.

Tom: We're looking at the paper: "The Illusion of Learning from Observational Data".

Jane: Here's the gist of the paper. Randomized experiments have long been the gold standard for scientists seeking to learn about cause and effect.

Tom: To shatter this illusion, the authors take an empirical Bayes perspective, showing that calibration studies can be used to learn the distribution of observational biases and allow observational studies to inform causal parameter estimation via empirical Bayes shrinkage.

Jane: The Illusion of Learning The paper formalizes the illusion of learning from observational data in a hierarchical Gaussian model for one experimental estimate and many observational estimates.

Tom: When assuming flat priors on both the effect and the biases, the posterior mean remains equal to the experimental estimate, and the posterior variance does not change, meaning observational studies offer no information beyond what is learned from the experiment alone.

Title and authors: Tom: So what does this paper actually do to break that illusion? They introduce empirical Bayes as a way to use those observational studies meaningfully, and they show that by applying it specifically to the variance of the biases, you can successfully break the illusion.

Jane: But wait, they run into a problem if you try to be too specific with what you know about those biases; assuming zero-mean bias is too strong because that brings back the illusion where observational studies just increase your overconfidence instead of helping.

Lu: That's a crucial distinction they make; fitting both the mean and variance of the biases using empirical Bayes brings the illusion right back because, in that case, those observational studies still don't actually inform your final estimate.

Meng: So so if you just use empirical Bayes on one part of the bias distribution, it helps but not fully? That means we have to be careful about which assumptions we make when trying to combine these different types of evidence.

Lalam: It suggests that simply applying a standard shrinkage technique isn't enough; you need a way to learn something about the shape of those biases across the studies.

The paper's summary: Tom: To fix this, they introduce calibration studies, which they use to target a known zero causal effect and figure out the actual distribution of biases, which then allows observational studies to improve estimation when you use calibrated empirical Bayes.

Jane: So calibration studies are the key here because they give us that information about the bias distribution that we were missing before, and with it, applying empirical Bayes lets those observational studies actually contribute to estimating the causal effect.

Lu: The main contributions of this paper are formalizing the illusion in a hierarchical Gaussian model, showing that empirical Bayes on variance breaks the illusion but requires knowing the mean bias distribution, and then showing that calibration studies restore learning by identifying that bias distribution so observational studies can improve estimation.

Meng: It sounds like they're proposing a calibrated empirical Bayes procedure where you use calibration studies to fit a prior over biases, and then you use that fitted prior to combine the experimental estimate with the observational estimates, which leads to vanishing risk as both observational and calibration studies grow.

Lalam: The core idea is that triangulation is necessary here; you need to learn something about how biases are distributed across all those observational studies for them to contribute meaningfully.

The paper's improvements: Tom: Beyond just breaking the illusion, the authors suggest some methodological extensions, moving beyond normal priors and into nonparametric priors, and even looking at regression settings instead of just simple one-experimental estimates.

Jane: And they also address resource allocation—how you can minimize the posterior variance of that target causal effect while balancing your budget between experimental studies, observational studies, and calibration studies.

Lu: They go further by suggesting extensions to nonparametric priors and moving beyond the normal means model into regression settings, which shows the idea can be applied in more complex scenarios than just a single effect estimation.

Meng: From my side, I'm interested in the resource allocation part; figuring out how many of each study type to run given limited resources is something that actually matters for deploying these kinds of models at scale.

Lalam: And they also discuss using moment matching estimators for estimating bias distribution parameters, like gamma two and mu, which the paper shows can satisfy risk consistency under certain conditions when you use more complex models than a simple normal prior.

Conclusion: Tom: So to wrap up, the main point is that to actually gain from combining experimental and observational methods, you have to assume or learn something concrete about how those biases are distributed across your observational studies.

Jane: Calibration studies are especially valuable because they quantify the bias of an observational research approach, which enables that approach to contribute meaningfully to estimating the target causal parameter when paired with calibrated empirical Bayes.

Lu: The strategy depends on calibration studies being subject to the same population of biases as your target observational studies, and they also provide a scalable approximate solution for resource allocation using a knapsack problem structure.

Meng: And those moment matching estimators they show are consistent in the L1 sense, which means even when you use more complex models for estimating those bias distribution parameters, you can still trust the posterior to converge to the true parameter.

Lalam: So ultimately, this paper on "The Illusion of Learning from Observational Data: An Empirical Bayes Perspective" shows that triangulation requires learning about the distribution of biases across observational studies, and it gives us a way to do that while optimizing how we allocate our research budget.

More episodes

← Home