The Illusion of Learning from Observational Data: An Empirical Bayes Perspective
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Statistics. The day in statistics: methods, theory and the inference under them.
Tom: We're looking at the paper: "The Illusion of Learning from Observational Data".
Jane: Here's the gist of the paper. Randomized experiments have long been the gold standard for scientists seeking to learn about cause and effect.
Tom: To shatter this illusion, the authors take an empirical Bayes perspective, showing that calibration studies can be used to learn the distribution of observational biases and allow observational studies to inform causal parameter estimation via empirical Bayes shrinkage.
Jane: The Illusion of Learning The paper formalizes the illusion of learning from observational data in a hierarchical Gaussian model for one experimental estimate and many observational estimates.
Tom: When assuming flat priors on both the effect and the biases, the posterior mean remains equal to the experimental estimate, and the posterior variance does not change, meaning observational studies offer no information beyond what is learned from the experiment alone.
Title and authors: Tom: So what does this paper actually do to break that illusion? They introduce empirical Bayes as a way to use those observational studies meaningfully, and they show that by applying it specifically to the variance of the biases, you can successfully break the illusion.
Jane: But wait, they run into a problem if you try to be too specific with what you know about those biases; assuming zero-mean bias is too strong because that brings back the illusion where observational studies just increase your overconfidence instead of helping.
Lu: That's a crucial distinction they make; fitting both the mean and variance of the biases using empirical Bayes brings the illusion right back because, in that case, those observational studies still don't actually inform your final estimate.
Meng: So so if you just use empirical Bayes on one part of the bias distribution, it helps but not fully? That means we have to be careful about which assumptions we make when trying to combine these different types of evidence.
Lalam: It suggests that simply applying a standard shrinkage technique isn't enough; you need a way to learn something about the shape of those biases across the studies.
The paper's summary: Tom: To fix this, they introduce calibration studies, which they use to target a known zero causal effect and figure out the actual distribution of biases, which then allows observational studies to improve estimation when you use calibrated empirical Bayes.
Jane: So calibration studies are the key here because they give us that information about the bias distribution that we were missing before, and with it, applying empirical Bayes lets those observational studies actually contribute to estimating the causal effect.
Lu: The main contributions of this paper are formalizing the illusion in a hierarchical Gaussian model, showing that empirical Bayes on variance breaks the illusion but requires knowing the mean bias distribution, and then showing that calibration studies restore learning by identifying that bias distribution so observational studies can improve estimation.
Meng: It sounds like they're proposing a calibrated empirical Bayes procedure where you use calibration studies to fit a prior over biases, and then you use that fitted prior to combine the experimental estimate with the observational estimates, which leads to vanishing risk as both observational and calibration studies grow.
Lalam: The core idea is that triangulation is necessary here; you need to learn something about how biases are distributed across all those observational studies for them to contribute meaningfully.
The paper's improvements: Tom: Beyond just breaking the illusion, the authors suggest some methodological extensions, moving beyond normal priors and into nonparametric priors, and even looking at regression settings instead of just simple one-experimental estimates.
Jane: And they also address resource allocation—how you can minimize the posterior variance of that target causal effect while balancing your budget between experimental studies, observational studies, and calibration studies.
Lu: They go further by suggesting extensions to nonparametric priors and moving beyond the normal means model into regression settings, which shows the idea can be applied in more complex scenarios than just a single effect estimation.
Meng: From my side, I'm interested in the resource allocation part; figuring out how many of each study type to run given limited resources is something that actually matters for deploying these kinds of models at scale.
Lalam: And they also discuss using moment matching estimators for estimating bias distribution parameters, like gamma two and mu, which the paper shows can satisfy risk consistency under certain conditions when you use more complex models than a simple normal prior.
Conclusion: Tom: So to wrap up, the main point is that to actually gain from combining experimental and observational methods, you have to assume or learn something concrete about how those biases are distributed across your observational studies.
Jane: Calibration studies are especially valuable because they quantify the bias of an observational research approach, which enables that approach to contribute meaningfully to estimating the target causal parameter when paired with calibrated empirical Bayes.
Lu: The strategy depends on calibration studies being subject to the same population of biases as your target observational studies, and they also provide a scalable approximate solution for resource allocation using a knapsack problem structure.
Meng: And those moment matching estimators they show are consistent in the L1 sense, which means even when you use more complex models for estimating those bias distribution parameters, you can still trust the posterior to converge to the true parameter.
Lalam: So ultimately, this paper on "The Illusion of Learning from Observational Data: An Empirical Bayes Perspective" shows that triangulation requires learning about the distribution of biases across observational studies, and it gives us a way to do that while optimizing how we allocate our research budget.
Columbia University
stat.ME
Submitted: 2026-04-10
Updated: 2026-11-05
Comments: arXiv admin note: This submission has been withdrawn because it does not meet arXiv's research content quality standards
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 90/100
The gist: Randomized experiments have long been the gold standard for scientists seeking to learn about cause and effect.
Key concepts
- Illusion of Learning
- When assuming flat priors on both the true effect and observational biases, the posterior mean remains fixed at the experimental estimate. This means observational studies provide no new information about the causal quantity of interest beyond what is learned from a single experiment alone, creating a misleading sense of improvement.
- Empirical Bayes Shrinkage
- This technique uses prior knowledge to adjust estimates from observational data. By learning the distribution of biases through calibration studies, shrinkage reduces the posterior variance and shifts the estimate towards the true causal effect, making observational studies informative.
- Calibration Studies
- These are experiments specifically designed to target a known zero causal effect. They serve as a tool to reveal how observational biases are distributed across different research approaches. This knowledge of bias is crucial for using empirical Bayes effectively in observational settings.
Terminology
Summary
Randomized experiments have long been the gold standard for scientists seeking to learn about cause and effect. p. 0 edu 4Departments of Computer Science and Statistics, Columbia University, e-mail:
blei@cs. edu
Abstract: Randomized experiments have long been the gold standard
for scientists seeking to learn about cause and effect. We show
that the distribution of observational biases can be learned from calibration
studies—experiments that target a causal effect that is known a priori to
be zero. Calibration identifies the distribution of observational bias and
allows observational studies to inform the estimation of causal parameters
via empirical Bayes shrinkage.Where the paper says it
Where the paper says it
The Illusion of Learning
The paper formalizes the illusion of learning from observational data in a hierarchical Gaussian model for one experimental estimate and many observational estimates. When assuming flat priors on both the effect and the biases, the posterior mean remains equal to the experimental estimate, and the posterior variance does not change, meaning observational studies offer no information beyond what is learned from the experiment alone. p. 4 As the next theorem shows, with flat priors on both
the effect and the biases, E [θ | ye, yo] = ye and the posterior variance remains
the same, Var [θ | ye] = σ
2
e
. The observational studies offer no information beyond
the experiment. As the next theorem shows, with flat priors on both
the effect and the biases, E [θ | ye, yo] = ye and the posterior variance remains
the same, Var [θ | ye] = σ
2
e
. The observational studies offer no information beyond
the experiment. , when the prior is flat, the observational studies receive no weight in the
posterior for the target causal effect. In this sense, the observational studies
create only an illusion of learning: they appear relevant but they do not improve
inference about the causal quantity of interest.Where the paper says it
Where the paper says it
Where the paper says it
Breaking the Illusion with Empirical Bayes
The authors show that applying empirical Bayes to the variance of the biases successfully breaks the illusion, as observational studies adjust the posterior expectation and reduce posterior variance, leading to vanishing risk as more observational studies are seen. However, assuming zero-mean bias is too strong; when fitting both the mean and variance of the biases using empirical Bayes, the illusion returns because observational studies do not inform the estimate and only increase overconfidence. p. 2 Third, we show that applying empirical Bayes to both the mean and variance of
the biases brings back the illusion, where the observational studies do not inform
the estimate. Finally, we show that calibration studies restore the possibility
of learning: they identify the bias distribution such that observational studies
can improve estimation of the causal effect.Where the paper says it
The Role of Calibration Studies
To break the illusion without making unrealistic assumptions about bias distributions, calibration studies are introduced, which target a known zero causal effect to reveal the distribution of biases. With calibrated empirical Bayes, the causal estimator achieves vanishing risk as both observational and calibration studies grow. p. 2 Finally, we show that calibration studies restore the possibility
of learning: they identify the bias distribution such that observational studies
can improve estimation of the causal effect. With calibrated empirical Bayes,
the causal estimator achieves vanishing risk as the number of observational and
calibration studies grows. Finally, we show that calibration studies restore the possibility
of learning: they identify the bias distribution such that observational studies
can improve estimation of the causal effect. With calibrated empirical Bayes,
the causal estimator achieves vanishing risk as the number of observational and
calibration studies grows. We study this method in theory, simulation, and a semi-synthetic analysis
based on a large field experiment. Calibrated empirical Bayes improves precision
relative to the experiment alone, enabling observational studies to contribute
meaningfully to the estimation of a causal effect.Where the paper says it
Where the paper says it
Where the paper says it
Methodological Extensions
The work extends beyond normal priors by suggesting extensions to nonparametric priors and moving beyond the normal means model into regression settings. The resource allocation problem is also addressed, showing how to minimize the posterior variance of the causal effect subject to a budget constraint between experimental, observational, and calibration studies. p. 1 The observational studies do not
move the posterior mean of the causal effect away from the experimental estimate. Even worse, they make the posterior variance smaller, creating unwarranted
confidence in the experimental results.Where the paper says it
Real Data Application
A semi-synthetic analysis based on a water-usage experiment is used to illustrate these findings, where the calibrated EB estimator achieves an estimate close to the oracle ATE and yields well-calibrated uncertainty compared to the illusion model. This demonstrates that calibration studies can unlock otherwise uninterpretable research findings in domains where observational studies abound. p. 9 Simulations
Simulations help illustrate the theoretical findings in Section 3. In particular,
we show that incorporating calibration studies leads to improved statistical
efficiency and accuracy of causal effect estimation when combining experimental
and observational studies, consistent with Theorem 4. Simulations
Simulations help illustrate the theoretical findings in Section 3. In particular,
we show that incorporating calibration studies leads to improved statistical
efficiency and accuracy of causal effect estimation when combining experimental
and observational studies, consistent with Theorem 4. 6)
With this model, we estimate the causal effect through its posterior p(θ | ye, yo). The posterior mean ˆθ provides a point estimate; the posterior variance vˆ provides (model-based) uncertainty.Where the paper says it
Where the paper says it
Where the paper says it
Conclusion on Triangulation
The analysis clarifies that to gain from combining methods, one must assume or learn something about how biases are distributed across observational studies. Calibration studies are especially valuable because they quantify the bias of an observational research approach, enabling that approach to contribute meaningfully to estimating the target causal parameter. p. 0 Observational studies are widely available and often large,
but they rely on untestable assumptions that, when violated, may result
in biased estimates. Uncertainty about bias leads to a phenomenon known
as the illusion of learning from observational research
(Gerber, Green and
Kaplan
, 2004a): absent prior information about bias, observational results
cannot meaningfully contribute to the estimation of a causal parameter. To shatter the illusion, we take an empirical Bayes perspective. We show
that the distribution of observational biases can be learned from calibration
studies—experiments that target a causal effect that is known a priori to
be zero. First, we formalize the
illusion of learning from observational data in a hierarchical Gaussian model for
one experimental estimate and many observational estimates. Second, we show
that applying empirical Bayes to the variance of the biases successfully breaks
the illusion but requires the unrealistic assumption that we know their mean. Because their true effect is known, any
nonzero estimate from a calibration study can be attributed to its bias. Calibration studies help solve our problem because they provide direct information about the distribution of biases associated with a given observational
design. In particular,
we show that incorporating calibration studies leads to improved statistical
efficiency and accuracy of causal effect estimation when combining experimental
and observational studies, consistent with Theorem 4. Setup and estimators.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
--- Page 1 ---
The Illusion of Learning from Observational Data: An Empirical Bayes Perspective
Randomized experiments have long been the gold standard for scientists seeking to learn about cause and effect. To shatter this illusion, we take an empirical Bayes perspective, showing that calibration studies can be used to learn the distribution of observational biases and allow observational studies to inform causal parameter estimation via empirical Bayes shrinkage. p. 0 We show
that the distribution of observational biases can be learned from calibration
studies—experiments that target a causal effect that is known a priori to
be zero. Calibration identifies the distribution of observational bias and
allows observational studies to inform the estimation of causal parameters
via empirical Bayes shrinkage.Where the paper says it
The authors formalize the illusion of learning from observational data in a hierarchical Gaussian model for one experimental estimate and many observational estimates. When assuming flat priors on both the effect and the biases, the posterior mean remains equal to the experimental estimate, and the posterior variance does not change, meaning observational studies offer no information beyond what is learned from the experiment alone.
To break the illusion without making unrealistic assumptions about bias distributions, calibration studies are introduced, which target a known zero causal effect to reveal the distribution of biases. With calibrated empirical Bayes, the causal estimator achieves vanishing risk as both observational and calibration studies grow.
A semi-synthetic analysis based on a water-usage experiment is used to illustrate these findings, where the calibrated EB estimator achieves an estimate close to the oracle ATE and yields well-calibrated uncertainty compared to the illusion model. This demonstrates that calibration studies can unlock otherwise uninterpretable research findings in domains where observational studies abound.
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Empirical Bayes shrinkage for causal inference allows observational studies to contribute meaningfully to causal parameter estimation by learning a prior distribution of observational biases from calibration studies. This can be applied in AI systems that rely on combining experimental data with large sets of observational data, improving precision when the bias distribution is known or estimated.
-
The system can implement a calibrated empirical Bayes procedure for causal estimation by fitting a prior over biases using calibration studies and then applying this prior to combine the experimental estimate with observational estimates, achieving
vanishing risk as the number of observational and calibration studies grows.
This allows AI models to leverage vast, but potentially biased, real-world data more effectively. -
The system can use internal calibration studies paired with specific observational studies to directly estimate study-specific biases rather than inferring them only through a prior. This capability enables the estimation of the causal effect within each study rather than just for a broader meta-analytic superpopulation, leading to more accurate causal inference from observational data.
-
The system can perform resource allocation between experimental, observational, and calibration studies by minimizing the posterior variance of the target causal effect subject to a budget constraint. This provides an optimized strategy for AI research teams deciding how many of each study type to conduct given limited resources.
-
The system can utilize moment matching estimators for estimating bias distribution parameters, such as gamma 2 and mu, which are shown to satisfy risk consistency under certain conditions. This allows the AI system to robustly estimate the unknown distribution of observational biases even when more complex models than a simple normal prior are used.
Abstract
Randomized experiments have long been the gold standard for scientists seeking to learn about cause and effect. When randomized experiments are infeasible, scientists often resort to observational studies, which are widely available and often large but rely on untestable assumptions that, when violated, may result in biased estimates. Uncertainty about bias leads to a phenomenon known as the illusion of learning from observational research (Gerber, Green and Kaplan, 2004a): absent prior information about bias, observational results cannot meaningfully contribute to the estimation of a causal parameter. To shatter the illusion, we take an empirical Bayes perspective. We show that the distribution of observational biases can be learned from calibration studies-experiments that target a causal effect that is known a priori to be zero. Calibration identifies the distribution of observational bias and allows observational studies to inform the estimation of causal parameters via empirical Bayes shrinkage. We formalize the illusion phenomenon in an empirical Bayes setting and show that, with an increasing number of calibration and observation studies, both the bias distribution and the causal effect can be consistently recovered. We illustrate our method through a simulation study and a semi-synthetic application based on Ferraro and Miranda (2013)'s water-usage experiment.
Sources
- Multi-Source Causal Inference Using Control Variates
- Introducing the b-value: combining unbiased and biased estimators from a sensitivity analysis perspective
- Cross-Validated Causal Inference: a Modern Method to Combine Experimental and Observational Data
Related papers
- Learning Robust Treatment Rules for Censored Data
- Neural Estimation of Summary Statistics for Simulation-Based Inference with Non-Asymptotic Guarantees
- Spectral divide-and-conquer MCMC for long stationary time series
- Improving the Efficiency of Subgroup Analysis in Randomized Controlled Trials with TMLE
- Personalized Federated Vector Autoregression with Personalization Diversity
- Doubly Robust and Efficient Calibration of Prediction Sets for Right-Censored Time-to-Event Outcomes