The Illusion of Learning from Observational Data: An Empirical Bayes Perspective

arXiv:2604.08853 · stat.ME · Submitted 2026-04-10 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Statistics. The day in statistics: methods, theory and the inference under them.

Tom: We're looking at the paper: "The Illusion of Learning from Observational Data".

Jane: Here's the gist of the paper. Randomized experiments have long been the gold standard for scientists seeking to learn about cause and effect.

Tom: To shatter this illusion, the authors take an empirical Bayes perspective, showing that calibration studies can be used to learn the distribution of observational biases and allow observational studies to inform causal parameter estimation via empirical Bayes shrinkage.

Jane: The Illusion of Learning The paper formalizes the illusion of learning from observational data in a hierarchical Gaussian model for one experimental estimate and many observational estimates.

Tom: When assuming flat priors on both the effect and the biases, the posterior mean remains equal to the experimental estimate, and the posterior variance does not change, meaning observational studies offer no information beyond what is learned from the experiment alone.

Title and authors: Tom: So what does this paper actually do to break that illusion? They introduce empirical Bayes as a way to use those observational studies meaningfully, and they show that by applying it specifically to the variance of the biases, you can successfully break the illusion.

Jane: But wait, they run into a problem if you try to be too specific with what you know about those biases; assuming zero-mean bias is too strong because that brings back the illusion where observational studies just increase your overconfidence instead of helping.

Lu: That's a crucial distinction they make; fitting both the mean and variance of the biases using empirical Bayes brings the illusion right back because, in that case, those observational studies still don't actually inform your final estimate.

Meng: So so if you just use empirical Bayes on one part of the bias distribution, it helps but not fully? That means we have to be careful about which assumptions we make when trying to combine these different types of evidence.

Lalam: It suggests that simply applying a standard shrinkage technique isn't enough; you need a way to learn something about the shape of those biases across the studies.

The paper's summary: Tom: To fix this, they introduce calibration studies, which they use to target a known zero causal effect and figure out the actual distribution of biases, which then allows observational studies to improve estimation when you use calibrated empirical Bayes.

Jane: So calibration studies are the key here because they give us that information about the bias distribution that we were missing before, and with it, applying empirical Bayes lets those observational studies actually contribute to estimating the causal effect.

Lu: The main contributions of this paper are formalizing the illusion in a hierarchical Gaussian model, showing that empirical Bayes on variance breaks the illusion but requires knowing the mean bias distribution, and then showing that calibration studies restore learning by identifying that bias distribution so observational studies can improve estimation.

Meng: It sounds like they're proposing a calibrated empirical Bayes procedure where you use calibration studies to fit a prior over biases, and then you use that fitted prior to combine the experimental estimate with the observational estimates, which leads to vanishing risk as both observational and calibration studies grow.

Lalam: The core idea is that triangulation is necessary here; you need to learn something about how biases are distributed across all those observational studies for them to contribute meaningfully.

The paper's improvements: Tom: Beyond just breaking the illusion, the authors suggest some methodological extensions, moving beyond normal priors and into nonparametric priors, and even looking at regression settings instead of just simple one-experimental estimates.

Jane: And they also address resource allocation—how you can minimize the posterior variance of that target causal effect while balancing your budget between experimental studies, observational studies, and calibration studies.

Lu: They go further by suggesting extensions to nonparametric priors and moving beyond the normal means model into regression settings, which shows the idea can be applied in more complex scenarios than just a single effect estimation.

Meng: From my side, I'm interested in the resource allocation part; figuring out how many of each study type to run given limited resources is something that actually matters for deploying these kinds of models at scale.

Lalam: And they also discuss using moment matching estimators for estimating bias distribution parameters, like gamma two and mu, which the paper shows can satisfy risk consistency under certain conditions when you use more complex models than a simple normal prior.

Conclusion: Tom: So to wrap up, the main point is that to actually gain from combining experimental and observational methods, you have to assume or learn something concrete about how those biases are distributed across your observational studies.

Jane: Calibration studies are especially valuable because they quantify the bias of an observational research approach, which enables that approach to contribute meaningfully to estimating the target causal parameter when paired with calibrated empirical Bayes.

Lu: The strategy depends on calibration studies being subject to the same population of biases as your target observational studies, and they also provide a scalable approximate solution for resource allocation using a knapsack problem structure.

Meng: And those moment matching estimators they show are consistent in the L1 sense, which means even when you use more complex models for estimating those bias distribution parameters, you can still trust the posterior to converge to the true parameter.

Lalam: So ultimately, this paper on "The Illusion of Learning from Observational Data: An Empirical Bayes Perspective" shows that triangulation requires learning about the distribution of biases across observational studies, and it gives us a way to do that while optimizing how we allocate our research budget.

Columbia University

stat.ME

Submitted: 2026-04-10

Updated: 2026-11-05

Comments: arXiv admin note: This submission has been withdrawn because it does not meet arXiv's research content quality standards

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 90/100

The gist: Randomized experiments have long been the gold standard for scientists seeking to learn about cause and effect.

Key concepts

Illusion of Learning
When assuming flat priors on both the true effect and observational biases, the posterior mean remains fixed at the experimental estimate. This means observational studies provide no new information about the causal quantity of interest beyond what is learned from a single experiment alone, creating a misleading sense of improvement.
Empirical Bayes Shrinkage
This technique uses prior knowledge to adjust estimates from observational data. By learning the distribution of biases through calibration studies, shrinkage reduces the posterior variance and shifts the estimate towards the true causal effect, making observational studies informative.
Calibration Studies
These are experiments specifically designed to target a known zero causal effect. They serve as a tool to reveal how observational biases are distributed across different research approaches. This knowledge of bias is crucial for using empirical Bayes effectively in observational settings.

Terminology

Summary

Randomized experiments have long been the gold standard for scientists seeking to learn about cause and effect. p. 0

Where the paper says it

edu 4Departments of Computer Science and Statistics, Columbia University, e-mail: blei@cs. edu Abstract: Randomized experiments have long been the gold standard for scientists seeking to learn about cause and effect.

To shatter this illusion, the authors take an empirical Bayes perspective, showing that calibration studies can be used to learn the distribution of observational biases and allow observational studies to inform causal parameter estimation via empirical Bayes shrinkage. p. 0
Where the paper says it

We show that the distribution of observational biases can be learned from calibration studies—experiments that target a causal effect that is known a priori to be zero. Calibration identifies the distribution of observational bias and allows observational studies to inform the estimation of causal parameters via empirical Bayes shrinkage.

The Illusion of Learning

The paper formalizes the illusion of learning from observational data in a hierarchical Gaussian model for one experimental estimate and many observational estimates. When assuming flat priors on both the effect and the biases, the posterior mean remains equal to the experimental estimate, and the posterior variance does not change, meaning observational studies offer no information beyond what is learned from the experiment alone. p. 4

Where the paper says it

As the next theorem shows, with flat priors on both the effect and the biases, E [θ | ye, yo] = ye and the posterior variance remains the same, Var [θ | ye] = σ 2 e . The observational studies offer no information beyond the experiment.

p. 4
Where the paper says it

As the next theorem shows, with flat priors on both the effect and the biases, E [θ | ye, yo] = ye and the posterior variance remains the same, Var [θ | ye] = σ 2 e . The observational studies offer no information beyond the experiment.

This demonstrates that without knowing how observational biases are distributed, they create an illusion of learning where they do not improve inference about the causal quantity of interest. p. 1
Where the paper says it

, when the prior is flat, the observational studies receive no weight in the posterior for the target causal effect. In this sense, the observational studies create only an illusion of learning: they appear relevant but they do not improve inference about the causal quantity of interest.

Breaking the Illusion with Empirical Bayes

The authors show that applying empirical Bayes to the variance of the biases successfully breaks the illusion, as observational studies adjust the posterior expectation and reduce posterior variance, leading to vanishing risk as more observational studies are seen. However, assuming zero-mean bias is too strong; when fitting both the mean and variance of the biases using empirical Bayes, the illusion returns because observational studies do not inform the estimate and only increase overconfidence. p. 2

Where the paper says it

Third, we show that applying empirical Bayes to both the mean and variance of the biases brings back the illusion, where the observational studies do not inform the estimate. Finally, we show that calibration studies restore the possibility of learning: they identify the bias distribution such that observational studies can improve estimation of the causal effect.

The Role of Calibration Studies

To break the illusion without making unrealistic assumptions about bias distributions, calibration studies are introduced, which target a known zero causal effect to reveal the distribution of biases. With calibrated empirical Bayes, the causal estimator achieves vanishing risk as both observational and calibration studies grow. p. 2

Where the paper says it

Finally, we show that calibration studies restore the possibility of learning: they identify the bias distribution such that observational studies can improve estimation of the causal effect. With calibrated empirical Bayes, the causal estimator achieves vanishing risk as the number of observational and calibration studies grows.

p. 2
Where the paper says it

Finally, we show that calibration studies restore the possibility of learning: they identify the bias distribution such that observational studies can improve estimation of the causal effect. With calibrated empirical Bayes, the causal estimator achieves vanishing risk as the number of observational and calibration studies grows.

Simulations illustrate that the calibrated EB estimator improves precision relative to the experiment alone, enabling observational studies to contribute meaningfully to causal estimation. p. 2
Where the paper says it

We study this method in theory, simulation, and a semi-synthetic analysis based on a large field experiment. Calibrated empirical Bayes improves precision relative to the experiment alone, enabling observational studies to contribute meaningfully to the estimation of a causal effect.

Methodological Extensions

The work extends beyond normal priors by suggesting extensions to nonparametric priors and moving beyond the normal means model into regression settings. The resource allocation problem is also addressed, showing how to minimize the posterior variance of the causal effect subject to a budget constraint between experimental, observational, and calibration studies. p. 1

Where the paper says it

The observational studies do not move the posterior mean of the causal effect away from the experimental estimate. Even worse, they make the posterior variance smaller, creating unwarranted confidence in the experimental results.

Real Data Application

A semi-synthetic analysis based on a water-usage experiment is used to illustrate these findings, where the calibrated EB estimator achieves an estimate close to the oracle ATE and yields well-calibrated uncertainty compared to the illusion model. This demonstrates that calibration studies can unlock otherwise uninterpretable research findings in domains where observational studies abound. p. 9

Where the paper says it

Simulations Simulations help illustrate the theoretical findings in Section 3. In particular, we show that incorporating calibration studies leads to improved statistical efficiency and accuracy of causal effect estimation when combining experimental and observational studies, consistent with Theorem 4.

p. 9
Where the paper says it

Simulations Simulations help illustrate the theoretical findings in Section 3. In particular, we show that incorporating calibration studies leads to improved statistical efficiency and accuracy of causal effect estimation when combining experimental and observational studies, consistent with Theorem 4.

The final results show that the calibrated EB posterior centers near the oracle estimate and yields a tighter confidence interval than the illusion model, demonstrating accurate point estimation and calibrated uncertainty. p. 3
Where the paper says it

6) With this model, we estimate the causal effect through its posterior p(θ | ye, yo). The posterior mean ˆθ provides a point estimate; the posterior variance vˆ provides (model-based) uncertainty.

Conclusion on Triangulation

The analysis clarifies that to gain from combining methods, one must assume or learn something about how biases are distributed across observational studies. Calibration studies are especially valuable because they quantify the bias of an observational research approach, enabling that approach to contribute meaningfully to estimating the target causal parameter. p. 0

Where the paper says it

Observational studies are widely available and often large, but they rely on untestable assumptions that, when violated, may result in biased estimates. Uncertainty about bias leads to a phenomenon known as the illusion of learning from observational research (Gerber, Green and Kaplan , 2004a): absent prior information about bias, observational results cannot meaningfully contribute to the estimation of a causal parameter.

The strategy depends on calibration studies being subject to the same population of biases as the target observational studies. p. 0
Where the paper says it

To shatter the illusion, we take an empirical Bayes perspective. We show that the distribution of observational biases can be learned from calibration studies—experiments that target a causal effect that is known a priori to be zero.

The paper suggests that triangulation requires learning about the distribution of biases across observational studies. p. 2
Where the paper says it

First, we formalize the illusion of learning from observational data in a hierarchical Gaussian model for one experimental estimate and many observational estimates. Second, we show that applying empirical Bayes to the variance of the biases successfully breaks the illusion but requires the unrealistic assumption that we know their mean.

The authors also provide a scalable approximate solution for resource allocation between different study types using knapsack problem structure. p. 1
Where the paper says it

Because their true effect is known, any nonzero estimate from a calibration study can be attributed to its bias. Calibration studies help solve our problem because they provide direct information about the distribution of biases associated with a given observational design.

The moment-matching estimators used in the calibration studies are shown to be consistent in the L1 sense, ensuring convergence of the empirical Bayes posterior to the true parameter. p. 9
Where the paper says it

In particular, we show that incorporating calibration studies leads to improved statistical efficiency and accuracy of causal effect estimation when combining experimental and observational studies, consistent with Theorem 4. Setup and estimators.

--- Page 1 ---

The Illusion of Learning from Observational Data: An Empirical Bayes Perspective

Randomized experiments have long been the gold standard for scientists seeking to learn about cause and effect. To shatter this illusion, we take an empirical Bayes perspective, showing that calibration studies can be used to learn the distribution of observational biases and allow observational studies to inform causal parameter estimation via empirical Bayes shrinkage. p. 0

Where the paper says it

We show that the distribution of observational biases can be learned from calibration studies—experiments that target a causal effect that is known a priori to be zero. Calibration identifies the distribution of observational bias and allows observational studies to inform the estimation of causal parameters via empirical Bayes shrinkage.

The authors formalize the illusion of learning from observational data in a hierarchical Gaussian model for one experimental estimate and many observational estimates. When assuming flat priors on both the effect and the biases, the posterior mean remains equal to the experimental estimate, and the posterior variance does not change, meaning observational studies offer no information beyond what is learned from the experiment alone.

To break the illusion without making unrealistic assumptions about bias distributions, calibration studies are introduced, which target a known zero causal effect to reveal the distribution of biases. With calibrated empirical Bayes, the causal estimator achieves vanishing risk as both observational and calibration studies grow.

A semi-synthetic analysis based on a water-usage experiment is used to illustrate these findings, where the calibrated EB estimator achieves an estimate close to the oracle ATE and yields well-calibrated uncertainty compared to the illusion model. This demonstrates that calibration studies can unlock otherwise uninterpretable research findings in domains where observational studies abound.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Empirical Bayes shrinkage for causal inference allows observational studies to contribute meaningfully to causal parameter estimation by learning a prior distribution of observational biases from calibration studies. This can be applied in AI systems that rely on combining experimental data with large sets of observational data, improving precision when the bias distribution is known or estimated.

  2. The system can implement a calibrated empirical Bayes procedure for causal estimation by fitting a prior over biases using calibration studies and then applying this prior to combine the experimental estimate with observational estimates, achieving vanishing risk as the number of observational and calibration studies grows. This allows AI models to leverage vast, but potentially biased, real-world data more effectively.

  3. The system can use internal calibration studies paired with specific observational studies to directly estimate study-specific biases rather than inferring them only through a prior. This capability enables the estimation of the causal effect within each study rather than just for a broader meta-analytic superpopulation, leading to more accurate causal inference from observational data.

  4. The system can perform resource allocation between experimental, observational, and calibration studies by minimizing the posterior variance of the target causal effect subject to a budget constraint. This provides an optimized strategy for AI research teams deciding how many of each study type to conduct given limited resources.

  5. The system can utilize moment matching estimators for estimating bias distribution parameters, such as gamma 2 and mu, which are shown to satisfy risk consistency under certain conditions. This allows the AI system to robustly estimate the unknown distribution of observational biases even when more complex models than a simple normal prior are used.

Abstract

Randomized experiments have long been the gold standard for scientists seeking to learn about cause and effect. When randomized experiments are infeasible, scientists often resort to observational studies, which are widely available and often large but rely on untestable assumptions that, when violated, may result in biased estimates. Uncertainty about bias leads to a phenomenon known as the illusion of learning from observational research (Gerber, Green and Kaplan, 2004a): absent prior information about bias, observational results cannot meaningfully contribute to the estimation of a causal parameter. To shatter the illusion, we take an empirical Bayes perspective. We show that the distribution of observational biases can be learned from calibration studies-experiments that target a causal effect that is known a priori to be zero. Calibration identifies the distribution of observational bias and allows observational studies to inform the estimation of causal parameters via empirical Bayes shrinkage. We formalize the illusion phenomenon in an empirical Bayes setting and show that, with an increasing number of calibration and observation studies, both the bias distribution and the causal effect can be consistently recovered. We illustrate our method through a simulation study and a semi-synthetic application based on Ferraro and Miranda (2013)'s water-usage experiment.

Sources

Related papers