Dangers of Bayesian analyses and how to address them
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Statistics. The day in statistics: methods, theory and the inference under them.
Tom: We're looking at the paper: "Dangers of Bayesian analyses and how to address them".
Jane: Here's the gist of the paper. Bayesian analyses are presented as extensions to conventional analyses rather than replacements because this treatment places prior construction and justification at the fore of the transition from frequentist to Bayesian analyses in teaching and practice The paper focuses on the well-known sensitivities of Bayesian results to biases embedded in their prior distributions, calling for special safety features such as detailed justification of informative priors based on empirical-research literature rather than singular expert opinions Pruning redundant reasoning tokens The authors argue that the information contributed by priors should be realistic in light of actual background data rather than merely judged so by available experts They illustrate their recommendations with using a textbook clinical-trial example which covered these points The requirements they impose for Bayesian reporting include that all informative priors must be presented in enough detail to allow reproduction by the reader Furthermore, informative priors must be given a full justification in relation to background topic information with citation to supporting empirical literature Finally, reports should show the impact of a prior by comparing its prior interval, the posterior interval, and the conventional frequentist interval or reference posterior interval derived from the same data model How the training signal is built The paper critiques common priors that violate realism requirements One strategy suggested to aid judgments about priors is to translate them into data to be added to the study data, and this source of prior data should be examined with the same critical attitude as the source of study data The authors critique some common priors that violate their realism requirements They discuss different types of priors, including expert opinions, default priors, and reference priors Expert opinions are susceptible to publication biases and entrenched positions Default priors may appear to be questionable devices to enable computations and statements of Bayesian form Reference priors can exhibit frequentist properties superior to conventional frequentist statistics when the data information is negligible Where the gains come from The paper emphasizes that Bayesian analyses should be treated as extensions or supplements to conventional analyses, rather than replace them A key distinction between frequentist and Bayesian results is that the latter reflect information added in the form of prior distributions on model parameters In most medical applications, Bayesian intervals and one-sided hypothesis probabilities differ non-negligibly from conventional frequentist interval estimates and one-sided P-values to the extent that the prior information used in the analysis is not negligible compared to the data information The authors provide a quantitative way to assess compatibility or conflict between the prior and the data by measuring their relative contribution using variance ratios and effective sample sizes Pruning redundant reasoning tokens The paper suggests that Bayesian analyses should be treated as extensions or supplements to conventional analyses, rather than replace them for that treatment puts prior construction and justification at the fore of the transition from frequentist to Bayesian analyses in teaching and practice The authors require reports of Bayesian analyses to give the priors and their justifications in detail in the main paper Specifically, they demand that rationales for the priors be explained if they contain more than negligible information and have no reasonable derivation from actual external information Reports should show the impact of a prior by comparing its prior interval, the posterior interval, and the conventional frequentist interval or reference posterior interval derived from the same data model How the training signal is built The paper suggests that Bayesian analyses should be treated as extensions or supplements to conventional analyses, rather than replace them for that treatment puts prior construction and justification at the fore of the transition from frequentist to Bayesian analyses in teaching and practice The authors require reports of Bayesian analyses to give the priors and their justifications in detail in the main paper Specifically, they demand that rationales for the priors be explained if they contain more than negligible information and have no reasonable derivation from actual external information Reports should show the impact of a prior by comparing its prior interval, the posterior interval, and the conventional frequentist interval or reference posterior interval derived from the same data model Where the gains come from The paper suggests that Bayesian analyses should be treated as extensions or supplements to conventional analyses, rather than replace them for that treatment puts prior construction and justification at the fore of the transition from frequentist to Bayesian analyses in teaching and practice The authors require reports of Bayesian analyses to give the priors and their justifications in detail in the main paper Specifically, they demand that rationales for the priors be explained if they contain more than negligible information and have no reasonable derivation from actual external information Reports should show the impact of a prior by comparing its prior interval, the posterior interval, and the conventional frequentist interval or reference posterior interval derived from the same data model Where the gains come from
Title and authors: Tom: So we're diving into "Dangers of Bayesian analyses and how to address them" by Sander Greenland and Jason Oke. The title itself sets a serious tone about the problems we need to talk about when we use this approach.
Jane: It sounds like they are not just throwing up some vague warnings, but they're pointing directly at the core issue: those prior distributions that people put in at the beginning of a Bayesian analysis.
Tom: Exactly. It’s not just about using a method; it’s about how we construct that starting point, and if it’s flawed, the whole result can be misleading.
Lu: They focus specifically on the well-known sensitivities of these results to biases embedded in the prior distributions, which is a really important technical detail.
Meng: So, are we talking about just any prior being bad? Because in production AI, we often use priors that are just defaults or based on some easy expert opinion.
Jane: That’s exactly what they're saying; using singular expert opinions isn't enough because those opinions can be biased or entrenched.
Tom: They call for something much more rigorous—detailed justification of informative priors based on actual empirical research, not just what one person says.
The paper's summary: Jane: So, what’s the main message here? Basically, they argue that Bayesian analyses shouldn't be treated as a complete replacement for conventional frequentist methods; they need to be seen as extensions of them.
Tom: Right. They make a key distinction: frequentist results are based only on the data we collect, but Bayesian results include that prior information we add about the parameters beforehand.
Lu: The paper emphasizes that in most medical applications, those prior intervals and one-sided probabilities often differ significantly from what you'd get from just looking at the data alone because the prior info isn't negligible compared to the actual data information.
Meng: That makes sense when you’re trying to deploy a model; if your initial assumptions are wrong, your deployment is going to be shaky.
Jane: They suggest we need quantitative ways to check how compatible or in conflict our prior is with the data information, using things like variance ratios and effective sample sizes.
Tom: And they give some practical requirements for reporting: you have to show the impact of that prior by comparing three intervals—the prior interval, the posterior interval, and that standard frequentist interval—to really show what it's doing.
The paper's improvements: Jane: They lay out some very concrete requirements for how Bayesian analyses should be reported if we want to make them safer.
Tom: It’s not just about being transparent; they demand that all informative priors have to be presented with enough detail so a reader could actually reproduce them.
Lu: And those priors need a full justification connected to the background topic, meaning you have to cite supporting empirical literature for why you chose that specific prior shape or value.
Meng: That sounds like a huge lift for someone shipping models; it means every initial assumption has to be backed by solid external research, not just an expert’s hunch.
Jane: They also suggest a strategy to critique common bad priors: try translating those priors into actual data you would add to the study and then scrutinize that new source of prior data with the same critical eye you'd use for your study data.
Tom: That’s a clever way to test the realism of an assumption, isn't it? And they critique different types like expert opinions or default priors, saying those are all risky.
Conclusion: Tom: So, wrapping up this paper "Dangers of Bayesian analyses and how to address them," the main point is that we need to treat Bayesian results as extensions rather than replacements for frequentist ones.
Jane: The key is prioritizing the construction and justification of priors, making sure they are realistic and backed by empirical research, not just expert consensus.
Lu: If you want to use Bayesian methods safely, you have to show the impact of that prior by comparing its interval to the posterior interval and the frequentist reference interval derived from the same data model.
Meng: From a deployment standpoint, it means we need rigorous checks on our initial assumptions before we even start running simulations at scale.
Lalam: I agree; my literature review shows that when priors are poorly justified, they can introduce significant uncertainty that isn't accounted for by just looking at the raw data.
University of California, Los Angeles · University of Oxford
stat.ME, stat.AP
Submitted: 2026-09-12
Updated: 2026-11-15
Comments: 35 pages, 4 figures
License: http://creativecommons.org/licenses/by-nc-nd/4.0/
Importance score: 91/100
The gist: Bayesian analyses are presented as extensions to conventional analyses rather than replacements because this treatment places prior construction and justification at the fore of the transition from…
Key concepts
- Prior Distributions
- These are the initial beliefs or assumptions about model parameters before seeing any data. The paper warns that these priors can introduce biases if they are based on singular expert opinions or default settings without strong, evidence-based justification.
- Informative Priors Justification
- When using specific prior knowledge, it is crucial to provide detailed justification for those priors. This justification must link the prior directly to background topic information and cite supporting empirical research, rather than relying solely on expert opinions.
- Prior vs Data Contribution
- The paper suggests quantifying how much influence the prior has compared to the actual study data. This comparison can be done using measures like variance ratios and effective sample sizes to assess compatibility or conflict between the assumed prior and observed data.
- Extension vs. Replacement
- Bayesian methods should be viewed as supplements to frequentist analyses, not substitutes for them. The key difference is that Bayesian results incorporate prior information, which can lead to non-negligible differences from traditional frequentist estimates.
Terminology
Summary
Bayesian analyses are presented as extensions to conventional analyses rather than replacements because this treatment places prior construction and justification at the fore of the transition from frequentist to Bayesian analyses in teaching and practice The paper focuses on the well-known sensitivities of Bayesian results to biases embedded in their prior distributions, calling for special safety features such as detailed justification of informative priors based on empirical-research literature rather than singular expert opinions
Pruning redundant reasoning tokens
The authors argue that the information contributed by priors should be realistic in light of actual background data rather than merely judged so by available experts They illustrate their recommendations with using a textbook clinical-trial example which covered these points The requirements they impose for Bayesian reporting include that all informative priors must be presented in enough detail to allow reproduction by the reader Furthermore, informative priors must be given a full justification in relation to background topic information with citation to supporting empirical literature Finally, reports should show the impact of a prior by comparing its prior interval, the posterior interval, and the conventional frequentist interval or reference posterior interval derived from the same data model
How the training signal is built
The paper critiques common priors that violate realism requirements One strategy suggested to aid judgments about priors is to translate them into data to be added to the study data, and this source of prior data should be examined with the same critical attitude as the source of study data The authors critique some common priors that violate their realism requirements They discuss different types of priors, including expert opinions, default priors, and reference priors Expert opinions are susceptible to publication biases and entrenched positions Default priors may appear to be questionable devices to enable computations and statements of Bayesian form Reference priors can exhibit frequentist properties superior to conventional frequentist statistics when the data information is negligible
Where the gains come from
The paper emphasizes that Bayesian analyses should be treated as extensions or supplements to conventional analyses, rather than replace them A key distinction between frequentist and Bayesian results is that the latter reflect information added in the form of prior distributions on model parameters In most medical applications, Bayesian intervals and one-sided hypothesis probabilities differ non-negligibly from conventional frequentist interval estimates and one-sided P-values to the extent that the prior information used in the analysis is not negligible compared to the data information The authors provide a quantitative way to assess compatibility or conflict between the prior and the data by measuring their relative contribution using variance ratios and effective sample sizes
The paper suggests that Bayesian analyses should be treated as extensions or supplements to conventional analyses, rather than replace them for that treatment puts prior construction and justification at the fore of the transition from frequentist to Bayesian analyses in teaching and practice The authors require reports of Bayesian analyses to give the priors and their justifications in detail in the main paper Specifically, they demand that rationales for the priors be explained if they contain more than negligible information and have no reasonable derivation from actual external information Reports should show the impact of a prior by comparing its prior interval, the posterior interval, and the conventional frequentist interval or reference posterior interval derived from the same data model
The paper suggests that Bayesian analyses should be treated as extensions or supplements to conventional analyses, rather than replace them for that treatment puts prior construction and justification at the fore of the transition from frequentist to Bayesian analyses in teaching and practice The authors require reports of Bayesian analyses to give the priors and their justifications in detail in the main paper Specifically, they demand that rationales for the priors be explained if they contain more than negligible information and have no reasonable derivation from actual external information Reports should show the impact of a prior by comparing its prior interval, the posterior interval, and the conventional frequentist interval or reference posterior interval derived from the same data model
The paper suggests that Bayesian analyses should be treated as extensions or supplements to conventional analyses, rather than replace them for that treatment puts prior construction and justification at the fore of the transition from frequentist to Bayesian analyses in teaching and practice The authors require reports of Bayesian analyses to give the priors and their justifications in detail in the main paper Specifically, they demand that rationales for the priors be explained if they contain more than negligible information and have no reasonable derivation from actual external information Reports should show the impact of a prior by comparing its prior interval, the posterior interval, and the conventional frequentist interval or reference posterior interval derived from the same data model
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Prior Construction Justification Requirement: AI systems should be equipped to demand that all informative priors
must be given a full justification in relation to background topic information with citation to supporting empirical literature (if extensive this might be in an appendix, but is essential).
-
Prior vs Data Information Comparison: The improved system can perform quantitative comparison by calculating the information ratio (1/sigma2)/(1/s2) and effective sample sizes (ESS) contributed by the prior versus the data.
-
Sensitivity Analysis Execution: AI systems must be able to conduct priorsensitivity analysis, testing results across a range of priors, such as comparing an expert prior with a completely neutral or in equipoise prior to gauge their relative impact on posterior results.
-
Interpretation of Frequentist vs Bayesian Results: The system should be trained to recognize the numerical differences between frequentist and Bayesian results and understand that
the two approaches can diverge widely in practice
due to priors, rather than assuming they are interchangeable. -
Misinterpretation Detection: The AI should flag common misinterpretations, such as
mistaking a P-value for a hypothesis probability
or confusing a95% confidence interval for the probability the effect falls between the reported interval limits.
Abstract
In light of the US FDA announcement supporting the use of Bayesian methods in clinical trials, we present a nontechnical review of problems with Bayesian analyses and methods to address them. Our focus is on the well-known sensitivities of Bayesian results to biases embedded in their prior distributions. This vulnerability calls for special safety features, including detailed justification of informative priors based on empirical-research literature rather than on singular expert opinions. Crucially, the information contributed by priors should be realistic in light of actual background data rather than merely judged so by available experts. We illustrate our recommendations with using a textbook clinical-trial example which covered these points. We then use the example describe basic diagnostic procedures and safety recommendations for Bayesian analyses and their presentation. These include reference analyses which exclude informative priors, and justification for informative priors based on empirical-research literature rather than on philosophical arguments. We also critique some common priors that violate our realism requirements. One strategy to aid judgments about priors is to translate them into data to be added to the study data. The source of these prior data should be examined with the same critical attitude as the source of study data. We illustrate some simple summary methods for comparing prior and data information, including prior predictions of data, variance ratios, and their translation to effective sample sizes (ESS). Finally, we argue that Bayesian analyses should be treated as extensions or supplements to conventional analyses, rather than replace them, for that treatment puts prior construction and justification at the fore of the transition from frequentist to Bayesian analyses in teaching and practice.
Sources
Related papers
- Learning Robust Treatment Rules for Censored Data
- Neural Estimation of Summary Statistics for Simulation-Based Inference with Non-Asymptotic Guarantees
- Spectral divide-and-conquer MCMC for long stationary time series
- Improving the Efficiency of Subgroup Analysis in Randomized Controlled Trials with TMLE
- Personalized Federated Vector Autoregression with Personalization Diversity
- Doubly Robust and Efficient Calibration of Prediction Sets for Right-Censored Time-to-Event Outcomes