A Fully Specified Bayes Factor Range for Hypothesis Testing and Sensitivity Analysis in Process Tracing
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Statistics. The day in statistics: methods, theory and the inference under them.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: We're looking at the paper: "A Fully Specified Bayes Factor Range for Hypothesis Testing and Sensitivity Analysis in Process Tracing".
Jane: Here's the gist of the paper. In process tracing, researchers seek to quantify how strongly their evidence supports a working theory over a rival theory using Bayes factors derived from a fully specified generative model of observation processes.
Tom: The working theory is defined as the claim that more than half of the evidence supports it (theta > 1/2), while the rival theory claims no more than half does (theta at most 1/2).
Jane: The Bayes factor is defined as the ratio of the probability of observations under the working theory's claim to their probability under the rival's claim.
Tom: The model specifies that each observation independently supports one theory with probability theta and the other with probability 1 - theta, where theta represents the share of evidence supporting the working theory.
Title and authors: Tom: So we're talking about "A Fully Specified Bayes Factor Range for Hypothesis Testing and Sensitivity Analysis in Process Tracing," and the authors are López, Bowers, and Gajardo Cooper.
Jane: They’re tackling that problem where previous methods of setting hand-set probabilities weren't robust enough for rigorous comparison between theories.
Lu: The central idea they present is moving away from those hand-set probabilities by deriving them from a fully specified generative model of observation processes, which gives us two Bayes factors to work with.
Meng: I’m still picturing that model—how exactly does it define every probability needed for the observations to happen in the first place?
Tom: The paper lays out this full model, moving beyond just comparing the two theories and into how observations are generated under each theory.
Jane: They show that this new framework allows researchers to report how much observation bias or re-coding would change a conclusion, which shifts the conversation from "how much do you trust this?" to "how much bias... would it take to change the conclusion?"
Lu: This is a huge shift because they are formalizing claims about the process so we can move from just discussing what's plausible or implausible into discussions about what's probable or improbable.
Meng: If I were deploying this, I’d want to know if this model breaks at scale, but for now, it seems like a strong way to get a number behind those subjective assessments.
Tom: And they do include some applications in six published studies, showing how these factors perform across different real-world scenarios.
Jane: It’s interesting that for two of those studies, both Bayes factors exceed twenty, while for the other four, the uniform weights don't reach that threshold at all.
Lu: That contrast is what makes it so rich; it shows where this method is working and where it might be hitting a wall with less evidence.
The paper's summary: Tom: Okay, to summarize the paper, they’re moving past older methods that assign probabilities based on the subjective probative value of each observation.
Jane: Instead, they use a fully specified model of observation processes where each observation independently supports one theory with probability theta.
Lu: This allows them to derive two key Bayes factors: the uniform-weights Bayes factor and the worst-case Bayes factor that grants the rival more benefit of doubt.
Meng: So, one is an average across all consistent values of theta, and the other is just checking against a single, most favorable point for the rival.
Tom: And they use this structure to formalize how we turn those vague claims like "more than half" into actual probabilities for comparison.
Jane: They propose two ways to do that: one that spreads the rival’s claim evenly over all possible values it allows, which is the uniform-weights Bayes factor.
Lu: And the other way gives you a single value based on where the actual observations are most probable under that rival theory's constraints.
Tom: They then use these factors to test how much observation bias or re-coding would shift a conclusion, using this new framework for sensitivity analysis.
Jane: This is really about making sure that when we talk about evidence, we are talking about the actual process of how evidence is made, not just a subjective calibration.
Lu: It’s formalizing claims so that discussions move from "plausible" to "probable," which is a necessary step for serious hypothesis testing.
The paper's improvements: Tom: The improvements they propose are pretty substantial because they let us compute these fully specified Bayes factors directly from a generative model of observation, instead of just relying on hand-set probabilities.
Jane: So, the first improvement is moving to a system that computes these for hypothesis testing by taking a generative model, like a binomial model of observation.
Lu: That means we're no longer stuck with guessing the probabilities; we can use formal models to define exactly what observations are possible and how they scale.
Meng: I like the idea of this because it lets us move beyond just calibration and state conclusions based on sensitivity analysis, which is something I really want to do in a production setting.
Tom: And that second improvement is calculating the worst-case Bayes factor as a lower bound, which is proven to be at least two point seven three for the example we looked at.
Jane: That guaranteed minimum value against any rival proponent's weights is a really important piece of evidence showing it has some mathematical robustness.
Lu: Plus, they can report the separation between theories when that worst-case Bayes factor is below twenty, which tells us exactly how far apart the claims about the evidence base would have to be to reach a twenty to one ratio.
Tom: And finally, they can incorporate probabilistic weights by treating them as effective replication, which computes both Bayes factors as if a weight w copies of an observation were observed.
Meng: That’s interesting; so we can argue about how many effective replications a piece of evidence represents, which helps determine how far search bias must go before a conclusion is overturned.
Conclusion: Tom: So, to wrap up the paper, the main implication is that with this fully specified Bayes factor range for hypothesis testing and sensitivity analysis in process tracing, we can conclude something for one hypothesis by comparing it to a threshold.
Jane: The key is reporting beside that conclusion what bias or re-coding or change of weight would reverse it. It shifts the conversation from "how much do you trust this?" to "how much bias... would it take to change the conclusion?"
Lu: Essentially, they're giving us a computable way to assess how much bias or change in weight is required to reverse a conclusion.
Meng: I just think that making the framework computable, instead of just theoretical, is what makes it deployable for real-world use.
Lalam: I’m confident this paper provides a formal structure for comparing theories based on evidence processes, and the online supplement gives formal details for the generative model and both Bayes factors.
Tom: So, we've looked at how these two Bayes factors compare—the uniform-weights factor is about twenty point six seven while the worst-case is about two point seven three.
Jane: It shows that a separation between theories must be quite large to get that twenty to one evidence ratio under this new framework.
Lu: The paper, "A Fully Specified Bayes Factor Range for Hypothesis Testing and Sensitivity Analysis in Process Tracing," offers a computable way to assess how much bias or change in weight would reverse a conclusion.
Matias Lopez, Jake Bowers, Daniel Gajardo Cooper
University of Illinois Urbana-Champaign · Pontificia Universidad Catolica de Chile
stat.ME, stat.OT
Submitted: 2026-06-15
Updated: 2026-11-06
Comments: Title and abstract revised, and the paper and supplement shortened to 59 pages. As in v2, the hypergeometric urn model of v1 is removed: the paper develops one binomial model yielding two Bayes factors and reanalyzes six published studies
Code: https://github.com/bowers-illinois-edu/DrWrinch
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Importance score: 85/100
The gist: In process tracing, researchers seek to quantify how strongly their evidence supports a working theory over a rival theory using Bayes factors derived from a fully specified generative model of…
Key concepts
- Bayes Factor
- A ratio comparing the probability of observing data under one theory versus another. It quantifies how much more strongly one theory's evidence supports it compared to a rival theory.
- Uniform-Weights Bayes Factor
- This factor averages the evidence across all possible values of support ($ heta$) for each theory, treating every possibility equally. For nine observations favoring the working theory and three favoring the rival, this yields a value of approximately 20.67.
- Worst-Case Bayes Factor
- This factor evaluates the rival's claim using only the single $ heta$ value most favorable to them (their best case). It serves as a guaranteed lower bound for any proponent of the rival theory, reported around 2.73.
- Sensitivity Analysis
- A method to test how robust a conclusion is by quantifying the required change in observation bias, re-coding, or smoking-gun weight needed to reverse it. This shifts focus from trusting evidence calibration to understanding necessary bias levels.
Terminology
Summary
In process tracing, researchers seek to quantify how strongly their evidence supports a working theory over a rival theory using Bayes factors derived from a fully specified generative model of observation processes. p. 0 The working theory is defined as the claim that more than half of the evidence supports it (theta > 1/2), while the rival theory claims no more than half does (theta at most 1/2). p. 2 The Bayes factor is defined as the ratio of the probability of observations under the working theory's claim to their probability under the rival's claim.
The model specifies that each observation independently supports one theory with probability theta and the other with probability 1 - theta, where theta represents the share of evidence supporting the working theory. p. 7 The paper derives two key Bayes factors from this model:
-
The Uniform-Weights Bayes Factor: This factor averages the probability of counts across all values of theta consistent with each theory's range, using equal weights for each value. For a specific instance with nine observations favoring the working theory and three favoring the rival, this yields a value of approximately 20. p. 767.
-
The Worst-Case Bayes Factor: This factor evaluates the rival’s claim at the single value of theta most favorable to it (the rival's best case). This is shown to be a lower bound on the Bayes factor against any proponent of the rival theory and is reported as approximately 2. p. 373 for nine observations favoring the working theory and three favoring the rival. p. 7
The framework incorporates sensitivity analysis by allowing researchers to compute how much observation bias, re-coding, or a smoking-gun weight would change a conclusion. p. 0 A decision threshold of Bayes factor at least 20 is adopted for setting aside the rival theory as strong evidence. p. 3 The model formalizes claims about the process by specifying how observations are made, enabling discussions about plausible
or implausible
into discussions of probable
or improbable.
The worst-case Bayes factor provides a crucial bound: it is at least 2.73 whenever the count supporting the working theory is at least as large as the count supporting the rival, and it bounds how often a researcher who sets aside the rival makes a mistake (a researcher who treats 20 as the value at which to set the rival aside errs at most one time in twenty when the rival is right
). The uniform-weights Bayes factor of 20. p. 267 sits just above this threshold, while the worst-case Bayes factor of 2. p. 273 falls well short. p. 0
The framework allows researchers to transition from relying solely on evidence calibration to making conclusions based on sensitivity analysis by quantifying how much bias or change in weight is required to alter a conclusion. p. 0 The model formalizes claims about the process by allowing discussions about plausible
or implausible
into discussions of probable
or improbable.
Furthermore, the paper demonstrates how to incorporate probative weights (e.g., assigning a weight to a smoking gun) by treating it as effective replication, where weighted counts replace unit counts in the Bayes factor formulas, showing that adding weight can substantially increase both Bayes factors and helping determine how far search bias must go before a conclusion is overturned. p. 0
Limitations include not modeling dependence between observations and leaving coding to the researcher's judgment. However, the framework provides a computable way to assess how much bias or change in weight would reverse a conclusion. p. 0 The worst-case Bayes factor is constructed so that it cannot overstate the case for H1 against any weights the rival’s proponent might hold, while the uniform-weights Bayes factor is not bounded. p. 3 The separation between these two factors measures how far apart the theories' claims about the evidence base must be to reach a 20-to-1 evidence ratio.
The paper concludes that with a fully specified Bayes factor, a researcher can conclude for one hypothesis by comparing it to a threshold and report beside that conclusion what bias, re-coding, or change of weight would reverse it, shifting the conversation from how much do you trust this?
to how much bias... would it take to change the conclusion? p. 4
The Online Supplement provides formal details for the generative model and both Bayes factors, along with applications to six published studies. Results indicate that for two studies, both Bayes factors exceed 20; for three studies, uniform weights yield values above 20; and for Pavone & Stiansen, no Bayes factor reaches 20. p. 0 The worst-case Bayes factor is consistently smaller than the uniform-weights one in every row. p. 3
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
The improved system can compute fully specified Bayes factors for hypothesis testing by taking a generative model of observation that produces every probability needed, moving beyond hand-set probabilities to formal models like
A binomial model of observation.
-
The system can perform sensitivity analysis by asking how much
observation bias, re-coding, or a smoking-gun weight would change a conclusion,
allowing researchers to move from reliance on calibration to stating conclusions on the grounds of sensitivity analysis. -
The improved system can calculate the worst-case Bayes factor as a lower bound, which is proven
at least 2.73
for the running example, providing a guaranteed minimum value against any rival proponent's weights. -
The system can report a separation between theories when the worst-case Bayes factor is below 20, determining how far apart the claims about the evidence base would have to be for a conclusion to reach 20-to-1 evidence.
-
The system can incorporate probabilistic weights into Bayes factors by treating them as
effective replication,
computing both Bayes factors as if a weight w copies of an observation were observed, thereby allowing researchers to argue abouthow many effective replications a piece of evidence represents.
Related papers
- Learning Robust Treatment Rules for Censored Data
- Neural Estimation of Summary Statistics for Simulation-Based Inference with Non-Asymptotic Guarantees
- Spectral divide-and-conquer MCMC for long stationary time series
- Improving the Efficiency of Subgroup Analysis in Randomized Controlled Trials with TMLE
- Personalized Federated Vector Autoregression with Personalization Diversity
- Doubly Robust and Efficient Calibration of Prediction Sets for Right-Censored Time-to-Event Outcomes