A Fully Specified Bayes Factor Range for Hypothesis Testing and Sensitivity Analysis in Process Tracing

summary

The gist

In process tracing, researchers seek to quantify how strongly their evidence supports a working theory over a rival theory using Bayes factors derived from a fully specified generative model of…

In short

The paper develops a framework using Bayes factors from a fully specified generative model of observation processes to quantify evidence strength between working and rival theories. It introduces two key factors: Uniform-Weights (around 20.67) and Worst-Case (around 2.73). This allows researchers to use sensitivity analysis, determining how much bias or weight change is needed to overturn a conclusion.

Key concepts

Bayes Factor
A ratio comparing the probability of observing data under one theory versus another. It quantifies how much more strongly one theory's evidence supports it compared to a rival theory.
Uniform-Weights Bayes Factor
This factor averages the evidence across all possible values of support ($ heta$) for each theory, treating every possibility equally. For nine observations favoring the working theory and three favoring the rival, this yields a value of approximately 20.67.
Worst-Case Bayes Factor
This factor evaluates the rival's claim using only the single $ heta$ value most favorable to them (their best case). It serves as a guaranteed lower bound for any proponent of the rival theory, reported around 2.73.
Sensitivity Analysis
A method to test how robust a conclusion is by quantifying the required change in observation bias, re-coding, or smoking-gun weight needed to reverse it. This shifts focus from trusting evidence calibration to understanding necessary bias levels.

Terminology used across episodes

This episode discusses

The paper

A Fully Specified Bayes Factor Range for Hypothesis Testing and Sensitivity Analysis in Process Tracing · Read on arXiv

Matias Lopez, Jake Bowers, Daniel Gajardo Cooper

University of Illinois Urbana-Champaign · Pontificia Universidad Catolica de Chile

Transcript

Introduction to the show: ident: Statistics. The day in statistics: methods, theory and the inference under them.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "A Fully Specified Bayes Factor Range for Hypothesis Testing and Sensitivity Analysis in Process Tracing".

Jane: Here's the gist of the paper. In process tracing, researchers seek to quantify how strongly their evidence supports a working theory over a rival theory using Bayes factors derived from a fully specified generative model of observation processes.

Tom: The working theory is defined as the claim that more than half of the evidence supports it (theta > 1/2), while the rival theory claims no more than half does (theta at most 1/2).

Jane: The Bayes factor is defined as the ratio of the probability of observations under the working theory's claim to their probability under the rival's claim.

Tom: The model specifies that each observation independently supports one theory with probability theta and the other with probability 1 - theta, where theta represents the share of evidence supporting the working theory.

Title and authors: Tom: So we're talking about "A Fully Specified Bayes Factor Range for Hypothesis Testing and Sensitivity Analysis in Process Tracing," and the authors are López, Bowers, and Gajardo Cooper.

Jane: They’re tackling that problem where previous methods of setting hand-set probabilities weren't robust enough for rigorous comparison between theories.

Lu: The central idea they present is moving away from those hand-set probabilities by deriving them from a fully specified generative model of observation processes, which gives us two Bayes factors to work with.

Meng: I’m still picturing that model—how exactly does it define every probability needed for the observations to happen in the first place?

Tom: The paper lays out this full model, moving beyond just comparing the two theories and into how observations are generated under each theory.

Jane: They show that this new framework allows researchers to report how much observation bias or re-coding would change a conclusion, which shifts the conversation from "how much do you trust this?" to "how much bias... would it take to change the conclusion?"

Lu: This is a huge shift because they are formalizing claims about the process so we can move from just discussing what's plausible or implausible into discussions about what's probable or improbable.

Meng: If I were deploying this, I’d want to know if this model breaks at scale, but for now, it seems like a strong way to get a number behind those subjective assessments.

Tom: And they do include some applications in six published studies, showing how these factors perform across different real-world scenarios.

Jane: It’s interesting that for two of those studies, both Bayes factors exceed twenty, while for the other four, the uniform weights don't reach that threshold at all.

Lu: That contrast is what makes it so rich; it shows where this method is working and where it might be hitting a wall with less evidence.

The paper's summary: Tom: Okay, to summarize the paper, they’re moving past older methods that assign probabilities based on the subjective probative value of each observation.

Jane: Instead, they use a fully specified model of observation processes where each observation independently supports one theory with probability theta.

Lu: This allows them to derive two key Bayes factors: the uniform-weights Bayes factor and the worst-case Bayes factor that grants the rival more benefit of doubt.

Meng: So, one is an average across all consistent values of theta, and the other is just checking against a single, most favorable point for the rival.

Tom: And they use this structure to formalize how we turn those vague claims like "more than half" into actual probabilities for comparison.

Jane: They propose two ways to do that: one that spreads the rival’s claim evenly over all possible values it allows, which is the uniform-weights Bayes factor.

Lu: And the other way gives you a single value based on where the actual observations are most probable under that rival theory's constraints.

Tom: They then use these factors to test how much observation bias or re-coding would shift a conclusion, using this new framework for sensitivity analysis.

Jane: This is really about making sure that when we talk about evidence, we are talking about the actual process of how evidence is made, not just a subjective calibration.

Lu: It’s formalizing claims so that discussions move from "plausible" to "probable," which is a necessary step for serious hypothesis testing.

The paper's improvements: Tom: The improvements they propose are pretty substantial because they let us compute these fully specified Bayes factors directly from a generative model of observation, instead of just relying on hand-set probabilities.

Jane: So, the first improvement is moving to a system that computes these for hypothesis testing by taking a generative model, like a binomial model of observation.

Lu: That means we're no longer stuck with guessing the probabilities; we can use formal models to define exactly what observations are possible and how they scale.

Meng: I like the idea of this because it lets us move beyond just calibration and state conclusions based on sensitivity analysis, which is something I really want to do in a production setting.

Tom: And that second improvement is calculating the worst-case Bayes factor as a lower bound, which is proven to be at least two point seven three for the example we looked at.

Jane: That guaranteed minimum value against any rival proponent's weights is a really important piece of evidence showing it has some mathematical robustness.

Lu: Plus, they can report the separation between theories when that worst-case Bayes factor is below twenty, which tells us exactly how far apart the claims about the evidence base would have to be to reach a twenty to one ratio.

Tom: And finally, they can incorporate probabilistic weights by treating them as effective replication, which computes both Bayes factors as if a weight w copies of an observation were observed.

Meng: That’s interesting; so we can argue about how many effective replications a piece of evidence represents, which helps determine how far search bias must go before a conclusion is overturned.

Conclusion: Tom: So, to wrap up the paper, the main implication is that with this fully specified Bayes factor range for hypothesis testing and sensitivity analysis in process tracing, we can conclude something for one hypothesis by comparing it to a threshold.

Jane: The key is reporting beside that conclusion what bias or re-coding or change of weight would reverse it. It shifts the conversation from "how much do you trust this?" to "how much bias... would it take to change the conclusion?"

Lu: Essentially, they're giving us a computable way to assess how much bias or change in weight is required to reverse a conclusion.

Meng: I just think that making the framework computable, instead of just theoretical, is what makes it deployable for real-world use.

Lalam: I’m confident this paper provides a formal structure for comparing theories based on evidence processes, and the online supplement gives formal details for the generative model and both Bayes factors.

Tom: So, we've looked at how these two Bayes factors compare—the uniform-weights factor is about twenty point six seven while the worst-case is about two point seven three.

Jane: It shows that a separation between theories must be quite large to get that twenty to one evidence ratio under this new framework.

Lu: The paper, "A Fully Specified Bayes Factor Range for Hypothesis Testing and Sensitivity Analysis in Process Tracing," offers a computable way to assess how much bias or change in weight would reverse a conclusion.

More episodes

← Home