Comparing randomized trials: a formal approach
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Statistics. The day in statistics: methods, theory and the inference under them.
Tom: We're looking at the paper: "Comparing randomized trials: a formal approach".
Jane: Here's the gist of the paper. Disagreements between randomized trials that address the same causal question pose challenges for the interpretation and synthesis of evidence<ref:2610.14895#pg2>.
Tom: This DRAFT presents WORK IN PROGRESS<ref:2610.14895#pg2>.
Jane: Causal Framework for Comparison The paper sketches a causal framework for comparing randomized trials that explicitly accommodates the possibility of trial engagement effects<ref:2610.14895#pg4>.
Tom: Using counterfactual outcomes defined under joint interventions on treatment assignment and trial-related actions, it compares trialnative treatment effects with the effects transported from one trial to the population of another in a common target population<ref:2610.14895#pg4>.
Title and authors: Tom: So we're looking at this paper today called "Comparing randomized trials: a formal approach." Jane, what are you seeing on the title and authors?
Jane: It's about comparing different randomized trials, and the authors are Yu-Han Chiu and Issa J. Dahabreh. It seems like they’re tackling how we can actually use these studies together when they try to answer the same causal question.
Tom: That sounds like a real headache for model training, trying to combine noisy data from different sources. What exactly is the core problem they're addressing with this paper?
Jane: The summary says that disagreements between randomized trials, even when they are testing the exact same thing, create big challenges for interpreting and putting all the evidence together. They are trying to sketch a way to compare them that accounts for something important.
Lu: What’s important is they are building a causal framework where you can explicitly allow for trial engagement effects. That means they aren't just assuming treatment assignment is the only thing affecting outcomes, which is often not true in real-world trials.
Meng: Trial engagement effects sound like all that extra stuff—monitoring, counseling, or ancillary interventions—that happens because someone is participating in a study and not just getting the treatment. But how do you actually measure that without running an infinite number of follow-up studies?
Lalam: The paper suggests they use counterfactual outcomes defined by joint interventions on both the treatment assignment and those trial-related actions to compare trial-native effects with effects transported across trials.
Tom: Transported effects, right? So they're looking at how an effect seen in one trial actually moves over to a different population when you look at a second trial. That’s where things can get tricky if the standard conditions don't hold up.
Jane: Exactly, and they show that just having the standard conditions for transporting treatment effects aren't enough to guarantee that the transported effect will match what you see in each trial individually. They need more specific conditions to ensure agreement.
Lu: They say agreement requires conditions about how those trial-related actions affect the outcome across different studies, like making sure those engagement effects are either absent, or they are equivalent across trials, or there's no interaction between the treatment assignment and the trial engagement.
Title and authors: Meng: That sounds incredibly complex to check in a production environment. If we’re trying to deploy a model based on these comparisons, how do you practically verify if those conditions about trial engagement effects are met for two different studies?
Lalam: They formalize two types of identification conditions: one that makes sure each individual trial can identify its own causal effect and that effects can be transported across trials, and another set of conditions to make sure the effects from different trial settings are comparable in a common target population.
Tom: So it’s not just about getting the effect right within one study, but also getting those individual studies to speak the same language when you try to compare them later on. That’s a huge hurdle for any big evidence synthesis project.
Jane: It sounds like they are giving us a blueprint for how to do that comparison rigorously, moving beyond just saying "these two studies don't agree." They are defining the necessary conditions for when those comparisons actually make sense.
Lu: The paper describes how these identification results support statistical comparisons of effect estimates across trials using nested expectation expressions, which is a very formal way to set up the math for this kind of comparison.
Tom: I see them using these expectation expressions to compare things like trial-specific effects versus the effect transported from one trial to another. It’s a clean way to structure the comparison for the reader, even if it’s dense mathematically.
Meng: But what does that actually mean for us when we think about deploying an AI model? Does this framework help us decide which trial results are most reliable for a specific user group?
Lalam: The paper uses nested expectation expressions to compare those quantities of interest, showing diagonal elements for the effects within a single trial and off-diagonal elements for those transported across trials.
Jane: So you get these diagonal numbers that tell you what happened in Trial A, and then you get off-diagonal numbers that tell you what happens when we move that information to Trial B’s population. It maps out the flow of knowledge between studies.
Title and authors: Tom: And they examine the null hypotheses at a conditional level by comparing the conditional mean difference between treatment groups across trials at the same covariate values, and then they have a marginal null hypothesis comparing an effect identified directly in one trial with the effect obtained by transporting information from another.
Lu: That marginal null hypothesis is really about testing if you can find that direct effect in one study versus trying to estimate it by transporting data from the other study, which is a key question for synthesis.
Meng: If we were building a system that needed to trust results across multiple studies, this paper tells us exactly what kind of data structure we need to handle—trial membership, treatment assignment, and trial-related actions all at once.
Jane: It really pushes the thinking toward needing these specific trial engagement conditions to bridge the gap between two different trial settings. Transportability alone isn't enough for things to line up properly.
Tom: So, it’s not just about transportability; it’s about controlling those engagement effects to make sure the transported and native effects actually coincide. That’s a lot of fine-tuning required in the model setup, I bet.
Lu: Right, that's what they conclude: additional conditions regarding trial engagement effects are required for transported and trial-native effects to coincide. This is the crucial piece of evidence for when we can trust cross-trial comparisons.
Meng: For me, that means any system trying to generalize results from one clinical setting to another needs a way to quantify or control those engagement factors precisely, otherwise the generalization breaks down at scale.
Lalam: One potential application they mention is the systematic evaluation of similarly designed trials that address the same clinical question, which is basically helping researchers compare their work more intelligently.
Tom: So this paper gives us a formal way to benchmark and compare studies even when they seem different on the surface, by accounting for these real-world trial dynamics. It’s a solid foundation for better evidence synthesis.
Jane: It sounds like a very rigorous tool for anyone trying to synthesize evidence from randomized trials, especially when you have multiple trials that might be slightly different in how they run things.
Lu: The whole point is providing this causal framework so we can move past just saying studies disagree and start asking *why* they disagree under these formal conditions. It’s about making the comparison more meaningful than just a simple difference calculation.
Title and authors: Meng: If we think about deploying a large language model to summarize evidence, this paper gives us the structure to know which comparisons are robust and which ones are just noise from unmeasured trial engagement factors.
Lalam: One limitation they point out is that these conditions are quite restrictive; they require specific assumptions about how trial-related actions influence outcomes across studies. If those assumptions don't hold, the comparison framework might not apply correctly.
Tom: So, it’s a powerful tool for structured comparison, but we have to be careful about applying those strict conditions in messy real-world data sets. It’s a great formal approach to wrestling with real complexity.
Jane: They end by showing how this framework helps us systematically evaluate how similar trials address the same clinical question, giving us a path forward for more trustworthy meta-analyses.
Lu: This paper lays out the structure for that systematic evaluation, which is what makes it useful in the context of synthesizing evidence across different trial designs.
Meng: For deployment, this means we need to build in parameters to estimate those engagement effects or risk having our transported results be unreliable when those conditions aren't met.
Lalam: We have the full picture now: comparing randomized trials using joint interventions on treatment assignment and trial-related actions, and understanding that agreement requires controlling for trial engagement effects.
Tom: So we’ve seen how they formalize the comparison, what the specific identification conditions are, and why just transportability isn't enough. It’s a lot of math to unpack here.
Jane: It is dense, but the goal is making sure that when we compare these studies on things like treatment effects, we are actually comparing apples to apples under more realistic circumstances.
Lu: The implication is that future work needs to focus on how to estimate those trial engagement effects themselves so that we can feed them into this framework for a more complete picture.
Meng: If the cost of estimating those engagement effects becomes too high, then this formal comparison might remain purely theoretical unless there's a scalable way to get that data.
Lalam: This paper is essentially about creating the language and the structure needed to compare randomized trials in a way that respects both their individual context and their relationship to each other.
The paper's summary: Tom: So, this paper is all about taking randomized trials and actually comparing them when they are trying to answer the same causal question.
Jane: Right, and what they’re doing is sketching out a way to do that comparison that lets you explicitly account for trial engagement effects.
Lu: They’re using counterfactual outcomes defined by joint interventions on both the treatment assignment and those specific trial actions to compare trial-native effects against effects transported from one study to another.
Meng: So, the big takeaway here is that just having standard conditions for transporting treatment effects isn't enough to guarantee that the results from different trials will actually match up.
Lalam: They show you need extra conditions—like making sure those trial engagement effects are absent, or they are equivalent across trials, or there’s no interaction between the treatment assignment and the engagement—to make those transported and native effects coincide.
Tom: That means for any real-world comparison of studies, you can’t just assume things line up based on basic rules; you need to check those specific conditions about how a trial behaves.
Jane: Exactly, and they formalize two identification conditions: one that lets each trial identify its own effect and get that effect transported, and another set of conditions to make sure the effects from different trial settings are comparable in one common target population.
Lu: The paper uses these formal identification results to support statistical comparisons of effect estimates across trials using nested expectation expressions, which is a very specific way to structure the math for this kind of comparison.
Meng: From my side, it’s interesting how they’re defining the null hypotheses at a conditional level by comparing the conditional mean difference between treatment groups across trials at the same covariate values.
Tom: And then they have this marginal null hypothesis that compares an effect identified directly in one trial against the effect you get by transporting information from another trial.
Jane: That last part is crucial because it tests whether you can find that direct effect versus relying on the transported estimate.
Lu: This framework suggests that transportability conditions alone don't guarantee agreement between trials, and those extra conditions about engagement effects are what finally make them match up in a common target population.
Tom: So, if you’re building a system to synthesize evidence from multiple randomized trials, this gives you the blueprint for how to test if those comparisons are actually meaningful.
Jane: It moves beyond just saying two studies disagree and starts asking what the exact conditions are for when those disagreements make sense in a causal way.
Lu: The implication is that we need to focus on estimating those trial engagement effects themselves so we can feed them into this framework for a more complete picture of the evidence.
The paper's improvements: Tom: Okay, so we just talked about how they compare trials using trial engagement effects to make sure transported results actually match native ones. Now, what are they suggesting as improvements on that framework?
Jane: The paper outlines three specific ways to improve the comparison process, focusing on how we estimate those effects.
Lu: They suggest first identifying trial-native effects within each study using observed data from that trial by calculating the expected value of the outcome given the treatment assignment and those trial actions.
Meng: So, you’re saying you can get a direct measure of what happened in one specific study just by looking at its own data, without needing some weird counterfactual modeling?
Tom: Yeah, that sounds like a clean way to anchor your baseline before you try to compare it to another study.
Lalam: Second, they suggest identifying transported effects across trials by using observed outcome information from trial one minus s to estimate the effect in trial s.
Jane: So, instead of just modeling the transport abstractly, they propose using actual outcome data from one trial as a starting point for estimating the effect in another study.
Lu: This enables you to estimate a causal effect for one trial’s population by looking at the data from another study entirely.
Meng: That sounds like it requires very specific access to the raw data from both trials, which is usually not something I have when I’m shipping models to production.
Tom: It definitely pushes the mechanics of what needs to be collected before we can even run these comparisons.
Jane: Third, they suggest systematically comparing marginal effects across trials by comparing the expected value of the outcome given those trial actions and treatment assignment for each study separately.
Lu: This lets you compare how the effect looks under different trial settings in one common target population, which is a lot more concrete than just abstract transportability.
Meng: So, it’s not just about comparing numbers, but about comparing the *types* of comparisons being made across different contexts.
Tom: It seems like they are tightening up the math so that the comparison isn't just some loose statistical check but a structured causal statement.
Jane: If you only listen to this show, this means when we look at two randomized trials, we need to ask not just "do these numbers match?" but "what specific conditions about trial engagement must be true for us to trust the comparison?"
Lu: They are essentially giving us a set of better tools for identifying those necessary conditions so that the transported and native effects actually line up.
Meng: I still have to ask what it costs to run all these kinds of complex calculations if we try to apply this at scale across thousands of trials.
Tom: That’s a fair point, Meng, because the complexity here is definitely high for a production environment.
Jane: But the point is that this framework helps us systematically evaluate similarly designed trials addressing the same clinical question in a much more rigorous way than we can right now.
Conclusion: Tom: So to wrap up, this paper, "Comparing randomized trials: a formal approach," gives us a rigorous way to handle disagreements between randomized trials when they’re trying to answer the same causal question.
Jane: It provides that framework by defining precise identification conditions that account for trial engagement effects when comparing results across different studies.
Lu: The core finding is that transportability alone isn't enough; you need those extra conditions about how trial-related actions affect outcomes to make sure the transported and native effects actually line up.
Meng: So, what does this mean for us deploying models? Does it give us a checklist for when we can trust a summary of evidence from multiple trials?
Tom: It gives you the structure to test if those comparisons are actually meaningful by checking those specific engagement conditions before you even start synthesizing the data.
Jane: Basically, it moves us from just saying studies disagree to asking what exact causal mechanisms must be controlled for when we try to compare them.
Lu: This framework is a tool for benchmarking and joint analysis that forces us to consider the real-world actions happening inside a trial.
Meng: I still worry about the cost of checking all those conditions, but if this helps filter out noisy comparisons, it might be worth the engineering effort.
Lalam: I'm sure this paper is very sure of its answer; it’s building the language for causal comparison in randomized trials that respects their context.
Tom: It sets a high bar for anyone trying to synthesize evidence from multiple sources in this field, because you can't just mix and match results anymore.
Jane: It changes how we think about evidence because it forces us to look at the trial engagement effects as a necessary part of the causal picture.
Lu: The next step would be figuring out how to estimate those trial engagement effects themselves so we can feed them into this framework for a more complete picture.
Department of Epidemiology, Brown University School of Public Health · Center for Gerontology and Health Care Research, Brown University School of Public Health · CAUSALab, Department of Epidemiology, Department of Biostatistics, Harvard T.H. Chan School of Public Health · Richard A. and Susan F. Smith Center for Outcomes Research, Beth Israel Deaconess Medical Center
stat.ME
Submitted: 2026-10-10
Updated: 2026-11-08
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 99/100
The gist: Disagreements between randomized trials that address the same causal question pose challenges for the interpretation and synthesis of evidence<ref:2610.14895#pg2>.
Key concepts
- Trial Engagement Effects
- These are unobserved factors related to how participants behave or interact with the trial environment. They can cause differences in outcomes between trials even if the treatment assignment is identical, meaning standard transportability assumptions often fail.
- Trial-Native Treatment Effects
- This refers to the causal effect of a treatment within a specific trial population, accounting for all actions and interventions unique to that study. It represents the true effect observed in that particular setting.
- Transported Effects
- These are causal effects estimated by using data from one randomized trial to infer what the outcome would be in another trial. The paper investigates when these transported effects match the native effects across different trials.
Terminology
Summary
Disagreements between randomized trials that address the same causal question pose challenges for the interpretation and synthesis of evidence<ref:2610. p. 0 and Susan F. Smith Center for Outcomes Research, Beth Israel Deaconess Medical Center,
Harvard Medical School, Boston, MA
Abstract
Disagreements between randomized trials that address the same causal question pose
challenges for the interpretation and synthesis of evidence. This DRAFT presents WORK IN PROGRESS. Please send comments to idahabreh@hsph.Where the paper says it
Where the paper says it
Causal Framework for Comparison
The paper sketches a causal framework for comparing randomized trials that explicitly accommodates the possibility of trial engagement effects<ref:2610.14895#pg4>. Using counterfactual outcomes defined under joint interventions on treatment assignment and trial-related actions, it compares trialnative treatment effects with the effects transported from one trial to the population of another in a common target population<ref:2610. p. 0 We sketch a causal framework for comparing the results of randomized trials that explicitly accommodates the
possibility of trial engagement effects. Using counterfactual outcomes defined under
joint interventions on treatment assignment and trial-related actions, we compare trialnative treatment effects with the effects transported from one trial to the population
of another in a common target population. Using counterfactual outcomes defined under
joint interventions on treatment assignment and trial-related actions, we compare trialnative treatment effects with the effects transported from one trial to the population
of another in a common target population. We show that standard conditions for
transporting treatment effects across trial populations are not sufficient to guarantee
agreement between transported and trial-native effects. We show that standard conditions for
transporting treatment effects across trial populations are not sufficient to guarantee
agreement between transported and trial-native effects. Agreement additionally requires conditions governing how trial-related actions affect outcome across studies—for
example, absence of trial engagement effects, equivalence of such effects across trials, or
lack of interaction between trial engagement and treatment assignment.Where the paper says it
Where the paper says it
Where the paper says it
Data Structure and Estimands
The data structure involves two separately conducted randomized trials, each enrolling participants from distinct and possibly different populations<ref:2610.14895#pg5>. The authors use S ∈ S = 0, 1 to denote trial membership, A ∈ A = 0, 1 to denote binary treatment assignment, Y to denote the outcome<ref:2610. p. 2 Data structure and sampling
Suppose we have data from two separately conducted randomized trials, each enrolling participants from distinct and possibly different populations. We use S ∈ {0, 1} to denote trial
membership, A ∈ {0, 1} to denote binary treatment assignment, Y to denote the outcome
1 To simplify the exposition, we assume no loss to follow-up and no
measurement error (in the sense that we consider each trial as able to measure the variables
it chooses to measure). Causal estimands for trial comparisons
We use Y
s,a to denote the potential (counterfactual) outcome under a joint intervention that
sets treatment assignment to A = a and sets the outcome-relevant trial actions and nonrandomized interventions to those of trial S = s [6, 11]. We use Y
a
to denote the
potential outcome under intervention only to set treatment A to a; these potential outcomes
are useful, for example, when trial engagement effects are absent. One family of causal estimands of interest is the average effect of assignment to treatment
versus control under the trial-related actions and interventions of a given trial s ∈ {0, 1},
2Where the paper says it
Where the paper says it
Where the paper says it
Identification Conditions
The paper formalizes two kinds of identification conditions: one ensuring each trial can identify causal effects and be transported across trials, and another ensuring causal effects under different trial settings are comparable in a common target population<ref:2610.14895#pg7>. The first kind includes A1 through A3, which are standard conditions for identification of causal effects within each trial<ref:2610. p. 6 That is,
Pr(A = a | X = x, S = s) > 0
for all x such that fX,S(x, s) ̸= 0 and a ∈ {0, 1}. Conditions A1–A3 are standard conditions for identification of causal effects within each
trial, E[Y
s,a=1 − Y
s,a=0 | S = s], and are expected to hold by design in a marginally or conditionally randomized trial. Conditions for comparability of causal effects across trials
To compare causal effects defined under different trial settings (i. , diagonal and off-diagonal
elements within the same row of Table 1), additional conditions about trial engagement effects are required.Where the paper says it
Where the paper says it
Comparison and Conclusion
The paper describes how the identification results support statistical comparisons of effect estimates across trials using nested expectation expressions<ref:2610.14895#pg11>. The framework clarifies that transportability conditions alone does not, in general, ensure agreement between trials<ref:2610. p. 0 Agreement additionally requires conditions governing how trial-related actions affect outcome across studies—for
example, absence of trial engagement effects, equivalence of such effects across trials, or
lack of interaction between trial engagement and treatment assignment. Our framework
clarifies potential sources of disagreement across trials and extends existing approaches
to study comparison to settings in which trial engagement effects may be present. , diagonal and off-diagonal
elements within the same row of Table 1), additional conditions about trial engagement effects are required. Assumptions B1–B3 below describe progressively weaker sufficient conditions under which causal effects defined under different trial-related conditions coincide
within the same population. Our framework
clarifies potential sources of disagreement across trials and extends existing approaches
to study comparison to settings in which trial engagement effects may be present. arXiv:2610.Where the paper says it
Where the paper says it
Where the paper says it
Table 1 Summary
Table 1 summarizes the causal quantities of interest in cross-trial comparison, showing diagonal elements for trial-specific effects and off-diagonal elements for effects obtained by transporting results across trials<ref:2610.14895#pg17>. The null hypotheses examined at the conditional level compare the conditional mean difference between treatment groups across trials at the same covariate values<ref:2610. p. 9 Under the conditions A1–A3 described above, it is wellknown that, for each trial s ∈ {0, 1}, the conditional average effect of treatment assignment
under the outcome-relevant actions and interventions of trial s, among participants in trial
s, is identified by
E[Y
s,a=1 − Y
s,a=0 | X, S = s] = E[Y | X, S = s, A = 1] − E[Y | X, S = s, A = 0]. (1)
The right-hand side of the expression above is the difference in the conditional mean
outcome between the two treatment groups within trial s, among participants with the same
covariate values X.Where the paper says it
References
[1] Issa J Dahabreh, James M Robins, and Miguel A Hern´an. Benchmarking observational methods by comparing randomized trials and their emulations. p. 2 Here, we sketch a causal framework for comparing randomized trials that explicitly accommodates the possibility of trial engagement effects [8]. The framework clarifies that
agreement between trials depends not only on identification conditions analogous to those
used in benchmarking observational analyses against trials [1], but also on additional conditions governing how trial-related actions influence outcomes across studies. Comparing randomized trials
Yu-Han Chiu1,2 and Issa J. Dahabreh3,4
1Department of Epidemiology, Brown University School of Public Health, Providence, RI
2Center for Gerontology and Health Care Research, Brown University School of Public Health, Providence,
RI
3CAUSALab, Department of Epidemiology, Department of Biostatistics, Harvard T.Where the paper says it
Where the paper says it
[2] Vivian C Wong, Kylie Anglin, and Peter M Steiner. Design-based approaches to causal replication studies. p. 0 Agreement additionally requires conditions governing how trial-related actions affect outcome across studies—for
example, absence of trial engagement effects, equivalence of such effects across trials, or
lack of interaction between trial engagement and treatment assignment. Our framework
clarifies potential sources of disagreement across trials and extends existing approaches
to study comparison to settings in which trial engagement effects may be present. Our framework
clarifies potential sources of disagreement across trials and extends existing approaches
to study comparison to settings in which trial engagement effects may be present. arXiv:2610.Where the paper says it
Where the paper says it
[3] Peter M Rothwell. External validity of randomised controlled trials:“to whom do the results of this trial apply? p. 2 The framework clarifies that
agreement between trials depends not only on identification conditions analogous to those
used in benchmarking observational analyses against trials [1], but also on additional conditions governing how trial-related actions influence outcomes across studies. More broadly,
the ideas developed here apply not only to comparisons between randomized trials, but also
to benchmarking and experimental replication efforts. Our framework
clarifies potential sources of disagreement across trials and extends existing approaches
to study comparison to settings in which trial engagement effects may be present. arXiv:2610.Where the paper says it
Where the paper says it
[4] OM Dekkers, E von Elm, Ale Algra, JA Romijn, and JP Vandenbroucke. How to assess the external validity of therapeutic trials: a conceptual approach. International journal of epidemiology, 39(1):89–94, 2010<ref:2610. p. 0 Comparing randomized trials
Yu-Han Chiu1,2 and Issa J. Dahabreh3,4
1Department of Epidemiology, Brown University School of Public Health, Providence, RI
2Center for Gerontology and Health Care Research, Brown University School of Public Health, Providence,
RI
3CAUSALab, Department of Epidemiology, Department of Biostatistics, Harvard T.Where the paper says it
[5] Lori E Dodd, Boris Freidlin, and Edward L Korn. Platform trials—beware the noncomparable control group. p. 3 We use Y
a
to denote the
potential outcome under intervention only to set treatment A to a; these potential outcomes
are useful, for example, when trial engagement effects are absent. One family of causal estimands of interest is the average effect of assignment to treatment
versus control under the trial-related actions and interventions of a given trial s ∈ {0, 1},
2 Our framework
clarifies potential sources of disagreement across trials and extends existing approaches
to study comparison to settings in which trial engagement effects may be present. arXiv:2610.Where the paper says it
Where the paper says it
[6] Issa J Dahabreh, James M Robins, Sebastien J-PA Haneuse, and Miguel A Hern´an. p. 0 Please send comments to idahabreh@hsph. Comparing randomized trials
Yu-Han Chiu1,2 and Issa J. We sketch a causal framework for comparing the results of randomized trials that explicitly accommodates the
possibility of trial engagement effects. Using counterfactual outcomes defined under
joint interventions on treatment assignment and trial-related actions, we compare trialnative treatment effects with the effects transported from one trial to the population
of another in a common target population. Our framework
clarifies potential sources of disagreement across trials and extends existing approaches
to study comparison to settings in which trial engagement effects may be present. arXiv:2610. Our framework
clarifies potential sources of disagreement across trials and extends existing approaches
to study comparison to settings in which trial engagement effects may be present. arXiv:2610.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
[7] Issa J Dahabreh, Sarah E Robertson, and Miguel A Hern´an. p. 0 Please send comments to idahabreh@hsph. Comparing randomized trials
Yu-Han Chiu1,2 and Issa J. Using counterfactual outcomes defined under
joint interventions on treatment assignment and trial-related actions, we compare trialnative treatment effects with the effects transported from one trial to the population
of another in a common target population. We show that standard conditions for
transporting treatment effects across trial populations are not sufficient to guarantee
agreement between transported and trial-native effects. Our framework
clarifies potential sources of disagreement across trials and extends existing approaches
to study comparison to settings in which trial engagement effects may be present. arXiv:2610. Our framework
clarifies potential sources of disagreement across trials and extends existing approaches
to study comparison to settings in which trial engagement effects may be present. arXiv:2610.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
[8] Lawson Ung, Tyler J VanderWeele, and Issa J Dahabreh. p. 0 Please send comments to idahabreh@hsph. Comparing randomized trials
Yu-Han Chiu1,2 and Issa J. Smith Center for Outcomes Research, Beth Israel Deaconess Medical Center,
Harvard Medical School, Boston, MA
Abstract
Disagreements between randomized trials that address the same causal question pose
challenges for the interpretation and synthesis of evidence. We sketch a causal framework for comparing the results of randomized trials that explicitly accommodates the
possibility of trial engagement effects. Comparing randomized trials
Yu-Han Chiu1,2 and Issa J. Dahabreh3,4
1Department of Epidemiology, Brown University School of Public Health, Providence, RI
2Center for Gerontology and Health Care Research, Brown University School of Public Health, Providence,
RI
3CAUSALab, Department of Epidemiology, Department of Biostatistics, Harvard T.Where the paper says it
Where the paper says it
Where the paper says it
[9] Issa J Dahabreh, Lucia C Petito, Sarah E Robertson, Miguel A Hern´an, and Jon A Steingrimsson. p. 0 Please send comments to idahabreh@hsph. Comparing randomized trials
Yu-Han Chiu1,2 and Issa J. Using counterfactual outcomes defined under
joint interventions on treatment assignment and trial-related actions, we compare trialnative treatment effects with the effects transported from one trial to the population
of another in a common target population. We show that standard conditions for
transporting treatment effects across trial populations are not sufficient to guarantee
agreement between transported and trial-native effects. Our framework
clarifies potential sources of disagreement across trials and extends existing approaches
to study comparison to settings in which trial engagement effects may be present. arXiv:2610. Our framework
clarifies potential sources of disagreement across trials and extends existing approaches
to study comparison to settings in which trial engagement effects may be present. arXiv:2610.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
[10] Issa J Dahabreh, Sebastien JP A Haneuse, James M Robins, Sarah E Robertson, Ashley L Buchanan, Elizabeth A Stuart, and Miguel A Hern´an. Study designs for extending causal inferences from a randomized trial to a target population. p. 0 We sketch a causal framework for comparing the results of randomized trials that explicitly accommodates the
possibility of trial engagement effects. Using counterfactual outcomes defined under
joint interventions on treatment assignment and trial-related actions, we compare trialnative treatment effects with the effects transported from one trial to the population
of another in a common target population. Comparing randomized trials
Yu-Han Chiu1,2 and Issa J. Dahabreh3,4
1Department of Epidemiology, Brown University School of Public Health, Providence, RI
2Center for Gerontology and Health Care Research, Brown University School of Public Health, Providence,
RI
3CAUSALab, Department of Epidemiology, Department of Biostatistics, Harvard T.Where the paper says it
Where the paper says it
[11] Laura B Balzer. “All generalizations are dangerous, even this one. p. 0 We sketch a causal framework for comparing the results of randomized trials that explicitly accommodates the
possibility of trial engagement effects. Using counterfactual outcomes defined under
joint interventions on treatment assignment and trial-related actions, we compare trialnative treatment effects with the effects transported from one trial to the population
of another in a common target population. Comparing randomized trials
Yu-Han Chiu1,2 and Issa J. Dahabreh3,4
1Department of Epidemiology, Brown University School of Public Health, Providence, RI
2Center for Gerontology and Health Care Research, Brown University School of Public Health, Providence,
RI
3CAUSALab, Department of Epidemiology, Department of Biostatistics, Harvard T.Where the paper says it
Where the paper says it
[12] Issa J Dahabreh, Sarah E Robertson, Jon A Steingrimsson, Elizabeth A Stuart, and Miguel A Hernan. p. 0 Please send comments to idahabreh@hsph. Comparing randomized trials
Yu-Han Chiu1,2 and Issa J. We sketch a causal framework for comparing the results of randomized trials that explicitly accommodates the
possibility of trial engagement effects. Using counterfactual outcomes defined under
joint interventions on treatment assignment and trial-related actions, we compare trialnative treatment effects with the effects transported from one trial to the population
of another in a common target population. Our framework
clarifies potential sources of disagreement across trials and extends existing approaches
to study comparison to settings in which trial engagement effects may be present. arXiv:2610.Where the paper says it
Where the paper says it
Where the paper says it
[13] Issa J Dahabreh, Jon A Steingrimsson, James M Robins, and Miguel A Hern´an. p. 0 Please send comments to idahabreh@hsph. Comparing randomized trials
Yu-Han Chiu1,2 and Issa J. Here, we sketch a causal framework for comparing randomized trials that explicitly accommodates the possibility of trial engagement effects [8]. The framework clarifies that
agreement between trials depends not only on identification conditions analogous to those
used in benchmarking observational analyses against trials [1], but also on additional conditions governing how trial-related actions influence outcomes across studies. Our framework
clarifies potential sources of disagreement across trials and extends existing approaches
to study comparison to settings in which trial engagement effects may be present. arXiv:2610. Our framework
clarifies potential sources of disagreement across trials and extends existing approaches
to study comparison to settings in which trial engagement effects may be present. arXiv:2610.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
[14] James Robins. A new approach to causal inference in mortality studies with a sustained exposure period—application to control of the healthy worker survivor effect. p. 3 We use Y
a
to denote the
potential outcome under intervention only to set treatment A to a; these potential outcomes
are useful, for example, when trial engagement effects are absent. One family of causal estimands of interest is the average effect of assignment to treatment
versus control under the trial-related actions and interventions of a given trial s ∈ {0, 1},
2 Our framework
clarifies potential sources of disagreement across trials and extends existing approaches
to study comparison to settings in which trial engagement effects may be present. arXiv:2610.Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Identify trial-native effects within each trial using observed data from that trial by calculating E[Y s,a=1 − Y s,a=0 X = x, S = s]. This allows for direct estimation of the treatment effect within a specific study population based only on its own data.
-
Identify transported effects across trials by using observed outcome information from trial 1 minus s to estimate the effect in trial s: E[Y 1−s,a=1 − Y 1−s,a=0 X = x, S = s]. This enables the estimation of a causal effect for one trial's population using data from another trial.
-
Systematically compare marginal effects across trials by comparing E[E[Y X, S = s, A = 1] − E[Y X, S = s, A = 0] S = s] with E[E[Y X, S = 1−s, A = 1] − E[Y X, S = 1−s, A = 0] S = s]. This allows for the comparison of marginal effects under the outcome-relevant actions and interventions of each trial in a common target population.
Abstract
Disagreements between randomized trials that address the same causal question pose challenges for the interpretation and synthesis of evidence. We sketch a causal framework for comparing the results of randomized trials that explicitly accommodates the possibility of trial engagement effects. Using counterfactual outcomes defined under joint interventions on treatment assignment and trial-related actions, we compare trial-native treatment effects with the effects transported from one trial to the population of another in a common target population. We show that standard conditions for transporting treatment effects across trial populations are not sufficient to guarantee agreement between transported and trial-native effects. Agreement additionally requires conditions governing how trial-related actions affect outcome across studies - for example, absence of trial engagement effects, equivalence of such effects across trials, or lack of interaction between trial engagement and treatment assignment. Our framework clarifies potential sources of disagreement across trials and extends existing approaches to study comparison to settings in which trial engagement effects may be present.
Sources
- Generalizing causal inferences from randomized trials: counterfactual and graphical identification
- Generalizing and transporting inferences about the effects of treatment assignment subject to non-adherence
- Towards causally interpretable meta-analysis: transporting inferences from multiple studies to a target population
- Randomized trials and their observational emulations: a framework for benchmarking and joint analysis
Related papers
- Learning Robust Treatment Rules for Censored Data
- Neural Estimation of Summary Statistics for Simulation-Based Inference with Non-Asymptotic Guarantees
- Spectral divide-and-conquer MCMC for long stationary time series
- Improving the Efficiency of Subgroup Analysis in Randomized Controlled Trials with TMLE
- Personalized Federated Vector Autoregression with Personalization Diversity
- Doubly Robust and Efficient Calibration of Prediction Sets for Right-Censored Time-to-Event Outcomes