Orthogonal factorial designs for trials of therapist-delivered interventions: Randomising intervention-therapist combinations to patients

arXiv:2601.16318 · stat.ME · Submitted 2026-01-22 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Statistics. The day in statistics: methods, theory and the inference under them.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "Orthogonal factorial designs for trials of therapist-delivered interventions".

Jane: Here's the gist of the paper. Orthogonal factorial designs for trials of therapist-delivered interventions: Randomising intervention-therapist combinations to patients is proposed to allow for therapist learning and centre too Here, interventions are a treatment factor, but therapists are not If the aim of a trial is to separate effects of therapists from those of interventions, they should be regarded as two potentially interacting treatment factors with a factorial structure The authors propose a family of orthogonal factorial experimental designs and their associated data analyses for trials of therapist-delivered interventions They outline three types of designs: completely randomised, randomised block, and multicentre randomised block factorial designs This formal approach leads to recommendations about how to randomise and analyse such trials.

Tom: Design Frameworks The paper introduces three specific orthogonal factorial experimental designs for therapist-delivered interventions, namely the completely randomised factorial design, the randomised block factorial design, and the multicentre randomised block factorial design The completely randomised factorial design considers all combinations of levels of two treatment factors, Interventions (I) and Therapists (T), with each treatment replicated a certain number of times The randomised block factorial design introduces a blocking factor B, representing categorical time periods or batches, to account for variation in when interventions commence Finally, the multicentre randomised block factorial design incorporates a centre factor C as another blocking factor to account for variations across different centres.

Jane: Statistical Analysis and Modeling The analysis employs an approach based on the Design of Experiments (DoE) methodology, utilizing Hasse diagrams to describe the randomisation structure The paper details the construction of ANOVA tables based on these diagrams, where strata represent subspaces associated with each factor For the completely randomised factorial design, the appropriate residual for testing the fixed effect of I is identified by finding the supremum of all random factors which occur below factor I in a combined Hasse diagram The regression models are presented as random coefficient models, defining variance components such as between-therapist variance and between-therapist variance in the intervention effect.

Tom: Simulation Study Findings A simulation study was conducted to explore the performance of the proposed randomisation methods on estimating key parameters like the intervention effect and its standard error The simulations compared five different randomisation methods, including our proposed method of randomising therapist-intervention combinations to patients The results indicated that the intervention effect was estimated without bias across all five randomisation methods, regardless of the size of the baseline patient covariate The between-therapist variance in the intervention effect was estimated to be about 0.15 when therapist allocation was randomized to patients The paper concludes that more purposeful trial design has the potential to lead to better evidence on a range of complex interventions The authors suggest that randomising therapists to patients (or at least concealing the intervention when assigning therapists to patients) is preferable for unbiased estimation of the between-therapist variance and between-therapist variance in the intervention effect They recommend that the primary data analysis should fully reflect the randomisation, including blocking by time Furthermore, they suggest that treatment contrasts are orthogonal to estimate main effects and interactions.

Paper summary: Tom: Okay, so to summarize what we've heard about "Orthogonal factorial designs for trials of therapist-delivered interventions: Randomising intervention-therapist combinations to patients", the authors are proposing a formal way to design these trials. They're suggesting that instead of just looking at the intervention, you need to consider both the intervention and the therapist as potentially interacting treatment factors with a factorial structure.

Jane: So, they aren't just looking at one thing; they are setting up a system where you can isolate what the therapy does from what the therapist does. They introduce three specific types of designs: completely randomized, randomized block, and multicentre randomized block factorial designs.

Lu: The completely randomized factorial design considers all combinations of levels for both factors, interventions and therapists, with each combination being replicated a certain number of times. It’s a broad starting point for setting up the structure.

Meng: And then you have the randomized block design where they introduce a blocking factor representing categorical time periods or batches to account for when interventions start happening. That’s more realistic than just one big trial, right?

Lalam: And finally, there's the multicentre randomized block factorial design which adds another blocking factor for variations across different centers. It’s like they are accounting for where the trial is happening geographically or institutionally.

Tom: The statistical analysis they use involves a Design of Experiments methodology, using Hasse diagrams to describe the randomization structure, and then constructing ANOVA tables based on those diagrams. They even identify specific residual values to test the fixed effect of the intervention.

Jane: That sounds very technical for someone who isn't in statistics. What does that actually mean for us when we think about how they analyze this data?

Lu: It means they are using these diagrams to visualize exactly how things are randomized, which helps them build models, like random coefficient models, that define variance components like the between-therapist variance in the intervention effect.

Meng: So if I were running this, I’d want to know what those variance components look like so I can budget for the necessary sample size. What are they saying about those estimates?

Lalam: A simulation study they did explored how these randomization methods perform when estimating key parameters like the intervention effect and its standard error. They found that the intervention effect was estimated without bias across five different randomization methods they tested, no matter how big the baseline patient covariate was.

Conclusion: Tom: So to wrap up on this paper, "Orthogonal factorial designs for trials of therapist-delivered interventions: Randomising intervention-therapist combinations to patients", the main implication is that more purposeful trial design can lead to better evidence on complex interventions. The authors strongly recommend randomizing therapists to patients directly, or at least concealing the intervention when assigning therapists because they think this helps get an unbiased estimate of the between-therapist variance and between-therapist variance in the intervention effect.

Jane: That sounds like a practical suggestion for people running these kinds of studies. It’s telling us that how you set up the randomization sequence has a real impact on what you can actually measure about therapist effects versus intervention effects.

Lu: The authors are suggesting that treatment contrasts should be orthogonal to estimate main effects and interactions in the regression models, which is a specific statistical tool for making sure the model estimates those things correctly.

Meng: From my side, I see this as a way to make sure we aren't just seeing noise when we look at therapist variation. If we can properly block for time and centers, that gives us more clean data on the intervention effect itself.

Lalam: It’s about making sure the statistical analysis fully reflects the randomization you did, including blocking by time, which is a key recommendation from this paper.

Tom: So in simple terms, they are pushing for a more structured way to randomize things so we can get clearer evidence on how much the therapist matters versus how much the therapy itself matters. That’s what this paper is all about.

Rebecca EA Walwyn, Rosemary A Bailey, Arpan Singh, Neil Corrigan, Steven G Gilmour

University of Leeds · University of St Andrews · King’s College London

stat.ME

Submitted: 2026-01-22

Updated: 2026-02-02

DOI: 10.1002/sim.70813

Code: https://github.com/arpan52/Randomising-IxT-combinations-to-patients

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 82/100

The gist: Orthogonal factorial designs for trials of therapist-delivered interventions: Randomising intervention-therapist combinations to patients is proposed to allow for therapist learning and centre too…

Key concepts

Orthogonal Factorial Designs
These are specific experimental structures used when testing two factors—like an intervention and a therapist—to ensure that the effects of one factor do not interfere with estimating the other. This formal approach helps separate the influence of therapists from treatments.
Randomized Block Design
This design introduces 'blocking' factors, such as time periods or different clinical centers, to account for known sources of variation. Blocking helps reduce noise in the data so that researchers can more accurately measure the true effects of the intervention and therapist.
Randomizing Therapists to Patients
The authors recommend randomizing which therapist is assigned to which patient. This method is statistically preferred because it helps achieve an unbiased estimate of how much variation comes from therapists versus how much comes from the intervention itself.

Terminology

Summary

Orthogonal factorial designs for trials of therapist-delivered interventions: Randomising intervention-therapist combinations to patients is proposed to allow for therapist learning and centre too Here, interventions are a treatment factor, but therapists are not If the aim of a trial is to separate effects of therapists from those of interventions, they should be regarded as two potentially interacting treatment factors with a factorial structure The authors propose a family of orthogonal factorial experimental designs and their associated data analyses for trials of therapist-delivered interventions They outline three types of designs: completely randomised, randomised block, and multicentre randomised block factorial designs This formal approach leads to recommendations about how to randomise and analyse such trials. p. 3

Where the paper says it

Using Bailey’s [42] approach, the aim of this paper is to propose a family of increasingly complex, orthogonal factorial experimental designs, and their associated data analyses, for trials of therapistdelivered interventions, incorporating therapist learning and centre. We outline two motivating examples in Section 2.

Design Frameworks

The paper introduces three specific orthogonal factorial experimental designs for therapist-delivered interventions, namely the completely randomised factorial design, the randomised block factorial design, and the multicentre randomised block factorial design The completely randomised factorial design considers all combinations of levels of two treatment factors, Interventions (I) and Therapists (T), with each treatment replicated a certain number of times The randomised block factorial design introduces a blocking factor B, representing categorical time periods or batches, to account for variation in when interventions commence Finally, the multicentre randomised block factorial design incorporates a centre factor C as another blocking factor to account for variations across different centres.

Statistical Analysis and Modeling

The analysis employs an approach based on the Design of Experiments (DoE) methodology, utilizing Hasse diagrams to describe the randomisation structure The paper details the construction of ANOVA tables based on these diagrams, where strata represent subspaces associated with each factor For the completely randomised factorial design, the appropriate residual for testing the fixed effect of I is identified by finding the supremum of all random factors which occur below factor I in a combined Hasse diagram The regression models are presented as random coefficient models, defining variance components such as between-therapist variance and between-therapist variance in the intervention effect.

Simulation Study Findings

A simulation study was conducted to explore the performance of the proposed randomisation methods on estimating key parameters like the intervention effect and its standard error The simulations compared five different randomisation methods, including our proposed method of randomising therapist-intervention combinations to patients The results indicated that the intervention effect was estimated without bias across all five randomisation methods, regardless of the size of the baseline patient covariate The between-therapist variance in the intervention effect was estimated to be about 0.15 when therapist allocation was randomized to patients

Conclusion and Recommendations

The paper concludes that more purposeful trial design has the potential to lead to better evidence on a range of complex interventions The authors suggest that randomising therapists to patients (or at least concealing the intervention when assigning therapists to patients) is preferable for unbiased estimation of the between-therapist variance and between-therapist variance in the intervention effect They recommend that the primary data analysis should fully reflect the randomisation, including blocking by time Furthermore, they suggest that treatment contrasts are orthogonal to estimate main effects and interactions.

References

The paper lists numerous references supporting the statistical methods and prior work on therapist-related clustering in trials. The cited literature includes foundational works on Design of Experiments, specific studies on therapist effects, and guidance on trial reporting. p. 2

Where the paper says it

More recently, it has been accepted that, likewise, patient outcomes are clustered by therapists, leading to intra-therapist correlation (for a review, see Walwyn and Roberts [10]). This consensus led to guidance on trial reporting [11].

--- Page 27 ---

Figure 2 illustrates the randomisation diagrams for the three proposed designs: completely-randomised factorial design, randomised block factorial design, and multicentre randomised block factorial design. These diagrams visually represent how treatments are allocated to patients in each experimental setting. p. 5

Where the paper says it

[Insert Figure 2 about here] Figure 2a is a two-tiered example of the randomisation diagrams introduced by Brien and Bailey [47] (see also [6]). The two boxes represent two sets of factors (or “tiers”), the set of treatments and the set of units.

--- Page 29 ---

Table 3 summarizes the ANOVA results for the completely randomised factorial design, showing expected mean squares for various strata and their degrees of freedom The table provides a clear structure for interpreting the variance components derived from the statistical model.

--- Page 30 ---

Table 2 details the data generating mechanism for all three designs, specifying parameters like mean intervention effect and various variance components The table shows how the true values of fixed and random effects are set for each example.

--- Page 31 ---

Table 4 presents the simulation results for the completely randomised factorial design, comparing different randomisation methods against Type I and Type II error rates The table provides quantitative evidence regarding the impact of different allocation strategies on statistical inference.

--- Page 32 ---

Table 5 presents the simulation results for the randomised block factorial design, showing how different randomisation methods affect variance component estimates and error rates This table is crucial for understanding the practical implications of blocking factors in complex designs.

--- Page 39 ---

The references section lists 58 specific sources that underpin the statistical and methodological framework of the paper. These sources cover aspects from foundational DoE to contemporary issues in psychotherapy research. p. 4

Where the paper says it

Shapiro et al [45 – 46] reported the design and results of the Second Sheffield Psychotherapy Project. This involved a single centre in the sense that the five therapists were all embedded in the research team.

--- Page 40 ---

The references include specific works on surgical trial design and therapist learning curves, which inform the context of the proposed designs. The inclusion of these sources highlights the practical considerations in applying statistical theory to complex clinical settings. p. 3

Where the paper says it

[Insert Figure 1 about here] There is a large body of statistical work on the Design of Experiments (DoE) and their data analysis that began with Fisher [41]. Perhaps apart from crossover designs, this has become separated from clinical trials, despite their common origins and its relevance.

--- Page 47 ---

The references include works on multiple randomizations and decomposition tables, which are essential for understanding the structure of the experimental designs. These sources help in interpreting the complex factor relationships described in Section 4. p. 2

Where the paper says it

Walwyn et al [6] have set out ten possible ways of designing a trial to separate the effects of the intervention and the therapist by including therapists as a factor when generating the randomisation sequence. These possibilities depend on the design of the complex intervention (the “therapist-intervention design” [6] here), which describes the structural relationships between the components of the complex intervention, irrespective of the randomisation.

--- Page 57 ---

The references include Satterthwaite's work on approximate distributions of variance component estimates, which is used to calculate degrees of freedom for F-tests This statistical tool is essential for obtaining accurate approximate F tests in the analysis.

--- Page 58 ---

The references include literature on non-orthogonal multistratum designs, which addresses the complexity arising from interactions between random factors in the analysis. This work is vital for understanding the statistical dependencies in the analysis of block structures. p. 3

Where the paper says it

Here, we suggest that time and centre are accounted for in the randomisation by including them as stratification (or “blocking”) factors (see Figure 1 for a summary of the complex intervention designs considered in this paper). [Insert Figure 1 about here] There is a large body of statistical work on the Design of Experiments (DoE) and their data analysis that began with Fisher [41].

--- Page 59 ---

The references include works on partially nested randomized controlled trials, which provide context for the complexity of the data structure being addressed. These sources contextualize the challenges in applying statistical methods to real-world clinical trial data. p. 3

Where the paper says it

[Insert Figure 1 about here] There is a large body of statistical work on the Design of Experiments (DoE) and their data analysis that began with Fisher [41]. Perhaps apart from crossover designs, this has become separated from clinical trials, despite their common origins and its relevance.

--- Page 65 ---

The references include works on generalized estimating equations for repeated outcome measurements, which are relevant for advanced modeling techniques discussed in Section 4.3. These sources support the use of more sophisticated regression models to handle complex data structures. p. 1

Where the paper says it

We set out the associated data analyses using ANOVA and regression and report the results of a small simulation study conducted to explore the performance of the proposed randomisation methods in estimating the intervention effect and its standard error, the between-therapist variance and the between-therapist variance in the intervention effect. We conclude that more purposeful trial design has the potential to lead to better evidence on a range of complex interventions and outline areas for further methodological research.

--- Page 74 ---

The references include meta-analyses on absolute mean differences from randomized trials with treatment-related clustering associated with care providers, which justify the need for this research. These sources emphasize the importance of disentangling therapist effects from intervention effects in clinical outcomes. p. 2

Where the paper says it

However, disentangling the effect of the therapists from the effect of the interventions has long been considered a challenge in individually-randomised, parallel-group trials [4 – 5]. Walwyn et al [6] have set out ten possible ways of designing a trial to separate the effects of the intervention and the therapist by including therapists as a factor when generating the randomisation sequence.

--- Page 75 ---

The references include works on standardized mean differences from randomized trials with treatment-related clustering associated with care providers, which further justify the need for this research These sources underscore the importance of precise measurement in evaluating complex interventions.

--- Page 77 ---

The references include works on intention-to-treat analysis in partially nested randomized controlled trials with real-world complexity, which provide context for the practical challenges discussed in Section 4.3. These sources help frame the discussion on handling factors that cannot be randomised due to practical constraints. p. 3

Where the paper says it

Here, we assume both the intervention and the therapist are treatment factors, with one combination of the levels of these two treatment factors (the “treatment”) randomised to each patient (see Example A in [6]). We acknowledge that, in many trials, there are other contextual factors affecting the complex intervention delivered to patients that cannot be randomised.

--- Page 79 ---

The references include works on optimal design approaches, which are suggested for future work in sample size implications of the proposed designs. These sources point towards a more rigorous approach to determining the necessary sample size for these complex designs. p. 2

Where the paper says it

For crossed designs, there was a consensus that a random coefficient model is appropriate, including a fixed effect for intervention, a random effect for therapist, an intervention-by-therapist random effect and a patient error [10, 22]. A method for calculating sample size was also proposed [10].

--- Page 80 ---

The references include literature on noncompliance and partial nesting, which is relevant for understanding real-world complexities in trial data These sources help frame the discussion on how to handle factors that would ideally be randomised but cannot be are feasible in practice.

--- Page 84 ---

The references include works on the necessity of a formal Design of Experiments approach, which justifies the paper's methodology This work is essential for establishing a better understanding of data analysis that follows the design.

--- Page 94 ---

The references include works on orthogonal contrasts, which are recommended for estimating main effects and interactions in the regression models. These sources support the use of orthogonal contrasts to ensure that the model estimates main effects and interactions correctly.

--- Page 95 ---

The references include works on software enabling ANOVA to be used more widely if there are ‘dependent’ random effects These sources suggest that using ANOVA for the data analysis of these trials could lead to fewer unintended consequences of boundary estimates on the degrees of freedom when testing the intervention effect.

--- Page 97 ---

The references include works advocating for more purposeful trial design, which is a key takeaway of the paper's discussion. This emphasizes that purposeful trial design has the potential to lead to better evidence on a range of complex interventions. p. 1

Where the paper says it

We set out the associated data analyses using ANOVA and regression and report the results of a small simulation study conducted to explore the performance of the proposed randomisation methods in estimating the intervention effect and its standard error, the between-therapist variance and the between-therapist variance in the intervention effect. We conclude that more purposeful trial design has the potential to lead to better evidence on a range of complex interventions and outline areas for further methodological research.

--- Page 98 ---

The references include recommendations for randomising therapists to patients or at least concealing the intervention when assigning therapists to patients These suggestions are based on statistical considerations for avoiding Type II error inflation.

--- Page 99 ---

The references include practical considerations regarding therapist capacity and turnover, which inform future work in making the proposed randomisation methods feasible These sources highlight the practical barriers to implementing ideal statistical designs in clinical practice.

--- Page 99 ---

The references include a final recommendation that more purposeful trial design has the potential to lead to better evidence on a range of complex interventions This reinforces the paper's central thesis.

--- Page 99 ---

The references include a final recommendation that randomising therapists to patients (or at least concealing the intervention when assigning therapists to patients) is preferable for unbiased estimation of the between-therapist variance and between-therapist variance in the intervention effect This reinforces the paper's central finding.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Orthogonal factorial designs for trials of therapist-delivered interventions can be used to propose a family of increasingly complex, orthogonal factorial experimental designs for trials involving therapist-delivered interventions, incorporating therapist learning and centre as blocking factors.

  2. The improved system can perform data analysis using ANOVA and regression by calculating appropriate residual mean squares based on Hasse diagrams derived from the design structure to obtain unbiased estimates of intervention effects and variance components.

  3. The AI system can perform a simulation study comparing five randomisation methods, including randomising the combination of therapist and intervention to patients (our proposed method) against others, to evaluate their performance in estimating the intervention effect and its standard error.

  4. The system can provide recommendations for trial design by suggesting that the primary data analysis should fully reflect the randomisation, including blocking by time, and using Satterthwaite’s method for calculating degrees of freedom routinely.

Abstract

It is recognised that treatment-related clustering should be allowed for in the sample size and analyses of individually-randomised parallel-group trials that evaluate therapist-delivered interventions such as psychotherapy. Here, interventions are a treatment factor, but therapists are not. If the aim of a trial is to separate effects of therapists from those of interventions, we propose that interventions and therapists should be regarded as two potentially interacting treatment factors (one fixed, one random) with a factorial structure. We consider the specific design where each therapist delivers each intervention (crossed therapist-intervention design), and the resulting therapist-intervention combinations are randomised to patients. We adopt a classical Design of Experiments (DoE) approach to propose a family of orthogonal factorial designs and their associated data analyses, which allow for therapist learning and centre too. We set out the associated data analyses using ANOVA and regression and report the results of a small simulation study conducted to explore the performance of the proposed randomisation methods in estimating the intervention effect and its standard error, the between-therapist variance and the between-therapist variance in the intervention effect. We conclude that more purposeful trial design has the potential to lead to better evidence on a range of complex interventions and outline areas for further methodological research.

Related papers