The Cochran--Mantel--Haenszel test as a test of the average treatment effect in stratified randomized trials
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Statistics. The day in statistics: methods, theory and the inference under them.
Tom: We're looking at the paper: "The Cochran--Mantel--Haenszel test as a test of the average treatment effect in stratified randomized trials".
Jane: Here's the gist of the paper. The Cochran-Mantel-Haenszel test can be interpreted as a standardized version of the augmented inverse probability weighted (AIPW) estimate of the causal average treatment effect, showing its utility beyond its traditional common-odds ratio interpretation CMH Test Interpretation The Cochran–Mantel–Haenszel (CMH) test statistic can be interpreted as a standardized version of the augmented inverse probability weighted (AIPW) estimate of the causal average treatment effect It shows that the CMH test statistic can be interpreted as a standardized version of the augmented inverse probability weighted (AIPW) estimate of the causal average treatment effect Consequently, it is not restricted to an interpretation as a test of the common-odds ratio in a stratified logistic regression model The note then discusses conditional and unconditional variances for this estimator as well as variance estimation and its implications for testing Connection to AIPW Estimation The connection between the CMH numerator and the AIPW estimator is established under stratified randomization with a fixed common treatment fraction q The estimator of the marginal risk difference coincides exactly with the AIPW estimate This exact equivalence between the CMH numerator and the AIPW estimator depends on stratified randomization with a fixed common treatment fraction within each stratum The reason is that the augmentation term Yi − mˆ ji (si) cancels exactly because treatment allocation is fixed within each stratum and there is no residual covariate imbalance to adjust for once conditioning on the stratum membership random variable S Variance Estimation and Testing The paper derives three notions of variance for the average treatment effect, including conditional variance given observed stratum sizes and unconditional variance The CMH test uses the variance conditional on the observed nk and r·k, yielding an exact conditional test if the exact hypergeometric distribution is used The classical CMH test uses a conditional null-based variance Consequences for Testing The comparison between different variance choices reveals that the CMH test is a conditional test that preserves the type I error under a strong null hypothesis of no treatment effect in any stratum The main point is not that the CMH test must have uniformly higher power than an unrestricted AIPW Wald test The CMH test provides a conditionally calibrated test based on the same stratified ATE estimator, without requiring the common-odds-ratio model often used to motivate it Its practical appeal lies in this conditional calibration, its robustness to that model assumption, and its close connection to the AIPW estimator in the present discrete-strata randomized setting Asymptotic Type I Error for Different Versions of Null Hypotheses For the null hypothesis H0: dk = 0 for all k versus the restricted one-sided alternative HA: dk ≥ 0 for all k, with strict inequality for at least one k, the one-sided version of the CMH test is an asymptotically type I error preserving test If the alternative to the strong H0 is the unrestricted alternative ”not H0” (i.e.
Tom: including cases with dk < 0 for some k), then the alternative contains situations where benefits in some strata and harms in others can cancel out such that ∆ = 0 For this null, the CMH test is no longer a valid level-α test in general In the practically important case of 1: 1-randomization, however, it is To show the latter claim, set Tn:= X K k=1 nk(p1k − p0k) Therefore, by the central limit theorem, Tn √ n d−→ N(0, σ2 S), with σ2 S = X K k=1 ψk π1k(− π1k) q + π0k(− π0k) 1 − q Since T squared n σ squared S·n ∼ χ squared 1, it follows that χ squared CMH = c · χ squared 1 where c = PK k=1 ψk
(− q)π1k(− π1k) + qπ0k(− π0k): Therefore, if some dk ≠ 0, then c < 1, rendering the test asymptotically conservative The main contribution of this note is to clarify that the CMH test can be understood not only through the traditional common-odds-ratio framework, but also through the lens of causal estimation and doubly robust methodology It also clarifies a practically important point: in stratified randomized trials with binary endpoints, the asymptotic elegance of unrestricted AIPW inference does not eliminate finite-sample concerns arising from sparsity, boundary estimates, and treatment-effect heterogeneity across strata In such settings, the conditional nature and pooled variance structure of the CMH test can remain attractive for testing Pruning redundant reasoning tokens The paper is organized as follows: In section 2, we quickly revisit the CMH test, representing it in a form that allows the connection with AIPW-estimation In section 3, this connection is made Section 4 considers different variance estimators of the AIPW point estimate and discusses their appropriateness for different situations Section 5 discusses the consequences of using these different variance estimates in testing and section 6 shows that the CMH test is essentially ”model-free” in the case of 1:1-randomization The paper concludes with a discussion How the training signal is built The CMH test statistic can then be written as X2 CMH = q(1 − q) hPK k=1 nk(p1k − p0k) i2 PK k=1 n2 k p·k(1−p·k) A slightly simpler large-sample approximation is X2 CMH ≈ q(1 − q) hPK k=1 nk(p1k − p0k) i2 PK k=1 nkp·k(1 − p·k) This formula shows the connection between the CMH numerator and the AIPW estimator Where the gains come from The first distinction, conditional versus unconditional inference, is the main focus of this note The classical CMH statistic combines both features: it conditions on the observed nk and r·k, and it uses a null-based pooled variance The main point is not that the CMH test must have uniformly higher power than an unrestricted AIPW Wald test The CMH test provides a conditionally calibrated test based on the same stratified ATE estimator, without requiring the common-odds-ratio model often used to motivate it Its practical appeal lies in this conditional calibration, its robustness to that model assumption, and its close connection to the AIPW estimator in the present discrete-strata randomized setting Where the gains come from The first distinction, conditional versus unconditional inference, is the main focus of this note The classical CMH statistic combines both features: it conditions on the observed nk and r·k, and it uses a null-based pooled variance The main point is not that the CMH test must have uniformly higher power than an unrestricted AIPW Wald test The CMH test provides a conditionally calibrated test based on the same stratified ATE estimator, without requiring the common-odds-ratio model often used to motivate it Where the gains come from
Title and authors: Tom: We've covered how this paper connects the Cochran--Mantel--Haenszel test to causal estimation, moving past its old role as just a common odds ratio check.
Jane: Right, and what that means in plain English is that this test isn't stuck needing that common odds ratio assumption anymore; it’s more flexible now for causal inference.
Lu: Exactly. The authors show an exact mathematical link between the Cochran--Mantel--Haenszel statistic and the augmented inverse probability weighted estimator when you have stratified randomization with a fixed common treatment fraction q.
Meng: So, if we're running a trial where we know exactly how many people are in each stratum, that link holds up perfectly for every sample.
Tom: That’s the exact equivalence they establish; it means the numerator of the CMH test is identical to that AIPW estimator under those conditions.
Jane: But then they don't just stop there; they get into how we measure uncertainty, looking at three different types of variance for that average treatment effect.
Lu: They look at conditional variance given observed stratum sizes, unconditional variance, and another one too. It’s about how the choice of variance affects the test itself.
Meng: I'm curious about that comparison; does it mean we can pick a specific way to test without worrying too much about what those variances are doing under different scenarios?
Tom: The main point they hammer home is that even though unrestricted AIPW point estimators might be more precise for the asymptotic behavior, the Cochran--Mantel--Haenszel test is still a conditionally calibrated test based on that same estimator.
Jane: It’s a conditional calibration; it doesn't need those restrictive common odds ratio models to work correctly in this stratified setting.
Lu: That robustness to the common odds ratio assumption is pretty key, especially when you’re dealing with discrete strata and binary endpoints.
Meng: That means we can rely on this conditional structure for testing without needing that extra layer of model assumption that the old CMH test required.
Tom: And they do look at the null hypothesis behavior too; they show that the one-sided version of the test actually preserves type one error when you have a strong null hypothesis saying there’s no treatment effect in any stratum.
Jane: That's a strong result for testing because it gives us confidence in the test's performance under that specific, most restrictive null condition.
Lu: But they also warn us about the alternative; if some strata get benefits and others get harms, meaning the average effect is zero anyway, then the CMH test isn't a valid level alpha test for all cases.
Meng: That’s a big caveat; it shows that while it’s good under very strict conditions, you can run into issues with heterogeneity across those strata.
Tom: So while the CMH test offers this conditional calibration based on the AIPW estimator, we have to be careful when the data doesn't fit that specific strong null hypothesis.
Lu: The paper does point out that in one to one randomization trials with binary outcomes, there’s a specific mathematical structure where the Cochran--Mantel--Haenszel statistic scales nicely with a chi squared one distribution.
Meng: That scaling is helpful for setting up tests because we know exactly how the statistic behaves under the null hypothesis of no effect across all strata.
Tom: So in summary, this paper lets us use the CMH test through the lens of causal estimation, offering conditional calibration based on an estimator that works well in stratified randomized trials.
Jane: It’s a tool for testing that respects stratification when you know your stratum composition beforehand, but you have to keep an eye on those finite sample concerns about heterogeneity across groups.
Lu: The next thing we're looking at is how these different variance choices actually play out in practice when we try to set up the formal hypothesis tests.
The paper's summary: Tom: We've covered how this test connects to the AIPW estimator and how it behaves under certain null hypotheses, and now we're talking about what they suggest improving about the original test framework itself.
Jane: Right, so what are these suggested improvements? I mean if this is a standardized version of something else, why not just use the better method directly?
Lu: The authors aren't suggesting you ditch it entirely; they’re focusing on how to handle the variance estimation part more rigorously. They show how different choices of variance estimators—like conditional versus unconditional ones—change what you get when you are actually testing something.
Meng: So, if I'm deploying a model, does this mean I have to be super careful about which variance structure I use? Because if the wrong one is picked, the test might look good when it isn't.
Tom: Exactly; they emphasize that the classical CMH statistic combines both conditional information and a null-based pooled variance, which is a specific choice for testing.
Jane: But they show how this compares to what you might get from an unrestricted or unconditional superpopulation variance used in Wald-type AIPW tests.
Lu: The improvement here is showing that the CMH test provides a conditionally calibrated test based on the same stratified average treatment effect estimator without needing that common odds ratio model.
Tom: That conditional calibration is what they are pushing; it means you get a valid test structure even if your data doesn't perfectly fit the common odds ratio assumption.
Meng: So, the practical gain is that you can use this structure for testing based on causal estimation without being tied to that old model, but I still have to worry about how those stratum sizes are observed in my real traffic.
Jane: That’s right; they highlight the trade-off between asymptotic precision and finite sample concerns arising from sparsity or heterogeneity across strata.
Lu: They aren't claiming it’s a universal fix for all problems, but they show that this conditional structure remains attractive for testing in settings with fixed stratum composition.
Tom: It seems like the main improvement is solidifying the link between the estimation and the testing structure, making it robust to those common odds ratio assumptions while acknowledging where those real-world finite sample issues still bite.
Jane: So, instead of just seeing it as a test for one thing—the common odds ratio—it’s being framed as a test derived from a causal average treatment effect estimator that is conditionally calibrated.
Lu: And this shifts the focus from just checking an assumption to building the actual testing framework around the causal quantity itself.
Meng: That means if I'm building my production pipeline, I can focus on making sure my data structure supports that conditional test setup instead of trying to force a model into an old framework.
Tom: So, we’re moving away from just checking for one specific hypothesis and toward using a test that respects the structure of the causal estimate under stratified randomization.
Jane: And this moves us toward understanding how to properly set up tests when you know the underlying data structure is fixed across groups.
Lu: The next thing we'll look at is how these variance choices actually impact the formal hypothesis testing and what that means for setting significance thresholds.
The paper's improvements: Tom: So we're wrapping up on this paper, "The Cochran--Mantel--Haenszel test as a test of the average treatment effect in stratified randomized trials," which essentially shows how to use that old CMH statistic through a causal estimation lens.
Jane: It’s clear they established that the CMH test isn't just for checking common odds ratios anymore; it’s being framed as a conditionally calibrated tool based on the same stratified average treatment effect estimator.
Lu: The main implication is that we get a testing structure that respects stratification when you know the stratum sizes, without needing to assume the common odds ratio model holds perfectly.
Meng: For me, what this means is I can deploy this test with confidence in randomized trials where I know my treatment groups are balanced across strata, because it’s calibrated to that specific setup.
Tom: It gives us a way to test for no effect across all strata that preserves type one error under a strong null hypothesis of zero treatment effect everywhere.
Jane: But we still have those caveats about heterogeneity; if some groups benefit and others harm, the test becomes conservative because it doesn't account for that imbalance in the alternative hypothesis.
Lu: And they do flag that while unrestricted AIPW point estimators might be more precise asymptotically, you still have to deal with those finite sample issues like sparsity or treatment effect differences across strata.
Meng: That means if I'm looking at real traffic data where I can't assume perfect stratification, this method still has those same finite sample headaches.
Tom: It’s a trade-off: you gain the conditional calibration and model flexibility but still have to manage the real-world data mess.
Jane: So we’re left with a conditionally calibrated test for stratified randomized trials, which is pretty robust, as long as you understand its limitations regarding heterogeneity.
Lu: The core contribution is showing this close connection to the augmented inverse probability weighted estimator and dissecting how different variance choices impact the formal hypothesis testing under those conditions.
Meng: I just want to know what happens when we try to run these tests on a huge scale with millions of observations; does it become too computationally expensive?
Tom: The paper focuses more on the theoretical structure than the specific computational cost at massive scale, but it points toward a framework that is robust for those fixed stratum settings.
Jane: It’s a solid tool for researchers who are dealing with randomized trials where the structure of their data is known in advance.
Conclusion: Tom: We're starting with this paper, "The Cochran--Mantel--Haenszel test as a test of the average treatment effect in stratified randomized trials." It looks like they are taking an old statistical tool and giving it a causal interpretation.
Jane: That sounds like they’re trying to take something that’s traditionally used for checking common odds ratios and show it works in a new way with causal inference methods.
Lu: Exactly. The paper shows that the CMH test can be interpreted as a standardized version of the augmented inverse probability weighted estimate of the causal average treatment effect.
Meng: So, if we're talking about something that’s traditionally used for checking odds ratios, what does this mean for someone who is actually running an experiment?
Tom: It means that instead of just looking at whether a common odds ratio is one, you can use the CMH test to test the average treatment effect directly.
Jane: That sounds like a big shift because it moves us away from relying on that specific model assumption.
Lu: The authors show an exact mathematical link between the CMH statistic and the augmented inverse probability weighted estimator when you have stratified randomization with a fixed common treatment fraction q, which is pretty neat mathematically.
Meng: So, if we know how many subjects are in each stratum, that link holds up perfectly for every sample.
Lu: But then they point out that if the alternative is the unrestricted alternative—meaning some strata benefit and others harm, so the average effect is zero anyway—the CMH test isn't a valid level alpha test in general.
Jane: And they leave us thinking about how this conditional nature of the test might
Novartis Pharma AG · Otto von Guericke University Magdeburg
stat.AP
Submitted: 2026-10-01
Updated: 2026-11-16
Comments: 20 pages
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 69/100
The gist: The Cochran-Mantel-Haenszel test can be interpreted as a standardized version of the augmented inverse probability weighted (AIPW) estimate of the causal average treatment effect…
Key concepts
- Cochran–Mantel–Haenszel (CMH) Test
- The CMH test statistic is interpreted as a standardized version of the AIPW estimate for the average treatment effect. It is not limited to testing a common odds ratio and offers a conditionally calibrated test based on the same estimator, making it robust to model assumptions.
- Augmented Inverse Probability Weighted (AIPW) Estimate
- The AIPW estimate is a method used to calculate the causal average treatment effect. The paper shows that the CMH numerator exactly matches this AIPW estimator under specific conditions like stratified randomization with fixed treatment fractions within each stratum.
- Conditional vs. Unconditional Variance
- The paper derives different variance measures for the average treatment effect, including conditional variance based on observed stratum sizes and unconditional variance. The CMH test utilizes a conditional test that preserves the Type I error under a strong null hypothesis, offering a practical advantage in testing.
Terminology
Summary
The Cochran-Mantel-Haenszel test can be interpreted as a standardized version of the augmented inverse probability weighted (AIPW) estimate of the causal average treatment effect, showing its utility beyond its traditional common-odds ratio interpretation
CMH Test Interpretation
The Cochran–Mantel–Haenszel (CMH) test statistic can be interpreted as a standardized version of the augmented inverse probability weighted (AIPW) estimate of the causal average treatment effect It shows that the CMH test statistic can be interpreted as a standardized version of the augmented inverse probability weighted (AIPW) estimate of the causal average treatment effect Consequently, it is not restricted to an interpretation as a test of the common-odds ratio in a stratified logistic regression model The note then discusses conditional and unconditional variances for this estimator as well as variance estimation and its implications for testing
Connection to AIPW Estimation
The connection between the CMH numerator and the AIPW estimator is established under stratified randomization with a fixed common treatment fraction q The estimator of the marginal risk difference coincides exactly with the AIPW estimate This exact equivalence between the CMH numerator and the AIPW estimator depends on stratified randomization with a fixed common treatment fraction within each stratum The reason is that the augmentation term Yi − mˆ ji (si) cancels exactly because treatment allocation is fixed within each stratum and there is no residual covariate imbalance to adjust for once conditioning on the stratum membership random variable S
Variance Estimation and Testing
The paper derives three notions of variance for the average treatment effect, including conditional variance given observed stratum sizes and unconditional variance The CMH test uses the variance conditional on the observed nk and r·k, yielding an exact conditional test if the exact hypergeometric distribution is used The classical CMH test uses a conditional null-based variance
Consequences for Testing
The comparison between different variance choices reveals that the CMH test is a conditional test that preserves the type I error under a strong null hypothesis of no treatment effect in any stratum The main point is not that the CMH test must have uniformly higher power than an unrestricted AIPW Wald test The CMH test provides a conditionally calibrated test based on the same stratified ATE estimator, without requiring the common-odds-ratio model often used to motivate it Its practical appeal lies in this conditional calibration, its robustness to that model assumption, and its close connection to the AIPW estimator in the present discrete-strata randomized setting
Asymptotic Type I Error for Different Versions of Null Hypotheses
For the null hypothesis H0: dk = 0 for all k versus the restricted one-sided alternative HA: dk ≥ 0 for all k, with strict inequality for at least one k, the one-sided version of the CMH test is an asymptotically type I error preserving test If the alternative to the strong H0 is the unrestricted alternative ”not H0” (i.e. including cases with dk < 0 for some k), then the alternative contains situations where benefits in some strata and harms in others can cancel out such that ∆ = 0 For this null, the CMH test is no longer a valid level-α test in general In the practically important case of 1: 1-randomization, however, it is To show the latter claim, set Tn:= X K k=1 nk(p1k − p0k) Therefore, by the central limit theorem, Tn √ n d−→ N(0, σ2 S), with σ2 S = X K k=1 ψk π1k(− π1k) q + π0k(− π0k) 1 − q Since T squared n σ squared S·n ∼ χ squared 1, it follows that χ squared CMH = c · χ squared 1 where c = PK k=1 ψk [(− q)π1k(− π1k) + qπ0k(− π0k)] Therefore, if some dk ≠ 0, then c < 1, rendering the test asymptotically conservative The main contribution of this note is to clarify that the CMH test can be understood not only through the traditional common-odds-ratio framework, but also through the lens of causal estimation and doubly robust methodology It also clarifies a practically important point: in stratified randomized trials with binary endpoints, the asymptotic elegance of unrestricted AIPW inference does not eliminate finite-sample concerns arising from sparsity, boundary estimates, and treatment-effect heterogeneity across strata In such settings, the conditional nature and pooled variance structure of the CMH test can remain attractive for testing
Pruning redundant reasoning tokens
The paper is organized as follows: In section 2, we quickly revisit the CMH test, representing it in a form that allows the connection with AIPW-estimation In section 3, this connection is made Section 4 considers different variance estimators of the AIPW point estimate and discusses their appropriateness for different situations Section 5 discusses the consequences of using these different variance estimates in testing and section 6 shows that the CMH test is essentially ”model-free” in the case of 1:1-randomization The paper concludes with a discussion
How the training signal is built
The CMH test statistic can then be written as X2 CMH = q(1 − q) hPK k=1 nk(p1k − p0k) i2 PK k=1 n2 k p·k(1−p·k) A slightly simpler large-sample approximation is X2 CMH ≈ q(1 − q) hPK k=1 nk(p1k − p0k) i2 PK k=1 nkp·k(1 − p·k) This formula shows the connection between the CMH numerator and the AIPW estimator
Where the gains come from
The first distinction, conditional versus unconditional inference, is the main focus of this note The classical CMH statistic combines both features: it conditions on the observed nk and r·k, and it uses a null-based pooled variance The main point is not that the CMH test must have uniformly higher power than an unrestricted AIPW Wald test The CMH test provides a conditionally calibrated test based on the same stratified ATE estimator, without requiring the common-odds-ratio model often used to motivate it Its practical appeal lies in this conditional calibration, its robustness to that model assumption, and its close connection to the AIPW estimator in the present discrete-strata randomized setting
The first distinction, conditional versus unconditional inference, is the main focus of this note The classical CMH statistic combines both features: it conditions on the observed nk and r·k, and it uses a null-based pooled variance The main point is not that the CMH test must have uniformly higher power than an unrestricted AIPW Wald test The CMH test provides a conditionally calibrated test based on the same stratified ATE estimator, without requiring the common-odds-ratio model often used to motivate it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Improved AI systems can interpret Cochran-Mantel-Haenszel test statistics as standardized versions of augmented inverse probability weighted estimates of causal average treatment effect, moving beyond its traditional interpretation as a common-odds ratio test.
-
The improved system can perform exact equivalence verification between the CMH numerator and the AIPW estimator in settings with equal treatment allocation and stratified randomization, ensuring that the connection holds exactly for every realized sample.
-
The system can distinguish between different variance choices for testing, specifically contrasting a conditional, null-based variance used by CMH with an unrestricted or unconditional superpopulation variance used by Wald-type AIPW tests to assess power and finite sample behavior.
-
Improved systems can leverage the CMH test's ability to remove variation due to random stratum composition by conditioning on observed stratum sizes, which is particularly useful in randomized clinical trials where stratum composition is fixed before outcome analysis.
-
The system can identify that while unrestricted AIPW point estimators might be more precise asymptotically, the CMH test remains a conditionally calibrated test based on the same estimator without requiring restrictive common-odds-ratio models.
Abstract
This note revisits the Cochran--Mantel--Haenszel (CMH) test in light of the increasing use of causal inference methods in clinical trials. It shows that the CMH test statistic can be interpreted as a standardized version of the augmented inverse probability weighted (AIPW) estimate of the causal average treatment effect. Consequently, it is not restricted to an interpretation as a test of the common-odds ratio in a stratified logistic regression model. The note then discusses conditional and unconditional variances for this estimator as well as variance estimation and its implications for testing. The results clarify the relationship between the classical conditional CMH test and Wald-type AIPW tests, and show that, under equal treatment allocation, the CMH test is an asymptotically exact test of the weak null hypothesis of zero average treatment effect, without any restriction on the heterogeneity of stratum-specific effects. Under unequal allocation, it can be liberal or conservative. A simulation study supports these findings.
Related papers
- Unraveling time-varying causal effects of multiple exposures: integrating Functional Data Analysis with Multivariable Mendelian Randomization
- Strategic Causal Policy Learning with Manipulable Eligibility
- BayesJudge: Uncertainty-Aware Bayesian Meta-Evaluation of Human and LLM Judgments
- Automating structural reliability analysis with a multi-agent large language model framework
- Improving Minority Population Sampling with BISG Probabilities: Evidence from a Survey of Jewish Americans
- When Do Sports Leagues Stop Being Competitive? Bayesian Detection of Competitive-Balance Eras from Match-Level Data