The inclusive Synthetic Control Method

arXiv:2403.17624 · econ.EM · Submitted 2024-03-26 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Economics. The day in economics: theory, mechanisms and the evidence beneath them.

Ruth: I'm Ruth.

Warren: And I'm Warren, here with Iris.

Ruth: We're looking at the paper: "The inclusive Synthetic Control Method".

Warren: Here's the gist of the paper. The inclusive synthetic control method (iSCM) is introduced as a modification of synthetic control methods that includes units in the donor pool potentially affected by an intervention, making it ideal for situations where including treated units or spillover effects are essential <ref:2403.17624#pg2>.

Ruth: The inclusive synthetic control method (iSCM) is a novel procedure that allows us to eliminate post-intervention effects from control units and safely include them in the donor pool <ref:2403.17624#pg4>.

Warren: The iSCM exploits the fact that synthetic control weights are estimated using only preintervention data, during which no unit is treated <ref:2403.17624#pg6>.

Ruth: By adding ”potentially affected” units to the donor pool, the synthetic control becomes a weighted average of all donor units’ outcomes, including any intervention effects on these units, weighted by their synthetic weights <ref:2403.17624#pg7>.

Paper summary: Ruth: We've talked about how this inclusive synthetic control method is a modification of standard synthetic control, focusing on including potentially affected units in the donor pool to handle spillover effects or include treated units <ref:two thousand four hundred three point one seven six two four#pg1.

Warren: It really comes down to using the pre-intervention data for weights, and then calculating the effect by comparing the main treated unit against this weighted average of all donor outcomes <ref:two thousand four hundred three point one seven six two four#pg7.

Iris: And they showed that they can decompose the estimation error into two sources, one from how well your chosen estimator fits the counterfactual and another from using those potentially affected units in the pool <ref:two thousand four hundred three point one seven six two four#pg9.

Ruth: The authors found that this method is almost certain to enhance performance compared to a restricted synthetic control when applied empirically, like on German reunification <ref:2403.17624#pg10>.

Warren: So the paper's main implication is that if you have spillover effects or need those treated units in your donor pool, this inclusive method offers a robust way to estimate the causal effect without needing a completely different estimation technique <ref:two thousand four hundred three point one seven six two four#pg1.

Iris: They also provide tools for checking the results with permutation and placebo tests, which gives you more confidence in what you're seeing <ref:2403.17624#pg10>.

Ruth: Ultimately, this paper shows that if you need to include other treated units or deal with indirect effects from an intervention, the inclusive synthetic control method is a practical tool because it's easy to implement and often performs better than simpler versions <ref:two thousand four hundred three point one seven six two four#pg1.

Warren: It’s about using that structure of the system of equations to solve for the treatment effect directly, which is pretty clean mathematically <ref:two thousand four hundred three point one seven six two four#pg3.

Conclusion: Ruth: So, we've been looking at how this inclusive synthetic control method handles those tricky spillover effects and including other treated units in the donor pool.

Warren: Yeah, it seems like the authors are really pushing this as a way to get a more complete picture of what actually happened after an intervention.

Iris: They’re essentially saying that by including these potentially affected units, you’re not just getting a simple comparison against a control group, but something closer to the real world outcome.

Ruth: The title itself is pretty straightforward—inclusive synthetic control method—suggesting it's about making sure you don't miss any relevant parts of the story.

Warren: It sounds like the authors are arguing that if you ignore those units, your estimate of the main treatment effect gets skewed, which is a big deal for anyone trying to judge policy impact.

Iris: The method solves this by using pre-intervention data for the weights and then weighting everything in the donor pool, including those affected units.

Ruth: So, what does this actually mean when we think about real-world situations? It means you can't just keep it simple when you're dealing with complex systems where things influence each other.

Warren: And they’re showing that this approach isn't just a theoretical exercise; in their German reunification example, it seemed to actually give a better fit than the more restricted versions.

Iris: That’s interesting because usually, adding more units complicates things, but here they claim it helps with identification and robustness.

Ruth: They also offer some ways to test if this method is really working using permutation tests and placebo tests in time and space to check for any bias or just luck.

Warren: It sounds like the authors are not just building a new tool, but providing a way to validate existing synthetic control ideas with this enhanced version.

Iris: It’s important because it gives us a clearer path for checking our assumptions about how these complex economic or social systems actually respond to changes.

Ruth: So, the big picture here is that if your research involves units that interact or get indirectly affected by an event, this method might be a much more reliable way to estimate the outcome.

Warren: It sets up a challenge for us all to think about whether our current models are capturing those indirect connections correctly when we try to estimate causal effects.

Roberta Di Stefano, Giovanni Mellace

Sapienza University of Rome · University of Southern Denmark

econ.EM

Submitted: 2024-03-26

Updated: 2026-11-02

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 81/100

The gist: The inclusive synthetic control method (iSCM) is introduced as a modification of synthetic control methods that includes units in the donor pool potentially affected by an intervention…

Key concepts

Inclusive Synthetic Control Method (iSCM)
A novel procedure that enhances synthetic control by adding 'potentially affected' units to the donor pool. This makes the synthetic control a weighted average of all donor outcomes, including intervention effects on these added units, using pre-intervention data.
Decomposition of Estimation Error
The estimation error is split into two parts: Bias B1sc, which relates to how well the chosen synthetic control estimator fits the counterfactual, and Bias B1te. The latter is specifically caused by including units that are 'potentially affected' by the intervention in the donor pool.
System of Equations and Identification
The method generates a system of $m$ equations with $m$ unknowns, solvable via Cramer’s rule. Under certain assumptions, this leads to a matrix form $ heta_t = eta_tar{oldsymbol{ ho}}^{-1}$, which allows for the estimation of the treatment effect on the main treated unit.

Terminology

Summary

The inclusive synthetic control method (iSCM) is introduced as a modification of synthetic control methods that includes units in the donor pool potentially affected by an intervention, making it ideal for situations where including treated units or spillover effects are essential <ref:2403. p. 017624#pg2>.

Introduction to the iSCM

The inclusive synthetic control method (iSCM) is a novel procedure that allows us to eliminate post-intervention effects from control units and safely include them in the donor pool <ref:2403.17624#pg4>. The iSCM exploits the fact that synthetic control weights are estimated using only preintervention data, during which no unit is treated <ref:2403. p. 1 p. 117624#pg6>. By adding ”potentially affected” units to the donor pool, the synthetic control becomes a weighted average of all donor units’ outcomes, including any intervention effects on these units, weighted by their synthetic weights <ref:2403. p. 1 p. 117624#pg7>. Comparing the ”main treated” unit and its synthetic control provides an estimate of the true effect minus the effects on other units, each weighted by their synthetic weights <ref:2403. p. 217624#pg10>.

Decomposition of Estimation Error

The estimation error of YbN1t can be decomposed into two distinct sources: Bias B1sc, related to how well the chosen SC-type estimators approximate the post-intervention counterfactual, and Bias B1te, induced by using units that are ”potentially affected” by the intervention in the donor pool <ref:2403.17624#pg9>. Assumption 2 allows for assuming an (approximately) perfect fit for the chosen SC-estimator to recover YN1t. p. 7 Lemma 1 shows that under Assumption 1 and 2, the treatment effect on the ”main treated” unit is approximately θb1t = θ1t − m∑j=2wbjγjt +op(1). p. 2

System of Equations and Identification

The method results in a system of m equations with m unknowns, which can be solved using Cramer’s rule. The resulting system is represented by the matrix form ϑt = βtomegab−1, where omegab is the (m×m) matrix of known quantities and βt is the vector of known quantities. Under Assumption 4, which states that omegab is non-singular, Theorem 1 establishes that θbiSCM t = omegab−1βt = ϑt +op(1). p. 7

Empirical Application and Comparison

In the empirical application on German reunification, the iSCM was found to be preferable to the ”restricted” SCM because it is almost certain to enhance performance compared to the ”unrestricted” SCM. The authors demonstrate that iSCM can serve as a robustness check even when excluding potentially affected units from the donor pool does not seem harmful. p. 3 p. 3 Table 1 shows that the “unrestricted” synthetic version of West Germany is much closer in terms of observable characteristics to West Germany than the “restricted” version. p. 0

Inference Procedures

For inference, permutation tests are suggested, including placebo tests in time and in-space. A specific in-space placebo test for θbiSCM t is defined by replacing Yit with Yit −γb iSCM it for all i = 2, to compute the test statistic. The authors also provide a data-driven procedure to determine whether iSCM could potentially outperform the ”restricted” SCM. p. 3

Conclusion

Our method is useful in applications where it is either important to include other treated units in the donor pool or where some of the units in the donor pool are affected indirectly by the intervention (spillover effects). A major advantage of iSCM is that it can be easily implemented using the synthetic control estimator or many of the new estimation methods available in the literature. p. 3 p. 3 Moreover, we demonstrate that iSCM is almost certain to enhance the performance compared to the ”unrestricted” SCM. p. 2 p. 2 Additionally, we introduce a data-driven approach to determine whether iSCM could potentially outperform the ”restricted” SCM. p. 3 p. 3 Finally, we illustrate the use of iSCM by re-estimating the impact of German reunification on GDP per capita. p. 0

References

ABADIE, A. (2021): “Using synthetic controls: Feasibility, data requirements, and methodological aspects,” Journal of Economic Literature, 59, 391–425. p. 1

ABADIE, A., A. DIAMOND, AND J. HAINMUELLER (2010): “Synthetic Control Methods for Comparative Case Studies: Estimating the Effect of California’s Tobacco Control Program,” Journal of the American Statistical Association, 105, 493–505. p. 1

ABADIE, A. AND J. GARDEAZABAL (2003): “The Economic Costs of Conflict: A Case Study of the Basque Country,” American Economic Review, 93, 113–132. p. 1

ABADIE, A. AND J. L’HOUR (2021): “A penalized synthetic control estimator for disaggregated data,” Journal of the American Statistical Association, 116, 1817–1834. p. 3

AMJAD, M., D. SHAH, AND D. SHEN (2018): “Robust synthetic control,” The Journal of Machine Learning Research, 19, 802–852. p. 7

BEN-MICHAEL, E. p. 2 p. 2, A. FELLER, AND J. ROTHSTEIN (2021): “The augmented synthetic control method,” Journal of the American Statistical Association, 116, 1789–1803. p. 2

BEN-MICHAEL, E., A. FELLER, AND J. ROTHSTEIN (2022): “Synthetic controls with staggered adoption,” Journal of the Royal Statistical Society Series B, 84, 351–381.

CAO, J. AND C. DOWD (2019): “Estimation and Inference for Synthetic Control Methods with Spillover Effects,” Papers 1902. p. 207343, arXiv. p. 0org.

CERQUA, A. AND G. PELLEGRINI (2017): “Industrial policy evaluation in the presence of spillovers,” Small Business Economics, 49, 671–686. p. 0

CHERNOZHUKOV, V., K. WUTHRICH ¨, AND Y. ZHU (2021): “An exact and robust conformal inference method for counterfactual and synthetic controls,” Journal of the American Statistical Association, 116, 1849–1864.

DOUDCHENKO, N. p. 2 AND G. W. IMBENS (2017): “Balancing, Regression, Difference-In-Differences and Synthetic Control Methods: A Synthesis,” arXiv:1610. p. 107748.

FERMAN, B. AND C. PINTO (2021): “Synthetic controls with imperfect pretreatment fit,” Quantitative Economics, 12, 1197–1221. p. 7

FIRPO, S. AND V. POSSEBOM (2018): “Synthetic control method: Inference, sensitivity analysis and confidence sets,” Journal of Causal Inference, 6, 1–26. p. 0

FORASTIERE, L. p. 2, E. M. AIROLDI, AND F. MEALLI (2021): “Identification and estimation of treatment and interference effects in observational studies on networks,” Journal of the American Statistical Association, 116, 901–918. p. 3

GEHLER, M. AND M. GRAF (2018): “Austria, German Unification, and European Integration: A Brief Historical Background,” working paper-Cold War International History Project.

GOBILLON, L. AND T. MAGNAC (2016): “Regional policy evaluation: interactive fixed effects and synthetic controls,” Review of Economics and Statistics, 98, 535–551.

GROSSI, G., M. MARIANI, A. MATTEI, P. LATTARULO, AND O. ¨ O¨ NER (2024): “Direct and spillover effects of a new tramway line on the commercial vitality of peripheral streets: a synthetic-control approach,” Journal of the Royal Statistical Society Series A: Statistics in Society, qnae032. p. 3

HUBER, M. AND A. STEINMAYR (2019): “A framework for separating individual-level treatment effects from spillover effects,” Journal of Business & Economic Statistics, 1–15. p. 2

KELLOGG, M. p. 2, M. MOGSTAD, G. A. POULIOT, AND A. TORGOVITSKY (2021): “Combining matching and synthetic control to tradeoff biases from extrapolation and interpolation,” Journal of the American statistical association, 116, 1804–1816. p. 3

LI, K. T. (2019): “Statistical inference for average treatment effects estimated by synthetic control methods,” Journal of the American Statistical Association, 1–16. p. 3

POWELL, D. (2018): Imperfect Synthetic Controls: Did the Massachusetts Health Care Reform Save Lives? p. 8, Santa Monica, CA: RAND Corporation.

ROTH, J. (2022): “Pretest with Caution: Event-Study Estimates after Testing for Parallel Trends,” American Economic Review: Insights, 4, 305–22.

VAZQUEZ-BARE, G. p. 2 (2022): “Identification and estimation of spillover effects in randomized experiments,” Journal of Econometrics. p. 2

XU, Y. (2017): “Generalized Synthetic Control Method: Causal Inference with Interactive Fixed Effects Models,” Political Analysis, 25, 57–76. p. 0

ZHANG, X., W. WANG, AND X. ZHANG (2022): “Asymptotic Properties of the Synthetic Control Method,”. p. 7

A Appendix

A.1 Non-singularity

Let omegabi j a generic element of omegab. As omegab is a square matrix, it is non-singular if, and only if, its determinant is different from zero. p. 2

A.2 RMSPEs ratios

Panels 3a and 3b of Figure 3 show the ratios between the RMSPEs in the post- and pre-reunification of West Germany’s donor pool and Austria’s donor pool, respectively.

A.3 A geometrical interpretation on why use iSCM

Figure 4 shows the scenario in which both the ”main treated” and the ”potentially affected” units lie inside the convex hull of the ”pure control” units. Figure 5 presents some scenarios in which only one unit between the ”main treated” and the “potentially affected” unit lies inside the convex hull of the other and the ”pure control” units. p. 6

Figure 5 shows a scenario in which using iSCM clearly makes things worse. p. 4

--- Page 5 ---

The inclusive synthetic control method (iSCM) is introduced as a modification of synthetic control methods that includes units in the donor pool potentially affected by an intervention, making it ideal for situations where including treated units or spillover effects are essential <ref:2403.17624#pg2>. The iSCM exploits the fact that synthetic control weights are estimated using only preintervention data, during which no unit is treated <ref:2403.17624#pg6>. By adding ”potentially affected” units to the donor pool, the synthetic control becomes a weighted average of all donor units’ outcomes, including any intervention effects on these units, weighted by their synthetic weights <ref:2403.17624#pg7>.

The estimation error of YbN1t can be decomposed into two distinct sources: Bias B1sc, related to how well the chosen SC-type estimators approximate the post-intervention counterfactual, and Bias B1te, induced by using units that are ”potentially affected” by the intervention in the donor pool <ref:2403.17624#pg9>.

The method results in a system of m equations with m unknowns, which can be solved using Cramer’s rule. The resulting system is represented by the matrix form ϑt = βtomegab−1, where omegab is the (m×m) matrix of known quantities and βt is the vector of known quantities.

In the empirical application on German reunification, the iSCM was found to be preferable to the ”restricted” SCM because it is almost certain to enhance performance compared to the ”unrestricted” SCM. The authors demonstrate that iSCM can serve as a robustness check even when excluding potentially affected units from the donor pool does not seem harmful.

For inference, permutation tests are suggested, including placebo tests in time and in-space. A specific in-space placebo test for θbiSCM t is defined by replacing Yit with Yit −γb iSCM it for all i = 2, to compute the test statistic.

Our method is useful in applications where it is either important to include other treated units in the donor pool or where some of the units in the donor pool are affected indirectly by the intervention (spillover effects). A major advantage of iSCM is that it can be easily implemented using the synthetic control estimator or many of the new estimation methods available in the literature. Moreover, we demonstrate that iSCM is almost certain to enhance the performance compared to the ”unrestricted” SCM. Additionally, we introduce a data-driven approach to determine whether iSCM could potentially outperform the ”restricted” SCM.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Develop an Inclusive Synthetic Control Method (iSCM) to estimate causal effects when including units in the donor pool that are potentially affected by an intervention, such as other treated units or spillover-affected control units. This allows for better pre-intervention fit, as the method allows for the direct inclusion of these affected units in the donor pool without modifying the underlying SCM estimator.

  2. Implement a data-driven procedure to decide whether iSCM is preferable to a restricted Synthetic Control Method (SCM). This involves comparing Root Mean Squared Prediction Errors (RMSPEs) and predictor biases between the methods, as suggested by Check whether the 'potentially affected' units receive substantial weights and comparing RMSPEs in the pre-intervention period of the 'restricted' SCM and 'unrestricted' SCM.

  3. Construct causal effect estimators using Cramer’s rule based on a system of equations derived from iSCM, which can be solved for the treatment effect on the main treated unit and effects on other affected units. This enables precise estimation of multiple treatment effects simultaneously, such as theta b1t = theta1t - wb2 gamma2t - wb3 gamma3t -... - wbm gamma mt.

  4. Design inference procedures using permutation tests, specifically in-space placebo tests, to measure the statistical significance of causal effects. This involves replacing the outcome of a unit with its estimated effect from iSCM when constructing the residual for testing, as detailed in Section 6.

Sources

Related papers