On the Use of Design-Based Simulations
summary
The gist
Design-based simulations—procedures that hold realized outcomes fixed and generate variation by resampling treatment assignment or shocks—are widely used in both methodological and applied work…
In short
Design-based simulations, which fix outcomes and vary treatment assignment or shocks, are used to test inference procedures in shift-share designs. Standard simulations can falsely inflate distortions caused by spatial correlation because they mix true treatment effects with error dependence. The paper proposes alternative simulation designs that are more informative about these spatial correlation problems.
Key concepts
- Design-based Simulations
- These are procedures where researchers keep the actual outcomes fixed and create variation by randomly changing which treatment a unit receives or by introducing shocks. They help assess how reliable statistical tests are when dealing with complex data structures like shift-share designs.
- Confounding Treatment Effects with Error Dependence
- This occurs when standard simulations fix the observed outcomes, but the true underlying data-generating process would naturally change if a treatment effect existed. This mismatch causes the simulation results to incorrectly attribute errors in inference to spatial correlation instead of actual treatment effects.
- Alternative Simulation Frameworks
- The paper suggests using different simulation setups, such as epsilon-fixed or placebo outcome simulations, instead of standard ones. These alternatives are better at revealing distortions caused by spatial correlation because they are less likely to confuse the true treatment effect with noise in the data.
Terminology used across episodes
This episode discusses
- On the Use of Design-Based Simulations · Paper Radio
- Randomization Inference Tests for Shift-Share Designs
- Assessing Inference Methods
The paper
On the Use of Design-Based Simulations · Read on arXiv
Bruno Ferman
Sao Paulo School of Economics – FGV
Transcript
Introduction to the show: ident: Economics. The day in economics: theory, mechanisms and the evidence beneath them.
Ruth: I'm Ruth.
Warren: And I'm Warren, here with Iris.
Ruth: We're looking at the paper: "On the Use of Design-Based Simulations".
Warren: Here's the gist of the paper. Design-based simulations—procedures that hold realized outcomes fixed and generate variation by resampling treatment assignment or shocks—are widely used in both methodological and applied work to assess inference procedures Focusing on shift-share designs, standard simulations that fix outcomes and resample shocks may rely on a data-generating process that is not aligned with the true one.
Ruth: In particular, these simulations confound true treatment effects with error dependence, potentially overstating inference distortions due to spatial correlation.
Warren: The Core Problem in Simulation Design The fundamental challenge with holding realized outcomes fixed is that the true data-generating process would not be given by the fixed outcomes if treatment effects are present.
Ruth: For example, if potential outcomes are symmetric and single-peaked, a design-based simulation might consider a DGP in which the distributions of potential outcomes may be dual peaked, thus differing fundamentally from the true DGP.
Title and authors: Ruth: So we're looking at this paper 'On the Use of Design-Based Simulations'. It’s about how these simulations, which fix the outcomes and then resample things like treatment assignment or shocks to test an inference procedure, actually work.
Warren: Yeah, and the authors Bruno Ferman and others are really digging into whether these procedures are actually informative about whether an inference method is valid. It seems like they’re setting up a framework to see if standard simulations can mislead us in some situations.
Iris: They focus specifically on shift-share designs, which is where these simulations have been used a lot lately, so we're seeing how this applies to real-world data structures.
Ruth: Exactly. It’s not just about the math; it’s about what those fixed outcomes and resampled shocks actually mean for the inference we get out of them.
Warren: If these simulations are flawed, they might be telling us something completely wrong about how much effect a treatment actually had versus just some kind of error structure in the data.
The paper's summary: Ruth: So, what this paper basically says is that standard design-based simulations, the ones where you fix the outcomes and then resample shocks, can be relying on a data-generating process that isn't actually like the true one we have in reality.
Warren: That means these simulations might confuse a real treatment effect with just some kind of error structure in our data, which could make us think there are more inference distortions than there actually are due to spatial correlation.
Iris: They point out that if you have true treatment effects, the DGP used in the simulation might be fundamentally different from the true one along dimensions that matter for your inference.
Ruth: It’s a critical distinction: whether these simulations give you useful information depends entirely on how closely the simulated data-generating process aligns with what's actually happening.
Warren: They show that when treatment effects are present, standard design-based simulations can mechanically confound those treatment effects with error dependence, which overstates the rejection of inference methods that don't account for spatial correlation.
The paper's improvements: Iris: The authors propose two alternative design-based simulation designs specifically to fix this problem where you have treatment effects and error dependence mixed up.
Ruth: These alternatives are meant to circumvent the issue of confounding, and they show how these different simulations can actually be more informative about distortions caused by spatial correlation than tests based on looking at significant effects in regressions using pre-treatment outcomes.
Warren: So it’s not just about throwing away the simulations; it’s about finding better ways to simulate that tell us something more precise about the errors in our data, especially when things are spatially correlated.
Ruth: They show that these alternative designs can flag spatial correlation problems whenever there is enough variation in those shocks, meaning some treatment assignments would generate big values of the difference between two potential outcomes, even if the specific assignment you actually get results in a small difference.
Conclusion: Warren: So to wrap up, the main point from this paper 'On the Use of Design-Based Simulations' is that we can't just trust standard simulations blindly because they might be relying on a DGP that doesn't match reality when treatment effects are present.
Ruth: The usefulness of these design-based simulations depends entirely on how well the simulated data-generating process matches the true one along the dimensions relevant for your inference.
Iris: And they suggest that using alternative designs, like epsilon-fixed or placebo outcome simulations, is a better way to avoid misleading conclusions about competing inference methods when you're dealing with shift-share designs.
Warren: So if you’re doing shift-share analysis and spatial correlation is a thing, this paper gives you guidance on how to choose your simulation design so you don't get those size distortions from being confused between treatment effects and error dependence.
Ruth: Yeah, it’s a warning that careful construction and interpretation of these simulations are absolutely essential for anyone using them in applied work.
Iris: That’s the main thing: the simulation design itself is a variable that can materially affect what you conclude about how reliable your inference method is.
More episodes
- Daily Summary for 2026-11-19
- 2608.21274-Recommendation Quality and the Concentration of Consumption: Experimental Evidence from Netflix
- 2408.02757-A nonparametric test for diurnal variation in spot correlation processes
- 2306.16393-High-Dimensional Canonical Correlation Analysis
- 2605.17299-Geometric Brownian motion with intermittent entries and exits
- 2607.04753-Fooling Yourself: how narratives shape beliefs
- 2408.14671-Inference on Treatment Effects with Repeatedly Mismeasured High-Dimensional Controls
- 2201.06898-Difference-in-Differences Estimators for Treatments Continuously Distributed at Every Period
- 2501.15307-Influence Functions: A Unified Differentiation-Based Framework
- The day in conversation for 2026-11-19