Visual Inference and Graphical Representation in Regression Discontinuity Designs

arXiv:2112.03096 · econ.EM · Submitted 2021-12-06 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Economics. The day in economics: theory, mechanisms and the evidence beneath them.

Ruth: I'm Ruth.

Warren: And I'm Warren, here with Iris.

Ruth: We're looking at the paper: "Visual Inference and Graphical Representation in Regression Discontinuity Designs".

Warren: Here's the gist of the paper. This study investigates "visual inference," a measure of readers' ability to process statistical information conveyed through graphs in regression discontinuity (RD) designs, which is often overlooked despite the widespread use of such graphs in empirical research.

Ruth: The researchers examine how accurately participants identify discontinuities in graphs generated from data generating processes (DGPs) calibrated on 11 published economics journal papers.

Warren: The core experimental work assesses the impact of various graphical representation methods using randomized experiments, finding that bin widths and fit lines have the most significant influence on whether participants correctly perceive a discontinuity.

Ruth: The researchers propose that using small bins without fit lines is a sensible starting point for constructing RD graphs.

Paper summary: Ruth: So this paper looks at visual inference, which is basically how well people can read the statistical stuff on graphs that show up in regression discontinuity designs. It says that even though these graphs are used everywhere, nobody really knows how good readers are at figuring out if there's a real break in the data or just noise.

Warren: Yeah, so they take this visual inference and test it by showing people graphs made from data generating processes based on eleven published economics journal papers. The main idea is to see how accurately participants can spot those discontinuities.

Iris: They are testing different ways of drawing these graphs using randomized experiments to figure out what actually affects whether someone sees that discontinuity or not. The authors found that the bin widths and the fit lines used in these graphs have the biggest impact on whether people correctly perceive a discontinuity >

Ruth: That’s interesting because it suggests that how you draw the graph matters more than just having data, right? They even suggested a starting point for constructing these graphs: using small bins without any fit lines is a sensible way to begin >

Warren: And they found that when you compare this visual inference to the actual econometric procedures used in regression discontinuity analysis, visual inference can achieve similar or even lower false positive rates >

Iris: That means it could actually work alongside the econometric methods, not just be an alternative. But they also pointed out that visual inference is less good at estimating the size of a discontinuity than it is at just spotting whether one exists >

Ruth: So, what this means for us in terms of real research is that before you build a graph, you should probably stick to small bins and skip the fit lines if you want people to actually see the discontinuity when they look at it >

Conclusion: Warren: So, to wrap up this paper, the authors are focusing on visual inference within regression discontinuity designs by testing how readers identify those breaks in graphs >

Iris: It’s about making evidence-based recommendations for how practitioners should actually construct these graphs because the way you draw it can potentially mislead people >

Ruth: The big implication is that visual inference isn't just some soft skill; it seems to be competitive when you use the recommended method, achieving lower false positive rates than traditional econometric methods at a five percent level >

Warren: But they also flag a limitation: the mechanism behind why visual inference works this way is still fuzzy, and they suggest future work needs to systematically check which data generating processes actually give better or worse performance when it comes to visual inference >

Iris: So, for anyone doing regression discontinuity work right now, the practical advice from this paper is to default to small bins with no fit lines and use evenly spaced bins as a sensible starting point for making these graphs >

University of Delaware · U.S. Census Bureau · Columbia University · Ž Cornell University and IZA

econ.EM

Submitted: 2021-12-06

Updated: 2023-01-28

Comments: Revised previous version following editor and reviewer suggestions. Changes are largely expositional

DOI: 10.1093/qje/qjad011

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 84/100

The gist: This study investigates "visual inference," a measure of readers' ability to process statistical information conveyed through graphs in regression discontinuity (RD) designs…

Key concepts

Visual Inference
This measures a reader's ability to correctly identify a sudden jump or break in data presented visually on a graph. It tests if people can accurately perceive the statistical signal of an RD design without using formal econometric calculations.
Regression Discontinuity (RD) Design
An experimental method where treatment assignment is determined by whether an observed variable crosses a specific cutoff point. Graphs are used to illustrate the relationship between this cutoff and the outcome, helping readers see if a sharp change occurs exactly at that threshold.
Type I Error
This is the risk of falsely concluding that a discontinuity exists when it actually does not (a false positive). The study compares visual inference rates against established econometric methods to show how reliable these different approaches are at making incorrect claims about the data.

Terminology

Summary

This study investigates visual inference, a measure of readers' ability to process statistical information conveyed through graphs in regression discontinuity (RD) designs, which is often overlooked despite the widespread use of such graphs in empirical research. The researchers examine how accurately participants identify discontinuities in graphs generated from data generating processes (DGPs) calibrated on 11 published economics journal papers. p. 0

The core experimental work assesses the impact of various graphical representation methods using randomized experiments, finding that bin widths and fit lines have the most significant influence on whether participants correctly perceive a discontinuity. p. 0 The researchers propose that using small bins without fit lines is a sensible starting point for constructing RD graphs. p. 0 Furthermore, when comparing visual inference on graphs made with their preferred method against established econometric inference procedures, they find that visual inference achieves similar or even lower type I error (false positive) rates and complements the econometric methods used in RD analysis. p. 0

The study rigorously tests five graphical parameters: bin width, bin spacing, axis scaling, polynomial fit lines, and a vertical line at the policy threshold. p. 5 The definitive recommendation is to default to small bins with no fit lines for generating RD graphs. p. 0 The researchers define type I and type II error probabilities based on classifying a continuous graph as having a discontinuity (type I error) or a discontinuous graph as continuous (type II error), proposing a conceptual framework to aggregate performance across subjects. p. 5

The study concludes that visual inference is competitively effective when using the recommended graphical method, achieving lower type I error rates than econometric inference at the 5% level, based on methods like Imbens and Kalyanaraman (2012) and Calonico, Cattaneo, and Titiunik (2014). p. 3 Crucially, the study finds that visual and econometric inferences appear to be complementary. p. 3 However, a key finding is that visual inference is less competitive at estimating the magnitude of a discontinuity than it is at identifying its existence.

The limitations acknowledged include the restricted set of parameters tested experimentally and reliance on a specific set of DGPs. p. 3 The mechanism underlying RD visual inference remains elusive, suggesting future work should systematically explore which DGPs yield better or worse visual inference performance. p. 1 The final practical recommendation is to use small bins with no fit lines, even spacing, default y-axis scaling, and a vertical line at the policy threshold as a sensible default setting for RD graph construction. p. 2

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Develop an AI system capable of selecting optimal graphical parameters for regression discontinuity designs by using a decision theoretic framework that incorporates classification accuracy as a metric to compare graphical methods. This system can recommend small bins with no fit lines as a starting point to construct RD graphs, as suggested by the authors.

  2. Enhance visual inference capabilities by training models that can assess the performance of different graphical representation methods against widely used econometric inference procedures, such as IK and CCT procedures, to determine which method achieves similar or lower type I error rates while maintaining competitive type II error rates.

  3. Implement an expert prediction module that can evaluate the ability of domain experts to predict which graphing options perform best for non-expert samples, allowing the AI to weigh empirical findings against expert intuition regarding visualization choices.

Sources

Related papers