“How Much Should We Trust Modern Difference-in-Differences Estimates?” Amanda Weiss, Yale

AMERICAN POLITICS & PUBLIC POLICY WORKSHOP

Abstract: When do modern difference-in-differences (DID)-style methods work for empirical political science? Scholars exploit the staggered roll-out of policies like election regulation, civil service reform, and healthcare across places to estimate causal effects - often using the two-way fixed effects (TWFE) estimator. However, recent literature has highlighted the TWFE estimator’s bias in the presence of heterogeneous treatment effects and tendency to make “forbidden comparisons” between treated units. In response, scholars have increasingly turned to modern DID estimators that promise greater robustness to real-world data problems. This paper asks how well these modern methods work for the empirical settings and sample sizes commonly used in political science, with the U.S. states as the running example. In particular, it provides a simulation study of the performance of seven DID methods under either constant or heterogeneous effects, in an N=50 setting that mimics the American federalism natural experiment. I find that many modern methods (1) produce confidence intervals that do not include the true average effect at the specified rate and (2) are underpowered. I show that many cases of coverage problems with modern DID estimators can be addressed using the block bootstrap to estimate standard errors. However, I also show that even where identification and estimation are straightforward, the fifty-state sample poses a power problem without large average effect sizes - at least 0.5 standard deviations. I illustrate the challenges of DID research with the fifty-state panel in the case of estimating the effects of strict voter identification laws on voter turnout.

Amanda Weiss is a Ph.D. candidate in political science, also getting an MA in statistics. She works on political methodology and American politics, often with a policy orientation. The first strand of her research takes up challenges in observational causal inference about policy effects. The second strand takes up challenges in experimental political behavior about policy attitudes and affective states.

Open to the Yale community only. Please visit this link to subscribe and receive regular announcements: https://csap.yale.edu/american-politics-public-policy-workshop.