MacMillan-CSAP Workshop on Quantitative Research Methods: Devin Caughey, “Permutation Tests of Bounded Null Hypotheses”
“Permutation Tests of Bounded Null Hypotheses”
Devin Caughey, Assistant Professor of Political Science, MIT
Abstract: Permutation (a.k.a. randomization) tests, despite their appealing nonparametric properties, are often dismissed as tests of an implausible and uninteresting null hypothesis: the sharp null of no effects whatsoever. On the contrary, we show that just as a directional test of a weak null hypothesis (e.g., μ = 0) is a conservative test of a much larger composite null (e.g., μ ≤ 0), so too can many permutation tests be interpreted as conservative tests of a bounded null hypothesis. Specifically, if a difference-of-means permutation test rejects the sharp null of a constant treatment effect τ0 at some level α, then any null under which all unit-level treatment effects are smaller (or larger) than τ0 can also be rejected at α. By inverting the test, we can form one-sided confidence intervals for the maximum (or minimum) treatment effect. These properties also hold for rank statistics and other potential-outcomes monotonic test statistics. An especially useful example is the Stephenson rank statistic, which is sensitive to large-but-rare effects and can detect negative effects even when the average or median effect is positive. We discuss the connection between the bounded null and the normative concept of Pareto efficiency, and we illustrate with a re-analysis of a well-known field experiment in Benin. (A paper is not being circulated for this seminar.)
Speaker Bio: Devin Caughey is a graduate of Yale (B.A.), Cambridge (M.Phil.), and UC Berkeley (Ph.D.) and is currently an assistant professor of political science at MIT. Devin’s primary fields are American politics, which he often studies from a historical angle, and political methodology, where his interests include design-based causal inference, Bayesian measurement models, survey weighting, and permutation tests. His work has been published in American Journal of Political Science, Political Analysis, and Studies in American Political Development. Devin is currently completing a book manuscript on representation in the one-party South. He lives in Cambridge, MA with his wife and son.