“Counterfactual Audit for Racial Bias in Police Traffic Stops,” Amanda Coston, Microsoft Research

QUANTITATIVE RESEARCH METHODS WORKSHOP

Abstract: Racial bias in criminal justice is a notoriously difficult problem to quantify because biases can affect where police choose to patrol and who they choose to stop. This talk focuses on assessing bias in who police choose to stop. A key challenge is that data about decisions are observed under outcome-dependent sampling. To address this, I propose a counterfactual audit for biased decision-making in settings with outcome-dependent data. This approach builds on the traditional “veil of darkness” design [Grogger and Ridgeway 2006] that uses the dark of night as a proxy for obfuscation of the driver’s race. We show that the odds-ratio targeted in veil of darkness analyses has a causal interpretation. Under the potential outcomes framework and certain assumptions, the odds ratio represents a relative measure of the covariate-conditional risk ratios for black versus white drivers. We provide a nonparametric estimator for this covariate-conditional measure. For an aggregated estimate of bias, we propose using the geometric mean to maintain collapsibility. We provide an efficient estimator for this aggregated measure using doubly-robust techniques that allow for flexible estimation. Using data from the Stanford Open Policing Project [Pierson et al. 2020], I demonstrate how this method can assess racial bias in police traffic stops in cities and states across the country.

Amanda Coston is a Postdoc with Microsoft Research in the Statistics and Machine Learning Team. Amanda earned her PhD in Machine Learning and Public Policy at Carnegie Mellon University, and she holds a BSE in Computer Science from Princeton. She will join the Department of Statistics at UC Berkeley in fall 2024 as an Assistant Professor. Her research investigates the validity, equity, and governance of data-driven algorithms used in societally consequential settings.

This workshop is open to the Yale community. To receive announcements and invitations to attend, please subscribe at https://csap.yale.edu/quantitative-research-methods-workshop.

This series is sponsored by the ISPS Center for the Study of American Politics and The Whitney and Betty MacMillan Center for International and Area Studies at Yale with support from the Edward J. and Dorothy Clarke Kempf Fund.