MacMillan-CSAP Workshop on Quantitative Research Methods: Justin Grimmer (Stanford), “The Unreliability of Measures of Intercoder Reliability, and What to Do About It”

Event time: 
Thursday, November 5, 2015 - 5:00pm through 6:15pm
Event description: 

“The Unreliability of Measures of Intercoder Reliability and What to Do About It”

Justin Grimmer, Associate Professor of Political Science, Stanford University

LINK TO PAPER (With Gary King and Chiara Superti)

Abstract: In both automated and traditional text analysis, human coders are regularly tasked with placing documents into one of a set of given categories. Researchers then evaluate the success of this crucial step in the research process via one of many measures of intercoder reliability, such as Cronbach’s alpha. Nearly universal practice across many literatures is to improve coding practices until intercoder reliability achieves some chosen arbitrary threshold, and to ignore remaining disagreements for subsequent analyses. We prove that this common practice generates severely biased estimates and misleading conclusions. Moreover, because the level of intercoder reliability is only loosely related to the accuracy of estimators of the quantities of interest, we show that the many measures of intercoder reliability in this literature are not helpful and should rarely if ever be used. Given the limitations of existing practices, we introduce a new method for properly incorporating coder uncertainty in statistical estimation. The method offers an interval estimate which we prove contains the true proportion of documents in each category, under reasonable assumptions. We then extend this method to situations with multiple coders, when one coder is trusted more than another, and when the resulting uncertain document codes are used as inputs to another statistical model. We also offer examples that demonstrate how basing research on widely used intercoder reliability measures, and ignoring the remaining coder disagreements, can lead to highly misleading substantive conclusions, and how our methods easily fix these problems.

Justin Grimmer is an associate professor of political science at Stanford University. His research examines how representation occurs in American politics using new statistical methods.

This workshop series is being sponsored by the ISPS Center for the Study of American Politics and The Whitney and Betty MacMillan Center for International and Area Studies at Yale with support from the Edward J. and Dorothy Clarke Kempf Fund.

Event type