Testing the Accuracy of Regression Discontinuity Analysis Using Experimental Benchmarks

Households in the state of Michigan: The targeting criteria used in this mailing campaign were developed by a political consultant, Mark Grebner who uses targeting criteria based on a combination of address information readily available from the Qualified Voter File (QVF) and a set of proprietary indices of partisanship and voting behavior developed by his consulting firm. Grebner’s targeting objective was to direct mailings to those who were thought to be especially responsive to them. Mailings were therefore sent to voters whose expected probability of voting was deemed to be moderate. Those believed to be strong Democrats were excluded on the grounds that they had little chance of voting in an election that was meaningful mainly to Republicans. Absentee voters were excluded because they were believed to vote early, before the receipt of these mailings. Sparsely populated streets were excluded because the Neighbors treatment requires the voting histories of several neighbors. Apartment addresses were excluded because apartment numbers are sometimes unreliable, and it is hard to be certain which voters belong to the same household... We removed everyone for whom we could not assign a valid 9-digit ZIP, people who live on blocks where more than 10% of the addresses included apartment numbers, people who live on streets with fewer than four addresses (or fewer than 10 voters). Prior to random assignment we also removed households with the following characteristics: all members of the household had over a 60% probability of voting by absentee ballot if they voted or all household members had a greater than a 60% probability of choosing the Democratic primary rather than the Republican primary. Absentees were removed because it was thought that many would have decided to vote or not prior to receipt of the experimental mailings, which were sent to arrive just a few days before the election. Those considered overwhelmingly likely to favor the Democratic primary were excluded because it was thought that, given the lack of contested primaries, these citizens would tend to ignore pre election mailings. We removed everyone who lived in a route where fewer than 25 households remained, because the production process depended on using carrier-route-presort standard mail. Finally, we removed all those who had abstained in the 2004 general election on the grounds that those not voting in this very high turnout election were likely to be “deadwood”—those who had moved, died, or registered under more than one name.