Designing Survey Experiments

A survey experiment is an experiment whose randomly assigned treatment is administered within the context of a survey. Survey experiments are attractive because they give researchers tight control over the details of implementation. The key downside of survey experiments is the artificial setting, which makes it difficult to establish clear links between the theoretical model, experimental design, and real-world phenomenon of interest. In this chapter, we discuss the design of survey experiments through the lens of their fundamental advantages and disadvantages. We first discuss the problem of selecting treatments that operate only via intended mechanisms. We then discuss the selection of outcome measures, emphasizing the desirability of specificity, real-world costs or stakes, and testing for persistence. We then consider issues that affect statistical power, including covariate adjustment, multi-item scales, and attention and quality checks. After briefly discussing sample representativeness, we conclude by summarizing our discussion through a set of recommendations.