Political Practitioners Poorly Predict Which Messages Persuade the Public

Do political practitioners have good intuitions about how to persuade the public? Longstanding theories that political elites’ messages have large effects on public opinion and the large sums spent to secure some practitioners’ messaging advice suggest they should. However, findings regarding the surprising limits of expert forecasts in other domains suggest they may not. In this paper, we evaluate how well political practitioners can predict which messages are most persuasive. We measured the effects of N=172 messages about 21 political issues using a large-sample survey experiment (N=67,215 respondent-message observations). We then asked both political practitioners (N=1,524 practitioners, N=22,763 predictions) and laypeople (N=21,247 respondents, N=63,442 predictions) to predict the efficacy of these messages. The practitioners we surveyed ranged widely in their experience and expertise with persuasive message design. We find that: (a) political practitioners and laypeople both performed barely better than chance at predicting persuasive effects; (b) once accounting for laypeople’s inflated expectations about the average size of effects, practitioners did not predict meaningfully better than laypeople; (c) these results held even for self-identified issue experts and highly experienced practitioners; and (d) practitioners’ experience, expertise, information environment, and demographics did not meaningfully explain variation in their accuracy. Our findings have theoretical implications for understanding the conditions likely to produce meaningful elite influence on public opinion as well as practical implications for practitioners.