“Using Experimentation to Manage Innovation Risk: An Experimental Design for Anytime-Valid Causal Inference on Multi-Armed Bandits,” Iavor Bojinov, Harvard Business School

QUANTITATIVE RESEARCH METHODS WORKSHOP

Abstract: Experimentation is now a core capability for most digital firms and conventional companies undergoing a digital transformation, with some running thousands of experiments per year on millions of customers. The rise of experimentation is due to two somewhat contradictory motivations (1) to obtain causal estimates of the impact of a change, enabling managers to learn customer preferences, identify improvement opportunities, and make data-informed deployment and investment decisions; and (2) to manage risk in the innovation process by limiting customer’s exposure to adverse changes and rapidly release beneficial offerings. Multi-armed bandits (MAB) have been proposed as a potential solution for balancing the two objectives by allowing managers to dynamically reduce the proportion of people assigned to the subpar treatment and still perform causal inference at the end of the study. However, in many applications, it is advantageous to continuously produce inference on the average treatment effect (ATE) between arms as new data arrive and determine a data-driven stopping time for the experiment. Existing work on continuous inference for adaptive experiments assumes that the treatment assignment probabilities are bounded away from zero and one, thus excluding nearly all standard bandit algorithms. In this work, we develop the Mixture Adaptive Design (MAD), a new experimental design for multi-armed bandits that enables continuous inference on the ATE with guarantees on statistical validity and power for nearly any bandit algorithm. We empirically show that the MAD improves the coverage and power of ATE inference in MAB experiments without significant losses in finite-sample reward.

Iavor Bojinov is an Assistant Professor of Business Administration and the Richard Hodgson Fellow at Harvard Business School. He is the co-PI of the AI and Data Science Operations Lab and a faculty affiliate in the Department of Statistics at Harvard University and the Harvard Data Science Initiative. His research and writings center on data science strategy and operations, aiming to understand how companies should overcome the methodological and operational challenges presented by the novel applications of AI. His work has been published in top academic journals such as Annals of Applied Statistics, Biometrika, Journal of the American Statistical Association, Journal of Econometrics, Quantitative Economics, Management Science, and Science, and has been cited in Forbes, The New York Times, The Washington Post, and Reuters, among other outlets. More broadly, as one of the few scholars who work at the intersection of data science and business, he was the first author to have spotlight featured articles in both the Harvard Business Review and the Harvard Data Science Review. Professor Bojinov is also the co-creator of the first-year required MBA course “Data Science for Managers” and has previously taught the “Competing in the Age of AI” and “Technology and Operations Management” courses. Before joining Harvard Business School, Professor Bojinov worked as a data scientist leading the causal inference effort within the Applied Research Group at LinkedIn. He holds a Ph.D. and an MA in Statistics from Harvard and an MSci in Mathematics from King’s College London.

This workshop is open to the Yale community. To receive announcements and invitations to attend, please subscribe at https://csap.yale.edu/quantitative-research-methods-workshop.

This series is sponsored by the ISPS Center for the Study of American Politics and The Whitney and Betty MacMillan Center for International and Area Studies at Yale with support from the Edward J. and Dorothy Clarke Kempf Fund.