Large-scale A/B testing is increasingly prevalent in many industries. We propose an empirical Bayes approach, which assumes that the treatment effects are realized from a “true prior”. This requires inferring the prior from previous experiments. Following Robbins, we estimate a family of marginal densities of empirical effects, indexed by the noise scale. We show that this family is characterized by the heat equation. We develop a spectral MLE based on Fourier series, which can be efficiently computed via convex optimization. We select hyperparameters and compare models using two model selection criteria. Our method is demonstrated on experimentation data from Amazon.com. The same method can also be applied to meta-analysis.
Note: This is joint work with James McQueen (Amazon.com) and Thomas Richardson (UW and Amazon.com).
Preprint: http://arxiv.org/abs/2002.02564
During the Spring 2020 academic quarter, the CSSS Seminar Series will be conducted online. Please contact Will Brown if you are interested in attending (brownw at uw dot edu).
