Abstract:
Economists often use balance tests to demonstrate that the treatment and control groups are comparable prior to an intervention. We show that typical implementations of balance tests have poor statistical properties. Pairwise t-tests leave it unclear how many rejections indicate overall imbalance. Omnibus tests of joint orthogonality, in which the treatment is regressed on all the baseline covariates, address this ambiguity but substantially over-reject the null hypothesis using the sampling-based p-values that are typical in the literature. This problem is exacerbated when the number of covariates is high compared to the number of observations. We examine the performance of alternative tests, and show that omnibus F-tests of joint orthogonality with randomization inference p-values have the correct size and reasonable power. We apply these tests to data from two prominent recent articles, where standard F-tests indicate imbalance, and show that the study arms are actually balanced when appropriate tests are used.
Jason Kerwin is an Associate Professor in the Department of Economics and a Brimmer Distinguished Scholar at the University of Washington, an Affiliated Professor at J-PAL, and a Research Fellow at IZA. He received his Ph.D. in Economics from the University of Michigan, where he was also an Economic Demography Trainee at Michigan’s Population Studies Center. Jason’s research uses randomized field experiments to understanding the choices people in developing countries make about health, education, employment, and savings
