Randomized Controlled Trials (RCTs) hold a privileged position in causal inference because of their strong internal validity. However, because of problems with external validity, the ability of RCTs to estimate population treatment effects is often questionable. Because the experimental sample is almost never a random sample of the population of interest, many turn to observational studies to estimate population treatment effects. Also, the treatment effect may inherently differ between observational and experimental studies. Observational studies, however, suffer from confounding because of the non-random assignment of treatment. We provide a new theoretical decomposition of the bias that can be induced when using RCTs to estimate population effects. We show the assumptions that are necessary to move from estimating sample to population effects, and most importantly we propose placebo tests that can be used to test the identifying assumptions. Our placebo tests provide an essential validation step for any estimating strategy. We present four different estimating strategies that can be used to extend the results of the randomized trial to a larger population, including BART, inverse propensity score weighting, and a new estimation strategy which combines a machine learning matching method with maximum entropy weighting to match population moments.
We apply our approach to a cost-effectiveness analysis of a clinical intervention of interest to policy makers, Pulmonary Artery Catheterization (PAC). By using both information from an RCT and a non-random study, we estimate the population average treatment effect on the treated. We show that although the overall effect on patient mortality is not distinguishable from zero, PAC is beneficial for elective surgical patients and patients on a ventilator. The population estimates are larger than those found in the RCT. Non-random studies do a poor job estimating the cost of PAC use. Implications for the fast growing literature on cost-effectiveness analysis, a component of health care reform, are considered.
