In recent years the Higher Education Funding Council for England (HEFCE) and the corresponding bodies in Scotland, Wales, and Northern Ireland have become increasingly interested in monitoring the quality with which universities carry out their public mandate. In December 2000 HEFCE published results on a number of performance indicators for 1996-97 and 97-98, including drop-out rates after the first year of undergraduate study, comparing observed and expected outcomes for all 175 government-funded higher education institutions in the UK (this publicly available report makes interesting reading; data for 1997-98 have also been published).
Interpreting the performance indicators effort using the language of causal inference, with a binary outcome such as dropout at the student level, (a) the process of students choosing and attending universities gives rise to an observational study (rather than a controlled experiment), meaning that measuring and controlling for potential confounding factors (PCFs) is crucial; (b) the supposedly causal factor at the university level is the underlying -- and *unobserved* -- quality of the university; and (c) there are many student-level PCFs (including age, entry qualifications, and subject of study). HEFCE's method of computing expected outcomes given the PCFs turns out to be a form of indirect standardization, which we have shown is almost equivalent to a version of hierarchical model-based inference in which the university effects are treated as fixed and the link function is linear.
In consulting work on behalf of HEFCE we have developed a method of shrinkage variance estimation that properly calibrates the HEFCE approach and have shown that the numbers of unusually under- and over-achieving universities, as far as dropout rates were concerned, in 1996-97 were both surprisingly high. In this talk I will describe our calibration work, compare fixed-effects and random-effects formulations of the model-based version of standardization on their ability to correctly identify "good" and "bad" universities, and explore the effects of unobserved student-level PCFs on the results.
