Answer to "How will I know if our selection processes have “worked” in terms of selecting the best applicants?"
Medical schools typically assess how good their selection process is by using metrics, such as students’ assessment performance, retention rates and alumni performance on later indicators of academic ability and clinical competence (such as College/Board examinations). Some useful papers reporting on this in different contexts are provided below.
MacKenzie RK, Dowell J, Ayansina D, Cleland JA. Do personal attributes assessed on medical school admission predict exit performance? A UK-wide longitudinal cohort study. Advances in Health Sciences Education 2017; 365–385.
MacKenzie RK, Cleland JA, Ayansina D, Nicholson S. Does the UKCAT predict performance on exit from medical school? A national cohort study. BMJ Open 2016; 6:e011313 doi:10.1136/bmjopen-2016-011313.
McManus IC, Elder AT, de Champlain A, Dacre JE, Mollon J, Chis L. Graduates of different UK medical schools show substantial differences in performance on MRCP(UK) Part 1, Part 2 and PACES examinations. BMC Medicine 2008 6:5 https://doi.org/10.1186/1741-7015-6-5
Big data approaches also bear promise in relating selection to in-training performance and potentially health outcomes data:
Dowell J, Cleland JA, Fitzpatrick S et al. The UK Medical Education Database (UKMED) What is it? Why and how might you use it? BMC Medical Education. (2018) 18:6 DOI 10.1186/s12909-017-1115-9.
Ellaway RH, Pusic MV, Galbraith RM, Cameron T. 2014. Developing the role of big data and analytics in health professional education. Medical Teacher. 36(3):216-222.
There is a common problem when wanting to know if selection processes have worked or not – record keeping! You will not know if your selection processes have worked unless you keep good records of these processes, how applicants perform on each selection tool, how they perform as they progress though medical school, and after graduation. The biggest barrier to assessing outcomes is poor routine data management. We urge medical schools to align their selection and assessment databases to enable easy evaluation and analysis.