Answer to "What training do those involved with selection need?"

Any selection process where applicants are assessed directly by examiners requires those examiners to make judgments based on formal observation of the candidate’s performance, usually in defined domains. These judgments are inherently subjective, at least to some extent, and thus open to error and bias, which may have the potential to influence the outcome of the selection process.  See the following paper for further discussion.

  • Till H, Myford C, Dowell J.  Mini-Interviews With Multifaceted Rasch Modeling. Academic Medicine 2013; 88:216–223.

The study by Till et al., clearly indicates a need for examiners to receive training to improve fairness and consistency in MMI ratings.  However, there is a paucity of literature on this topic.  Thus, taking the stance that selection for medical school is the first assessment in a medical career, we looked to the assessment literature for useful references.  Those we identified as potentially useful relate to either using patients or medical students as assessors:

  • Thomson FC, MacKenzie RK, Denison AR, Currie GP.  Incorporating patient partner scores into high stakes assessment: An observational study into opinions and attitudes. BMC Medical Education 2017: 17(1); 214.

  • Iblher P, Zupanic M, Karsten J, Brauer K. May student examiners be reasonable substitute examiners for faculty in an undergraduate OSCE on medical emergencies? Medical Teacher 2015: 37(4); 374-378.

Or to the role of the assessor in making judgements (i.e., the impact of the assessor on assessment variance).

  • Pell G, Homer MS, Roberts TE.  Assessor training: Its effects on criterion-based assessment in a medical context. International Journal of Research and Method in Education 2008: 31(2); 143-154.

  • Gingerich A, van der Vleuten CPM, Eva KW, Regehr G. More consensus than idiosyncrasy: Categorizing social judgments to examine variability in Mini-CEX ratings. Academic Medicine 2014: 89; 1510–1519

One area of increasing importance in medicine and medical education generally is the notion of unconscious or implicit bias. Unconscious bias is a term used to describe the associations that we hold which, despite being outside our conscious awareness, can have a significant influence on our attitudes and behaviour. Regardless of how fair minded we believe ourselves to be, all people have some degree of unconscious bias. The means that we automatically respond to others (e.g., people from different racial/ethnic groups, or gender, or sexual orientation) in positive or negative ways. There is some work looking directly at the existence of unconscious bias in selection:

  • Capers Q, Clinchot D, McDougle L, Greenwald AG. Implicit Racial Bias in Medical School Admissions. Academic Medicine 2017: 92(3); 365-369.

There are methods and techniques for reducing implicit/unconscious bias, which may be useful for assessor training.  This document gives a good overview of the issues and ways to reduce implicit/unconscious biases:

Given that people traditionally select in their same image (Rivera. American Sociological Review 2012; 77(6): 999–1022), this issue can be addressed practically by assuring your assessors are diverse in terms of gender, ethnic group and socio-economic background.