Answer to "Is there evidence that entrustment as assessment is more valid than regular workplace-based assessment?"

 

Workplace-based assessment (WBA) in clinical training is different from other assessments in higher education. Traditional psychometric measures, applied to current methods of WBA are not very favorable. Typical problems documented are generosity error (i.e. too high scores, failure to fail), halo effect (generalizing from observing one feature), non-reproducibility (context differences), idiosyncrasy (observer/rater differences), unclear standards (often no standards), and scores unclearly related to proficiency, effort, development, or reference group performance. (Albanese, 2000; Govaerts et al., 2007; Eva, 2018)

Crossley and colleagues (2011) and Eva (2018) have pointed to the fact that clinicians, fully invested in the care for patients, and cognitively geared to thinking about task distribution in health care teams, including trainees, have difficulty shifting to thinking as observers and raters of the competence of others. Assessment should, therefore, be integrated with the provision of supervision (Kogan et al., 2014). Weller et al. (2014, 2017) found that, if the question "how much supervision is needed?” is translated to assessment scales, the reliability of these ratings increases. That is exactly what happens in entrustment scales (Rekman et al., 2016); supervision scales linked to EPA entrustment decisions (ten Cate, 2016) show increased validity (Mink et al., 2018). In the surgical domain, reliability benefits have been reported (George et al., 2014; Sandhu et al., 2018). Warm et al. (2016) provided support for the construct validity of entrustment scales over time in internal medicine. So, in general, this question may be answered with: yes.

References

Albanese, M. A. (2000) ‘Challenges in using rater judgements in medical education’, Journal of Evaluation in Clinical Practice, 6(3), pp. 305–319. https://doi.org/10.1046/j.1365-2753.2000.00253.x.

ten Cate, O. (2016) ‘Entrustment as Assessment: Recognizing the Ability, the Right, and the Duty to Act’, Journal of Graduate Medical Education, 8(2), pp. 261–262. https://doi.org/10.4300/JGME-D-16-00097.1.

Crossley, J., Johnson, G., Booth, J. and Wade, W. (2011) ‘Good questions, good answers: construct alignment improves the performance of workplace-based assessment scales’, Medical Education, 45(6), pp. 560–569. https://doi.org/10.1111/j.1365-2923.2010.03913.x.

Eva, K. W. (2018) ‘Cognitive Influences on Complex Performance Assessment: Lessons from the Interplay between Medicine and Psychology’, Journal of Applied Research in Memory and Cognition, 7(2), pp. 177–188. https://doi.org/10.1016/j.jarmac.2018.03.008.

George, B. C., Teitelbaum, E. N., Meyerson, S. L., Schuller, M. C., et al. (2014) ‘Reliability, Validity, and Feasibility of the Zwisch Scale for the Assessment of Intraoperative Performance’, Journal of Surgical Education, 71(6), pp. e90–e96. https://doi.org/10.1016/j.jsurg.2014.06.018.

Govaerts, M. J. B., van der Vleuten, C. P. M., Schuwirth, L. W. T. and Muijtjens, A. M. M. (2007) ‘Broadening Perspectives on Clinical Performance Assessment: Rethinking the Nature of In-training Assessment’, Advances in Health Sciences Education, 12(2), pp. 239–260. https://doi.org/10.1007/s10459-006-9043-1.

Kogan, J. R., Conforti, L. N., Iobst, W. F. and Holmboe, E. S. (2014) ‘Reconceptualizing Variable Rater Assessments as Both an Educational and Clinical Care Problem’, Academic Medicine, 89(5), p. 721. https://doi.org/10.1097/ACM.0000000000000221.

Mink, R. B., Schwartz, A., Herman, B. E., Turner, D. A., et al. (2018) ‘Validity of Level of Supervision Scales for Assessing Pediatric Fellows on the Common Pediatric Subspecialty Entrustable Professional Activities’, Academic Medicine, 93(2), pp. 283–291. https://doi.org/10.1097/ACM.0000000000001820.

Rekman, J., Gofton, W., Dudek, N., Gofton, T., et al. (2016) ‘Entrustability Scales: Outlining Their Usefulness for Competency-Based Clinical Assessment’, Academic Medicine, 91(2), p. 186. https://doi.org/10.1097/ACM.0000000000001045.

Sandhu, G., Nikolian, V. C., Magas, C. P. M., Stansfield, R. B., et al. (2018) ‘OpTrust:  Validity of a Tool Assessing Intraoperative Entrustment Behaviors’, Annals of Surgery, 267(4), pp. 670–676. https://doi.org/10.1097/SLA.0000000000002235.

Warm, E. J., Held, J. D., Hellmann, M., Kelleher, M., et al. (2016) ‘Entrusting Observable Practice Activities and Milestones Over the 36 Months of an Internal Medicine Residency’, Academic Medicine, 91(10), pp. 1398–1405. https://doi.org/10.1097/ACM.0000000000001292.

Weller, J. M., Castanelli, D. J., Chen, Y. and Jolly, B. (2017) ‘Making robust assessments of specialist trainees’ workplace performance’, BJA: British Journal of Anaesthesia, 118(2), pp. 207–214. https://doi.org/10.1093/bja/aew412.

Weller, J. M., Misur, M., Nicolson, S., Morris, J., et al. (2014) ‘Can I leave the theatre? A key to more reliable workplace-based assessment’, BJA: British Journal of Anaesthesia, 112(6), pp. 1083–1091. https://doi.org/10.1093/bja/aeu052.

© 2019 AMEE

 

Privacy