Answer to "Several entrustment scales have been published, which is best?"

Entrustment decisions, based on EPAs, signify the moments when a supervisor decides to have a learner act without supervision. That sounds like a dichotomous decision (yes or no), or, as clinicians frequently call this, 'hands on' versus 'hands off' (Gingerich et al., 2018). In practice, however, clinical supervision has many forms.  Supervisors may watch while a learner works, be quickly available, only debrief with a learner afterwards, and there are many shades in between. Different names have been given for scales representing these levels of supervision. Linguistically best is 'supervision scale' or 'entrustment scale', but 'entrustability scale' also has been proposed (Rekman, Gofton, et al., 2016; Rekman, Hamstra, et al., 2016) and is often used. See FAQ #2] .

The most widely used, generic supervision-entrustment scale was mentioned in FAQ#11 (scale 1). It provides a good start when thinking about the design of a scale. Next, adaptations can be made. For undergraduate education, level 4 (permission to act unsupervised) is not relevant; level 3 (indirect, reactive supervision, i.e. with a supervisor not in the room but quickly available when needed) is, in general, the target for undergraduate education. Level 1 (permission to be present, but not to act) generally does not require an entrustment decision. So for UME, levels 2 and 3 are relevant. They are somewhat crude. The Chen-adapted scale, designed for UME, expands these levels with more gradations of trainee autonomy:

  • 2a As co-activity with supervisor

  • 2b With supervisor in room ready to step in as needed

  • 3a With supervisor immediately available, all findings and decisions double checked

  • 3b With supervisor immediately available, key findings and decisions double checked

  • 3c With supervisor distantly available (e.g. by phone), findings and decisions promptly reviewed

Some scales have been created for a specific specialty (e.g. surgery, anesthesiology - see the references for more details). This is not unimportant. The essence of entrustment scales is that the assessment aligns with daily concerns of clinicians and is targeted to patient care (Crossley et al., 2011; Kogan et al., 2014; ten Cate, 2017); such scales appear to have more favorable psychometric properties (George et al., 2014; Weller et al., 2014, 2017; Mink et al., 2018). As specialties differ, those scales may be adapted to improve alignment with specialty needs.

Two issues remain to be discussed: the prospective versus retrospective nature of the scale, and use of the scale for entrustment decisions.

Prospective versus retrospective scales

Prospective scales ask the question: Given what we know about this learner and this EPA: would we be comfortable deciding that the learner may now act with indirect supervision only (cf. the endstage for UME) or act without supervision (cf the endstage for GME)? This requires an estimation of risks for future performance. Several such observations may support a summative entrustment decision. Multiple sources of information may be merged to arrive at such a decision, such as short observations (e.g. evaluating performance after an ad hoc entrustment decision) and entrustment-based discussions with the learner (ten Cate and Hoff, 2017). When clinicians use such a scale in an one-on-one observation of a learner, the scale should be read as a recommendation: "based on my observation, I believe this trainee is ready for supervision level..." (ten Cate et al., 2015). This recommendation can subsequently be weighed by a program director, an entrustment committee or a clinical competency committee as one of the sources to discuss trainee progress and to inform summative entrustment decisions to award a STAR, and, of course, serve to provide feedback to the trainee.

Retrospective scales just report the experiences of the observing clinician. In surgery, such scales are increasingly used. In intra-operative evaluations, these scales acknowledge how much supervision was actually provided during the operation. The Zwisch scale reports:

  1. 'show and tell' (I had to show and tell the trainee everything),

  2. 'active help' (I had to actively guide the trainee during the operation),

  3. 'passive help' (I could safely wait until the trainee asked for guidance,

  4. supervision only (I did not need to provide help).

These retrospective reports can be conveniently documented with mobile devices, using the PASS or the SIMPL tool, designed to help faculty complete competency-based performance assessments within seconds and without interrupting their work flow (Mellinger et al., 2017). These reports, of course, can also add to a larger documentation to prepare for summative entrustment decisions. They are, however, more sensitive to context influences (e.g. an extremely complex case may require active help for a resident who may generally be trusted to work without help).

The use of scales for entrustment decisions

Decisions are, by their nature, dichotomous (yes or no). So one may wonder then, why entrustment decisions actually utilize scales, as half a decision cannot be made. Even ad-hoc decisions (do I leave the operating theatre, do I leave the consultation room or the ward, or not?) are ultimately dichotomous, just as the awarding of a certificate or diploma, or the confirmation of completion of a training period is dichotomous.

In addition to standardizing the language around levels of supervision, there are three instances where having entrustment scales are useful. First, ad hoc decisions are often not planned in advance. A situation on the clinical ward may prompt a supervisor to consider asking a learner to attend to a patient. The supervisor must then judge how much help will be needed, or, in entrustability terms, what level of supervision would be appropriate to provide. Second, trainee assessments after an activity (e.g. a procedure) may be evaluated by stating in retrospect how much supervision the learner actually needed. Third, supervisors (or any attending), may be asked to recommend a level of supervision that would be appropriate for a trainee in the future. Subsequently, the supervisor judgment may be taken into account when a clinical competency committee convenes and considers a summative entrustment decision.

What makes entrustment scales different from regular workplace-based assessment (WBA) scales is that they are used with decisions in mind that have direct consequences for levels of trainee responsibility and patient safety. Entrustment scales may prompt supervising clinicians to think "Would I feel comfortable if one of my family members were admitted and assigned to this trainee for care?" In contrast, regular WBA tend to focus on an evaluation of the trainee without direct consequences for the workplace (such as determining whether a trainee is "above average" or "meeting expectations" or a "B+" or a 7.5 on a 10-point scale). This is a relevant issue as illustrated in the following example from medical student clerkships where EPAs are the dominant focus of training. Here, an expectation could be that a summative entrustment decision will be made for the EPA, ‘General procedures of the physician’ (which includes ‘inserting an IV’) at indirect supervision (level 3). If a student does complete the clerkship with a level 3 STAR for the EPA, the STAR holds a promise for subsequent clerkships. That student can now be asked to insert IVs without direct supervision, and there is no reason to make a distinction between adequate and excellent performance. Some authors have argued that medical education should follow this competency based approach and move away from traditional grading scales in clinical clerkships (Dannefer and Henson, 2007; Hanson, Rosenberg and Lane, 2013; Fazio, Torre and DeFer, 2016).

 

References

ten Cate, O. (2017) ‘Entrustment Decisions: Bringing the Patient Into the Assessment Equation’, Academic Medicine, 92(6), p. 736. https://doi.org/10.1097/ACM.0000000000001623.

ten Cate, O., Chen, H. C., Hoff, R. G., Peters, H., et al. (2015) ‘Curriculum development for the workplace using Entrustable Professional Activities (EPAs): AMEE Guide No. 99’, Medical Teacher, 37(11), pp. 983–1002. https://doi.org/10.3109/0142159X.2015.1060308.

ten Cate, O. and Hoff, R. G. (2017) ‘From case-based to entrustment-based discussions’, The Clinical Teacher, 14(6), pp. 385–389. https://doi.org/10.1111/tct.12710.

ten Cate, O. and Scheele, F. (2007) ‘Viewpoint: Competency-Based Postgraduate Training: Can We Bridge the Gap between Theory and Clinical Practice?’, Academic Medicine, 82(6), p. 542. https://doi.org/10.1097/ACM.0b013e31805559c7.

Chen, H. C., van den Broek, W. E. S. and ten Cate, O. (2015) ‘The Case for Use of Entrustable Professional Activities in Undergraduate Medical Education’, Academic Medicine, 90(4), p. 431. https://doi.org/10.1097/ACM.0000000000000586.

Chen, H. C. and ten Cate, O. (2018) ‘Assessment through Entrustable Professional Activities’, in Learning and Teaching in Clinical Contexts: A Practical Guide. 2nd edn. Chatswood: Elsevier Health Sciences.

Crossley, J., Johnson, G., Booth, J. and Wade, W. (2011) ‘Good questions, good answers: construct alignment improves the performance of workplace-based assessment scales’, Medical Education, 45(6), pp. 560–569. https://doi.org/10.1111/j.1365-2923.2010.03913.x.

Dannefer, E. F. and Henson, L. C. (2007) ‘The Portfolio Approach to Competency-Based Assessment at the Cleveland Clinic Lerner College of Medicine’, Academic Medicine, 82(5), p. 493. https://doi.org/10.1097/ACM.0b013e31803ead30.

DaRosa, D. A., Zwischenberger, J. B., Meyerson, S. L., George, B. C., et al. (2013) ‘A Theory-Based Model for Teaching and Assessing Residents in the Operating Room’, Journal of Surgical Education, 70(1), pp. 24–30. https://doi.org/10.1016/j.jsurg.2012.07.007.

Fazio, S. B., Torre, D. M. and DeFer, T. M. (2016) ‘Grading Practices and Distributions Across Internal Medicine Clerkships’, Teaching and Learning in Medicine, 28(3), pp. 286–292. https://doi.org/10.1080/10401334.2016.1164605.

George, B. C., Teitelbaum, E. N., Meyerson, S. L., Schuller, M. C., et al. (2014) ‘Reliability, Validity, and Feasibility of the Zwisch Scale for the Assessment of Intraoperative Performance’, Journal of Surgical Education, 71(6), pp. e90–e96. https://doi.org/10.1016/j.jsurg.2014.06.018.

Gingerich, A., Daniels, V., Farrell, L., Olsen, S.-R., et al. (2018) ‘Beyond hands-on and hands-off: supervisory approaches and entrustment on the inpatient ward’, Medical Education, 52(10), pp. 1028–1040. https://doi.org/10.1111/medu.13621.

Gofton, W. T., Dudek, N. L., Wood, T. J., Balaa, F., et al. (2012) ‘The Ottawa Surgical Competency Operating Room Evaluation (O-SCORE): A Tool to Assess Surgical Competence’, Academic Medicine, 87(10), p. 1401. https://doi.org/10.1097/ACM.0b013e3182677805.

Hanson, J. L., Rosenberg, A. A. and Lane, J. L. (2013) ‘Narrative descriptions should replace grades and numerical ratings for clinical performance in medical education in the United States’, Frontiers in Psychology, 4. https://doi.org/10.3389/fpsyg.2013.00668.

Kennedy, T. J. T., Lingard, L., Baker, G. R., Kitchen, L., et al. (2007) ‘Clinical Oversight: Conceptualizing the Relationship Between Supervision and Safety’, Journal of General Internal Medicine, 22(8), pp. 1080–1085. https://doi.org/10.1007/s11606-007-0179-3.

Kogan, J. R., Conforti, L. N., Iobst, W. F. and Holmboe, E. S. (2014) ‘Reconceptualizing Variable Rater Assessments as Both an Educational and Clinical Care Problem’, Academic Medicine, 89(5), p. 721. https://doi.org/10.1097/ACM.0000000000000221.

MacEwan, M. J., Dudek, N. L., Wood, T. J. and Gofton, W. T. (2016) ‘Continued Validation of the O-SCORE (Ottawa Surgical Competency Operating Room Evaluation): Use in the Simulated Environment’, Teaching and Learning in Medicine, 28(1), pp. 72–79. https://doi.org/10.1080/10401334.2015.1107483.

Mellinger, J. D., Williams, R. G., Sanfey, H., Fryer, J. P., et al. (2017) ‘Teaching and assessing operative skills: From theory to practice’, Current Problems in Surgery, 54(2), pp. 44–81. https://doi.org/10.1067/j.cpsurg.2016.11.007.

Mink, R. B., Schwartz, A., Herman, B. E., Turner, D. A., et al. (2018) ‘Validity of Level of Supervision Scales for Assessing Pediatric Fellows on the Common Pediatric Subspecialty Entrustable Professional Activities’, Academic Medicine, 93(2), pp. 283–291. https://doi.org/10.1097/ACM.0000000000001820.

Peters, H., Holzhausen, Y., Boscardin, C., Cate, O. ten, et al. (2017) ‘Twelve tips for the implementation of EPAs for assessment and entrustment decisions’, Medical Teacher, 39(8), pp. 802–807. https://doi.org/10.1080/0142159X.2017.1331031.

Rekman, J., Gofton, W., Dudek, N., Gofton, T., et al. (2016) ‘Entrustability Scales: Outlining Their Usefulness for Competency-Based Clinical Assessment’, Academic Medicine, 91(2), p. 186. https://doi.org/10.1097/ACM.0000000000001045.

Rekman, J., Hamstra, S. J., Dudek, N., Wood, T., et al. (2016) ‘A New Instrument for Assessing Resident Competence in Surgical Clinic: The Ottawa Clinic Assessment Tool’, Journal of Surgical Education, 73(4), pp. 575–582. https://doi.org/10.1016/j.jsurg.2016.02.003.

Sandhu, G., Magas, C. P. M., Robinson, A. B., Scally, C. P., et al. (2017) ‘Progressive Entrustment to Achieve Resident Autonomy in the Operating Room: A National Qualitative Study With General Surgery Faculty and Residents’, Annals of Surgery, 265(6), pp. 1134–1140. https://doi.org/10.1097/SLA.0000000000001782.

Sandhu, G., Nikolian, V. C., Magas, C. P. M., Stansfield, R. B., et al. (2018) ‘OpTrust:  Validity of a Tool Assessing Intraoperative Entrustment Behaviors’, Annals of Surgery, 267(4), pp. 670–676. https://doi.org/10.1097/SLA.0000000000002235.

Sandhu, G., Thompson-Burdine, J., Nikolian, V. C., Sutzko, D. C., et al. (2018) ‘Association of Faculty Entrustment With Resident Autonomy in the Operating Room’, JAMA Surgery, 153(6), pp. 518–524. https://doi.org/10.1001/jamasurg.2017.6117.

Weller, J. M., Castanelli, D. J., Chen, Y. and Jolly, B. (2017) ‘Making robust assessments of specialist trainees’ workplace performance’, BJA: British Journal of Anaesthesia, 118(2), pp. 207–214. https://doi.org/10.1093/bja/aew412.

Weller, J. M., Misur, M., Nicolson, S., Morris, J., et al. (2014) ‘Can I leave the theatre? A key to more reliable workplace-based assessment’, BJA: British Journal of Anaesthesia, 112(6), pp. 1083–1091. https://doi.org/10.1093/bja/aeu052.

Williams, R. G., George, B. C., Bohnen, J. D., Meyerson, S. L., et al. (2018) ‘Is the operative autonomy granted to a resident consistent with operative performance quality’, Surgery, 164(3), pp. 566–570. https://doi.org/10.1016/j.surg.2018.04.034.

© 2019 AMEE

 

Privacy