An exploration of “real time” assessments as a means to better understand preceptors’ judgments of student performance
Tóm tắt
Clinical supervisors are known to assess trainee performance idiosyncratically, causing concern about the validity of their ratings. The literature on this issue relies heavily on retrospective collection of decisions, resulting in the risk of inaccurate information regarding what actually drives raters’ perceptions. Capturing in-the-moment information about supervisors’ impressions could yield better insight into how to intervene. The purpose of this study, therefore, was to gather “real-time” judgments to explore what drives preceptors’ judgments of student performance. We performed a prospective study in which physicians were asked to adjust a rating scale in real-time while watching two video-recordings of trainee clinical performances. Scores were captured in 1-s increments, examined for frequency, direction, and magnitude of adjustments, and compared to assessors’ final entrustability judgment as measured by the modified Ottawa Clinic Assessment Tool. The standard deviation in raters’ judgment was examined as a function of time to determine how long it takes impressions to begin to vary. 20 participants viewed 2 clinical vignettes. Considerable variability in ratings was observed with different behaviours triggering scale adjustments for different raters. That idiosyncrasy occurred very quickly, with the standard deviation in raters’ judgments rapidly increasing within 30 s of case onset. Particular moments appeared to generally be influential, but their degree of influence still varied. Correlations between the final assessment and (a) score assigned upon first adjustment of the scale, (b) upon last adjustment, and (c) the mean score, were r = 0.13, 0.32, and 0.57 for one video and r = 0.30, 0.50, and 0.52 for the other, indicating the degree to which overall impressions reflected accumulation of raters’ idiosyncratic moment-by-moment observations. Our results demonstrated that variability in raters’ impressions begins very early in a case presentation and is associated with different behaviours having different influence on different raters. More generally, this study outlines a novel methodology that offers a new path for gaining insight into factors influencing assessor judgments.
Tài liệu tham khảo
Bargh, J. A., & Chartrand, T. L. (1999). The unbearable automaticity of being. American Psychologist, 54(7), 462–479. https://doi.org/10.1037/0003-066X.54.7.462
Bates, J., Schrewe, B., Ellaway, R. H., Teunissen, P. W., & Watling, C. (2019). Embracing standardisation and contextualisation in medical education. Medical Education, 53(1), 15–24. https://doi.org/10.1111/medu.13740
Boydstun, A. E., Glazier, R. A., Pietryka, M. T., & Resnik, P. (2014). Real-time reactions to a 2012 presidential debate: A method for understanding which messages matter. Public Opinion Quarterly, 78(S1), 33–343. https://doi.org/10.1093/poq/nfu007
Byrne, A., Tweed, N., & Halligan, C. (2014). A pilot study of the mental workload of objective structured clinical examination examiners. Medical Education, 48(3), 262–267. https://doi.org/10.1111/medu.12387
Choo, K. J., Arora, V. M., Barach, P., Johnson, J. K., & Farnan, J. M. (2014). How do supervising physicians decide to entrust residents with unsupervised tasks? A qualitative analysis. Journal of Hospital Medicine, 9(3), 169–175. https://doi.org/10.1002/jhm.2150
Eva, K. (2018). Cognitive influences on complex performance assessment: Lessons from the interplay between medicine and psychology. Journal of Applied Research in Memory and Cognition, 7, 177–188.
Eva, K. W., Armson, H., Holmboe, E., Lockyer, J., Loney, E., Mann, K., & Sargeant, J. (2012). Factors influencing responsiveness to feedback: On the interplay between fear, confidence, and reasoning processes. Advances in Health Sciences Education: Theory and Practice, 17(1), 15–26. https://doi.org/10.1007/s10459-011-9290-7
Eva, K., Brooks, L., & Norman, G. (2002). Forward reasoning as a hallmark of expertise in medicine: logical, psychological and phenomenological inconsistencies. Advances in Psychological Research, 8, 41–69.
Eva, K. W., & Hodges, B. D. (2012). Scylla or Charybdis? Can we navigate between objectification and judgement in assessment? Medical Education, 46(9), 914–919. https://doi.org/10.1111/j.1365-2923.2012.04310.x
Gingerich, A. (2015). What if the "trust’ in entrustable were a social judgement? Medical Education, 49(8), 750–752. https://doi.org/10.1111/medu.12772
Gingerich, A., Kogan, J., Yeates, P., Govaerts, M., & Holmboe, E. (2014). Seeing the “black box” differently: Assessor cognition from three research perspectives. Medical Education, 48(11), 1055–1068. https://doi.org/10.1111/medu.12546
Gingerich, A., Regehr, G., & Eva, K. W. (2011). Rater-based assessments as social judgments: Rethinking the etiology of rater errors. Academic Medicine, 86(10), S2–S8. https://doi.org/10.1097/ACM.0b013e31822a6cf8
Ho, K., Yao, C., Novak Lauscher, H., Koehler, B. E., Shojania, K., Jamal, S., & Eva, K. (2019). Remote assessment via video evaluation (RAVVE): A pilot study to trial video-enabled peer feedback on clinical performance. BMC Medical Education, 19(1), 466. https://doi.org/10.1186/s12909-019-1905-3
Holmboe, E. S., Sherbino, J., Long, D. M., Swing, S. R., Frank, J. R., & Collaborators, I. C. (2010). The role of assessment in competency-based medical education. Medical Teacher, 32(8), 676–682. https://doi.org/10.3109/0142159x.2010.500704
Kogan, J. R., Conforti, L., Bernabeo, E., Iobst, W., & Holmboe, E. (2011). Opening the black box of clinical skills assessment via observation: A conceptual model. Medical Education, 45(10), 1048–1060. https://doi.org/10.1111/j.1365-2923.2011.04025.x
Lurie, S. J., Mooney, C. J., & Lyness, J. M. (2009). Measurement of the general competencies of the accreditation council for graduate medical education: A systematic review. Academic Medicine, 84(3), 301–309. https://doi.org/10.1097/ACM.0b013e3181971f08
Maier, J., Hampe, J. F., & Jahn, N. (2016). Breaking out of the lab: Measuring real-time responses to political content in real-world settings. Public Opinion Quarterly, 80(2), 542–553. https://doi.org/10.1093/poq/nfw010
Meissner, C. A., Brigham, J. C., & Kelley, C. M. (2001). The influence of retrieval processes in verbal overshadowing. Memory & Cognition, 29(1), 176–186. https://doi.org/10.3758/bf03195751
Motavalli, A., & Nestel, D. (2016). Complexity in simulation-based education: Exploring the role of hindsight bias. Advances in Simulation (lond), 1, 3. https://doi.org/10.1186/s41077-015-0005-7
Peterman, J. (1940). The “program analyzer”: A new technique in studying liked and disliked items in radio programs. Journal of Applied Psychology, 24, 728–741. https://doi.org/10.1037/h0056834
Rekman, J., Hamstra, S. J., Dudek, N., Wood, T., Seabrook, C., & Gofton, W. (2016). A new instrument for assessing resident competence in surgical clinic: The ottawa clinic assessment tool. Journal of Surgical Education, 73(4), 575–582. https://doi.org/10.1016/j.jsurg.2016.02.003
Santen, S. A., Wolff, M. S., Saxon, K., Juneja, N., & Bassin, B. (2019). Factors affecting entrustment and autonomy in emergency medicine: “How much rope do I give them?” The Western Journal of Emergency Medicine, 20(1), 58–63. https://doi.org/10.5811/westjem.2018.10.39843
Surgeons, R. C. o. P. a. (2017). Entrustable Professional Activities for Otolaryngology—Head and Neck Surgery. Retrieved from https://www.royalcollege.ca/rcsite/documents/cbd/epa-guide-ohns-oto-v3-e.pdf
Tavares, W., & Eva, K. W. (2014). Impact of rating demands on rater-based assessments of clinical competence. Education for Primary Care, 25(6), 308–318. https://doi.org/10.1080/14739879.2014.11730760
Tiyyagura, G., Balmer, D., Chaudoin, L., Kessler, D., Khanna, K., Srivastava, G., & Auerbach, M. (2014). The greater good: How supervising physicians make entrustment decisions in the pediatric emergency department. Academic Pediatrics, 14(6), 597–602. https://doi.org/10.1016/j.acap.2014.06.001
Williams, R. G., Klamen, D. A., & McGaghie, W. C. (2003). Cognitive, social and environmental sources of bias in clinical performance ratings. Teaching and Learning in Medicine, 15(4), 270–292. https://doi.org/10.1207/S15328015tlm1504_11
Wood, T. J., Pugh, D., Touchie, C., Chan, J., & Humphrey-Murto, S. (2018). Can physician examiners overcome their first impression when examinee performance changes? Advances in Health Sciences Education, 23(4), 721–732. https://doi.org/10.1007/s10459-018-9823-4
Yeates, P., O’Neill, P., Mann, K., & Eva, K. (2013). Seeing the same thing differently mechanisms that contribute to assessor differences in directly-observed performance assessments. Advances in Health Sciences Education, 18(3), 325–341. https://doi.org/10.1007/s10459-012-9372-1