Predicting future meaningful autonomy using prior autonomy ratings for pediatric surgery fellows
Tóm tắt
Assessment of trainees’ operative autonomy is challenging. However, workplace-based assessment (WBA) systems have made it possible to capture longitudinal data on trainees’ operative autonomy. We evaluated the ability of prior WBA operative autonomy ratings to predict future autonomy ratings for pediatric surgery fellows. WBA data from two pediatric surgery training programs were analyzed using Bayesian mixed effects models to evaluate the relationship between prior cumulative autonomous operative experiences and the probability of being granted meaningful autonomy in a subsequent operation rated using a WBA. Cumulative autonomous experience was modeled as a fixed effect, while procedure, complexity, fellow, and attending surgeon were all modeled as random effects. Marginal predictions were generated and visualized to identify the number of prior autonomous operative experiences needed to achieve a 95% probability of being granted autonomy on a subsequent rating for three procedures: laparoscopic gastrostomy, laparoscopic inguinal hernia repair, and laparoscopic pyloromyotomy. At site one, 843 of 1111 (76%) evaluations were rated as meaningfully autonomous for 9 fellows. At site two, 201 of 234 evaluations (86%) were rated as meaningfully autonomous for 3 fellows. Both sites identified similar expectations for autonomy based on cumulative autonomous experiences. The number of previously autonomous ratings needed to achieve a 95% probability of being granted meaningful autonomy for a subsequent procedure were 12 (site one and two) for laparoscopic gastrostomy, 10 (site one) and 14 (site two) for laparoscopic inguinal hernia repair, and 9 (site one) and 13 (site two) for laparoscopic pyloromyotomy. Prior operative autonomy ratings appear effective in predicting the probability of being granted meaningful autonomy in subsequent procedures across multiple faculty and fellow groups. The approach demonstrated in this paper could support establishing minimum case number requirements and monitoring of fellows’ developing entrustability.
Tài liệu tham khảo
DaRosa DA, Zwischenberger JB, Meyerson SL, George BC, Teitelbaum EN, Soper NJ, et al. A theory-based model for teaching and assessing residents in the operating room. J Surg Educ. 2013;70(1):24–30.
Krumm AE, Marcotte KM, Thelen AE, Gates RS, George BC. Connecting quantity and quality: an innovative statistical method for linking ACGME case logs and surgical resident autonomy. Global Surg Educ. 2023;2(1):34.
Hashimoto DA, Bynum WE, Lillemoe KD, Sachdeva AK. See more, do more, teach more: surgical resident autonomy and the transition to independent practice. Acad Med. 2016;91(6):757–60.
ACGME Program Requirements for Graduate Medical Education in General Surgery [Internet]. ACGME; 2020. https://www.acgme.org/globalassets/pfassets/programrequirements/440_generalsurgery_2020.pdf.
Beasley SW. The challenges facing training in pediatric surgery worldwide. Front Pediatr. 2013. https://doi.org/10.3389/fped.2013.00024.
Bohnen JD, George BC, Williams RG, Schuller MC, DaRosa DA, Torbeck L, et al. The feasibility of real-time intraoperative performance assessment with SIMPL (system for Improving and measuring procedural learning): early experience from a multi-institutional trial. J Surg Educ. 2016;73(6):e118–30.
George BC, Bohnen JD, Schuller MC, Fryer JP. Using smartphones for trainee performance assessment: a SIMPL case study. Surgery. 2020;167(6):903–6.
Cox ML, Weaver ML, Johnson C, Chen X, Carter T, Yee CC, et al. Early findings and strategies for successful implementation of SIMPL workplace-based assessments within vascular surgery residency and fellowship programs. J Vasc Surg. 2023;S0741–5214(23):01147–53.
Aryal K, Hamed M, Currow C. The usefulness of work-based assessments in higher surgical training: A systematic review. Int J Surg. 2021;94: 106127.
George BC, Teitelbaum EN, Meyerson SL, Schuller MC, DaRosa DA, Petrusa ER, et al. Reliability, validity, and feasibility of the Zwisch scale for the assessment of intraoperative performance. J Surg Educ. 2014;71(6):e90-96.
Kay M. tidybayes: tidy data and geoms for Bayesian models [Internet]. Zenodo; 2023 https://zenodo.org/record/1308151. Cited 26 May 2023.
Bürkner PC. Brms : an R package for bayesian multilevel models using stan. J Stat Soft. 2017;80(1):i01.
Krumm AE, Marcotte K, George BC. Model-based operative performance expectations for quantifying competency in general surgery. JAMA Surg. 2023;158(5):515.
Pardos ZA. Big data in education and the models that love them. Curr Opin Behav Sci. 2017;18:107–13.
Pusic MV, Boutis K, Hatala R, Cook DA. Learning curves in health professions education. Acad Med. 2015;90(8):1034–42.
Levy BE, MacDonald M, Bontrager N, Castle JT, Draus JM, Worhunsky DJ. Evaluation of the learning curve for laparoscopic pyloromyotomy. Surg Endosc. 2023;37:5101–8.
Moulton CAE, Dubrowski A, MacRae H, Graham B, Grober E, Reznick R. Teaching surgical skills: what kind of practice makes perfect? A randomized controlled trial. Ann Surg. 2006;244(3):400–9.
Lindeman B, Sarosi GA. Competency-based resident education: the United States perspective. Surgery. 2020;167(5):777–81.
Wagner JP, Lewis CE, Tillou A, Agopian VG, Quach C, Donahue TR, et al. Use of entrustable professional activities in the assessment of surgical resident competency. JAMA Surg. 2018;153(4):335.
Brasel KJ, Klingensmith ME, Englander R, Grambau M, Buyske J, Sarosi G, et al. Entrustable professional activities in general surgery: development and implementation. J Surg Educ. 2019;76(5):1174–86.
Ten Cate O. Nuts and bolts of entrustable professional activities. J Grad Med Educ. 2013;5(1):157–8.
Williams RG, George BC, Bohnen JD, Dunnington GL, Fryer JP, Klamen DL, et al. A proposed blueprint for operative performance training, assessment, and certification. Ann Surg. 2021;273(4):701–8.
Williams RG, George BC, Bohnen JD, Meyerson SL, Schuller MC, Meier AH, et al. Is the operative autonomy granted to a resident consistent with operative performance quality. Surgery. 2018;164(3):566–70.