Effectiveness of a focused educational intervention on resident evaluations from faculty
Tóm tắt
OBJECTIVE: To improve the quality and specificity of written evaluations by faculty attendings of internal medicine residents during inpatient rotations. DESIGN: Prospective randomized controlled trial. SETTING: Four hospitals: tertiary care university hospital, Veterans’ Administration hospital, and two community hospitals. PARTICIPANTS: Eighty-eight faculty and 157 residents from categorical and primary-care internal medicine residency training programs rotating on inpatient general medicine teams. INTERVENTION: Focused 20-minute educational session on evaluation and feedback, accompanied by 3 by 5 reminder card and diary, given to faculty at the start of their attending month. MEASUREMENTS AND MAIN RESULTS: Primary outcomes: 1) number of written comments from faculty specific to unique, preselected dimensions of competence; 2) number of written comments from faculty describing a specific resident behavior or providing a recommendation; and 3) resident Likert-scale ratings of the quantity and effect of feedback received from faculty. Faculty in the intervention group provided more written comments specific to defined dimensions of competence, a median of three comments per evaluation form versus two in the control group, but when adjusted for clustering by faculty, the difference was not statistically significant (P=.09). Regarding feedback, residents in the intervention group rated the quantity significantly higher (P=.04) and were significantly more likely to make changes in clinical management of patients than residents in the control group (P=.04). CONCLUSIONS: A brief, focused educational intervention delivered to faculty prior to the start of a ward rotation appears to have a modest effect on faculty behavior for written evaluations and promoted higher quality feedback given to house staff.
Tài liệu tham khảo
Eisenberg JM. Evaluating internists’ clinical competence. J Gen Intern Med. 1989;4:139–43.
Miller GE. The assessment of clinical skills/competence/performance. Acad Med. 1990;65(suppl):63–7.
American Board of Internal Medicine. Project Professionalism. Philadelphia: American Board of Internal Medicine; 1994.
Turnbull J, Gray J, MacFadyen J. Improving in-training evaluation programs. J Gen Intern Med. 1998;13:317–23.
American Board of Internal Medicine. Attending Physicians: Your Role in Evaluating Residents. Philadelphia: American Board of Internal Medicine; 1995.
American Board of Internal Medicine. Guide to the Evaluation of Residents in Internal Medicine 1990–92. Philadelphia: American Board of Internal Medicine; 1992.
Gray JD. Globa rating scales in residency education. Acad Med. 1996;71:S55–63.
Thompson WG, Lipkin M, Gilbert DA, Guzzo RA, Roberson L. Evaluating evaluation: assessment of the American Board of Internal Medicine resident evaluation form. J Gen Intern Med. 1990;5:214–7.
Haber RJ, Avins AL. Do ratings on the American Board of Internal Medicine resident evaluation form detect differences in clinical competence? J Gen Intern Med. 1994;9:140–5.
Borman WC. Effects of instructions to avoid halo error on reliability and validity of performance evaluation ratings. J Appl Psychol. 1975;60:556–60.
Ende J. Feedback in clinical medical education. JAMA. 1983;250:777–81.
Ende J. The evaluation product: putting it to use. In: Lloyd JS, Langley DG, eds. How to Evaluate Residents. Chicago: American Board of Medical Specialties; 1986:99–116.
Weinholtz D. Teaching during attending rounds: a manual for attending physicians. Des Moines, Iowa: University of Iowa; 1987.
Skeff K, Stratos G. Feedback. From the Standford Faculty Development Program, Clinical Teaching. Standford, Calif: Stanford University; 1996.
Neufeld V, Norman GR, eds. Assessing Clinical Competence. New York: Springer; 1985.
Huber PJ. The behavior of maximum likelihood estimates under non-standard conditions. Proceedings of the Fifth Berkeley Symposium in Mathematical Probability. Berkeley, Calif: University of California Press; 1967.
White H. A heteroskedasticity-consistent covariance matrix estimator and a direct test for heteroskedasticity. Econometrica. 1980;48:817–30.
Holmboe ES, Hawkins RE. Methods for evaluating the clinical competence of residents in Internal Medicine: a review. Ann Intern Med. 1998;129:42–8.
Veet LL, Shea JA, Ende J. Our continuing interest in manuscripts about education. J Gen Intern Med. 1997;12:583–5.
Education Group for Guidelines on Evaluation. Guideline for evaluating papers on educational interventions. BMJ. 1999;318:1265–7.
Davis JK, Inamdar S, Stone RK. Interrater agreement and predictive validity of faculty ratings of pediatric residents. J Med Educ. 1986;61:901–5.
Maxim BR, Dielman TE. Dimensionality, internal consistency, and interrater reliability of clinical performance ratings. Med Educ. 1996;21:130–7.
Carline JD, Paauw DS, Thiede KW, Ramsey PG. Factors affecting the reliability of ratings of students’ clinical skills in a medicine clerkship. J Gen Intern Med. 1992;7:506–10.
Norcini JJ. Standards and reliability in evaluation: when rules of thumb don’t apply. Acad Med. 1999;74:1088–90.
Kassirer JP. Performance results on the 1995 Internal Medicine certifying examination. ABIM News Update. 1995;Fall/Winter:2–4.
Noel GL, Herbers JE, Caplow MP, Cooper GS, Pangaro LN, Harvey J. How well do Internal Medicine faculty members evaluate the clinical skills of residents? Ann Intern Med. 1992;117:757–65.
Herbers JE, Noel GL, Cooper GS, Harvey J, Pangaro LN, Weaver MJ. How accurate are faculty evaluations of clinical competence? J Gen Intern Med. 1989;4:202–8.
Kroboth FJ, Hanusa BH, Parker S, et al. The inter-rater reliability and internal consistency of a clinical examination exercise. J Gen Intern Med. 1992;7:174–9.
Norcini JJ, Blank LL, Arnold GK, Kimball HR. The mini-CEX (clinical evaluation exercise): a preliminary investigation. Ann Intern Med. 1995;123:795–9.
Kalet A, Earp J, Kowlowitz V. How well do the faculty evaluate the interviewing skills of medical students? J Gen Intern Med. 1992;7:499–505.