Using a Longitudinal Database to Assess the Validity of Preceptors' Ratings of Clerkship Performance
Tóm tắt
Purpose. To examine the validity of using scores from a clinical evaluation form as an assessment of clinical competence. Method. Investigators collected a longitudinal clinical skills assessment database that included scores reflecting performance on standardized patient interactions, case-based learning performance, scores on multiple-choice clinical examinations, and preceptors' ratings of students during their clerkships. Pearson correlation coefficients and coefficients corrected for attenuation were calculated between the mean preceptor rating CEF score and the other measures collected during years one and two. Results. Estimates from an earlier across-clerkship generalizability study of the CEF demonstrated that a mean rating computed across 28 forms and four clerkships (the average measurement frequencies observed in this study), yielded an estimated G-coefficient of 0.62 (Kreiter and Ferguson, 2001). Reliabilities for the other measures ranged from 0.23 to 0.56. Nine of the twelve clinical skill measures correlated with the CEF. For those significantly correlated measures where reliabilities could be calculated, correlations corrected for attenuation ranged from 0.46 to 0.58. Conclusion. This study indicates that the skills measured by the CEF are related to other clinical performance measures and, conversely, that pre-clinical measures of skills that are believed to be important during clinical years are in fact predictors of preceptors' ratings of clinical performance later in medical school. In addition, the magnitude of the disattenuated coefficients suggests that ratings on the CEF are dependent on important aspects such as clinical knowledge. This study suggests that when averaged over a large number of observations, mean CEF scores demonstrate validity coefficients large enough to support their use as part of an evaluation of students' clinical performance.
Tài liệu tham khảo
Brennan, R.L. (2001). Generalizability Theory. New York: Springer Publishing.
Callahan, C.A., Erdmann, J.B., Hojat, M., Veloski, J.J., Rattner, S., Nasca, T.J. & Gonnella, J.S. (2000). Validity of faculty ratings of students' clinical competence in core clerkships in relation to scores on licensing examinations and supervisors' ratings in residency. Academic Medicine 75: S71-S73.
Ferguson, K.J., Kreiter, C.D., Densen, P. & Woodhead, J.C. (1999). Personal Characteristics of Medical Students as Assessed by an Adjective Checklist. Poster/Abstract presentation at Central Group on Educational Affairs / American Association of Medical Colleges, Kansas City.
Ferguson, K.J., Steiner, L.A., Kreiter, C.D. & Pomrehn, P.R. (1997). Assessment of case-based learning within a hybrid curriculum. Poster Presentation at the Association of American Medical Colleges/Research in Medical Education Annual Meeting, Washington, D.C.
Gough, H.G., Hall, W.B. & Harris, R.D. (1964). Evaluation of performance in medical training. Journal of Medical Education 39: 679-92.
Gregg, A. & Dehn, R. (1999). An analysis of clinical evaluation methods in physician assistant education. Perspective on P.A. Education 10(4): 185-193.
Kreiter, C., Ferguson, K., Lee,W., Brennan, R. & Densen, P. (1998). A generalizability study of a new standardized rating form used to evaluate students' clinical clerkship performances. Academic Medicine 73(12): 1294-1298.
Kreiter, C.D., Ferguson, K.J. (2001). Examining the generalizability of ratings across clerkships using a clinical evaluation form. Evaluation and the Health Professions 24: 36-46.
Linn, R.L. (1986). Comments on the g factor in employment testing. Journal of Vocational Behavior 29: 438-444.
Magarian, G.L. & Mazur, D.J. (1990). Evaluation of students in medicine clerkships. Academic Medicine 65(5): 341-345.
Muchinsky, P.M. (1996). The correction for attenuation. Educational and Psychological Measurement 56(1): 63-75.
Nunnally, J.C. (1978). Psychometric Theory (2nd ed.). New York: McGraw-Hill.
Pedhazur, E.J. (1982). Multiple Regression in Behavioral Research-Explanation and Prediction (2nd ed.) (pp. 112-114). Holt, Rinehart and Winston.
Petrusa, E.R, Blackwell, T.A. & Ainsworth, M.A. (1990). Reliability and validity of an objective structured clinical examination for assessing the clinical performance of residents. Archives of Internal Medicine 150: 573-577.