Inter-rater reliability and generalizability of patient note scores using a scoring rubric based on the USMLE Step-2 CS format

AERA, APA, & NCME. (2014). The standards for educational and psychological testing. Washington, DC: American Educational Research Association.

Boulet, J. R., Ben-David, M. F., et al. (1998). An investigation of the sources of measurement error in the post-encounter written scores from standardized patient examinations. Advances in Health Sciences Education: Theory and Practice, 3, 89–100.

Boulet, J. R., Rebbecchi, T. A., et al. (2004). Assessing the written communication skills of medical school graduates. Advances in Health Sciences Education: Theory and Practice, 9, 47–60.

Brennan, R. L. (2001). Generalizability theory. New York, NY: Springer.

Cianciolo, A. T., Williams, R. G., et al. (2013). Biomedical knowledge, clinical cognition and diagnostic justification: A structural equation model. Medical Education, 47, 309–316.

Clauser, B. E., Harik, P., et al. (2008). The generalizability of documentation scores from the USMLE step 2 clinical skills examination. Academic Medicine, 83, S41–S44.

Downing, S. M. (2003). Validity: On meaningful interpretation of assessment data. Medical Education, 37, 830–837.

Federation of the State Medical Boards & National Board of Medical Examiners. (2015). Step 2 clinical skills (CS) content description and general information. Philadelphia, PA: United States Medical Licensing Examination. Retrieved June 25, 2015 from http://www.usmle.org/pdfs/step-2-cs/cs-info-manual.pdf

Fleiss, J. L., & Cohen, J. (1973). The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability. Educational and Psychological Measurement, 33, 613–619.

Gilliland, W. R., La Rochelle, J., et al. (2008). Changes in clinical skills education resulting from the introduction of the USMLE step 2 clinical skills (CS) examination. Medical Teacher, 30, 325–327.

Gingerich, A., Kogan, J., et al. (2014). Seeing the ‘black box’ differently: Assessor cognition from three research perspectives. Medical Education, 48, 1055–1068.

Haist, S. A., Katsufrakis, P. J., et al. (2013). The evolution of the United States Medical Licensing Examination (USMLE): Enhancing assessment of practice-related competencies. Journal of the American Medical Association, 310, 2245–2246.

Hombo, C. M., Donoghue, J. R., et al. (2001). A simulation study of the effect of rater designs on ability estimation (ETS Research Report No. RR-01-05). Princeton, NJ: ETS. Retrieved June 25, 2015 from http://www.ets.org/research/policy_research_reports/publications/report/2001/hseq

Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33, 159–174.

Messick, S. (1995). Standards of validity and the validity of standards in performance assessment. Educational Measurement: Issues and Practice, 14, 5–8.

National Board of Medical Examiners. (2015). NBME 2013 Annual Report. Philadelphia, PA: NBME. Retrieved June 25, 2015 from http://www.nbme.org/PDF/Publications/2014Annual-Report.pdf

Park, Y. S., Lineberry, M., et al. (2013). Validity evidence for a patient note scoring rubric based on the new patient note format of the United States Medical Licensing Examination. Academic Medicine, 88, 1552–1557.

Southern Illinois University. (2015). DX Justification Scoring Form. Carbondale, IL: SIU School of Medicine. Retrieved June 25, 2015 from http://www.siumed.edu/oec/CCX_ASSESSMENTS/2015/DX%20Justification_scoring%20form.pdf

United States Medical Licensing Examination. (2015). 2014 Performance Data: Step 2 CS. Philadelphia, PA: NBME. Retrieved June 25, 2015, from http://www.usmle.org/performance-data/default.aspx#2014_step-2-cs

Whelan, G. P. (1999). Educational commission for foreign medical graduates: Clinical skills assessment prototype. Medical Teacher, 21, 156–160.

Williams, R. G., Klamen, D. A., et al. (2003). Special Article: Social and environmental sources of bias in clinical performance ratings. Teaching and Learning in Medicine, 15, 270–292.

Williams, R. G., Klamen, D. L., et al. (2014). Variations in senior medical student diagnostic justification ability. Academic Medicine, 89, 790–798.

Yudkowsky, R., Park, Y. S., et al. (2015). Characteristics and implications of diagnostic justification scores based on the new patient note format of the USMLE Step 2 CS exam. Academic Medicine, 90, S56–S62.