Clinical measurements obtained from point-of-care ultrasound images to assess acquisition skills
Tóm tắt
Current methods of assessing competence in acquiring point-of-care ultrasound images are inadequate. They rely upon cumbersome rating systems that do not depend on the actual outcome measured and lack evidence of validity. We describe a new method that uses a rigorous statistical model to assess performance of individual trainees based on the actual task, image acquisition. Measurements obtained from the images acquired (the actual desired outcome) are themselves used to validate effective training and competence acquiring ultrasound images. We enrolled a convenience sample of 21 spontaneously breathing adults from a general medicine ward. In random order, two trainees (A and B) and an instructor contemporaneously acquired point-of-care ultrasound images of the inferior vena cava and the right internal jugular vein from the same patients. Blinded diameter measurements from each ultrasound were analyzed quantitatively using a multilevel model. Consistent mean differences between each trainee’s and the instructor’s images were ascribed to systematic acquisition errors, indicative of poor measurement technique and a need for further training. Wider variances were attributed to sporadic errors, indicative of inconsistent application of measurement technique across patients. In addition, the instructor recorded qualitative observations of each trainee’s performance during image acquisition. For all four diameters, the means and variances of measurements from trainee A’s images differed significantly from the instructor’s, whereas those from trainee B’s images were comparable. Techniques directly observed by the instructor supported these model-derived findings. For example, mean anteroposterior diameters of the internal jugular vein obtained from trainee A’s images were 3.8 mm (90% CI 2.3–5.4) smaller than from the instructor’s; this model-derived finding matched the instructor’s observation that trainee A compressed the vein during acquisition. Instructor summative assessments agreed with model-derived findings, providing internal validation of the descriptive and quantitative assessments of competence acquiring ultrasound images. Clinical measurements obtained from point-of-care ultrasound images acquired contemporaneously by trainees and an instructor can be used to quantitatively assess the image acquisition competence of specific trainees. This method may obviate resource-intensive qualitative rating systems that are based on ultrasound image quality and direct observation, while also helping instructors guide remediation.
Tài liệu tham khảo
Liebo MJ, Israel RL, Lillie EO, Smith MR, Rubenson DS, Topol EJ (2011) Is pocket mobile echocardiography the next-generation stethoscope? A cross-sectional comparison of rapidly acquired images with standard transthoracic echocardiography. Ann Intern Med 155:33–38
Kane MT (1994) Validating interpretive arguments for licensure and certification arguments. Eval Health Prof 17:133–159
Barnhart HX, Haber MJ, Lin LI (2007) An overview on assessing agreement with continuous measurements. J Biopharm Stat 17:529–569
Tartière D, Seguin P, Juhel C, Laviolle B, Mallédant Y (2009) Estimation of the diameter and cross-sectional area of the internal jugular veins in adult patients. Crit Care 13:R197
Tchernodrinski S, Artfield R (2015) Inferior vena cava. In: Soni NJ, Arntfield R, Kory P (eds) Point of care ultrasound. Elsevier Saunders, Philadelphia
Lucas BP, Candotti C, Margeta B, Evans AT, Mba B, Baru J, Asbury JK, Asmar A, Kumapley R, Patel M, Borkowsky S, Fung S, Charles-Damte M (2009) Diagnostic accuracy of hospitalist-performed hand-carried ultrasound echocardiography after a brief training program. J Hosp Med 4:340–349
Lucas BP, D’Addio A, Clark J, Block C, Manning H, Remillard B, Leiter JC (2017) Reproducibility of point-of-care ultrasonography for central vein diameter measurement: separating image acquisition from interpretation. J Clin Ultrasound 45:488–496
Grant E, Rendano F, Sevinc E, Gammelgaard J, Holm HH, Grønvall S (1980) Normal inferior vena cava: caliber changes observed by dynamic ultrasound. AJR Am J Roentgenol 135:335–338
Fraser CG (2001) Biological variation: from principles to practice. AACC Press, Washington, DC
Molenberghs G, Laenen A, Vangeneugden T (2007) Estimating reliability and generalizability from hierarchical biomedical data. J Biopharm Stat 17:595–627
Hopkins WG (2000) Measures of reliability in sports medicine and science. Sports Med 30:1–15
Bartlett JW, Frost C (2008) Reliability, repeatability and reproducibility: analysis of measurement errors in continuous variables. Ultrasound Obstet Gynecol 31:466–475
Dunn G (1992) Design and analysis of reliability studies. Stat Method Med Res 1:123–157
Taylor JR (1997) An introduction to error analysis: the study of uncertainties in physical measurements, 2nd edn. University Science Books, Sausalito
Allen MJ, Yen WM (1979) Introduction to measurement theory. Waveland Press, Long Grove
Fraser CG, Petersen PH, Libeer JC, Ricos C (1997) Proposals for setting generally applicable quality goals solely based on biology. Ann Clin Biochem 34:8–12
Lyon M, Blaivas M, Brannam L (2005) Sonographic measurement of the inferior vena cava as a marker of blood loos. Am J Emerg Med 23:45–50
Browne WJ, Draper D (2006) A comparison of Bayesian and likelihood-based methods for fitting multilevel models. Bayesian Anal 1:473–514
Guldbrand Nielsen D, Jensen SL, O’Neill L (2015) Clinical assessment of transthoracic echocardiography skills: a generalizability study. BMC Med Educ 15:9
McGraw KO, Wong SP (1996) Forming inferences about some intraclass correlation coefficients. Psychol Methods 1:30–46
Leckie G, Charlton C (2013) runmlwin—A program to run the MLwiN multilevel modelling software from within Stata. J Stat Softw 52:1–40
Jang MJ, Lee Y, Lawson AB, Browne WJ (2007) A comparison of the hierarchical likelihood and Bayesian approaches to spatial epidemiological modelling. Environmetrics 18:809–821
Rudski LG, Lai WW, Afilalo J, Hua L, Handschumacher MD, Chandrasekaran K, Solomon SD, Louie EK, Schiller NB (2010) Guidelines for the echocardiographic assessment of the right heart in adults: a report from the American Society of Echocardiography. J Am Soc Echocardiogr 23:685–713
Keller AS, Melamed R, Malinchoc M, Reverly J, Tierney DM, Gajic O (2009) Diagnostic accuracy of simple ultrasound measurement to estimate central venous pressure in spontaneously breathing, critically ill patients. J Hosp Med 4:350–355
Tchernodrinski S, Lucas BP, Athavale A, Candotti C, Margeta B, Katz A, Kumapley R (2015) Inferior vena cava diameter change after intravenous furosemide in patients diagnosed with acute decompensated heart failure. J Clin Ultrasound 43:187–193
Malbrain ML, Marik PE, Witters I, Cordemans C, Kirkpatrick AW, Roberts DJ, Van Regenmortel N (2014) Fluid overload, de-resuscitation, and outcomes in critically ill or injured patients: a systematic review with suggestions for clinical practice. Anaesthesiol Intensive Ther 46:361–380
Bremer ML (2016) Relationship of sonographer credentialing to intersocietal accreditation commission echocardiography case study image quality. J Am Soc Echocardiogr 29:43–48
Bahner DP, Adkins EJ, Nagel R, Way D, Werman HA, Royall NA (2011) Brightness mode quality ultrasound imaging examination technique. J Ultrasound Med 30:1649–1655
Wass V, Van der Vleuten C, Shatzer J, Jones R (2001) Assessment of clinical competence. Lancet 357:945–949
Downing SM (2003) Validity: on the meaningful interpretation of assessment data. Med Educ 37:830–837
Lewiss RE, Pearl M, Nomura JT, Baty G, Bengiamin R, Duprey K, Stone M, Theodoro D, Akhtar S (2013) CORD-AEUS: consensus document for the emergency ultrasound milestone project. Acad Emerg Med 20:740–745
Gibbs V (2014) A proposed new clinical assessment framework for diagnostic medical ultrasound students. Ultrasound 22:113–117
Parker PC, Byass OR (2015) Successful implementation of a performance-related audit tool for sonographers. Ultrasound 23:97–102
Boulet JR, McKinley DW, Whelan GP, Hambleton RK (2003) Quality assurance methods for performance-based assessments. Adv Health Sci Educ Theory Pract 8:27–47
Haber M, Barnhart HX (2006) Coefficients of agreement for fixed observers. Stat Methods Med Res 15:255–271
Setodji CM, Shwartz M (2013) Fixed-effect or random-effect models: what are the key inference issues? Med Care 51:25–27
Harris EF, Smith RN (2009) Accounting for measurement error: a critical but often overlooked process. Arch Oral Biol 54:S107–S117