Selection of subtle cases for observer-performance studies: The importance of knowing the true diagnosis

Metz, 1989, Some practical issues of experimental design and data analysis in radiological ROC studies, Invest Radiol, 24, 234, 10.1097/00004424-198903000-00012 Thaete, 1994, Digital radiography and conventional imaging of the chest: a comparison of observer performance, AJR, 162, 575, 10.2214/ajr.162.3.8109499 Cox, 1990, Chest radiography: comparison of high-resolution digital displays with conventional and digital film, Radiology, 176, 771, 10.1148/radiology.176.3.2389035 Rockette, 1997, Relationship of subjective ratings of image quality and observer performance, 3036, 152 Swets, 1991, Enhancing and evaluating diagnostic accuracy, Med Decis Making, 11, 9, 10.1177/0272989X9101100102 Rockette, 1995, Image systems evaluation: effect of subtle cases on the design and analysis of ROC studies, AJR, 165, 679, 10.2214/ajr.165.3.7645495 King, 1995, Variability in reader selection of subtle cases for inclusion in a reduced size ROC study, 2436, 74 Gur, 1992, On the selection of “subtle” cases for ROC studies, 1654, 322 Slasky, 1990, Receiver operating characteristic analysis of chest image interpretation with conventional, laser-printed, and high-resolution workstation images, Radiology, 174, 775, 10.1148/radiology.174.3.2305061 Cohen, 1960, A coefficient of agreement for nominal scales, Educ Psychol Meas, 20, 37, 10.1177/001316446002000104 Aickin, 1990, Maximum likelihood estimation of agreement in the constant predictive probability model, and its relation to Cohen's Kappa, Biometrics, 46, 293, 10.2307/2531434 Block, 1989, 2 × 2 Kappa coefficients: measures of agreement or association, Biometrics, 45, 269, 10.2307/2532052 Maclure, 1987, Misinterpretation and misuse of the Kappa statistic, Am J Epidemiol, 126, 161, 10.1093/aje/126.2.161 Metz, 1990, New methods for estimating a binomial ROC curve from continuously distributed test results Metz, 1990, New methods for estimating a binormal ROC curve from continuously-distributed test results