Answers to 20 Questions About Interrater Reliability and Interrater Agreement
Tóm tắt
The use of interrater reliability (IRR) and interrater agreement (IRA) indices has increased dramatically during the past 20 years. This popularity is, at least in part, because of the increased role of multilevel modeling techniques (e.g., hierarchical linear modeling and multilevel structural equation modeling) in organizational research. IRR and IRA indices are often used to justify aggregating lower-level data used in composition models. The purpose of the current article is to expose researchers to the various issues surrounding the use of IRR and IRA indices often used in conjunction with multilevel models. To achieve this goal, the authors adopt a question-and-answer format and provide a tutorial in the appendices illustrating how these indices may be computed using the SPSS software.
Từ khóa
Tài liệu tham khảo
Bliese, P.D. (2000). Within-group agreement, non-independence, and reliability: Implications for data aggregation and analysis. In K. J. Klein & S. W. Kozlowski (Eds.), Multilevel theory, research, and methods in organizations (pp. 349-381). San Francisco: Jossey-Bass.
Borman, W.C. (1991). Job behavior, performance, and effectiveness. In M. D. Dunnette & L. M. Hough (Eds.), Handbook of industrial and organizational psychology (2nd ed., vol. 2, pp. 271-326). Palo Alto, CA: Consulting Psychologists Press.
Bryk, A.S., 1992, Hierarchical linear models: Application and data analysis methods
Cardy, R.L., 1994, Performance appraisal: Alternative perspectives
Cohen, J., 1988, Statistical power analysis for the behavioral sciences, 2
Cronbach, L.J., 1972, The dependability of behavioral measurements
Hofmann, D.A. (2002). Issues in multilevel research: Theory development, measurement, and analysis. In S. G. Rogelberg (Ed.), Handbook of research methods in industrial and organizational psychology (pp. 247-274). New York: Blackwell .
Hofmann, D.A., Griffin, M.A. & Gavin, M.B. (2000). The application of hierarchical linear modeling to organizational research. In K. J. Klein & S. W. Kozlowski (Eds.), Multilevel theory, research, and methods in organizations (pp. 467-511). San Francisco: Jossey-Bass.
Kozlowski, S.W.J. & Klein, K.J. (2000). A multilevel approach to theory and research in organizations: Contextual, temporal, and emergent processes. In K. J. Klein & S. W. Kozlowski (Eds.), Multilevel theory, research, and methods in organizations (pp. 349-381). San Francisco: Jossey-Bass.
Lord, F.M., 1968, Statistical theories of mental test scores
Murphy, K.R., 1995, Understanding performance appraisal: Social, organizational, and goal based perspectives
Murphy, K.R., 1998, Statistical power analysis: A simple and general model for traditional and modern hypothesis tests
Newman, D.A., Organizational Research Methods
Nunnally, J.C., 1978, Psychometric theory, 2
Nunnally, J.C., 1994, Psychometric theory, 3
Roberson, Q.M., Organizational Research Methods
Schriesheim, C.A., Paper presented the annual meeting of the Academy of Management
Smith, G.M., 1970, A simplified guide to statistics for psychology and education, 4