The Restriction of Variance Hypothesis and Interrater Reliability and Agreement: Are Ratings from Multiple Sources Really Dissimilar?

Organizational Research Methods - Tập 6 Số 1 - Trang 80-128 - 2003
James M. LeBreton1, Jennifer R. D. Burgess2, Robert B. Kaiser3, E. Kate Atchley4, Lawrence R. James4
1Wayne State U, Detroit, MI, US
2Bradley University,
3Kaplan DeVries Inc.
4#N##TAB##TAB##TAB##TAB# University of Tennessee#N##TAB##TAB##TAB#

Tóm tắt

The fundamental assumption underlying the use of 360-degree assessments is that ratings from different sources provide unique and meaningful information about the target manager’s performance. Extant research appears to support this assumption by demonstrating low correlations between rating sources. This article reexamines the support of this assumption, suggesting that past research has been distorted by a statistical artifact—restriction of variance in job performance. This artifact reduces the amount of between-target variance in ratings and attenuates traditional correlation-based estimates of rating similarity. Results obtained from a Monte Carlo simulation and two field studies support this restriction of variance hypothesis. Noncorrelation-based methods of assessing interrater agreement indicated that agreement between sources was about as high as agreement within sources. Thus, different sources did not appear to be furnishing substantially unique information. The authors conclude by questioning common practices in 360-degree assessments and offering suggestions for future research and application.

Từ khóa


Tài liệu tham khảo

Allen, M. J. & Yen, W. M. (1979). Introduction to measurement theory. Prospect Heights, IL: Waveland Press.

Antonioni, D. (1994). The effects of feedback accountability on upward appraisal ratings. Personnel Psychology, 47, 249-256.

10.1037/0021-9010.77.6.836

10.1006/obhd.2000.2897

Barber, A. E. (1998). Recruiting employees: Individual and organizational perspectives. Thousand Oaks, CA: Sage.

10.1111/j.1744-6570.1991.tb00688.x

10.1037/0033-2909.83.5.762

Bernardin, H. J. & Buckley, M. R. (1981). Strategies in rater training. Academy of Management Review, 6, 205-212.

Bernardin, H. J. & Kane, J. S. (1980). A second look at behavioral observation scales. Personnel Psychology, 33, 509-513.

10.1037/0021-9010.66.4.458

10.1037/0021-9010.74.3.478

Borman, W. C. (1991). Job behavior, performance, and effectiveness. In M. D. Dunnette & L. M. Hough (Eds.), Handbook of industrial and organizational psychology (2nd ed., Vol. 2.). Palo Alto, CA: Consulting Psychologists Press.

10.1016/S1053-4822(97)90010-3

10.1037/0021-9010.86.5.965

10.1037/0021-9010.60.5.561

10.1177/001316445001000201

Campbell, J. P. (1990). Modeling the performance prediction problem in industrial and organizational psychology. In M. D. Dunnette & L. M. Hough (Eds.), Handbook of industrial and organizational psychology (Vol. 1, 2nd ed.). Palo Alto, CA: Consulting Psychologists Press.

Cardy, R. L. & Dobbins, G. H. (1994). Performance appraisal: Alternative perspectives. Cincinnati, OH: South-Western.

10.1037/1082-989X.6.3.297

Cohen, J. (1988). Statistical power analysis for the behavioral sciences. Hillsdale, NJ: Erlbaum.

Cohen, J. & Cohen, P. (1983). Applied multivariate analysis/linear regression (2nd ed.). Hillsdale, NJ: Erlbaum.

10.1207/s15327043hup1004_2

10.1037/0033-2909.90.2.218

10.1177/1094428102239425

Dalessio, A. T. (1998). Using multi-source feedback for employee development and personnel decisions. In J. W. Smither (Ed.), Performance appraisal. San Francisco: Jossey-Bass.

10.1037/0021-9010.81.6.717

10.1037/0021-9010.66.2.127

10.1037/0021-9010.73.2.239

10.1177/014662169001400407

10.1177/0013164494054002010

10.1016/S1048-9843(96)90003-X

10.1037/0033-2909.101.1.75

Gatewood, R. D. & Feild, H. S. (2001). Human resource selection (5th ed.). New York: Harcourt College.

10.1037/0021-9010.75.2.107

Goldstein, I. L. & Ford, J. K. (2002). Training in organizations: Needs assessment, development, and evaluation (4th ed.). Belmont, CA: Wadsworth/Thomson Learning.

10.1037/0021-9010.83.6.960

Guilford, J. P. (1954). Psychometric methods (2nd ed.). New York: McGraw-Hill.

10.1111/j.1744-6570.1961.tb00930.x

Gulliksen (1950). Theory of mental tests. New York: John Wiley.

Harris, M. M. (1994). Rater motivation in the performance appraisal context: A theoretical framework. Journal of Management, 20, 737-756.

10.1111/j.1744-6570.1988.tb00631.x

10.1037/0021-9010.73.1.68

10.1037/0021-9010.79.2.184

10.1037/0021-9010.86.5.897

Hunter, J. E. (1983). Test validation for 12,000 jobs: An application of job classification and validity generalization to the General Aptitude Test Battery (U.S. Employment Service Test Research Report No. 45). Washington, DC: U.S. Department of Labor.

10.1037/0033-2909.96.1.72

10.1006/obhd.1993.1015

10.1037/h0034627

10.1037/0021-9010.67.2.219

10.1037/0021-9010.69.1.85

10.1037/0021-9010.78.2.306

James, L. R. & LeBreton, J. M. (2001). Disentangling issues of agreement, disagreement, and lack of agreement using rWG, rWGj, and r*WGj. Unpublished manuscript, University of Tennessee, Knoxville.

James, L. R. & Mazerolle, M. (2001). Personality in work organizations: An integrative approach. Thousand Oaks, CA: Sage.

10.1111/j.1744-6570.1997.tb01487.x

10.1037/0021-9010.86.1.80

10.1037/0033-2909.127.3.376

Kaiser, R. B., Craig, S. B., Kaplan, R. E. & McArthur (2002). Practical science and the development of Motorola’s leadership standards. In K. B. Brookhouse (Chair), Transforming leadership at Motorola. Practitioner forum presented at the 17th annual meeting of the Society for Industrial and Organizational Psychology, Toronto, Canada.

Kanfer, R. (1990). Motivation theory and Industrial/Organizational psychology. In M. D. Dunnette & L. Hough (Eds.), Handbook of industrial and organizational psychology (2nd ed., Vol. 1). Palo Alto, CA: Consulting Psychologists Press.

10.1111/j.1744-6570.1981.tb00942.x

10.1037/0021-9010.77.2.161

Kraiger, K. (1986). Self, peer, and supervisory ratings of performance. Paper presented at the annual meeting of the Society for Industrial and Organizational Psychology, Chicago.

10.1111/j.1744-6570.1996.tb01790.x

10.1037/0021-9010.78.5.821

10.1037/0033-2909.127.1.162

10.1177/014920639402000404

10.1037/0003-066X.41.11.1183

10.1037/0033-2909.87.1.72

Latham, G. P. & Wexley, K. N. (1977). Behavioral observation scales for performance appraisal purposes. Personnel Psychology, 30, 225-268.

10.1037/h0025095

10.1037/0021-9010.85.5.799

Lindsey, E. H., Homes, V. & McCall, M. W. (1987). Key events in executive’s lives. Greensboro, NC: Center for Creative Leadership.

Locke, E. A. (1999). The essence of leadership. New York: Lexington Books.

Lombardo, M. & McCauley, C. D. (1994). Benchmarks: A manual and trainer’s guide. Greensboro, NC: Center for Creative Leadership.

Lord, F. M. & Novick, M. R. (1968). Statistical theories of mental test scores. Reading, MA: Addison-Wesley.

10.1037/0021-9010.78.2.226

McCall, M., Lombardo, M. M. & Morrison, A. M. (1988). The lessons of experience: How successful executives develop on the job. New York: Free Press.

McCauley, C. D. & Lombardo, M. (1990). Benchmarks: An instrument for diagnosing managerial strengths and weaknesses. In K. E. Clark & M. B. Clark (Eds.), Measures of leadership. West Orange, NJ: Leadership Library of America.

10.1037/1082-989X.1.1.30

Mintzberg, H. (1989). Mintzberg on management: Inside our strange world of organizations. New York: Free Press.

10.1207/s15327043hup1002_1

10.1111/j.1744-6570.1998.tb00251.x

10.1037/0021-9010.74.4.619

Murphy, K. R. & Cleveland, J. N. (1995). Understanding performance appraisal: Social, organizational, and goal based perspectives. Thousand Oaks, CA: Sage.

Murphy, K. R., Cleveland, J. N. & Mohler, C. J. (2001). Reliability, validity, and meaningfulness of multisource ratings. In D. W. Bracken & C. W. Timmreck (Eds.), The handbook of multisource feedback: The comprehensive resource for designing and implementing MSF processes. San Francisco: Jossey-Bass.

10.1111/j.1744-6570.2000.tb02421.x

10.1111/j.1744-6570.1997.tb01484.x

Nunnally, J. C. (1978). Psychometric theory (2nd ed.). New York: McGraw-Hill.

Nunnally, J. C. & Bernstein, I. H. (1994). Psychometric theory (3rd ed.). New York: McGraw-Hill.

10.1037/0021-9010.81.6.660

10.1126/science.146.3642.347

10.1037/0021-9010.75.3.322

10.1037/0021-9010.85.1.112

10.1037/0033-2909.124.2.262

10.1037/0021-9010.61.4.473

10.1111/j.1744-6570.1971.tb00365.x

10.1111/j.1744-6570.1999.tb00169.x

10.1111/j.1744-6570.2000.tb02422.x

Schmitt, N. & Chan, D. (1998). Personnel selection: A theoretical approach. Thousand Oaks, CA: Sage.

10.1111/j.1744-6570.1987.tb00609.x

10.1111/j.1744-6570.1995.tb01780.x

10.1037/0021-9010.82.6.880

10.1037/0021-9010.85.6.956

10.1037/0033-2909.86.2.420

Smith, G. M. (1970). A simplified guide to statistics for psychology and education (4th ed.). New York: Holt, Rinehart & Winston.

Smith, P. C. (1976). Behaviors, results, and organizational effectiveness: The problem of criteria. In M. D. Dunnette (Ed.), Handbook of industrial and organizational psychology (pp. 745-775). Chicago: Rand McNally.

10.1037/0021-9010.73.3.497

10.1037/0021-9010.79.4.535

Tears, R. S., O’Leary, R. S. & Facteau, J. D. (2001, April). Do rating sources matter? A replication of Mount, Judge, Scullen, Systma, and Hezlett (1998). Presented in R. B. Kaiser & S. B. Craig (Co-chairs), Modern analytic techniques in the study of 360-degree performance ratings. Symposium presented a the 16th annual conference of the Society for Industrial and Organizational Psychology, San Diego, CA.

10.1002/hrm.3930320203

10.1111/j.1744-6570.1988.tb00654.x

Van Velsor, E. & Leslie, J. B. (1993). Feedback to managers (Vol. II): A review and comparison of sixteen multi-rater feedback instruments. Greensboro, NC: Center for Creative Leadership.

10.1111/j.1744-6570.1996.tb01803.x

10.1037/0021-9010.83.4.586

10.1037/0021-9010.81.5.557

10.1037/0021-9010.87.2.345

10.1037/0021-9010.79.4.525

10.1111/j.2044-8325.1994.tb00562.x

10.1177/014920639702300505

Yukl, G. (1998). Leadership in organizations (4th ed.). Upper Saddle River, NJ: Prentice Hall.

Zedeck, S. (1995). [Review of Benchmarks]. In J. Conoley & J. Impara (Eds.), The twelfth mental measurements yearbook (Vol. 1, pp. 128-129). Lincoln, NE: Buros Institute of Mental Measurements.

10.1037/0021-9010.67.6.752