Many Labs 2: Investigating Variation in Replicability Across Samples and Settings

Advances in Methods and Practices in Psychological Science - Tập 1 Số 4 - Trang 443-490 - 2018
Richard Klein1, Michelangelo Vianello2, Byron G. Adams3,4, Reginald B. Adams5, Sinan Alper6, Mark Aveyard7, Jordan Axt8, Mayowa T. Babalola9, Štěpán Bahník10, Rishtee Batra11, Mihály Berkics12, Michael J. Bernstein13, Olga Białobrzeska14, Evans Dami Binan15, Konrad Bocian16, Mark J. Brandt17, Anna Cabak Rédei18, Huajian Cai19, Ingrid Poncin20,21, Katarzyna Cantarero22, Cheryl L. Carmichael23, Francisco Cerić24,25, Jesse Chandler26,27, Jen‐Ho Chang28,29, Armand Chatard30,31, Eva E. Chen32, Winnee Cheong33, David C. Cicero34, Sharon Coen35, Jennifer A. Coleman36, Brian Collisson37, Morgan Conway38, Katherine S. Corker39, Paul Curran39, Fiery Cushman40, Zubairu Kwambo Dagona15, Sophia Christin Weißgerber41, Anna Dalla Rosa2, William E. Davis42, Leander De Schutter17, Thierry Devos43, Marieke de Vries3,44,45, Canay Doğulu46, Nerisa Dozo47, Kristin Nicole Dukes48, Kevin Durrheim49, Charles R. Ebersole50, John Edlund51, Anja Eller52, Carolyn Finck53, Natalia Frankowska54, Miguel-Ángel Freyre52, Michael Friedman20,21, Elisa Maria Galliani55, Tanuka Ghoshal56, Steffen R. Giessner57, Tripat Gill58, Timo Gnambs59,60, Ángel Gómez61, Roberto González62, Jesse Graham63, Ivan Grahek64, Eva G. T. Green65, Matthew Haigh66, Elizabeth L. Haines67, Michael P. Hall68, Marie E. Heffernan69, Joshua A. Hicks70, Petr Houdek71, Jeffrey R. Huntsinger72, Ho Phi Huynh73, Hans IJzerman1, Yoel Inbar74, Åse Innes-Ker75, William Jiménez‐Leal53, Melissa-Sue John76, Jennifer Joy-Gaba36, Roza Gizem Kamiloglu77, Heather Barry Kappes78, Serdar Karabatı79, Haruna Karick15,54, Victor N. Keller80, Anna Kende81, Nicolas Kervyn20,21, Goran Knežević82, Carrie Kovacs83, Lacy E. Krueger84, German Kurapov85, Jamie Kurtz86, Daniël Lakens87, Ljiljana B. Lazarević88, Carmel Levitan89, Neil A. Lewis90, Samuel Lins91, Nikolette P. Lipsey38, Joy E. Losee38, Esther Maassen92, Angela T. Maitner7, Winfrida Malingumu93, Robyn K. Mallett72, Satia A. Marotta94, Janko Međedović95,96, Fernando Mena-Pacheco97, Taciano L. Milfont98, Wendy L. Morris99, Sean C. Murphy100, Andriy Myachykov101, Nick Neave101, Koen Ilja Neijenhuijs102,103, A. Nelson5, Félix Neto91, Austin Nichols104, Aaron Ocampo97, Susan O’Donnell105, Haruka Oikawa106, Masanori Oikawa106, Elsie Ong107, Gábor Orosz108, Małgorzata Osowiecka54, Grant Packard58, Rolando Pérez Sánchez109, Boban Petrović96, Ronaldo Pilati80, Brad Pinter5, Lysandra Podesta3,4, Gabrielle Pogge38, Monique Pollmann110, Abraham M. Rutchick111, Patricio Saavedra112, Alexander K. Saeri113, Erika Salomon114, Kathleen Schmidt115, Felix D. Schönbrodt116, Maciej Sekerdej117, David Sirlopú25, Jeanine Skorinko76, Michael Smith101, Vanessa Smith‐Castro109, K.C.H.J. Smolders87, Agata Sobków118, Walter Sowden119, Philipp Spachtholz116, Manini Srivastava120, Troy G. Steiner5, Jeroen Stouten121, Chris Street122, Oskar Sundfelt75, Stephanie Szeto35, Ewa Szumowska117, Andrew C. W. Tang107, Norbert K. Tanzer123, Morgan J. Tear113, Jordan E. Theriault124, Manuela Thomae125, David Torres126, Jakub Traczyk118, Joshua M. Tybur127, Adrienn Ujhelyi81, Robbie C. M. van Aert92, Marcel A. L. M. van Assen92, Marije van der Hulst128, Paul A. M. Van Lange127, Anna van 't Veer129, Alejandro Vásquez Echeverría130, Leigh Ann Vaughn131, Alexandra Vázquez61, Luis Diego Vega97, Catherine Verniers132, Mark Verschoor133, Ingrid Voermans4, Marek Vranka134, Cheryl Alyssa Welch86, Aaron L. Wichman135, Lisa Williams136, Michael Wood125, Julie A. Woodzicka137, Marta K. Wronska16, Liane Young138, John M. Zelenski139, Zhijia Zeng140, Brian A. Nosek141,50
1Laboratoire Inter-universitaire de Psychologie, Personnalité, Cognition, Changement Social (LIP/PC2S), Université Grenoble Alpes
2Department of Philosophy, Sociology, Education and Applied Psychology, University of Padua
3Behavioural Science Institute, Radboud University, Nijmegen
4School of Pedagogical and Educational Sciences, Radboud University Nijmegen
5Department of Psychology, The Pennsylvania State University
6Department of Psychology, Yasar University
7Department of International Studies, American University of Sharjah
8Center for Advanced Hindsight, Duke University
9College of Business and Economics United Arab Emirates University
10Department of Management, Faculty of Business Administration, University of Economics, Prague
11Erivan K. Haub School of Business, Saint Joseph’s University
12Institute of Psychology, ELTE Eötvös Loránd University
13Psychological and Social Sciences Program, Pennsylvania State University Abington
14Department of Psychology, California State University, San Marcos
15Department of General and Applied Psychology, University of Jos
16Sopot Faculty of Psychology, SWPS University of Social Sciences and Humanities
17Department of Social Psychology, Tilburg University
18Department of Psychology, University of Potsdam
19Institute of Psychology, Chinese Academy of Sciences
20Center on Consumers and Marketing Strategy (CCMS), Université catholique de Louvain
21Louvain Research Institute in Management and Organizations (LouRIM), Université catholique de Louvain
22Social Behavior Research Centre, Wroclaw Faculty of Psychology, SWPS University of Social Sciences and Humanities
23Department of Psychology, Brooklyn College & Graduate Center, CUNY
24Centro de Apego y Regulacion Emocional, Universidad del Desarrollo
25Facultad de Psicologia, Universidad del Desarrollo
26Institute for Social Research, University of Michigan
27Mathematica Policy Research, Princeton, New Jersey
28Department of Psychology, National Taiwan University
29Institute of Ethnology, Academia Sinica
30CNRS Unité Mixte de Recherche 7295, Poitiers, France
31Department of Psychology, Poitiers University
32Division of Social Science, The Hong Kong University of Science and Technology
33Department of Psychology, HELP University
34Department of Psychology, University of Hawaii at Manoa
35Directorate of Psychology and Public Health, University of Salford
36Department of Psychology, Virginia Commonwealth University
37Department of Psychology, Azusa Pacific University
38Department of Psychology, University of Florida
39Department of Psychology, Grand Valley State University
40Department of Psychology, Harvard University
41Department of Psychology, Middle East Technical University
42Department of Psychology, Wittenberg University
43Department of Psychology, San Diego State University
44Institute for Computing and Information Sciences, Radboud University Nijmegen
45Tilburg Institute for Behavioral Economics Research, Tilburg University
46Department of Psychology, Başkent University
47School of Psychology, The University of Queensland
48Office of Institutional Diversity, Allegheny College
49Department of Psychology, Yale University
50Department of Psychology, University of Virginia
51Department of Psychology, Rochester Institute of Technology
52Facultad de Psicología, Universidad Nacional Autónoma de México
53Departamento de Psicología, Universidad de los Andes, Colombia
54Warsaw Faculty of Psychology, SWPS University of Social Sciences and Humanities
55Department of Political and Juridical Sciences and International Studies, University of Padua
56Department of Marketing and International Business, Baruch College, CUNY
57Department of Organisation and Personnel Management, Rotterdam School of Management, Erasmus University
58Lazaridis School of Business and Economics, Wilfrid Laurier University
59Educational Measurement, Leibniz Institute for Educational Trajectories, Bamberg, Germany
60Institute of Education and Psychology, Johannes Kepler University Linz
61Departamento de Psicología Social y de las Organizaciones, Universidad Nacional de Educación a Distancia
62Escuela de Psicología, Pontificia Universidad Católica de Chile
63Eccles School of Business, University of Utah
64Psychology, Pacific Lutheran University
65Institute of Psychology, Faculty of Social and Political Sciences, University of Lausanne
66Amity Institute of Psychology and Allied Sciences, Amity University
67Department of Psychology, William Paterson University
68Department of Psychology, University of Michigan
69Smith Child Health Research, Outreach, and Advocacy Center, Ann & Robert H. Lurie Children’s Hospital of Chicago, Chicago, Illinois
70Department of Psychological & Brain Sciences, Texas A&M University
71Department of Economics and Management, Faculty of Social and Economic Studies, Jan Evangelista Purkyne University
72Department of Psychology, Loyola University, Chicago
73Department of Science and Mathematics, Texas A&M University-San Antonio
74Department of Psychology, University of Toronto Scarborough
75Department of Psychology, Lund University
76Department of Social Science and Policy Studies, Worcester Polytechnic Institute
77Department of Psychology, University of Amsterdam
78Department of Management, London School of Economics and Political Science
79Department of Business Administration, Istanbul Bilgi University
80Department of Social and Work Psychology, Institute of Psychology, University of Brasilia
81Department of Social Psychology, ELTE Eötvös Loránd University
82Department of Psychology, Faculty of Philosophy, University of Belgrade
83Department of Work, Organizational and Media Psychology, Johannes Kepler University Linz
84Department of Psychology & Special Education, Texas A&M University-Commerce
85International Victimology Institute Tilburg, Tilburg University
86Department of Psychology, James Madison University
87School of Innovation Science, Eindhoven University of Technology
88Institute of Psychology, Faculty of Philosophy, University of Belgrade
89Department of Cognitive Science, Occidental College
90Department of Communication, Cornell University
91Department of Psychology, University of Porto
92Department of Methodology and Statistics, Tilburg University
93Department of Education Policy Planning and Administration, Faculty of Education, Open University of Tanzania
94Department of Occupational Therapy, Tufts University
95Faculty of Media and Communications, Singidunum University
96Institute of Criminological and Sociological Research, Belgrade, Serbia
97Department of Psychology, Universidad Latina de Costa Rica
98Centre for Applied Cross-Cultural Research, Victoria University of Wellington
99Department of Psychology, McDaniel College
100Melbourne School of Psychological Sciences, The University of Melbourne
101Department of Psychology, Northumbria University
102Amsterdam Public Health Research Institute, Amsterdam, the Netherlands
103Department of Clinical, Neuro- and Developmental Psychology, Vrije Universiteit Amsterdam
104Department of Psychology, University of Central Florida
105Department of Psychology, George Fox University
106Department of Psychology, Doshisha University
107Li Ka Shing Institute of Professional and Continuing Education (LiPACE), The Open University of Hong Kong
108Department of Psychology, Stanford University
109Institute for Psychological Research, University of Costa Rica
110Department of Communication and Cognition, Tilburg University
111Department of Psychology, California State University, Northridge
112School of Psychology, University of Sussex
113BehaviourWorks Australia, Monash Sustainable Development Institute, Monash University
114Department of Computer Science, University of Chicago
115Department of Psychology, Southern Illinois University, Carbondale
116Department of Psychology, Ludwig-Maximilians-Universität München
117Institute of Psychology, Jagiellonian University in Kraków
118Wroclaw Faculty of Psychology, SWPS University of Social Sciences and Humanities
119Center for Military Psychiatry and Neuroscience, Walter Reed Army Institute of Research, Silver Spring, Maryland
120Faculty of Psychology and Educational Science and Physical Education, University of Regensburg
121Occupational & Organisational Psychology and Professional Learning, KU Leuven
122Department of Psychology, University of Huddersfield
123Institute of Psychology, University of Graz
124Department of Psychology, Northeastern University
125Department of Psychology, University of Winchester
126Department of Psychology, Universidad de Iberoamerica
127Department of Experimental and Applied Psychology, Vrije Universiteit Amsterdam
128Department of Obstetrics and Gynaecology, Erasmus MC, Rotterdam, The Netherlands
129Methodology and Statistics Unit, Institute of Psychology, Leiden University
130Centro de Investigación Básica en Psicología, Universidad de la República
131Department of Psychology, Ithaca College
132Institute of Psychology, Paris Descartes University - Sorbonne Paris Cité
133Department of Social Psychology, University of Groningen
134Department of Psychology, Faculty of Arts, Charles University
135Department of Psychological Science, Western Kentucky University
136School of Psychology, University of New South Wales
137Department of Psychology, Washington and Lee University
138Department of Psychology, Boston College
139Department of Psychology, Carleton University
140Zhejiang University of Finance and Economics
141Center for Open Science, Charlottesville, Virginia

Tóm tắt

We conducted preregistered replications of 28 classic and contemporary published findings, with protocols that were peer reviewed in advance, to examine variation in effect magnitudes across samples and settings. Each protocol was administered to approximately half of 125 samples that comprised 15,305 participants from 36 countries and territories. Using the conventional criterion of statistical significance ( p < .05), we found that 15 (54%) of the replications provided evidence of a statistically significant effect in the same direction as the original finding. With a strict significance criterion ( p < .0001), 14 (50%) of the replications still provided such evidence, a reflection of the extremely high-powered design. Seven (25%) of the replications yielded effect sizes larger than the original ones, and 21 (75%) yielded effect sizes smaller than the original ones. The median comparable Cohen’s ds were 0.60 for the original findings and 0.15 for the replications. The effect sizes were small (< 0.20) in 16 of the replications (57%), and 9 effects (32%) were in the direction opposite the direction of the original effect. Across settings, the Q statistic indicated significant heterogeneity in 11 (39%) of the replication effects, and most of those were among the findings with the largest overall effect sizes; only 1 effect that was near zero in the aggregate showed significant heterogeneity according to this measure. Only 1 effect had a tau value greater than .20, an indication of moderate heterogeneity. Eight others had tau values near or slightly above .10, an indication of slight heterogeneity. Moderation tests indicated that very little heterogeneity was attributable to the order in which the tasks were performed or whether the tasks were administered in lab versus online. Exploratory comparisons revealed little heterogeneity between Western, educated, industrialized, rich, and democratic (WEIRD) cultures and less WEIRD cultures (i.e., cultures with relatively high and low WEIRDness scores, respectively). Cumulatively, variability in the observed effect sizes was attributable more to the effect being studied than to the sample or setting in which it was studied.

Từ khóa


Tài liệu tham khảo

10.1037/0003-066X.49.1.15

10.1037/0278-6133.19.6.586

10.1037/0096-3445.136.4.569

10.1177/0956797611434537

10.1111/j.1467-9280.2009.02299.x

10.1177/0956797616647519

10.1177/0956797611429579

10.1177/1745691616652878

10.1038/s41562-017-0189-z

10.1146/annurev.psych.55.090902.141601

10.1002/9780470743386

10.1177/135910457000100301

10.1038/nrn3475

10.1126/science.aaf0918

10.1038/s41562-018-0399-z

10.3758/s13428-013-0365-7

10.1007/s11136-014-0726-4

10.1037/0022-3514.93.3.415

10.1037/0278-6133.27.2.268

Coppock A., Political Science Research Methods

10.3758/s13423-015-0913-5

10.1027/1864-9335/a000202

10.1002/bdm.586

10.3758/BRM.40.2.428

10.1371/journal.pone.0029081

10.1073/pnas.1516179112

10.1016/j.jesp.2015.10.012

10.1016/j.paid.2009.07.012

10.1027/1864-9335/a000202

10.1037/a0019106

Forsell E., Predicting replication outcomes in the Many Labs 2 study, 10.1016/j.joep.2018.10.009

10.1371/journal.pone.0109019

10.1037/0022-0167.51.1.115

10.1257/089533005775196732

10.1016/j.obhdp.2006.10.001

10.1126/science.aad7243

10.1037/0033-2909.117.1.21

10.1016/j.jrp.2014.06.003

Goldberg L. R., 1981, Review of personality and social psychology, 2, 141

10.1016/S0092-6566(03)00046-1

10.1037/a0015141

10.1037/a0013748

10.1037/h0076157

10.1177/1745691616652873

10.1016/0191-8869(94)90212-7

10.1111/j.1468-0017.2006.00297.x

10.1177/0956797614553121

10.1017/S0140525X0999152X

10.3758/s13428-015-0678-9

10.1002/(SICI)1099-0771(199806)11:2<107::AID-BDM292>3.0.CO;2-Y

10.1002/ejsp.2013

10.1037/a0015960

10.1177/0956797611430953

10.1027/1864-9335/a000186

10.1016/0022-1031(67)90034-0

10.1016/j.jesp.2016.04.002

10.1037/a0034462

10.1027/1864-9335/a000178

Klein R. A., 2014, Social Psychology, 45, 307, 10.1027/1864-9335/a000178

10.1111/1467-8284.00419

10.1007/s11098-004-4510-0

10.1017/xps.2014.7

10.1037/10019-000

10.1007/s11205-011-9783-z

10.1037/0033-295X.98.2.224

10.1037/1082-989X.2.1.3

10.1037/a0028085

10.1126/science.1245317

10.1037/0022-3514.83.5.1239

10.5281/zenodo.838685

10.1016/0022-1031(85)90020-4

10.1017/XPS.2015.19

10.2139/ssrn.1804189

10.1207/s15516709cog2605_4

10.1126/science.aab2374

10.1073/pnas.1708274114

10.1027/1864-9335/a000192

10.1037/1040-3590.19.3.281

10.1126/science.aac4716

10.1016/j.jesp.2009.03.009

10.1126/science.1180606

10.1037/0022-3514.95.2.293

10.1177/0146167201272002

10.1177/1073191113517260

10.1515/9781400876136

10.1037/0033-2909.86.3.638

10.1016/0022-1031(77)90049-X

Ross L., 1991, The person and the situation: Perspectives of social psychology

10.1111/1467-9280.00334

10.1186/1471-2288-8-79

10.1111/j.1467-6494.1993.tb00283.x

10.1177/0956797609359908

10.1177/0956797616653102

10.1027/1864-9335/a000202

10.1086/269239

10.3758/BF03197186

10.1177/0956797611417632

10.1177/0956797614567341

10.1037/0022-3514.37.10.1660

10.1177/1745691613514450

10.1177/0956797610392929

Tversky A., 1978, Cognition and Categorization, 1, 79

10.1126/science.7455683

10.1073/pnas.1521897113

10.1037/0022-3514.73.4.733

10.1007/978-1-4020-8569-7_5

10.18637/jss.v036.i03

10.1177/1745691612463078

10.1038/nclimate2093

10.1126/science.1130726