Many Analysts, One Data Set: Making Transparent How Variations in Analytic Choices Affect Results

Advances in Methods and Practices in Psychological Science - Tập 1 Số 3 - Trang 337-356 - 2018
Raphael Silberzahn1, Eric Luis Uhlmann2, Daniel P. Martin3, Pasquale Anselmi4, Frederik Aust5, Eli Awtrey6, Štěpán Bahník7, Feng Bai8, Colin Bannard9, Evelina Bonnier10, Rickard Carlsson11, Felix Cheung12, Garret Christensen13, Russ Clay14, Maureen A. Craig15, Anna Dalla Rosa4, Lammertjan Dam16, Mathew H. Evans17, Ismael Flores Cervantes18, Nathan M. Fong19, Monica Gamez-Djokic20, Andreas Glenz21, Shauna Gordon-McKeon22, Timothy Heaton23, Karin Hederos24, Moritz Heene25, Alicia Hofelich Mohr26, Fabia Högden5, Kent Ngan‐Cheung Hui27, Magnus Johannesson10, Jonathan Kalodimos28, Erikson Kaszubowski29, Deanna M. Kennedy30, Ryan Lei15, Thomas Lindsay26, Silvia Liverani31, Christopher R. Madan32, Daniel C. Molden33, Eric Molleman16, Richard D. Morey34, Laetitia B. Mulder16, B. R. Nijstad16, Nolan Pope35, Bryson Pope36, Jason M. Prenoveau37, Floor Rink16, Egidio Robusto4, Hadiya Roderique38, Anna Sandberg24, Elmar Schlüter39, Felix D. Schönbrodt25, Martin F. Sherman37, S. Amy Sommer40, Kristin Lee Sotak41, Seth M. Spain42, Christoph Spörlein43, Tom Stafford44, Luca Stefanutti4, Susanne Täuber16, Johannes Ullrich21, Michelangelo Vianello4, Eric‐Jan Wagenmakers45, Maciej Witkowiak46, Sangsuk Yoon19, Brian A. Nosek47,3
1Organisational Behaviour, University of Sussex Business School
2Organisational Behaviour Area, INSEAD Asia Campus
3Department of Psychology, University of Virginia
4Department of Philosophy, Sociology, Education and Applied Psychology, University of Padua
5Department of Psychology, University of Cologne
6Department of Management, University of Cincinnati
7Department of Management, Faculty of Business Administration, University of Economics, Prague
8Department of Management and Marketing, Hong Kong Polytechnic University
9Department of Psychology, University of Liverpool
10Department of Economics, Stockholm School of Economics
11Department of Psychology, Linnaeus University
12School of Public Health, University of Hong Kong
13Berkeley Institute for Data Science, University of California, Berkeley
14Department of Psychology, College of Staten Island, City University of New York
15Department of Psychology, New York University
16Faculty of Economics and Business, University of Groningen
17Division of Neuroscience and Experimental Psychology, University of Manchester
18Westat, Rockville, Maryland
19Department of Marketing and Supply Chain Management, Temple University
20Department of Management and Organizations, Kellogg School of Management, Northwestern University
21Department of Psychology, University of Zurich
22Washington, D. C.
23School of Mathematics and Statistics, University of Sheffield
24Swedish Institute for Social Research (SOFI), Stockholm University
25Department of Psychology, Ludwig-Maximilians-Universität München
26College of Liberal Arts, University of Minnesota
27School of Management, Xiamen University
28College of Business, Oregon State University
29Department of Psychology, Federal University of Santa Catarina
30School of Business, University of Washington Bothell
31School of Mathematical Sciences, Queen Mary University of London
32School of Psychology, University of Nottingham,
33Department of Psychology, Northwestern University
34School of Psychology, Cardiff University
35Department of Economics, University of Maryland
36Department of Economics, Brigham Young University
37Department of Psychology, Loyola University Maryland
38Rotman School of Management, University of Toronto
39Department of Social Sciences and Cultural Studies, Institute of Sociology, Justus Liebig University, Giessen
40United States Military Academy at West Point
41Department of Marketing and Management, SUNY Oswego
42John Molson School of Business, Concordia University
43Lehrstuhl für Soziologie, insb. Sozialstrukturanalyse, Otto-Friedrich-Universität Bamberg
44Department of Psychology, University of Sheffield
45Department of Psychological Methods, University of Amsterdam
46Poznań, Poland
47Center for Open Science, Charlottesville, Virginia

Tóm tắt

Twenty-nine teams involving 61 analysts used the same data set to address the same research question: whether soccer referees are more likely to give red cards to dark-skin-toned players than to light-skin-toned players. Analytic approaches varied widely across the teams, and the estimated effect sizes ranged from 0.89 to 2.93 ( Mdn = 1.31) in odds-ratio units. Twenty teams (69%) found a statistically significant positive effect, and 9 teams (31%) did not observe a significant relationship. Overall, the 29 different analyses used 21 unique combinations of covariates. Neither analysts’ prior beliefs about the effect of interest nor their level of expertise readily explained the variation in the outcomes of the analyses. Peer ratings of the quality of the analyses also did not account for the variability. These findings suggest that significant variation in the results of analyses of complex data may be difficult to avoid, even by experts with honest intentions. Crowdsourcing data analysis, a strategy in which numerous research teams are recruited to simultaneously investigate the same research question, makes transparent how defensible, yet subjective, analytic choices influence research results.

Từ khóa


Tài liệu tham khảo

10.1073/pnas.1414026112

10.1177/1745691612459060

10.1037/0022-3514.55.5.726

10.1002/9780470743386.ch7

10.3389/fnins.2012.00149

10.1016/j.neuroimage.2012.07.004

10.1037/0022-3514.83.6.1314

10.1177/0956797613504966

10.1016/j.jesp.2015.10.012

10.1001/jama.2014.9646

10.1037/0022-3514.54.1.74

10.1511/2014.111.460

10.1046/j.0956-7976.2003.psci_1478.x

10.3389/fncom.2012.00063

10.1073/pnas.1313476110

10.1287/mnsc.2014.1967

10.1027/1864-9335/a000178

10.1001/jama.2014.9647

10.1002/ejsp.214

10.1177/0146167202282010

10.1111/j.1540-5982.2008.00509.x

10.1257/aer.101.4.1410

10.1162/qjec.2010.125.4.1859

10.1177/1745691614549257

10.1111/0162-895X.00264

10.1177/0956797613480366

10.1037/a0033242

Simonsohn U., 2016, Specification curve: Descriptive and inferential statistics for all plausible specifications

10.1177/0956797611417632

10.1177/1745691616658637

Twine F. W., 1998, Racism in a racial democracy: The maintenance of White supremacy in Brazil

10.1097/SMJ.0b013e31817a7ee4

10.1177/1745691612463078

10.1037/0003-066X.61.7.726