Data Collection in a Flat World: The Strengths and Weaknesses of Mechanical Turk Samples

Journal of Behavioral Decision Making - Tập 26 Số 3 - Trang 213-224 - 2013
Joseph K. Goodman1, Cynthia Cryder1, Amar Cheema2
1Olin Business School Washington University in St. Louis Saint Louis MO USA
2McIntire School of Commerce, University of Virginia, Charlottesville, VA, USA

Tóm tắt

ABSTRACTMechanical Turk (MTurk), an online labor system run by Amazon.com, provides quick, easy, and inexpensive access to online research participants. As use of MTurk has grown, so have questions from behavioral researchers about its participants, reliability, and low compensation. In this article, we review recent research about MTurk and compare MTurk participants with community and student samples on a set of personality dimensions and classic decision‐making biases. Across two studies, we find many similarities between MTurk participants and traditional samples, but we also find important differences. For instance, MTurk participants are less likely to pay attention to experimental materials, reducing statistical power. They are more likely to use the Internet to find answers, even with no incentive for correct responses. MTurk participants have attitudes about money that are different from a community sample's attitudes but similar to students' attitudes. Finally, MTurk participants are less extraverted and have lower self‐esteem than other participants, presenting challenges for some research domains. Despite these differences, MTurk participants produce reliable results consistent with standard decision‐making biases: they are present biased, risk‐averse for gains, risk‐seeking for losses, show delay/expedite asymmetries, and show the certainty effect—with almost no significant differences in effect sizes from other samples. We conclude that MTurk offers a highly valuable opportunity for data collection and recommend that researchers using MTurk (1) include screening questions that gauge attention and language comprehension; (2) avoid questions with factual answers; and (3) consider how individual differences in financial and social domains may influence results. Copyright © 2012 John Wiley & Sons, Ltd.

Từ khóa


Tài liệu tham khảo

10.1002/bdm.443

10.1177/1745691610393980

10.1002/(SICI)1099-0771(200001/03)13:1<61::AID-BDM343>3.0.CO;2-J

Cryder C. &Loewenstein G.(2010).The time versus money scale. Unpublished data Olin Business School Washington University in St. Louis.

Eriksson K., 2010, Emotional reactions to losing explain gender differences in entering a risky lottery, Judgment and Decision Making, 5, 159, 10.1017/S1930297500001030

10.1257/089533005775196732

Friedman T. L., 2005, The world is flat: A brief history of the twenty‐first century

10.1016/S0092-6566(03)00046-1

10.1111/j.0956-7976.2004.00715.x

10.1007/s10683-011-9273-9

Ipeirotis P.(2010).Demographics of Mechanical Turk. (CeDER Working Paper‐10‐01). New York University. Retrieved fromhttp://hdl.handle.net/2451/29585.

John O. P., 1999, Handbook of personality: Theory and research, 102

10.2307/1914185

10.1002/bdm.433

10.1287/mnsc.34.2.200

10.1086/208919

10.1509/jmkr.43.4.618

10.3758/s13428-011-0124-6

10.1509/jmkr.45.6.633

Nenkov G., 2008, A short form of the maximization scale: Factor structure, reliability, and validity studies, Judgment and Decision Making, 3, 371, 10.1017/S1930297500000395

10.1016/j.jesp.2009.03.009

10.1016/j.cognition.2006.12.008

10.1017/S1930297500002205

10.1002/bdm.481

10.1086/323732

10.1016/j.jtbi.2011.03.004

10.1086/383436

10.1086/523285

10.1177/0146167201272002

Rosenthal R., 2000, Contrasts and effect sizes in behavioral research: A correlational approach

10.1177/0146167211398138

10.1371/journal.pone.0016836

10.1016/0165-1765(81)90067-7

10.1037/0022-3514.84.1.205

Tversky A., 1974, Judgment under uncertainty: Heuristics and biases, Science, 211, 453

10.1126/science.7455683

10.1007/BF00122574

10.1177/0092070399273005

Thông tin
Thông tin xuất bản

Journal of Behavioral Decision Making

Tập 26 Số 3

213-224

Thông tin tác giả