Conducting Clinical Research Using Crowdsourced Convenience Samples
Tóm tắt
Crowdsourcing has had a dramatic impact on the speed and scale at which scientific research can be conducted. Clinical scientists have particularly benefited from readily available research study participants and streamlined recruiting and payment systems afforded by Amazon Mechanical Turk (MTurk), a popular labor market for crowdsourcing workers. MTurk has been used in this capacity for more than five years. The popularity and novelty of the platform have spurred numerous methodological investigations, making it the most studied nonprobability sample available to researchers. This article summarizes what is known about MTurk sample composition and data quality with an emphasis on findings relevant to clinical psychological research. It then addresses methodological issues with using MTurk—many of which are common to other nonprobability samples but unfamiliar to clinical science researchers—and suggests concrete steps to avoid these issues or minimize their impact.
Từ khóa
Tài liệu tham khảo
Allan NP, Norr AM, Macatee RJ, Gajewska A, Schmidt NB. 2015. Interactive effects of anxiety sensitivity and emotion regulation on anxiety symptoms.J. Psychopathol. Behav.37:663–72
Arditte KA, Çek D, Shaw AM, Timpano KR. 2015. The importance of assessing clinical phenomena in Mechanical Turk research.Psychol. Assess.doi:10.137/pas0000217
Barnhoorn JS, 2014, Behav. Res. Methods, 530, 1
Benoit K, Conway D, Lauderdale BE, Laver M, Mikhaylov S. 2015. Crowd-sourced text analysis: reproducible and agile reproduction of political data.Am. Polit. Sci. Rev.In press
Beramendi P, Duch RM, Matsuo A. 2014. When lab subjects meet real people: comparing different modes of experiments. Presented atAsian Polit. Methodol. Conf., Taipai
Bernal DR. 2014.Acculturation, acculturative stress, social status, and well-being among English language proficient immigrants.PhD Thesis, Univ. Miami, Coral Gables
Bui DC, Myerson J, Hale S. 2015. Age-related slowing in online samples.Psychol. Record.doi: 10.1007/s40732-015-0135-2
Carr A. 2014. An exploration of Mechanical Turk as a feasible recruitment platform for cancer survivors. Undergrad. honors thesis: Univ. Colo., Boulder
Casey L, Chandler J, Levine AS, Proctor A, Strolovitch D. 2015. Demographic characteristics of a large sample of US workers.Mathematica Policy Res. Rep.In press
Cavanagh TM. 2014. Cheating on online assessment tests: prevalence and impact on validity. PhD Thesis, Colo. State Univ., Ft. Collins
Cooper EA, Farid H. 2014. Does the Sun revolve around the Earth? A comparison between the general public and online survey respondents in basic scientific knowledge.Public Underst. Sci.doi: 10.1177/0963662514554354
Daly TM, Nataraajan R. 2015. Swapping bricks for clicks: crowdsourcing longitudinal data on Amazon Turk.J. Bus. Res.68:2603–9
Downes-Le Guin T, 2006, ESOMAR World Res. Conf. Panel Res.
Farrell AM, Grenier JH, Leiby J. 2014. Scoundrels or stars? Theory and evidence on the quality of workers in online labor markets. SSRN. doi: 10.2139/ssrn.2424718
Gates GJ. 2014.LGBT demographics: comparisons among population-based surveys.Williams Inst., UCLA Sch. Law
Gerber AS, 2012, Field Experiments: Design, Analysis, and Interpretation
Goldin G, Darlow A. 2013. TurkGate, Version 0.4.0.Software.http://www.gideongoldin.github.com/TurkGate/
Gootzeit JH. 2014.ACT process measures: specificity and incremental value. PhD Thesis, Univ. Iowa, Iowa City
Greenblatt JB. 2013.US residential consumer product information: validation of methods for post-stratification weighting of Amazon Mechanical Turk surveys.Lawrence Berkeley Natl. Lab., LBNL Pap. LBNL-6163E, Berkeley, CA
Greenblatt JB. 2014.US residential miscellaneous refrigeration products: results from Amazon Mechanical Turk surveys.Lawrence Berkeley Natl. Lab., LBNL Pap. LBNL-6537E, Berkeley, CA
Groves RM, 2004, Survey Errors and Survey Costs, 536
Hauser DJ, Schwarz N. 2015a. Attentive Turkers: MTurk participants perform better on online attention checks than do subject pool participants.Behav. Res. Methods.In press
Hawkins RXD. 2014. Conducting real-time multiplayer experiments on the web.Behav. Res. Methods.doi: 10.3758/s13428-014-0515-6
Higgins C, 2010, Proc. NAACL HLT 2010 Workshop on Creat. Speech and Language Data with Amazon's Mechanical Turk, Los Angeles, 89
Huff C, Tingley D. 2014.“Who are these people?”: evaluating the demographic characteristics and political preferences of MTurk survey respondents. Work. Pap., Dep. Gov., Harvard Univ.
Ipeirotis P. 2010.Demographics of Mechanical Turk. Work. Pap., Stern Sch. Bus., New York Univ.
Jiang L, Wagner C. 2014.Participation in micro-task crowdsourcing markets as work and leisure: the impact of motivation and micro-time structuring.Presented at Collect. Intell. Conf., June 10–12, MIT, Cambridge, MA
Kang R, Brown S, Dabbish L, Kiesler S. 2014.Privacy attitudes of Mechanical Turk workers and the US public.Presented at SOUPS Workshop on Privacy Personas and Segmentation, 10th, Menlo Park, CA, July 9–11, pp. 37–47. Berkeley, CA: USENIX Assoc.
Kim J. 2014.Does stigma against smokers really motivate cessation? A moderated mediation model on the effect of anti-smoking campaigns promoting smoker-related stigma on cessation intentions.Master's thesis, Univ. Wis., Milwaukee
Krische SD. 2014. Who is the average individual investor? Numerical skills and implications for accounting research. SSRN.http://www.dx.doi.org/10.2139/ssrn.2426570
Lang ASID, 2011, Code4Lib J., 15, 1
Lease M, Hullman J, Bigham JP, Bernstein M, Kim J, et al. 2013. Mechanical Turk is not anonymous. SSRN. doi: 10.2139/ssrn.2228728
Lebreton P, 2015, Proc. SPIE Conf. Hum. Vis. Electron. Imaging, 20th, San Francisco
Lynn BMD. 2014.Shared sense of purpose and well-being among veterans and non-veterans. PhD Thesis, North Carolina State Univ., Raleigh
Marietta G, Viola J, Ibekwe N, Claremon J, Gehlbach H. 2014.Improving relationships through virtual environments: how seeing the world through victims' eyes may prevent bullying. Work. Pap., Grad. Sch. Educ., Harvard Univ.
Mullinix K, Drukman J, Freese J. 2014.The generalizability of survey experiments.Inst. Policy Res., Work. Pap Ser., WP-14–19, Northwest. Univ., Evanston, IL
O'Connell BH, O'Shea D, Gallagher S. 2016. Enhancing social relationships through positive psychology activities: a randomised controlled trial.J. Posit. Psychol.11:149–62
Parent J, McKee LG, Rough JN, Forehand R. 2015. The association of parent mindfulness with parenting and youth psychopathology across three developmental stages.J. Abnorm. Child Psychol.doi: 10.1007/s10802-015-9978-x
Pearl RL, Puhl RM, Dovidio JF. 2014. Differential effects of weight bias experiences and internalization on exercise among women with overweight and obesity.J. Health Psychol.doi: 10.1177/1359105313520338
Peer E, Paolacci G, Chandler J, Mueller P. 2012. Selectively recruiting participants from Amazon Mechanical Turk using Qualtrics. SSRN.http://www.ssrn.com/abstract=092100631
Pew Res. Cent. 2010.Blacks upbeat about black progress, prospects: a year after Obama's election.http://www.pewsocialtrends.org/2010/01/12/blacks-upbeat-about-black-progress-prospects/
Reese ED, Veilleux JC. 2016. Relationships between craving beliefs and abstinence self-efficacy are mediated by smoking motives and moderated by nicotine dependence.Nicotine Tob. Res.18:48–55
Schnorf S, 2014, A comparison of six sample providers regarding online privacy benchmarks.
Shao W, 2015, Digit. Cult. Educ., 7
Sina S, Kraus S, Rosenfeld A. 2014. Using the crowd to generate content for scenario-based serious-games. arXiv:1402.5034 [cs.AI]
Suri S, 2011, AAAI Workshops, 25th, San Francisco, 61
Usinger T. 2014.Effect of internet administered mindfulness training on anxiety and sleep quality.Undergraduate thesis, Univ. Colo., Boulder
Voracek M, 2001, Dimensions of Internet Science, 91
Wong CM. 2015. 50 Percent of millennials believe gender is a spectrum, Fusion's Massive Millennial Poll finds.The Huffington Post.http://www.huffingtonpost.com/2015/02/05/fusion-millennial-poll-gender_n_6624200.html
Wymbs BT, Dawson AE. 2015. Screening Amazon's Mechanical Turk for adults with ADHD.J. Attention Disord.doi: 10.1177/1087054715597471