Conducting interactive experiments online
Tóm tắt
Online labor markets provide new opportunities for behavioral research, but conducting economic experiments online raises important methodological challenges. This particularly holds for interactive designs. In this paper, we provide a methodological discussion of the similarities and differences between interactive experiments conducted in the laboratory and online. To this end, we conduct a repeated public goods experiment with and without punishment using samples from the laboratory and the online platform Amazon Mechanical Turk. We chose to replicate this experiment because it is long and logistically complex. It therefore provides a good case study for discussing the methodological and practical challenges of online interactive experimentation. We find that basic behavioral patterns of cooperation and punishment in the laboratory are replicable online. The most important challenge of online interactive experiments is participant dropout. We discuss measures for reducing dropout and show that, for our case study, dropouts are exogenous to the experiment. We conclude that data quality for interactive experiments via the Internet is adequate and reliable, making online interactive experimentation a potentially valuable complement to laboratory studies.
Tài liệu tham khảo
Abeler, J., & Nosenzo, D. (2015). Self-selection into laboratory experiments: pro-social motives versus monetary incentives. Experimental Economics, 18(2), 195–214. doi:10.1007/s10683-014-9397-9.
Amir, O., Rand, D. G., & Gal, Y. K. (2012). Economic games on the internet: The effect of $1 stakes. PLoS ONE. doi:10.1371/journal.pone.0031461.
Anderhub, V., Muller, R., & Schmidt, C. (2001). Design and evaluation of an economic experiment via the internet. Journal of Economic Behavior & Organization, 46(2), 227–247. doi:10.1016/S0167-2681(01)00195-0.
Anderson, J., Burks, S. V., Carpenter, J., Gotte, L., Maurer, K., Nosenzo, D., et al. (2013). Self-selection and variations in the laboratory measurement of other-regarding preferences across subject pools: evidence from one college student and two adult samples. Experimental Economics, 16(2), 170–189. doi:10.1007/s10683-012-9327-7.
Behrend, T. S., Sharek, D. J., Meade, A. W., & Wiebe, E. N. (2011). The viability of crowdsourcing for survey research. Behavior Research Methods, 43(3), 800–813. doi:10.3758/s13428-011-0081-0.
Belot, M., Duch, R., & Miller, L. (2010). Who should be called to the lab? A comprehensive comparison of students and non-students in classic experimental games. University of Oxford, Nuffield College Discussion Papers.((2010-001)).
Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012). Evaluating online labor markets for experimental research: Amazon.com’s Mechanical Turk. Political Analysis, 20(3), 351–368. doi:10.1093/pan/mpr057.
Bock, O., Baetge, I., & Nicklisch, A. (2014). hroot: hamburg registration and organization online tool. European Economic Review, 71, 117–120. doi:10.1016/j.euroecorev.2014.07.003.
Buhrmester, M., Kwang, T., & Gosling, S. D. (2011). Amazon’s Mechanical Turk: A new source of inexpensive, yet high-quality, data? Perspectives on Psychological Science, 6(1), 3–5. doi:10.1177/1745691610393980.
Carpenter, J., & Seki, E. (2011). Do social preferences increase productivity? field experimental evidence from fishermen in toyama bay. Economic Inquiry, 49(2), 612–630. doi:10.1111/j.1465-7295.2009.00268.x.
Chandler, J., Mueller, P., & Paolacci, G. (2014). Nonnaïveté among Amazon Mechanical Turk workers: Consequences and solutions for behavioral researchers. Behavior Research Methods, 46(1), 112–130. doi:10.3758/s13428-013-0365-7.
Chandler, J., Paolacci, G., Peer, E., Mueller, P., & Ratliff, K. A. (2015). Using nonnaive participants can reduce effect sizes. Psychological Science, 26(7), 1131–1139.
Chesney, T., Chuah, S. H., & Hoffmann, R. (2009). Virtual world experimentation: An exploratory study. Journal of Economic Behavior & Organization, 72(1), 618–635. doi:10.1016/j.jebo.2009.05.026.
Cleave, B. L., Nikiforakis, N., & Slonim, R. (2013). Is there selection bias in laboratory experiments? The case of social and risk preferences. Experimental Economics, 16(3), 372–382. doi:10.1007/s10683-012-9342-8.
Difallah, D. E., Catasta, M., Demartini, G., Ipeirotis, P. G., & Cudre-Mauroux, P. (2015). The Dynamics of Micro-Task Crowdsourcing The Case of Amazon MTurk. Www’15 Companion: Proceedings of the 24th International Conference on World Wide Web, 617–617, doi:10.1145/2740908.2744109.
Egas, M., & Riedl, A. (2008). The economics of altruistic punishment and the maintenance of cooperation. Proceedings of the Royal Society B-Biological Sciences, 275(1637), 871–878. doi:10.1098/rspb.2007.1558.
Fehr, E., & Gächter, S. (2000). Cooperation and punishment in public goods experiments. American Economic Review, 90(4), 980–994. doi:10.1257/aer.90.4.980.
Fehr, E., & Gächter, S. (2002). Altruistic punishment in humans. Nature, 415(6868), 137–140.
Fischbacher, U. (2007). z-Tree: Zurich toolbox for ready-made economic experiments. Experimental Economics, 10(2), 171–178. doi:10.1007/s10683-006-9159-4.
Gächter, S., & Herrmann, B. (2011). The limits of self-governance when cooperators get punished: Experimental evidence from urban and rural Russia. European Economic Review, 55(2), 193–210. doi:10.1016/j.euroecorev.2010.04.003.
Gächter, S., Herrmann, B., & Thoni, C. (2004). Trust, voluntary cooperation, and socio-economic background: survey and experimental evidence. Journal of Economic Behavior & Organization, 55(4), 505–531. doi:10.1016/j.jebo.2003.11.006.
Gallo, E., & Yan, C. (2015). The effects of reputational and social knowledge on cooperation. Proceedings of the National Academy of Sciences of the United States of America, 112(12), 3647–3652. doi:10.1073/pnas.1415883112.
Goodman, J. K., Cryder, C. E., & Cheema, A. (2013). Data collection in a flat world: The strengths and weaknesses of mechanical turk samples. Journal of Behavioral Decision Making, 26(3), 213–224. doi:10.1002/bdm.1753.
Greiner, B. (2015). Subject pool recruitment procedures: organizing experiments with ORSEE. Journal of the Economic Science Association, 1(1), 114–125. doi:10.1007/s40881-015-0004-4.
Guillen, P., & Veszteg, R. F. (2012). On “lab rats”. Journal of Socio-Economics, 41(5), 714–720. doi:10.1016/j.socec.2012.07.002.
Haigh, M. (2016). Has the standard cognitive reflection test become a victim of its own success? Advances in Cognitive Psychology, 12(3), 145–149. doi:10.5709/acp-0193-5.
Hauser, O. P., Hendriks, A., Rand, D. G., & Nowak, M. A. (2016). Think global, act local: Preserving the global commons. [Article]. Scientific reports, 6, 36079, doi:10.1038/srep36079.http://www.nature.com/articles/srep36079#supplementary-information.
Hergueux, J., & Jacquemet, N. (2015). Social preferences in the online laboratory: A randomized experiment. Experimental Economics, 18(2), 251–283. doi:10.1007/s10683-014-9400-5.
Herrmann, B., Thoni, C., & Gächter, S. (2008). Antisocial punishment across societies. Science, 319(5868), 1362–1367.
Horton, J. J., Rand, D. G., & Zeckhauser, R. J. (2011). The online laboratory: conducting experiments in a real labor market. Experimental Economics, 14(3), 399–425. doi:10.1007/s10683-011-9273-9.
Ipeirotis, P. G. (2010). Analyzing the amazon mechanical turk marketplace. XRDS: Crossroads, The ACM Magazine for Students, 17(2), 16–21.
Jenkins, S. P. (1995). Easy estimation methods for discrete-time duration models. Oxford Bulletin of Economics and Statistics, 57(1), 129–138. doi:10.1111/j.1468-0084.1995.tb00031.x.
Klein, R. A., Ratliff, K. A., Vianello, M., Adams, R. B., Bahnik, S., Bernstein, M. J., et al. (2014). Investigating variation in replicability a “many labs’’ replication project. Social Psychology, 45(3), 142–152. doi:10.1027/1864-9335/a000178.
Krawczyk, M. (2011). What brings your subjects to the lab? A field experiment. Experimental Economics, 14(4), 482–489. doi:10.1007/s10683-011-9277-5.
Krupnikov, Y., & Levine, A. S. (2014). Cross-sample comparisons and external validity. Journal of Experimental Political Science, 1(1), 59.
List, J. A. (2004). Young, selfish and male: Field evidence of social preferences. Economic Journal, 114(492), 121–149. doi:10.1046/j.0013-0133.2003.00180.x.
Mullinix, K. J., Leeper, T. J., Druckman, J. N., & Freese, J. (2015). The generalizability of survey experiments. Journal of Experimental Political Science, 2(02), 109–138. doi:10.1017/XPS.2015.19.
Nishi, A., Shirado, H., Rand, D. G., & Christakis, N. A. (2015). Inequality and visibility of wealth in experimental social networks. Nature, 526(7573), 426. doi:10.1038/nature15392.
Paolacci, G., & Chandler, J. (2014). Inside the turk: Understanding mechanical turk as a participant pool. Current Directions in Psychological Science, 23(3), 184–188. doi:10.1177/0963721414531598.
Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010). Running experiments on Amazon Mechanical Turk. Judgment and Decision Making, 5(5), 411–419.
Peer, E., Brandimarte, L., Samat, S., & Acquisti, A. (2017). Beyond the Turk: Alternative platforms for crowdsourcing behavioral research. Journal of Experimental Social Psychology, 70, 153–163. doi:10.1016/j.jesp.2017.01.006.
Peer, E., Vosgerau, J., & Acquisti, A. (2014). Reputation as a sufficient condition for data quality on Amazon Mechanical Turk. Behavior Research Methods, 46(4), 1023–1031. doi:10.3758/s13428-013-0434-y.
Pfattheicher, S., & Schindler, S. (2015). Understanding the dark side of costly punishment: The impact of individual differences in everyday sadism and existential threat. European Journal of Personality, 29(4), 498–505. doi:10.1002/per.2003.
Rand, D. G. (2012). The promise of Mechanical Turk: How online labor markets can help theorists run behavioral experiments. Journal of Theoretical Biology, 299, 172–179. doi:10.1016/j.jtbi.2011.03.004.
Rand, D. G., Peysakhovich, A., Kraft-Todd, G. T., Newman, G. E., Wurzbacher, O., Nowak, M. A., et al. (2014). Social heuristics shape intuitive cooperation. Nature Communications. doi:10.1038/ncomms4677.
Reips, U.-D. (2000). The Web experiment method: Advantages, disadvantages, and solutions. In M. H. Birnbaum (Ed.), Psychological experiments on the internet (pp. 89–117). San Diego, CA: Academic Press.
Schmelz, K., & Ziegelmeyer, A. (2015). Social distance and control aversion: Evidence from the Internet and the laboratory. Research Paper Series Thurgau Institute of Economics and Department of Economics at the University of Konstanz, TWI-RPS 100.
Stagnaro, M. N., Arechar, A. A., & Rand, D. G. (2017). From good institutions to generous citizens: Top-down incentives to cooperate promote subsequent prosociality but not norm enforcement. Cognition. doi:10.1016/j.cognition.2017.01.017.
Stewart, N., Ungemach, C., Harris, A. J. L., Bartels, D. M., Newell, B. R., Paolacci, G., et al. (2015). The average laboratory samples a population of 7,300 Amazon Mechanical Turk workers. Judgment and Decision Making, 10(5), 479–491.
Stoop, J., Noussair, C. N., & Van Soest, D. (2012). From the lab to the field: Cooperation among fishermen. Journal of Political Economy, 120(6), 1027–1056. doi:10.1086/669253.
Suri, S., & Watts, D. J. (2011). Cooperation and contagion in web-based, networked public goods experiments. PLoS ONE. doi:10.1371/journal.pone.0016836.
Wang, J., Suri, S., & Watts, D. J. (2012). Cooperation and assortativity with dynamic partner updating. Proceedings of the National Academy of Sciences of the United States of America, 109(36), 14363–14368. doi:10.1073/pnas.1120867109.
Weinberg, J. D., Freese, J., & McElhattan, D. (2014). Comparing data characteristics and results of an online factorial survey between a population-based and a crowdsource-recruited sample. Sociological Science, 1, 292–310.
Zhou, H. T., & Fishbach, A. (2016). The pitfall of experimenting on the web: How unattended selective attrition leads to surprising (Yet False) research conclusions. Journal of Personality and Social Psychology, 111(4), 493–504. doi:10.1037/pspa0000056.