Reproducibility in Human-Robot Interaction: Furthering the Science of HRI
Tóm tắt
To discuss the current state of reproducibility of research in human-robot interaction (HRI), challenges specific to the field, and recommendations for how the community can support reproducibility. As in related fields such as artificial intelligence, robotics, and psychology, improving research reproducibility is key to the maturation of the body of scientific knowledge in the field of HRI. The ACM/IEEE International Conference on Human-Robot Interaction introduced a theme on Reproducibility of HRI to their technical program in 2020 to solicit papers presenting reproductions of prior research or artifacts supporting research reproducibility. This review provides an introduction to the topic of research reproducibility for HRI and describes the state of the art in relation to the HRI 2020 Reproducibility theme. As a highly interdisciplinary field that involves work with technological artifacts, there are unique challenges to reproducibility in HRI. Biases in research evaluation and practice contribute to challenges in supporting reproducibility, and the training of researchers could be changed to encourage research reproduction. The authors propose a number of solutions for addressing these challenges that can serve as guidelines for the HRI community and related fields.
Tài liệu tham khảo
Sprunk C, Roewekaemper J, Parent G, Spinello L, Tipaldi GD, Burgard W, et al. An experimental protocol for benchmarking robotic indoor navigation. In: Hsieh MA, Khatib O, Kumar V, editors. Experimental robotics. vol. 109 of Springer Tracts in Advanced Robotics. Springer International Publishing; 2015. p. 487-504.
Bonsignorio F, del Pobil AP. Toward replicable and measurable robotics research [from the guest editors]. IEEE Robot Autom Mag. 2015;22(3):32–5.
Henrich J, Heine SJ, Norenzayan A. The weirdest people in the world? Behav Brain Sci. 2010;33(2–3):61–83.
Lee HR, Sabanović S. Culturally variable preferences for robot design and use in South Korea, Turkey, and the United States. In: Proceedings of the 2014 ACM/IEEE International Conference on Human-Robot Interaction. HRI ’14. New York, NY, USA: Association for Computing Machinery; 2014. p. 17–24. Available from: https://doi.org/10.1145/2559636.2559676.
Pereira A, Oertel C, Fermoselle L, Mendelson J, Gustafson J. Effects of different interaction contexts when evaluating gaze models in HRI. In: Proceedings of the 2020 ACM/IEEE International Conference on Human-Robot Interaction. HRI ’20. New York, NY, USA: Association for Computing Machinery; 2020. p. 131–139. Available from: https://doi.org/10.1145/3319502.3374810.
Strait M, Urry HL, Muentener P. Children’s responding to humanlike agents reflects an uncanny valley. In: 2019 14th ACM/IEEE International Conference on Human-Robot Interaction (HRI); 2019. p. 506-15.
Lee HR, Tan H, Šabanović S. That robot is not for me: addressing stereotypes of aging in assistive robot design. In: 2016 25th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN); 2016. p. 312-7.
Strait M, Vujovic L, Floerke V, Scheutz M, Urry H. Too much humanness for human-robot interaction: exposure to highly humanlike robots elicits aversive responding in observers. In: Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. CHI ’15. New York, NY, USA: Association for Computing Machinery; 2015. p. 3593–3602. Available from: https://doi.org/10.1145/2702123.2702415.
Institute of Education Sciences (IES), US Department of Education & National Science Foundation (NSF). Companion guidelines on replication reproducibility in education research: a supplement to the common guidelines for education research and development; 2018. https://www.nsf.gov/pubs/2019/nsf19022/nsf19022.pdf. Accessed 1 June 2022.
Strait M, Lier F, Bernotat J, Wachsmuth S, Eyssel F, Goldstone R, et al. A three-site reproduction of the joint Simon effect with the NAO robot. In: Proceedings of the 2020 ACM/IEEE International Conference on Human-Robot Interaction. HRI ’20. New York, NY, USA: Association for Computing Machinery; 2020. p. 103–111. Available from: https://doi.org/10.1145/3319502.3374783.
Sandygulova A, Johal W, Zhexenova Z, Tleubayev B, Zhanatkyzy A, Turarova A, et al. CoWriting Kazakh: learning a new script with a robot. In: Proceedings of the 2020 ACM/IEEE International Conference on Human-Robot Interaction. HRI ’20. New York, NY, USA: Association for Computing Machinery; 2020. p. 113–120. Available from: https://doi.org/10.1145/3319502.3374813.
Kubota A, Peterson EIC, Rajendren V, Kress-Gazit H, Riek LD. JESSIE: synthesizing social robot behaviors for personalized neurorehabilitation and beyond. In: Proceedings of the 2020 ACM/IEEE International Conference on Human-Robot Interaction. HRI ’20. New York, NY, USA: Association for Computing Machinery; 2020. p. 121–130. Available from: https://doi.org/10.1145/3319502.3374836.
Li J, Currano R, Sirkin D, Goedicke D, Tennent H, Levine A, et al. On-road and online studies to investigate beliefs and behaviors of Netherlands, US and Mexico pedestrians encountering hidden-driver vehicles. In: Proceedings of the 2020 ACM/IEEE International Conference on Human-Robot Interaction. HRI ’20. New York, NY, USA: Association for Computing Machinery; 2020. p. 141–149. Available from: https://doi.org/10.1145/3319502.3374790.
Stange S, Kopp S. Effects of a social robot’s self-explanations on how humans understand and evaluate its behavior. In: Proceedings of the 2020 ACM/IEEE International Conference on Human-Robot Interaction. HRI ’20. New York, NY, USA: Association for Computing Machinery; 2020. p. 619–627. Available from: https://doi.org/10.1145/3319502.3374802.
Walker N, Weatherwax K, Allchin J, Takayama L, Cakmak M. Human perceptions of a curious robot that performs off-task actions. In: Proceedings of the 2020 ACM/IEEE International Conference on Human-Robot Interaction. HRI ’20. New York, NY, USA: Association for Computing Machinery; 2020. p. 529–538. Available from: https://doi.org/10.1145/3319502.3374821.
Nanavati A, Doering M, Brščić D, Kanda T. Autonomously learning one-to-many social interaction logic from human-human interaction data. In: Proceedings of the 2020 ACM/IEEE International Conference on Human-Robot Interaction. HRI ’20. New York, NY, USA: Association for Computing Machinery; 2020. p. 419–427. Available from: https://doi.org/10.1145/3319502.3374798.
Suguitan M, Hoffman G. Blossom: a handcrafted open-source robot. ACM Transactions on Human-Robot Interaction (THRI). 2019;8(1):1–27.
• Cervera E. Try to start it! The challenge of reusing code in robotics research. IEEE Robotics and Automation Letters. 2019;4(1):49-56. This article analyzes source code provided with publications at a large recent robotics conference (ICRA 2017) and highlights numerous practical obstacles to research software reuse.
Quigley M, Gerkey B, Conley K, Faust J, Foote T, Leibs J, Berger E, Wheeler R, Ng A. ROS: an open-source robot operating system. In: Workshop on Open Source Software (Collocated with ICRA 2009), Kobe, Japan, May 2009.
Gerkey B, Vaughan RT, Howard A. The player/stage project: tools for multi-robot and distributed sensor systems. In: Proceedings of the 11th international conference on advanced robotics. vol. 1; 2003. p. 317-23.
Lier F, Hanheide M, Natale L, Schulz S, Weisz J, Wachsmuth S, et al. Towards automated system and experiment reproduction in robotics. In: 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE; 2016. p. 3298-305.
Wagner J, Lingenfelser F, Baur T, Damian I, Kistler F, André E. The Social Signal Interpretation (SSI) framework: multimodal signal processing and recognition in real-time. In: Proceedings of the 21st ACM international conference on Multimedia; 2013. p. 831-4.
Skantze G, Al Moubayed S. IrisTK: a statechart-based toolkit for multi-party face-to-face interaction. In: Proceedings of the 14th ACM international conference on Multimodal interaction; 2012. p. 69-76.
Bohus D, Andrist S, Jalobeanu M. Rapid development of multimodal interactive systems: a demonstration of platform for situated intelligence. In: Proceedings of the 19th ACM International Conference on Multimodal Interaction; 2017. p. 493-4.
• Cockburn A, Dragicevic P, Besançon L, Gutwin C. Threats of a replication crisis in empirical computer science. Commun ACM. 2020 Jul;63(8):70–79. This article reviews the use of statistical significance in the discipline of computer science and discusses practices which make the field vulnerable to replication crisis. Available from: https://doi.org/10.1145/3360311.
Gelman A, Loken E. The statistical crisis in science: data-dependent analysis-a “garden of forking paths’’-explains why many statistically significant comparisons don’t hold up. Am Sci. 2014;102(6):460–6.
Baxter P, Kennedy J, Senft E, Lemaignan S, Belpaeme T. From characterising three years of HRI to methodology and reporting recommendations. In: 2016 11th ACM/IEEE International Conference on Human-Robot Interaction (HRI). IEEE; 2016. p. 391-8.
Elson M, Huff M, Utz S. Metascience on peer review: testing the effects of a study’s originality and statistical significance in a field experiment. Adv Methods Pract Psychol Sci. 2020;3(1):53–65.
Feil-Seifer D, Haring KS, Rossi S, Wagner AR, Williams T. Where to next? The impact of COVID-19 on human-robot interaction research. J Hum-Robot Interact. 2020 Jun;10(1). Available from: https://doi.org/10.1145/3405450.
Nielsen M, Haun D, Kärtner J, Legare CH. The persistent sampling bias in developmental psychology: a call to action. J Exp Child Psychol. 2017;162:31–8.
Strait M, Ramos AS, Contreras V, Garcia N, Robots racialized in the likeness of marginalized social identities are subject to greater dehumanization than those racialized as white. In, . 27th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN). IEEE. 2018;2018:452–7.
Riek LD, Howard D. A code of ethics for the human-robot interaction profession. In: Proc. We Robot, Apr. 2014, pp. 1–10.
Aguinis H, Solarino AM. Transparency and replicability in qualitative research: the case of interviews with elite informants. Strateg Manag J. 2019;40(8):1291-315. Available from: https://onlinelibrary.wiley.com/doi/abs/10.1002/smj.3015.