On multi-human multi-robot remote interaction: a study of transparency, inter-human communication, and information loss in remote interaction

Admoni, H., & Scassellati, B. (2017). Social eye gaze in human–robot interaction: A review. Journal of Human-Robot Interaction, 6(1), 25–63. Ayanian, N., Spielberg, A., Arbesfeld, M., Strauss, J., & Rus, D. (2014). Controlling a team of robots with a single input. In 2014 IEEE International Conference on Robotics and Automation (ICRA) (pp. 1755–1762). IEEE. http://ieeexplore.ieee.org/abstract/document/6907088/ Bhaskara, A., Skinner, M., & Loft, S. (2020). Agent transparency: A review of current theory and evidence. IEEE Transactions on Human-Machine Systems. https://doi.org/10.1109/THMS.2020.2965529. Black, D. (1976). Partial justification of the borda count. Public Choice, 28, 1–15. Breazeal, C., Kidd, C. D., Thomaz, A. L., Hoffman, G., & Berlin, M. (2005). Effects of nonverbal communication on efficiency and robustness in human–robot teamwork. In 2005 IEEE/RSJ international conference on intelligent robots and systems (pp. 708–713). IEEE. Capelli, B., Secchi, C., & Sabattini, L. (2019). Communication through motion: Legibility of multi-robot systems. In 2019 international symposium on Multi-Robot and Multi-Agent Systems (MRS) (pp. 126–132). IEEE. https://doi.org/10.1109/MRS.2019.8901100 Chakraborti, T., Kulkarni, A., Sreedharan, S., Smith, D. E., & Kambhampati, S. (2019). Explicability? Legibility? Predictability? Transparency? Privacy? Security? The emerging landscape of interpretable agent behavior. In Proceedings of the international conference on automated planning and scheduling, Vol. 29, pp. 86–96. Che, Y., Okamura, A. M., & Sadigh, D. (2020). Efficient and trustworthy social navigation via explicit and implicit robot–human communication. IEEE Transactions on Robotics, 36, 692–707. Chen, J.Y., Procci, K., Boyce, M., Wright, J., Garcia, A., & Barnes, M. (2014). Situation awareness-based agent transparency. Tech. rep., Defense Technical Information Center, Fort Belvoir, VA. https://doi.org/10.21236/ADA600351 Chen, J. Y. C., Haas, E. C., & Barnes, M. J. (2007). Human performance issues and user interface design for teleoperated robots. IEEE Transactions on Systems, Man and Cybernetics, Part C (Applications and Reviews), 37(6), 1231–1245. https://doi.org/10.1109/TSMCC.2007.905819. Chen, J. Y. C., Lakhmani, S. G., Stowers, K., Selkowitz, A. R., Wright, J. L., & Barnes, M. (2018). Situation awareness-based agent transparency and human-autonomy teaming effectiveness. Theoretical Issues in Ergonomics Science, 19(3), 259–282. Cheung, Y., & Chung, J. H. (2011). Semi-autonomous control of single-master multi-slave teleoperation of heterogeneous robots for multi-task multi-target pairing. International Journal of Control and Automation, 4(3), 17. Daily, M., Cho, Y., Martin, K., & Payton, D. (2003). World embedded interfaces for human–robot interaction. In Proceedings of the 36th annual Hawaii International Conference on System Sciences, 2003 (p. 6). IEEE. Dardona, T., Eslamian, S., Reisner, L. A., & Pandya, A. (2019). Remote presence: Development and usability evaluation of a head-mounted display for camera control on the da Vinci surgical system. Robotics, 8(2), 31. Dimitoglou, G. (2019). Telepresence: Evaluation of robot stand-ins for remote student learning. Journal of Computing Sciences in Colleges, 35, 15. Do, N. D., Yamashina, Y., & Namerikawa, T. (2011). Multiple cooperative bilateral teleoperation with time-varying delay. SICE Journal of Control, Measurement, and System Integration, 4(2), 89–96. Dong, G. L., Gun, R. C., Min, S. L., Kim, B.-S., Oh, S., & Son H. I. (2013). Human-centered evaluation of multi-user teleoperation for mobile manipulator in unmanned offshore plants. In 2013 IEEE/RSJ international conference on intelligent robots and systems (pp. 5431–5438). IEEE. https://doi.org/10.1109/IROS.2013.6697142 Ellis, S. R., Mania, K., Adelstein, B. D., & Hill, M. I. (2004). Generalizeability of latency detection in a variety of virtual environments. In Proceedings of the human factors and ergonomics society annual meeting (Vol. 48, pp. 2632–2636). SAGE Publications. Endsley, M. R., & Kiris, E. O. (1995). The out-of-the-loop performance problem and level of control in automation. Human Factors, 37(2), 381–394. Esfahlani, S. S. (2019). Mixed reality and remote sensing application of unmanned aerial vehicle in fire and smoke detection. Journal of Industrial Information Integration. https://doi.org/10.1016/j.jii.2019.04.006. Ferreira, L. R. N., & Pereira, L. T. (2020). Immersive mobile telepresence systems: A systematic literature review. Journal of Mobile Multimedia, 15, 16. Feth, D., Tran, B. A., Groten, R., Peer, A., & Buss, M. (2009). Shared-control paradigms in multi-operator-single-robot teleoperation. In R. Dillmann, D. Vernon, Y. Nakamura, S. Schaal, H. Ritter, G. Sagerer, R. Dillmann, M. Buss (Eds.), Human centered robot systems. Series Title: Cognitive Systems Monographs (Vol. 6, pp. 53–62). Springer. https://doi.org/10.1007/978-3-642-10403-9_6 Friedman, M. (1937). The use of ranks to avoid the assumption of normality implicit in the analysis of variance. Journal of the American Statistical Association, 32(200), 675–701. Ghiringhelli, F., Guzzi, J., Di Caro, G. A., Caglioti, V., Gambardella, L. M., & Giusti, A. (2014). Interactive augmented reality for understanding and analyzing multi-robot systems. In 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2014) (pp. 1195–1201). IEEE. Goertz, R. C., & Thompson, W. M. (1954). Electronically controlled manipulator (Vol. 12). Nucleonics (US) Ceased Publication. Gouraud, J., Delorme, A., & Berberian, B. (2017). Autopilot, mind wandering, and the out of the loop performance problem. Frontiers in Neuroscience, 11, 541. Hart, S. G., & Staveland, L. E. (1988). Development of nasa-tlx (task load index): Results of empirical and theoretical research. In Advances in psychology (Vol. 52, pp. 139–183). Elsevier. Hokayem, P. F., & Spong, M. W. (2006). Bilateral teleoperation: An historical survey. Automatica, 42(12), 2035–2057. Holdcroft, D. (1976). Forms of indirect communication: An outline. Philosophy& Rhetoric, 9, 147–161. Hong, A., Bulthoff, H. H., & Son, H. I. (2013). A visual and force feedback for multi-robot teleoperation in outdoor environments: A preliminary result. In 2013 IEEE International Conference on Robotics and Automation (pp. 1471–1478). IEEE. https://doi.org/10.1109/ICRA.2013.6630765 Jingtai, L., Sun, L., Chen, T., Huang, X., & Zhao, C. (2005). Competitive multi-robot teleoperation. In Proceedings of the 2005 IEEE International Conference on Robotics and Automation (pp. 75–80). IEEE. https://doi.org/10.1109/ROBOT.2005.1570099 Jung, H., & Song, Y. E. (2018). Robotic remote control based on human motion via virtual collaboration system: A survey. Journal of Advanced Mechanical Design, Systems, and Manufacturing, 12(7), JAMDSM0126. https://doi.org/10.1299/jamdsm.2018jamdsm0126. Kim, J., Oh, S., & Kim, D. (2012). Implementation of a work distribution function for tele-operation under multi-user and multi-robot environments. In 2012 12th international conference on control, automation and systems p. 4. Kim, J., Yun, S., Yoo, J., & Kim, D. (2013). Implementation of N:1/1:N work distribution function for tele-operation under multi-user and multi-robot environments. In 2013 10th international conference on Ubiquitous Robots and Ambient Intelligence (URAI) (pp. 328–330). IEEE. https://doi.org/10.1109/URAI.2013.6677378 Kolling, A., Sycara, K., Nunnally, S., & Lewis, M. (2013). Human swarm interaction: An experimental study of two types of interaction with foraging swarms. Journal of Human-Robot Interaction. https://doi.org/10.5898/JHRI.2.2.Kolling. Lakhmani, S. G., Wright, J. L., Schwartz, M. R., & Barber, D. (2019). Exploring the effect of communication patterns and transparency on performance in a human–robot team. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 63(1), 160–164. Lane, J. C., Carignan, C. R., Sullivan, B. R., Akin, D. L., Hunt, T., & Cohen, R. (2002). Effects of time delay on telerobotic control of neutral buoyancy vehicles. In Proceedings 2002 IEEE international conference on robotics and automation (Cat. No. 02CH37292) (Vol. 3, pp. 2874–2879). IEEE. Lee, D., Franchi, A., Son, H. I., Ha, C., Bulthoff, H. H., & Giordano, P. R. (2013). Semiautonomous haptic teleoperation control architecture of multiple unmanned aerial vehicles. IEEE/ASME Transactions on Mechatronics, 18(4), 1334–1345. Li, H., Zhang, L., & Kawashima, K. (2018). Operator dynamics for stability condition in haptic and teleoperation system: A survey. The International Journal of Medical Robotics and Computer Assisted Surgery, 14(2), e1881. Lichiardopol, S. (2007). A survey on teleoperation. Technische Universitat Eindhoven, DCT report, Vol. 20, p. 34. Likert, R. (1932). A technique for the measurement of attitudes. Archives of Psychology, 140, 55. Liu, Z., Wu, M., Cao, W., Chen, L., Xu, J., Zhang, R., et al. (2017). A facial expression emotion recognition based human–robot interaction system. IEEE/CAA Journal of Automatica Sinica, 4, 668–676. Lunghi, G., Marin, R., Di Castro, M., Masi, A., & Sanz, P. J. (2019). Multimodal human–robot interface for accessible remote robotic interventions in hazardous environments. IEEE Access, 7, 127290–127319. Lyons, J. B. (2013). Being transparent about transparency: A model for human–robot interaction. In 2013 AAAI Spring Symposium Series. Ma, L., Yan, J., Zhao, J., Chen, Z., & Cai, H. (2010). Teleoperation system of Internet-based multi-operator multi-mobile-manipulator. In 2010 International Conference on Electrical and Control Engineering (pp. 2236–2240). IEEE. https://doi.org/10.1109/iCECE.2010.551 MacKenzie, I. S., & Ware, C. (1993). Lag as a determinant of human performance in interactive systems. In Proceedings of the INTERACT’93 and CHI’93 conference on Human factors in computing systems, pp. 488–493. Mavridis, N. (2015). A review of verbal and non-verbal human–robot interactive communication. Robotics and Autonomous Systems, 63, 22–35. Mercado, J. E., Rupp, M. A., Chen, J. Y. C., Barnes, M. J., Barber, D., & Procci, K. (2016). Intelligent agent transparency in human-agent teaming for multi-UxV management. Human Factors: The Journal of the Human Factors and Ergonomics Society, 58(3), 401–415. Miller, G. A. (1956). The magical number seven, plus or minus two: Some limits on our capacity for processing information. Psychological Review, 63(2), 81. Murphy, R. R. (2012). A decade of rescue robots. In 2012 IEEE/RSJ international conference on intelligent robots and systems (pp. 5448–5449). IEEE. Music, S., Salvietti, G., Dohmann, P. B., Chinello, F., Prattichizzo, D., & Hirche, S. (2019). Human–robot team interaction through wearable haptics for cooperative manipulation. IEEE Transactions on Haptics, 12(3), 350–362. Nak Y. C., Kotoku, T., Ohba, K., Komoriya, K., Matsuhira, N., & Tanie, K. (2000). Remote coordinated controls in multiple telerobot cooperation. In Proceedings 2000 ICRA. Millennium Conference. IEEE International Conference on Robotics and Automation. Symposia Proceedings (Cat. No.00CH37065) (Vol. 4, pp. 3138–3143). IEEE. https://doi.org/10.1109/ROBOT.2000.845146 Patel, J., Ramaswamy, T., Li, Z., & Pinciroli, C. (2021). Direct and indirect communication in multi-human multi-robot interaction. IEEE Transactions on Human-Machine Systems (Submitted). Patel, J., Xu, Y., & Pinciroli, C. (2019). Mixed-granularity human-swarm interaction. In 2019 IEEE International Conference on Robotics and Automation (ICRA). IEEE. http://arxiv.org/abs/1901.08522 Patel, T. M., Shah, S. C., & Pancholy, S. B. (2019). Long distance tele-robotic-assisted percutaneous coronary intervention: A report of first-in-human experience. EClinicalMedicine, 14, 53–58. Pinciroli, C., Trianni, V., O’Grady, R., Pini, G., Brutschy, A., Brambilla, M., et al. (2012). ARGoS: A modular, parallel, multi-engine simulator for multi-robot systems. Swarm Intelligence, 6(4), 271–295. Rakita, D., Mutlu, B., & Gleicher, M. (2019). Remote telemanipulation with adapting viewpoints in visually complex environments. In Robotics: Science and Systems XV. Robotics: Science and Systems Foundation. https://doi.org/10.15607/RSS.2019.XV.068 Ricks, B., Nielsen, C. W., & Goodrich, M. A. (2004). Ecological displays for robot interaction: A new perspective. In 2004 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (IEEE Cat. No. 04CH37566) (Vol. 3, pp. 2855–2860). IEEE. Roldan, J. J., Pena-Tapia, E., Garcia-Aunon, P., Del Cerro, J., & Barrientos, A. (2019). Bringing adaptive and immersive interfaces to real-world multi-robot scenarios: Application to surveillance and intervention in infrastructures. IEEE Access, 7, 86319–86335. Roundtree, K. A., Goodrich, M. A., & Adams, J. A. (2019). Transparency: Transitioning from human-machine systems to human-swarm systems. Journal of Cognitive Engineering and Decision Making. https://doi.org/10.1177/1555343419842776. Sartorato, F., Przybylowski, L., & Sarko, D. K. (2017). Improving therapeutic outcomes in autism spectrum disorders: Enhancing social communication and sensory processing through the use of interactive robots. Journal of Psychiatric Research, 90, 1–11. Saunderson, S., & Nejat, G. (2019). How robots influence humans: A survey of nonverbal communication in social human–robot interaction. International Journal of Social Robotics, 11(4), 575–608. Schauß, T., Groten, R., Peer, A., & Buss, M. (2010). Evaluation of a coordinating controller for improved task performance in multi-user teleoperation. In D. Hutchison, T. Kanade, J. Kittler, J. M. Kleinberg, F. Mattern, J. C. Mitchell, M. Naor, O. Nierstrasz, C. Pandu Rangan, B. Steffen, M. Sudan, D. Terzopoulos, D. Tygar, M. Y. Vardi, G. Weikum, A. M. L. Kappers, J. B. F. van Erp, W. M. Bergmann Tiest, F. C. T. van der Helm (Eds.), Haptics: Generating and perceiving tangible sensations. Series Title: Lecture Notes in Computer Science (Vol. 6191, pp. 240–247). Springer. https://doi.org/10.1007/978-3-642-14064-8_35 Sheridan, T. B., & Ferrell, W. R. (1963). Remote manipulative control with transmission delay. IEEE Transactions on Human Factors in Electronics, 1, 25–29. Taylor, R. M. (1990). Situational awareness rating technique (SART): The development of a tool for aircrew systems design. In Situational awareness (pp. 111–128). Routledge. Tomasello, M. (2010). Origins of human communication. MIT Press. Tulli, S., Correia, F., Mascarenhas, S., Gomes, S., Melo, F. S., & Paiva, A. (2019). Effects of agents’ transparency on teamwork. In International workshop on explainable, transparent autonomous agents and multi-agent systems (pp. 22–37). Springer. Uggirala, A., Gramopadhye, A. K., Melloy, B. J., & Toler, J. E. (2004). Measurement of trust in complex and dynamic systems using a quantitative approach. International Journal of Industrial Ergonomics, 34(3), 175–186. Varkonyi, T. A., Rudas, I. J., Pausits, P., & Haidegger, T. (2014). Survey on the control of time delay teleoperation systems. In IEEE 18th International Conference on Intelligent Engineering Systems INES 2014 (pp. 89–94). IEEE. https://doi.org/10.1109/INES.2014.6909347 Watson, B., Walker, N., Ribarsky, W., & Spaulding, V. (1998). Effects of variation in system responsiveness on user performance in virtual environments. Human Factors, 40(3), 403–414. Wohleber, R. W., Stowers, K., Chen, J. Y., & Barnes, M. (2017). Effects of agent transparency and communication framing on human-agent teaming. In 2017 IEEE International Conference on Systems, Man, and Cybernetics (SMC) (pp. 3427–3432). IEEE. https://doi.org/10.1109/SMC.2017.8123160