Usability Evaluation of Artificial Intelligence-Based Voice Assistants: The Case of Amazon Alexa

Statista, “Smart Speaker Market Value Worldwide 2014–2025,” [Online]. https://www.statista.com/statistics/1022823/worldwide-smart-speaker-market-revenue/, Accessed 8 Sep 2020.

Ki CW, Cho E, Lee JE. Can an intelligent assistant (IPA) be your friend? Para-friendship development mechanism between IPAs and their users. Comput Hum Behav. 2020;111:1–10. https://doi.org/10.1016/j.chb.2020.106412.

Statista, “Factors surrounding preference of voice assistants over websites and applications, worldwide,” [Online]. https://www.statista.com/statistics/801980/worldwide-preference-voice-assistant-websites-app/, Accessed 8 Sep 2020.

McLean G, Frimpong KO. Hey Alexa … examine the variables influencing the use of artificial intelligent in-home voice assistants. Comput Hum Behav. 2019;99:28–37. https://doi.org/10.1016/j.chb.2019.05.009.

Pal D, Arpnikanondt C, Funilkul S, Chutimaskul W. The adoption analysis of voice based smart IoT products. IEEE Internet Things J. 2020;7(11):10852–67. https://doi.org/10.1109/JIOT.2020.2991791.

Feng L, Wei W. An empirical study on user experience evaluation and identification of critical UX issues. Sustainability. 2019;11(8):1–19. https://doi.org/10.3390/su11082432.

Oliver RL. A cognitive model of the antecedents and consequences of satisfaction decisions. J Mark Res. 1980;17(4):460–9. https://doi.org/10.2307/3150499.

Bhattacharjee A. Understanding information systems continuance: an expectation-confirmation model. MIS Q. 2001;25(3):351–70. https://doi.org/10.2307/3250921.

Parasuraman A, Zeithaml VA, Berry LL. A conceptual model of service quality and its implications for future research. J Mark. 1985;49(4):41–50. https://doi.org/10.2307/1251430.

Kocaballi AB, Laranjo L, Coiera E. Measuring user experience in conversational interfaces: a comparison of six questionnaires. In: Proc. of 32nd international BCS human computer interaction conference (HCI) July 4–6; 2018. pp. 1–12.

Lewis JR. Standardized questionnaires for voice interaction design. ACIXD J. 2016;1(1):1–16.

Lewis JR. Measuring perceived usability: SUS, UMUX, and CSUQ ratings for four everyday products. Int J Hum-Comput Interact. 2018;35(15):1404–19. https://doi.org/10.1080/10447318.2018.1533152.

Lewis JR. The system usability scale: past, present, and future. Int J Hum Comput Interact. 2018;34(7):577–90. https://doi.org/10.1080/10447318.2018.1455307.

Kocaballi AB, Laranjo L, Coiera E. Understanding and measuring user experience in conversational interfaces. Interact Comput. 2019;31(2):192–207. https://doi.org/10.1093/iwc/iwz015.

Murad C, Munteanu C, Cowan BR, Clark L. Revolution or evolution? Speech interaction and HCI design guidelines. IEEE Pervasive Comput. 2019;18(2):33–45. https://doi.org/10.1109/MPRV.2019.2906991.

Cowan BR, et al. What can i help you with?: Infrequent users’ experiences of intelligent personal assistants. In: Proc. 19th international conference on human-computer interaction with mobile devices and services; 2017. pp. 1–12. https://doi.org/10.1145/3098279.3098539.

Silva AB, et al. Intelligent personal assistants: a systematic literature review. Expert Syst Appl. 2020;147:1–14. https://doi.org/10.1016/j.eswa.2020.113193.

Kawase T, Okamoto M, Fukutomi T, Takahashi Y. Speech enhancement parameter adjustment to maximize accuracy of automatic speech recognition. IEEE Trans Consum Electron. 2020;66(2):125–33. https://doi.org/10.1109/TCE.2020.2986003.

Kumar AJ, Schmidt C, Kohler J. A knowledge graph based speech interface for question answering systems. Speech Commun. 2017;92:1–12. https://doi.org/10.1016/j.specom.2017.05.001.

Guo L, Wang L, Dang J, Liu Z, Guan H. Exploration of complementary features for speech emotion recognition based on kernel extreme learning machine. IEEE Access. 2019;7:75798–809. https://doi.org/10.1109/ACCESS.2019.2921390.

Nath RK, Bajpai R, Thapliyal H. IoT based indoor location detection system for smart home environment. In: 2018 IEEE international conference on consumer electronics (ICCE), Las Vegas, NV; 2018. pp. 1–3. https://doi.org/10.1109/ICCE.2018.8326225.

Greene S, Thapliyal H, Carpenter D. IoT-based fall detection for smart home environments. In: 2016 IEEE international symposium on nanoelectronic and information systems (iNIS), Gwalior; 2016. pp. 23–28. https://doi.org/10.1109/iNIS.2016.017.

Sun T. End-to-end speech emotion recognition with gender information. IEEE Access. 2020;8:152423–38. https://doi.org/10.1109/ACCESS.2020.3017462.

Park J, Son H, Lee J, Choi J. Driving assistant companion with voice interface using long short-term memory networks. IEEE Trans Ind Inform. 2019;15(1):582–90. https://doi.org/10.1109/TII.2018.2861739.

Jia J, et al. Inferring emotions from large-scale internet voice data. IEEE Trans Multimedia. 2019;21(7):1853–66. https://doi.org/10.1109/TMM.2018.2887016.

Alepis E, Patsakis C. Monkey says, monkey does: security and privacy on voice assistants. IEEE Access. 2017;5:17841–51. https://doi.org/10.1109/ACCESS.2017.2747626.

Zhang R, Chen X, Wen S, Zheng X, Ding Y. Using AI to attack VA: a stealthy spyware against voice assistances in smart phones. IEEE Access. 2019;7:153542–54. https://doi.org/10.1109/ACCESS.2019.2945791.

Malik KM, Javed A, Malik H, Irtaza A. A light-weight replay detection framework for voice controlled IoT devices. IEEE J Sel Top Signal Process. 2020;14(5):982–96. https://doi.org/10.1109/JSTSP.2020.2999828.

Yan C, Zhang G, Ji X, Zhang T, Zhang T, Xu W. The feasibility of injecting inaudible voice commands to voice assistants. IEEE Trans Dependable Secure Comput. 2019. https://doi.org/10.1109/TDSC.2019.2906165.

Thapliyal H, Ratajczak N, Wendroth O, Labrado C. Amazon echo enabled IoT home security system for smart home environment, 2018. In: IEEE international symposium on smart electronic systems (iSES) (Formerly iNiS), Hyderabad, India; 2018. pp. 31–36. https://doi.org/10.1109/iSES.2018.00017.

Nguyen QN, Ta A, Prybutok V. An integrated model of voice-user interface continuance intention: the gender effect. Int J Hum-Comput Interact. 2019;35(15):1362–77. https://doi.org/10.1080/10447318.2018.1525023.

Yang H, Lee H. Understanding user behavior of virtual personal assistant devices. IseB. 2019;17:65–87. https://doi.org/10.1007/s10257-018-0375-1.

Pal D, Arpnikanondt C, Funilkul S, Razzaque MA. Analyzing the adoption and diffusion of voice-enabled smart-home systems: empirical evidence from Thailand. Univers Access Inf Soc. 2020. https://doi.org/10.1007/s10209-020-00754-3.

Maguire, M. Development of a heuristic evaluation tool for voice user interfaces. In: Proc. of international conference on human-computer interaction (HCII’19), Orlando, USA; 2019. pp. 212–25. https://doi.org/10.1007/978-3-030-23535-2_16.

López G, Quesada L, Guerrero LA. Alexa vs. Siri vs. Cortana vs. Google assistant: a comparison of speech-based natural user interfaces. In: Proc. of 2017 international conference on applied human factors and ergonomics (AHFE 2017), Los Angeles, USA; 2017. pp. 241–50. https://doi.org/10.1007/978-3-319-60366-7_23.

Bogers T, et al. A study of usage and usability of intelligent personal assistants in Denmark. In: Proc. of international conference on information in contemporary society (iConference 19), Washington, USA; 2019. pp. 79–90. https://doi.org/10.1007/978-3-030-15742-5_7.

Pal D, Arpnikanondt C, Funilkul S, Varadarajan V. User experience with smart voice assistants: the accent perspective. In: Proc. of 2019 10th international conference on computing, communication and networking technologies (ICCCNT), Kanpur, India; 2019, pp. 1–6. https://doi.org/10.1109/ICCCNT45670.2019.8944754.

Ghosh D, Foong PS, Zhang S, Zhao S. Assessing the utility of the system usability scale for evaluating voice-based user interfaces. In: Proc. of the sixth international symposium of Chinese CHI (ChineseCHI '18), association for computing machinery, New York, NY, USA. pp. 11–15. https://doi.org/10.1145/3202667.3204844.

Yang C, Chen X, Xu Q, Hu L, Jin C, Wang C. A questionnaire for subjective evaluation of the intelligent speech system. In: Proc. of 2018 first Asian conference on affective computing and intelligent interaction (ACII Asia), Beijing; 2018. pp. 1–6. https://doi.org/10.1109/ACIIAsia.2018.8470339.

Brooke J. SUS: a quick and dirty usability scale. 1st ed. London: Taylor & Francis; 1996.

Bangor A, Kortum PT, Miller JT. An empirical evaluation of the system usability scale. Int J Hum-Comput Interact. 2008;24(6):574–94. https://doi.org/10.1080/10447310802205776.

Ghosh D, Foong PS, Zhao S, Chen D, Fjeld M. EDITalk: Towards designing eyes-free interactions for mobile word processing. In: Proc. of the 2018 CHI conference on human factors in computing systems—CHI’18, association for computing machinery, New York, NY, USA. pp. 1–10. https://doi.org/10.1145/3173574.3173977.

Babel M, McGuire G, King J. Towards a more nuanced view of vocal attractiveness. PLoS ONE. 2014;9(2):e88616. https://doi.org/10.1371/journal.pone.0088616.

Koreman J, Pützer M. The usability of perceptual ratings of voice quality. In: Proc. 6th international conference on advances in quantitative laryngology, voice and speech research (AQL), Hamburg, Germany; 2003.

Pfeuffer N, Benlian A, Gimpel H, Hinz O. Anthropomorphic information systems. Bus Inf Sys Eng. 2019;61:523–33. https://doi.org/10.1007/s12599-019-00599-y.

Wei Z, Landay JA. Evaluating speech-based smart devices using new usability heuristics. IEEE Pervasive Comput. 2018;17(2):84–96. https://doi.org/10.1109/MPRV.2018.022511249.

Bangor A, Kortum P, Miller J. Determining what Individual SUS scores mean: adding an adjective rating scale. J Usability Stud. 2009;4(3):114–23.

Field A. Discovering statistics using IBM SPSS statistics. 4th ed. Thousand Oaks: Sage Publications; 2013.