Speech emotion recognition based on emotion perception
Tóm tắt
Speech emotion recognition (SER) is a hot topic in speech signal processing. With the advanced development of the cheap computing power and proliferation of research in data-driven methods, deep learning approaches are prominent solutions to SER nowadays. SER is a challenging task due to the scarcity of datasets and the lack of emotion perception. Most existing networks of SER are based on computer vision and natural language processing, so the applicability for extracting emotion is not strong. Drawing on the research results of brain science on emotion computing and inspired by the emotional perceptive process of the human brain, we propose an approach based on emotional perception, which designs a human-like implicit emotional attribute classification and introduces implicit emotional information through multi-task learning. Preliminary experiments show that the unweighted accuracy (UA) of the proposed method has increased by 2.44%, and weighted accuracy (WA) 3.18% (both absolute values) on the Interactive Emotional Dyadic Motion Capture (IEMOCAP) dataset, which verifies the effectiveness of our method.
Tài liệu tham khảo
L.S.A. Low, N.C. Maddage, M. Lech, L.B. Sheeber, N.B. Allen, Detection of clinical depression in adolescents’ speech during family interactions. IEEE Trans. Biomed. Eng. 58(3), 574–586 (2010)
X. Huahu, G. Jue, Y. Jian, in Proceedings of the 2010 International Conference on Artificial Intelligence and Computational Intelligence, vol. 1. Application of speech emotion recognition in intelligent household robot, (IEEE, Sanya, 2010), pp. 537–541
W.J. Yoon, Y.H. Cho, K.S. Park, in International Conference on Ubiquitous Intelligence and Computing. A study of speech emotion recognition and its application to mobile services (Springer, Hong Kong China, 2007), pp. 758–766
K. Han, D. Yu, I. Tashev, in Proceedings of Interspeech 2014. Speech emotion recognition using deep neural network and extreme learning machine (ISCA, Singapore, 2014)
M. Chen, X. He, J. Yang, H. Zhang, 3-d convolutional recurrent neural networks with attention model for speech emotion recognition. IEEE Signal Process. Lett. 25(10), 1440–1444 (2018)
X. Wu, S. Liu, Y. Cao, X. Li, J. Yu, D. Dai, X. Ma, S. Hu, Z. Wu, X. Liu, et al., in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). Speech emotion recognition using capsule networks (IEEE, Brighton UK, 2019), pp. 6695–6699
Y. Xu, H. Xu, J. Zou, in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). Hgfm: a hierarchical grained and feature model for acoustic emotion recognition (IEEE, Barcelona, 2020), pp. 6499–6503
D. Priyasad, T. Fernando, S. Denman, S. Sridharan, C. Fookes, in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). Attention driven fusion for multi-modal emotion recognition (IEEE, Barcelona, 2020), pp. 3227–3231
A. Nediyanchath, P. Paramasivam, P. Yenigalla, in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). Multi-head attention for speech emotion recognition with auxiliary learning of gender recognition (IEEE, Barcelona, 2020), pp. 7179–7183
C.H. Park, D.W. Lee, K.B. Sim, Emotion recognition of speech based on rnn. Nurse Lead. 4, 2210–2213 (2002). https://doi.org/10.1109/ICMLC.2002.1175432
J. Niu, Y. Qian, K. Yu, in The 9th International Symposium on Chinese Spoken Language Processing. Acoustic emotion recognition using deep neural network (IEEE, Singapore, 2014), pp. 128–132
Q. Mao, M. Dong, Z. Huang, Y. Zhan, Learning salient features for speech emotion recognition using convolutional neural networks. IEEE Trans. Multimedia 16(8), 2203–2213 (2014)
J. Lee, I. Tashev, in Proceedings of Interspeech 2015. High-level feature representation using recurrent neural network for speech emotion recognition (ISCA, Dresden Germany, 2015)
M.A. Jalal, E. Loweimi, R.K. Moore, T. Hain, in Proceedings of Interspeech 2019. Learning temporal clusters using capsule routing for speech emotion recognition (ISCA, Graz, 2019), pp. 1701–1705
R. Shankar, H.W. Hsieh, N. Charon, A. Venkataraman, in Proceedings of Interspeech 2019. Automated emotion morphing in speech based on diffeomorphic curve registration and highway networks(ISCA, Graz, 2019), pp. 4499–4503
S. Siriwardhana, T. Kaluarachchi, M. Billinghurst, S. Nanayakkara, Multimodal emotion recognition with transformer-based self supervised feature fusion. IEEE Access 8, 176274–176285 (2020)
S. Costantini, G. De Gasperis, P. Migliarini, in 2019 IEEE Second International Conference on Artificial Intelligence and Knowledge Engineering (AIKE). Multi-agent system engineering for emphatic human-robot interaction (IEEE, Sardinia Italy, 2019), pp. 36–42
H. Okon-Singer, T. Hendler, L. Pessoa, A.J. Shackman, The neurobiology of emotion-cognition interactions: fundamental questions and strategies for future research. Front. Hum. Neurosci. 9, 58 (2015)
Q. Ma, D. Guo, Research on brain mechanisms of emotion. Adv. Psychol. Sci. 11(03), 328 (2003)
S. Lee, S. Yildirim, A. Kazemzadeh, S. Narayanan, in Ninth European Conference on Speech Communication and Technology. An articulatory study of emotional speech production (ISCA, Lisbon Portugal, 2005)
J. LeDoux, Rethinking the emotional brain. Neuron 73(4), 653–676 (2012)
V.R. Rao, K.K. Sellers, D.L. Wallace, M.B. Lee, M. Bijanzadeh, O.G. Sani, Y. Yang, M.M. Shanechi, H.E. Dawes, E.F. Chang, Direct electrical stimulation of lateral orbitofrontal cortex acutely improves mood in individuals with symptoms of depression. Curr. Biol. 28(24), 3893–3902 (2018)
P. Fusar-Poli, A. Placentino, F. Carletti, P. Landi, P. Allen, S. Surguladze, F. Benedetti, M. Abbamonte, R. Gasparotti, F. Barale et al., Functional atlas of emotional faces processing: a voxel-based meta-analysis of 105 functional magnetic resonance imaging studies. J. Psychiatry Neurosci. 34(6), 418–432 (2009)
F. Ahs, C.F. Davis, A.X. Gorka, A.R. Hariri, Feature-based representations of emotional facial expressions in the human amygdala. Soc. Cogn. Affect. Neurosci. 9(9), 1372–1378 (2014)
M.D. Pell, Recognition of prosody following unilateral brain lesion: influence of functional and structural attributes of prosodic contours. Neuropsychologia 36(8), 701–715 (1998)
B. McFee, C. Raffel, D. Liang, D.P. Ellis, M. McVicar, E. Battenberg, O. Nieto, in Proceedings of the 14th python in science conference, vol. 8. librosa: audio and music signal analysis in python (SciPy, Texas US, 2015), pp. 18–25
C. Busso, M. Bulut, C.C. Lee, A. Kazemzadeh, E. Mower, S. Kim, J.N. Chang, S. Lee, S.S. Narayanan, Iemocap: interactive emotional dyadic motion capture database. Lang. Resour. Eval. 42(4), 335–359 (2008)
P. Li, Y. Song, I.V. McLoughlin, W. Guo, L.R. Dai, in Proceedings of Interspeech 2018. An attention pooling based representation learning method for speech emotion recognition (ISCA, Hyderabad India, 2018)
Z. Zhao, Z. Bao, Z. Zhang, N. Cummins, H. Wang, B. Schuller, Attention-enhanced connectionist temporal classification for discrete speech emotion recognition (2019)
M. Neumann, N.T. Vu, in ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). Improving speech emotion recognition with unsupervised representation learning on unlabeled speech (IEEE, Brighton UK, 2019), pp. 7390–7394
L. Tarantino, P.N. Garner, A. Lazaridis, et al., in Proceedings of Interspeech 2019. Self-attention for speech emotion recognition (ISCA, Graz, 2019), pp. 2578–2582