STSNet: a novel spatio-temporal-spectral network for subject-independent EEG-based emotion recognition
Tóm tắt
How to use the characteristics of EEG signals to obtain more complementary and discriminative data representation is an issue in EEG-based emotion recognition. Many studies have tried spatio-temporal or spatio-spectral feature fusion to obtain higher-level representations of EEG data. However, these studies ignored the complementarity between spatial, temporal and spectral domains of EEG signals, thus limiting the classification ability of models. This study proposed an end-to-end network based on ManifoldNet and BiLSTM networks, named STSNet. The STSNet first constructed a 4-D spatio-temporal-spectral data representation and a spatio-temporal data representation based on EEG signals in manifold space. After that, they were fed into the ManifoldNet network and the BiLSTM network respectively to calculate higher-level features and achieve spatio-temporal-spectral feature fusion. Finally, extensive comparative experiments were performed on two public datasets, DEAP and DREAMER, using the subject-independent leave-one-subject-out cross-validation strategy. On the DEAP dataset, the average accuracy of the valence and arousal are 69.38% and 71.88%, respectively; on the DREAMER dataset, the average accuracy of the valence and arousal are 78.26% and 82.37%, respectively. Experimental results show that the STSNet model has good emotion recognition performance.
Tài liệu tham khảo
Ali M, Mosa AH, Al Machot F, Kyamakya K, EEG-based emotion recognition approach for e-healthcare applications. In: 2016 eighth international conference on ubiquitous and future networks (ICUFN). IEEE; 2016. p. 946–50.
Vij A, Pruthi J. An automated psychometric analyzer based on sentiment analysis and emotion recognition for healthcare. Procedia Comput Sci. 2018;132:1184–91.
Rivas JJ, del Carmen Lara M, Castrejon L, Hernandez-Franco J, Orihuela-Espina F, Palafox L, Williams A, Berthouze N, Sucar E. Multi-label and multimodal classifier for affective states recognition in virtual rehabilitation. IEEE Trans Affect Comput. 2021. https://doi.org/10.1109/TAFFC.2021.3055790.
Deniz E, Sobahi N, Omar N, Sengur A, Acharya UR. Automated robust human emotion classification system using hybrid EEG features with ICBrainDB dataset. Health Inf Sci Syst. 2022;10(31):1–14. https://doi.org/10.1007/s13755-022-00201-y.
Roy Y, Banville H, Albuquerque I, Gramfort A, Falk TH, Faubert J. Deep learning-based electroencephalography analysis: a systematic review. J Neural Eng. 2019;16(5): 051001.
Hamann S, Canli T. Individual differences in emotion processing. Curr Opin Neurobiol. 2004;14(2):233–8.
Song T, Zheng W, Song P, Cui Z. EEG emotion recognition using dynamical graph convolutional neural networks. IEEE Trans Affect Comput. 2018;11(3):532–41.
Li Y, Zheng W, Wang L, Zong Y, Cui Z. From regional to global brain: a novel hierarchical spatial-temporal neural network model for EEG emotion recognition. IEEE Trans Affect Comput. 2022;13(02):568–78.
Zhang X, Liu J, Shen J, Li S, Hou K, Hu B, Gao J, Zhang T. Emotion recognition from multimodal physiological signals using a regularized deep fusion of kernel machine. IEEE Trans Cybern. 2020;51(9):4386–99.
Bhosale S, Chakraborty R, Kopparapu SK. Calibration free meta learning based approach for subject independent EEG emotion recognition. Biomed Signal Process Control. 2022;72: 103289.
Cui H, Liu A, Zhang X, Chen X, Liu J, Chen X. EEG-based subject-independent emotion recognition using gated recurrent unit and minimum class confusion. IEEE Trans Affect Comput. 2022. https://doi.org/10.1109/TAFFC.2022.3179717.
Cohen MX. Where does EEG come from and what does it mean? Trends Neurosci. 2017;40(4):208–18.
Lindquist KA, Barrett LF. A functional architecture of the human brain: emerging insights from the science of emotion. Trends Cognit Sci. 2012;16(11):533–40.
Zheng W-L, Lu B-L. Investigating critical frequency bands and channels for EEG-based emotion recognition with deep neural networks. IEEE Trans Auton Ment Dev. 2015;7(3):162–75.
Li P, Liu H, Si Y, Li C, Li F, Zhu X, Huang X, Zeng Y, Yao D, Zhang Y, et al. EEG based emotion recognition by combining functional connectivity network and local activations. IEEE Trans Biomed Eng. 2019;66(10):2869–81.
Jiang D, Yu M, Yuanyuan W. Sleep stage classification using covariance features of multi-channel physiological signals on Riemannian manifolds. Comput Methods Programs Biomed. 2019;178:19–30.
Arsigny V, Fillard P, Pennec X, Ayache N. Log-Euclidean metrics for fast and simple calculus on diffusion tensors. Magn Reson Med. 2006;56(2):411–21.
Yger F, Berar M, Lotte F. Riemannian approaches in brain-computer interfaces: a review. IEEE Trans Neural Syst Rehabil Eng. 2016;25(10):1753–62.
Lotte F, Bougrain L, Cichocki A, Clerc M, Congedo M, Rakotomamonjy A, Yger F. A review of classification algorithms for EEG-based brain-computer interfaces: a 10 year update. J Neural Eng. 2018;15(3): 031005.
Abdel-Ghaffar EA, Wu Y, Daoudi M. Subject-dependent emotion recognition system based on multidimensional electroencephalographic signals: a riemannian geometry approach. IEEE Access. 2022;10:14993–5006.
Gao Y, Sun X, Meng M, Zhang Y. EEG emotion recognition based on enhanced SPD matrix and manifold dimensionality reduction. Comput Biol Med. 2022;146: 105606.
Chakraborty R, Bouza J, Manton J, Vemuri BC. Manifoldnet: a deep neural network for manifold-valued data with applications. IEEE Trans Pattern Anal Mach Intell. 2020;44(2):799–810.
Yu Z, Ramanarayanan V, Suendermann-Oeft D, Wang X, Zechner K, Chen L, Tao J, Ivanou A, Qian Y: Using bidirectional LSTM recurrent neural networks to learn high-level abstractions of sequential features for automated scoring of non-native spontaneous speech. In: 2015 IEEE workshop on automatic speech recognition and understanding (ASRU). IEEE; 2015. p. 338–45.
Koelstra S, Muhl C, Soleymani M, Lee J-S, Yazdani A, Ebrahimi T, Pun T, Nijholt A, Patras I. DEAP: a database for emotion analysis; using physiological signals. IEEE Trans Affect Comput. 2011;3(1):18–31.
Katsigiannis S, Ramzan N. DREAMER: a database for emotion recognition through EEG and ECG signals from wireless low-cost off-the-shelf devices. IEEE J Biomed Health Inform. 2017;22(1):98–107.
Jenke R, Peer A, Buss M. Feature extraction and selection for emotion recognition from EEG. IEEE Trans Affect Comput. 2014;5(3):327–39.
Li X, Zhang Y, Tiwari P, Song D, Hu B, Yang M, Zhao Z, Kumar N, Marttinen P. EEG based emotion recognition: a tutorial and review. ACM Comput Surv. 2022. https://doi.org/10.1145/3524499.
Li X, Song D, Zhang P, Zhang Y, Hou Y, Hu B. Exploring EEG features in cross-subject emotion recognition. Front Neurosci. 2018;12:162.
Khateeb M, Anwar SM, Alnowami M. Multi-domain feature fusion for emotion classification using DEAP dataset. IEEE Access. 2021;9:12134–42.
Mohammadi Z, Frounchi J, Amiri M. Wavelet-based emotion recognition system using EEG signal. Neural Comput Appl. 2017;28(8):1985–90.
Rahman MM, Sarkar AK, Hossain MA, Moni MA. EEG-based emotion analysis using non-linear features and ensemble learning approaches. Expert Syst Appl. 2022;207: 118025.
Liang Z, Zhou R, Zhang L, Li L, Huang G, Zhang Z, Ishii S. EEGFuseNet: hybrid unsupervised deep feature characterization and fusion for high-dimensional EEG with an application to emotion recognition. IEEE Trans Neural Syst Rehabil Eng. 2021;29:1913–25.
Wang H, Zhu X, Chen P, Yang Y, Ma C, Gao Z. A gradient-based automatic optimization CNN framework for EEG state recognition. J Neural Eng. 2022;19(1): 016009.
Li C, Wang B, Zhang S, Liu Y, Song R, Cheng J, Chen X. Emotion recognition from EEG based on multi-task learning with capsule network and attention mechanism. Comput Biol Med. 2022;143: 105303.
Zhao H, Liu J, Shen Z, Yan J. SCC-MPGCN: self-attention coherence clustering based on multi-pooling graph convolutional network for EEG emotion recognition. J Neural Eng. 2022;19(2): 026051.
Wang Z, Wang Y, Zhang J, Hu C, Yin Z, Song Y. Spatial-temporal feature fusion neural network for EEG-based emotion recognition. IEEE Trans Instrum Meas. 2022;71:1–12.
Ding Y, Robinson N, Zhang S, Zeng Q, Guan C. TSception: capturing temporal dynamics and spatial asymmetry from EEG for emotion recognition. IEEE Trans Affect Comput. 2022. https://doi.org/10.1109/TAFFC.2022.3169001.
Liu S, Wang X, Zhao L, Li B, Hu W, Yu J, Zhang Y-D. 3DCANN: a spatio-temporal convolution attention neural network for EEG emotion recognition. IEEE J Biomed Health Inform. 2021;26(11):5321–31.
Li J, Wu X, Zhang Y, Yang H, Wu X. DRS-Net: a spatial-temporal affective computing model based on multichannel EEG data. Biomed Signal Process Control. 2022;76: 103660.
Mithra U, Aravinth, J.: Spatial spectral based 3D feature map for EEG emotion recognition. In: 2022 3rd international conference on electronics and sustainable communication systems (ICESC). IEEE; 2022. p. 247–52.
Li R, Ren C, Li C, Zhao N, Lu D, Zhang X. SSTD: a novel spatio-temporal demographic network for EEG-based emotion recognition. IEEE Trans Comput Soc Syst. 2023;10(1):376–87.
Sabbagh D, Ablin P, Varoquaux G, Gramfort A, Engemann DA. Predictive regression modeling with MEG/EEG: from source power to signals and cognitive states. Neuroimage. 2020;222: 116893.
Schuster M, Paliwal KK. Bidirectional recurrent neural networks. IEEE Trans Signal Process. 1997;45(11):2673–81.
Walter S, Kim J, Hrabal D, Crawcour SC, Kessler H, Traue HC. Transsituational individual-specific biopsychological classification of emotions. IEEE Trans Syst Man Cybern Syst. 2013;43(4):988–95.
Pedregosa F, Varoquaux G, Gramfort A, Michel V, Thirion B, Grisel O, Blondel M, Prettenhofer P, Weiss R, Dubourg V, et al. Scikit-learn: machine learning in Python. J Mach Learn Res. 2011;12:2825–30.
Saha, P., Fels, S., Abdul-Mageed, M.: Deep learning the EEG manifold for phonological categorization from active thoughts. In: ICASSP 2019-2019 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE; 2019. p. 2762–6.
Pan B, Zheng W. Emotion recognition based on EEG using generative adversarial nets and convolutional neural network. Comput Math Methods Med. 2021;2021:2520394.
Lawhern VJ, Solon AJ, Waytowich NR, Gordon SM, Hung CP, Lance BJ. EEGNet: a compact convolutional neural network for EEG-based brain-computer interfaces. J Neural Eng. 2018;15(5): 056013.
Huang D, Chen S, Liu C, Zheng L, Tian Z, Jiang D. Differences first in asymmetric brain: a bi-hemisphere discrepancy convolutional neural network for EEG emotion recognition. Neurocomputing. 2021;448:140–51.
Arjun, Rajpoot AS, Panicker MR. Subject independent emotion recognition using EEG signals employing attention driven neural networks. Biomed Signal Process Control 2022;75:103547
Zhang G, Yu M, Liu Y, Zhao G, Zhang D, Zheng W. SparseDGCNN: recognizing emotion from multichannel EEG signals. IEEE Trans Affect Comput. 2021. https://doi.org/10.1109/TAFFC.2021.3051332.
Wang Y, Qiu S, Ma X, He H. A prototype-based SPD matrix network for domain adaptation EEG emotion recognition. Pattern Recogn. 2021;110: 107626.
Priyasad D, Fernando T, Denman S, Sridharan S, Fookes C. Affect recognition from scalp-EEG using channel-wise encoder networks coupled with geometric deep learning and multi-channel feature fusion. Knowl-Based Syst. 2022;250: 109038.
He Z, Zhong Y, Pan J. Joint temporal convolutional networks and adversarial discriminative domain adaptation for EEG-based cross-subject emotion recognition. In: ICASSP 2022-2022 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE; 2022. p. 3214–8.