Emotion recognition based on convolutional neural networks and heterogeneous bio-signal data sources

Ekman, 1994 Susanto, 2020, The hourglass model revisited, IEEE Intell. Syst., 35, 96, 10.1109/MIS.2020.2992799 Scherer, 2001 J. De Lemos, G.R. Sadeghnia, I. Ólafsdóttir, O. Jensen, Measuring emotions using eye tracking, in: Proceedings of Measuring Behavior, Vol. 226, 2008, pp. 225–226. Khosrowabadi, 2013, ERNN: A biologically inspired feedforward neural network to discriminate emotion from EEG signal, IEEE Trans. Neural Netw. Learn. Syst., 25, 609, 10.1109/TNNLS.2013.2280271 Yao, 2014, 133 Zhang, 2016, An approach to EEG-based emotion recognition using combined feature extraction method, Neurosci. Lett., 633, 152, 10.1016/j.neulet.2016.09.037 Lu, 2015, Combining eye movements and eeg to enhance emotion recognition, 1170 Huang, 2019, Combining facial expressions and electroencephalography to enhance emotion recognition, Future Internet, 11, 105, 10.3390/fi11050105 T. Van Huynh, H.-J. Yang, G.-S. Lee, S.-H. Kim, I.-S. Na, Emotion recognition by integrating eye movement analysis and facial expression model, in: Proceedings of the 3rd International Conference on Machine Learning and Soft Computing, 2019, pp. 166–169. Aracena, 2015, Neural networks for emotion recognition based on eye tracking data, 2632 M. Benovoy, J.R. Cooperstock, J. Deitcher, Biosignals analysis and its application in a performance setting, in: Proceedings of the International Conference on Bio-Inspired Systems and Signal Processing, 2008, pp. 253–258. Hosseinifard, 2013, Classifying depression patients and normal subjects using machine learning techniques and nonlinear features from EEG signal, Comput. Methods Programs Biomed., 109, 339, 10.1016/j.cmpb.2012.10.008 Kumar, 2016, Bispectral analysis of EEG for emotion recognition, Procedia Comput. Sci., 84, 31, 10.1016/j.procs.2016.04.062 Russell, 1980, A circumplex model of affect, J. Personal. Soc. Psychol., 39, 1161, 10.1037/h0077714 Huang, 2014, Deep learning for monaural speech separation, 1562 Young, 2018, Recent trends in deep learning based natural language processing, IEEE Comput. Intell. Mag., 13, 55, 10.1109/MCI.2018.2840738 Kermany, 2018, Identifying medical diagnoses and treatable diseases by image-based deep learning, Cell, 172, 1122, 10.1016/j.cell.2018.02.010 Pekrun, 2014, Emotions and learning, Educ. Pract. Ser., 24, 2 Grawemeyer, 2017, Affective learning: improving engagement and enhancing learning with affect-aware feedback, User Model. User-Adapt. Interact., 27, 119, 10.1007/s11257-017-9188-z Finch, 2015, Managing emotions: A case study exploring the relationship between experiential learning, emotions, and student performance, Int. J. Manage. Educ., 13, 23, 10.1016/j.ijme.2014.12.001 Harms, 2010, Facial emotion recognition in autism spectrum disorders: a review of behavioral and neuroimaging studies, Neuropsychol. Rev., 20, 290, 10.1007/s11065-010-9138-6 Kumari, 2015, Facial expression recognition: A survey, Procedia Comput. Sci., 58, 486, 10.1016/j.procs.2015.08.011 Kanade, 2000, Comprehensive database for facial expression analysis, 46 Zeng, 2018, Facial expression recognition via learning deep sparse autoencoders, Neurocomputing, 273, 643, 10.1016/j.neucom.2017.08.043 Mollahosseini, 2016, Going deeper in facial expression recognition using deep neural networks, 1 Ding, 2017, Facenet2expnet: Regularizing a deep face recognition net for expression recognition, 118 Lucey, 2010, The extended cohn-kanade dataset (ck+): A complete dataset for action unit and emotion-specified expression, 94 Muhammad, 2017, User emotion recognition from a larger pool of social network data using active learning, Multimedia Tools Appl., 76, 10881, 10.1007/s11042-016-3912-2 Martin, 2006, The eNTERFACE’05 audio-visual emotion database, 8 Salama, 2018, EEG-based emotion recognition using 3D convolutional neural networks, Int. J. Adv. Comput. Sci. Appl., 9, 329 Wang, 2014, Emotional state classification from EEG data using machine learning approach, Neurocomputing, 129, 94, 10.1016/j.neucom.2013.06.046 Bradley, 2008, The pupil as a measure of emotional arousal and autonomic activation, Psychophysiology, 45, 602, 10.1111/j.1469-8986.2008.00654.x H.-W. Ng, V.D. Nguyen, V. Vonikakis, S. Winkler, Deep learning for emotion recognition on small datasets using transfer learning, in: Proceedings of the 2015 ACM on International Conference on Multimodal Interaction, 2015, pp. 443–449. Guo, 2016, Deep neural networks with relativity learning for facial expression recognition, 1 Soleymani, 2011, Multimodal emotion recognition in response to videos, IEEE Trans. Affect. Comput., 3, 211, 10.1109/T-AFFC.2011.37 Nguyen, 2018, Facial expression recognition using a multi-level convolutional neural network, 217 Li, 2019, Classification of five emotions from EEG and eye movement signals: Discrimination ability and stability over time, 607 Zhao, 2019, Classification of five emotions from EEG and eye movement signals: complementary representation properties, 611 Sourina, 2011, A fractal-based algorithm of emotion recognition from EEG using arousal-valence model, 209 Lan, 2016, Real-time EEG-based emotion monitoring using stable features, Vis. Comput., 32, 347, 10.1007/s00371-015-1183-y Murugappan, 2011, Combining spatial filtering and wavelet transform for classifying human emotions using EEG signals, J. Med. Biol. Eng., 31, 45, 10.5405/jmbe.710 Jie, 2014, Emotion recognition based on the sample entropy of EEG, Bio-Med. Mater. Eng., 24, 1185, 10.3233/BME-130919 Chen, 2017, Novel algorithm for measuring the complexity of electroencephalographic signals in emotion recognition, J. Med. Imaging Health Inform., 7, 203, 10.1166/jmihi.2017.2007 Soroush, 2018, A novel method of eeg-based emotion recognition using nonlinear features variability and Dempster–Shafer theory, Biomed. Eng.: Appl. Basis Commun., 30, 1 Quesada-Tabares, 2017, Emotions detection based on a single-electrode EEG device, 89 Van Dyk, 2001, The art of data augmentation, J. Comput. Graph. Statist., 10, 1, 10.1198/10618600152418584 Pan, 2020, Loss functions of generative adversarial networks (GANs): Opportunities and challenges, IEEE Trans. Emerg. Top. Comput. Intell., 4, 500, 10.1109/TETCI.2020.2991774 Antoniou, 2017 Liang, 2018, Universal affective model for Readers’ emotion classification over short texts, Expert Syst. Appl., 114, 322, 10.1016/j.eswa.2018.07.027 Huang, 2020, Sentiment strength detection with a context-dependent lexicon-based convolutional neural network, Inform. Sci., 520, 389, 10.1016/j.ins.2020.02.026 Lang, 2005 LeCun, 1995, Convolutional networks for images, speech, and time series, 255 S. Tripathi, S. Acharya, R.D. Sharma, S. Mittal, S. Bhattacharya, Using deep and convolutional neural networks for accurate emotion classification on DEAP dataset, in: Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, 2017, pp. 4746–4752. Santamaria-Granados, 2018, Using deep convolutional neural network for emotion detection on a physiological signals dataset (AMIGOS), IEEE Access, 7, 57, 10.1109/ACCESS.2018.2883213 Simonyan, 2014, 1409 Tyagi, 2012, A review of EEG sensors used for data acquisition, 13 Shields, 2016, Are electrode caps worth the investment? An evaluation of EEG methods in undergraduate neuroscience laboratory courses and research, J. Undergrad. Neurosci. Educ., 15, A29 Ijjina, 2014, Facial expression recognition using kinect depth sensor and convolutional neural networks, 392 Komogortsev, 2010, Standardization of automated analyses of oculomotor fixation and saccadic behaviors, IEEE Trans. Biomed. Eng., 57, 2635, 10.1109/TBME.2010.2057429 Lang, 1980, Behavioral treatment and biobehavioral assessment: Computer applications, 119 Srivastava, 2014, Dropout: a simple way to prevent neural networks from overfitting, J. Mach. Learn. Res., 15, 1929 Kulkarni, 2009, Facial expression (mood) recognition from facial images using committee neural networks, Biomed. Eng. Online, 8, 1, 10.1186/1475-925X-8-16 Krizhevsky, 2012, Imagenet classification with deep convolutional neural networks, 1097 Nair, 2010, Rectified linear units improve restricted boltzmann machines, 807 Cowie, 2003, Describing the emotional states that are expressed in speech, Speech Commun., 40, 5, 10.1016/S0167-6393(02)00071-7 X. Glorot, Y. Bengio, Understanding the difficulty of training deep feedforward neural networks, in: Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, Vol. 9, 2010, pp. 249–256. Ruder, 2016 Huang, 2017, Fusion of facial expressions and EEG for multimodal emotion recognition, Comput. Intell. Neurosci., 2017, 1 Carrier, 2013 Dhall, 2012, Collecting large, richly annotated facial-expression databases from movies, IEEE Multimedia, 34, 10.1109/MMUL.2012.26 Soroush, 2018, Emotion classification through nonlinear EEG analysis using machine learning methods, Int. Clin. Neurosci. J., 5, 135, 10.15171/icnj.2018.26 Tao, 2015, L1-norm locally linear representation regularization multi-source adaptation learning, Neural Netw., 69, 80, 10.1016/j.neunet.2015.01.009 Tao, 2016, Multi-source adaptation joint kernel sparse representation for visual classification, Neural Netw., 76, 135, 10.1016/j.neunet.2016.01.008 Tao, 2017, Robust multi-source adaptation visual classification using supervised low-rank representation, Pattern Recognit., 61, 47, 10.1016/j.patcog.2016.07.006