Landmark triangulation-induced Altitude Signature for change detection of human emotion from face image sequence

Md Nasir1, Paramartha Dutta1, Avishek Nandi1
1Visva-Bharati University, Santiniketan, India

Tóm tắt

Video-based facial expression recognition is a potential alternative for detecting the transitional processes of human emotion. In this paper, we present the landmark triangulation-induced altitude signatures to depict gradual changes in human emotion from face video frames. In our proposed approach, we consider the geometry-based triangulation technique which results from the triangles formed by important landmark points on face images. In this work, we measure three altitudes from all triangles and generate three different variations of altitude signature viz. AS1, AS2 and AS3 for the classification of six basic emotions viz. anger (AN), disgust (DI), fear (FE), happiness (HA), sadness (SA) and surprise (SU) in several ways. The performance capability of our proposed recognition system is tested on three benchmark face video datasets: Extended Cohn–Kanade (CK+), M&M Initiative (MMI), and Multimedia Understanding Group (MUG) through the application of Multilayer Perceptron (MLP) classifier and also validated by tenfold cross-validation method. Experimental results, 98.77%, 93.06%, and 98.75% on CK+, MMI, and MUG, respectively, are found very impressive. System efficacy is vindicated by showing the superiority of the proposed technique over the other existing techniques.

Tài liệu tham khảo

Zabih R, Woodfill J (1994) Non-parametric local transforms for computing visual correspondence. In: European conference on computer vision, pp 151–158. Springer Mehrabian A (2007) Nonverbal communication. New Brunswick Tao J, Tan T (2005) Affective computing: a review. In: International conference on affective computing and intelligent interaction. Springer, pp 981–995 Beat F, Juergen L (2003) Automatic facial expression analysis: a survey. Pattern Recognit 36(1):259–275 Ekman P, Friesen Wallace V (1971) Constants across cultures in the face and emotion. J Personal Soc Psychol 17(2):124 Caifeng S, Shaogang G, McOwan Peter W (2009) Facial expression recognition based on local binary patterns: a comprehensive study. Image Vis Comput 27(6):803–816 Pantic M, Rothkrantz LJM (2004) Facial action recognition for facial expression analysis from static face images. IEEE Trans Syst Man Cybern Part B (Cybernetics) 34(3):1449–1461 Naveen Kumar HN, Jagadeesha S, Jain Amith K (2016) Human facial expression recognition from static images using shape and appearance feature. In: 2016 2nd international conference on applied and theoretical computing and communication technology (iCATccT), pp 598–603. IEEE Akkoca BS, Gökmen M (2014) Facial expression recognition from static images. In: 2014 22nd signal processing and communications applications conference (SIU), pp 1291–1294. IEEE Ghimire D, Jeong S, Lee J, Park SH (2017) Facial expression recognition based on local region specific features and support vector machines. Multimed Tools Appl 76(6):7803–7821 Irene K, Ioannis P (2006) Facial expression recognition in image sequences using geometric deformation features and support vector machines. IEEE Trans Image Process 16(1):172–187 Lajevardi SM, Lech M (2008) Facial expression recognition from image sequences using optimized feature selection. In: 2008 23rd international conference image and vision computing New Zealand, pp 1–6. IEEE Haris RA (2018) Face expression recognition using artificial neural network (ANN) model back propagation. In: Proceedings of the joint workshop KO2PI and the 1st international conference on advance & scientific innovation, pp 126–134. ICST (Institute for Computer Sciences, Social-Informatics and ...) Yang S, Bhanu B (2012) Understanding discrete facial expressions in video using an emotion avatar image. IEEE Trans Syst Man Cybern Part B (Cybernetics) 42(4):980–992 Zavaschi Thiago HH, Britto Jr Alceu S, Oliveira Luiz ES, Koerich Alessandro L (2013) Fusion of feature sets and classifiers for facial expression recognition. Expert Syst Appl 40(2):646–655 Wan C, Tian Y, Liu S (2012) Facial expression recognition in video sequences. In: Proceedings of the 10th world congress on intelligent control and automation, pp 4766–4770. IEEE Liu X, Cheng T (2003) Video-based face recognition using adaptive hidden Markov models. In: 2003 IEEE computer society conference on computer vision and pattern recognition, 2003. Proceedings, vol 1, p I. IEEE Pantic M, Patras I (2006) Dynamics of facial expression: recognition of facial actions and their temporal segments from face profile image sequences. IEEE Trans Syst Man Cybern Part B (Cybernetics) 36(2):433–449 Khairuni AKS, Hisham JM, Jussi P, Rajendran P (2016) Spatiotemporal feature extraction for facial expression recognition. IET Image Process 10(7):534–541 Yaddaden Y, Adda M, Bouzouane A, Gaboury S, Bouchard B (2017) Facial expression recognition from video using geometric features Anwar S, Ayoub A-H, Robert N, Moftah E (2014) Frame-based facial expression recognition using geometrical features. Adv Human-Comput Interact 2014:4 Nuri Ö, Mehmet BG (2021) New geometric based features for facial expression recognition. Technical report, EasyChair Murugappan M, Mutawa A (2021) Facial geometric feature extraction based emotional expression classification using machine learning algorithms. Plos One 16(2):e0247131 Singh A, Khan MA, Baghel N (2020) Face emotion identification by fusing neural network and texture features: facial expression. In: 2020 international conference on contemporary computing and applications (IC3A), pp 187–190. IEEE Chang L, Kaoru H, Junjie M, Zhiyang J, Yaping D (2021) Facial expression recognition using hybrid features of pixel and geometry. IEEE Access 9:18876–18889 Deepak G, Joonwhoan L, Ze-Nian L, Sunghwan J (2017) Recognition of facial expressions based on salient geometric features and support vector machines. Multimed Tools Appl 76(6):7921–7946 Asit B, Paramartha D (2019) Influence of shape and texture features on facial expression recognition. IET Image Process 13(8):1349–1363 Asit B, Paramartha D (2019) Facial expression recognition using distance and texture signature relevant features. Appl Soft Comput 77:88–105 Lei Z, Zengcai W, Zhang G (2017) Facial expression recognition from video sequences based on spatial-temporal motion local binary pattern and gabor multiorientation fusion histogram. Mathematical Problems in Engineering Deepak G, Joonwhoan L (2014) Extreme learning machine ensemble using bagging for facial expression recognition. J Inf Process Syst 10(3):443–458 Happy SL, Routray A (2015) Robust facial expression classification using shape and appearance features. In: 2015 eighth international conference on advances in pattern recognition (ICAPR), pp 1–5. IEEE Cootes Timothy F, Taylor Christopher J, Cooper David H, Jim G (1995) Active shape models-their training and application. Comput Vis Image Underst 61(1):38–59 Tzimiropoulos G, Pantic M (2013) Optimization problems for fast AAM fitting in-the-wild. In: Proceedings of the IEEE international conference on computer vision, pp 593–600 Barman A, Dutta P (2021) Facial expression recognition using distance and shape signature features. Pattern Recognit Lett 145:254–261 Ying-Ming W, Wang H-W, Lu Y-L, Yen S, Hsiao Y-T (2012) Facial feature extraction and applications: a review. In: Asian conference on intelligent information and database systems. Springer, pp 228–238 Lucey P, Cohn JF, Kanade T, Saragih J, Ambadar Z, Matthews I (2010) The extended cohn-kanade dataset (ck+): a complete dataset for action unit and emotion-specified expression. In: 2010 IEEE computer society conference on computer vision and pattern recognition-workshops, pp 94–101. IEEE Aifanti N, Papachristou C, Delopoulos A (2010) The mug facial expression database. In: 11th international workshop on image analysis for multimedia interactive services WIAMIS 10, pp 1–4. IEEE Valstar M, Pantic M (2010) Induced disgust, happiness and surprise: an addition to the mmi facial expression database. In: Proceedings of 3rd international workshop on EMOTION (satellite of LREC): Corpora for research on emotion and affect, p 65. Paris, France Amin M, Hassan A, Farzad T (2015) Video-based facial expression recognition by removing the style variations. IET Image Process 9(7):596–603 Yongqiang L, Shangfei W, Yongping Z, Qiang J (2013) Simultaneous facial feature tracking and facial expression recognition. IEEE Trans Image Process 22(7):2559–2573 Yogachandran R, Phan Raphael C-W, Chambers Jonathon A, Parish David J (2012) Facial expression recognition in the encrypted domain based on local fisher discriminant analysis. IEEE Trans Affect Comput 4(1):83–92 Shiqing Z, Xianzhang P, Yueli C, Xiaoming Z, Limei L (2019) Learning affective video features for facial expression recognition via hybrid deep learning. IEEE Access 7:32297–32304 Anima M, Laxmidhar B, Subramanian Venkatesh K (2014) Emotion recognition from geometric facial features using self-organizing map. Pattern Recognit 47(3):1282–1293