Modified chess patterns: handcrafted feature descriptors for facial expression recognition

Complex & Intelligent Systems - Tập 7 - Trang 3303-3322 - 2021
Mukku Nisanth Kartheek1,2, Munaga V. N. K. Prasad1, Raju Bhukya2
1Center for Affordable Technologies, Institute for Development and Research in Banking Technology, Hyderabad, India
2Department of Computer Science and Engineering, National Institute of Technology Warangal, Warangal, India

Tóm tắt

Facial expressions are predominantly important in the social interaction as they convey the personal emotions of an individual. The main task in Facial Expression Recognition (FER) systems is to develop feature descriptors that could effectively classify the facial expressions into various categories. In this work, towards extracting distinctive features, Radial Cross Pattern (RCP), Chess Symmetric Pattern (CSP) and Radial Cross Symmetric Pattern (RCSP) feature descriptors have been proposed and are implemented in a 5 $$\times $$ 5 overlapping neighborhood to overcome some of the limitations of the existing methods such as Chess Pattern (CP), Local Gradient Coding (LGC) and its variants. In a 5 $$\times $$ 5 neighborhood, the 24 pixels surrounding the center pixel are arranged into two groups, namely Radial Cross Pattern (RCP), which extracts two feature values by comparing 16 pixels with the center pixel and Chess Symmetric Pattern (CSP) extracts one feature value from the remaining 8 pixels. The experiments are conducted using RCP and CSP independently and also with their fusion RCSP using different weights, on a variety of facial expression datasets to demonstrate the efficiency of the proposed methods. The results obtained from the experimental analysis demonstrate the efficiency of the proposed methods.

Tài liệu tham khảo

Aifanti N, Papachristou C, Delopoulos A (2010) The mug facial expression database. In: 11th international workshop on image analysis for multimedia interactive services WIAMIS 10, pp 1–4. IEEE Alphonse AS, Shankar K, Rakkini MJ, Ananthakrishnan S, Athisayamani S, Singh AR, Gobi R (2020)A multi-scale and rotation-invariant phase pattern (MRIPP) and a stack of restricted Boltzmann machine (RBM) with preprocessing for facial expression classification. J Ambient Intell Human Comput 20: 1–17 Aneja D, Colburn A, Faigin G, Shapiro L, Mones B (2016) Modeling stylized character expressions via deep learning. In: Asian conference on computer vision. Springer, pp 136–153 Belhumeur PN, Hespanha JP, Kriegman DJ (1997) Eigenfaces vs fisherfaces: recognition using class specific linear projection. IEEE Trans Pattern Anal Mach Intell 19(7):711–720 Chan TH, Jia K, Gao S, Lu J, Zeng Z, Ma Y (2015) Pcanet: a simple deep learning baseline for image classification. IEEE Trans Image Process 24(12):5017–5032 Chen LF, Yen YS (2007) Taiwanese facial expression image database. Brain Mapping Laboratory, Institute of Brain Science, National Yang-Ming University, Taipei Cootes TF, Taylor CJ, Cooper DH, Graham J (1995) Active shape models-their training and application. Comput Vis Image Underst 61(1):38–59 Donato G, Bartlett MS, Hager JC, Ekman P, Sejnowski TJ (1999) Classifying facial actions. IEEE Trans Pattern Anal Mach Intell 21(10):974–989 Ekmen B, Ekenel HK (2019) From 2d to 3d real-time expression transfer for facial animation. Multimed Tools Appl 78(9):12519–12535 Feutry C, Piantanida P, Bengio Y, Duhamel P (2018) Learning anonymized representations with adversarial neural networks. arXiv:1802.09386 (arXiv preprint) Goeleven E, De Raedt R, Leyman L, Verschuere B (2008) The karolinska directed emotional faces: a validation study. Cogn Emot 22(6):1094–1118 He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778 Hong H, Neven H, Von der Malsburg C (1998) Online facial expression recognition based on personalized galleries. In: Proceedings third IEEE international conference on automatic face and gesture recognition, pp 354–359. IEEE Howard AG, Zhu M, Chen B, Kalenichenko D, Wang W, Weyand T, Andreetto M, Adam H (2017) Mobilenets: efficient convolutional neural networks for mobile vision applications. arXiv:1704.04861 (arXiv preprint) Huang Z, Song G, Zhao Y, Han J, Zhao X (2018) Smile recognition based on support vector machine and local binary pattern. In: 2018 IEEE 8th annual international conference on CYBER technology in automation, control, and intelligent systems (CYBER), pp 938–942. IEEE Iqbal MTB, Abdullah-Al-Wadud M, Ryu B, Makhmudkhujaev F, Chae O (2018) Facial expression recognition with neighborhood-aware edge directional pattern (nedp). IEEE Trans Affect Comput 11(1):125–137 Jabid T, Kabir MH, Chae O (2010) Robust facial expression recognition based on local directional pattern. ETRI J 32(5):784–794 Ji Y, Hu Y, Yang Y, Shen F, Shen HT (2019) Cross-domain facial expression recognition via an intra-category common feature and inter-category distinction feature fusion network. Neurocomputing 333:231–239 Jung H, Lee S, Park S, Kim B, Kim J, Lee I, Ahn C (2015) Development of deep learning-based facial expression recognition system. In: 2015 21st Korea-Japan joint workshop on frontiers of computer vision (FCV), pp 1–4. IEEE Jung H, Lee S, Park S, Lee I, Ahn C, Kim J (2015) Deep temporal appearance-geometry network for facial expression recognition. arXiv:1503.01532 (arXiv preprint) Kartheek MN, Prasad MVNK, Bhukya R (2020) Local optimal oriented pattern for person independent facial expression recognition. In: Twelfth international conference on machine vision (ICMV 2019), vol. 11433. International Society for Optics and Photonics, p 114330R1–8 Kas M, Ruichek Y, Messoussi R et al (2020) Multi level directional cross binary patterns: new handcrafted descriptor for SVM-based texture classification. Eng Appl Artif Intell 94:103743 Kola DGR, Samayamantula SK (2020) A novel approach for facial expression recognition using local binary pattern with adaptive window. Multimed Tools Appl 20:1–20 Kola DGR, Samayamantula SK (2021) Facial expression recognition using singular values and wavelet-based LGC-HD operator. IET Biom 20:20 Kumar RJR, Sundaram M (2020) Arumugam N Facial emotion recognition using subband selective multilevel stationary wavelet gradient transform and fuzzy support vector machine. Vis Comput 20:1–15 Kumar YR, Narayanappa C, Dayananda P (2020) Weighted full binary tree-sliced binary pattern: an RGB-D image descriptor. Heliyon 6(5):e03751 Lai CC, Ko CH (2014) Facial expression recognition based on two-stage features extraction. Optik-Int J Light Electron Opt 125(22):6678–6680 Li H, Xu H (2020) Deep reinforcement learning for robust emotional classification in facial expression recognition. Knowl Based Syst 20:106172 Li S, Deng W (2018) Reliable crowdsourcing and deep locality-preserving learning for unconstrained facial expression recognition. IEEE Trans Image Process 28(1):356–370 Lucey P, Cohn JF, Kanade T, Saragih J, Ambadar Z, Matthews I (2010) The extended cohn-kanade dataset (ck+): a complete dataset for action unit and emotion-specified expression. In: 2010 IEEE computer society conference on computer vision and pattern recognition-workshops, pp 94–101. IEEE Lyons M, Akamatsu S, Kamachi M, Gyoba J (1998) Coding facial expressions with gabor wavelets. In: Proceedings of third IEEE international conference on automatic face and gesture recognition, pp 200–205. IEEE Maheswari VU, Varaprasad G, Raju SV (2020) Local directional maximum edge patterns for facial expression recognition. J Ambient Intell Human Comput 20:1–9 Makhmudkhujaev F, Abdullah-Al-Wadud M, Iqbal MTB, Ryu B, Chae O (2019) Facial expression recognition with local prominent directional pattern. Signal Process Image Commun 74:1–12 Makhmudkhujaev F, Iqbal MTB, Ryu B, Chae O (2019) Local directional-structural pattern for person-independent facial expression recognition. Turk J Electr Eng Comput Sci 27(1):516–531 Mandal M, Verma M, Mathur S, Vipparthi SK, Murala S, Kumar DK (2019) Regional adaptive affinitive patterns (RADAP) with logical operators for facial expression recognition. IET Image Proc 13(5):850–861 Minaee S, Abdolrashidi A (2019) Deep-emotion: facial expression recognition using attentional convolutional network. arXiv:1902.01019 (arXiv preprint) Olszanowski M, Pochwatko G, Kuklinski K, Scibor-Rylski M, Lewinski P, Ohme RK (2015) Warsaw set of emotional facial expression pictures: a validation study of facial display photographs. Front Psychol 5:1–8 Reddy PCS, Rao PVP, Reddy PKK, Sridhar M (2019) Motif shape primitives on Fibonacci weighted neighborhood pattern for age classification. Soft computing and signal processing. Springer, Berlin, pp 273–280 Revina IM, Emmanuel WS (2019) MDTP: a novel multi-directional triangles pattern for face expression recognition. Multimed Tools Appl 78(18):26223–26238 Rivera AR, Castillo JR, Chae O (2015) Local directional texture pattern image descriptor. Pattern Recogn Lett 51:94–100 Rivera AR, Castillo JR, Chae OO (2012) Local directional number pattern for face analysis: face and expression recognition. IEEE Trans Image Process 22(5):1740–1752 Ryu B, Rivera AR, Kim J, Chae O (2017) Local directional ternary pattern for facial expression recognition. IEEE Trans Image Process 26(12):6006–6018 Sadeghi H, Raie AA (2019) Human vision inspired feature extraction for facial expression recognition. Multimed Tools Appl 78(21):30335–30353 Saurav S, Gidde P, Saini R, Singh S (2021) Dual integrated convolutional neural network for real-time facial expression recognition in the wild. Vis Comput 20:1–14 Shan C, Gong S, McOwan PW (2009) Facial expression recognition based on local binary patterns: a comprehensive study. Image Vis Comput 27(6):803–816 Shen S, Si S (2017) Facial expression recognition based on LGC in 5 \(\times \) 5 neighborhood. Intell Comput Appl 7:47–48 Shojaeilangari S, Yau WY, Nandakumar K, Li J, Teoh EK (2015) Robust representation and recognition of facial emotions using extreme sparse learning. IEEE Trans Image Process 24(7):2140–2152 Shojaeilangari S, Yau WY, Teoh EK (2016) Pose-invariant descriptor for facial emotion recognition. Mach Vis Appl 27(7):1063–1070 Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556 (arXiv preprint) Subramanian K, Radhakrishnan VB, Ramasamy S (2014) Database independent human emotion recognition with meta-cognitive neuro-fuzzy inference system. In: 2014 IEEE ninth international conference on intelligent sensors, sensor networks and information processing (ISSNIP), pp 1–6. IEEE Sun Z, Zp Hu, Wang M, Zhao S (2017) Individual-free representation-based classification for facial expression recognition. Signal Image Video Process 11(4):597–604 Sun Z, Hu Zp, Wang M, Zhao SH (2019) Dictionary learning feature space via sparse representation classification for facial expression recognition. Artif Intell Rev 51(1):1–18 Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D. Erhan D, Vanhoucke V, Rabinovich A (2015) Going deeper with convolutions. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1–9 Tong Y, Chen R (2019) Local dominant directional symmetrical coding patterns for facial expression recognition. Comput Intell Neurosci 2019:1–13 Tong Y, Chen R, Cheng Y (2014) Facial expression recognition algorithm using LGC based on horizontal and diagonal prior principle. Optik 125(16):4186–4189 Tuncer T, Dogan S, Abdar M, Plawiak P (2020) A novel facial image recognition method based on perceptual hash using quintet triple binary pattern. Multimed Tools Appl 1:1–21 Tuncer T, Dogan S, Ataman V (2019) A novel and accurate chess pattern for automated texture classification. Phys A Stat Mech Appl 536:122584 Turk M, Pentland A (1991) Face recognition using eigenfaces. In: Proceedings. 1991 IEEE computer society conference on computer vision and pattern recognition, pp 586–587 Van Der Schalk J, Hawk ST, Fischer AH, Doosje B (2011) Moving faces, looking places: validation of the Amsterdam dynamic facial expression set (ADFES). Emotion 11(4):907–920 Verma M, Vipparthi SK, Singh G (2019) Hinet: hybrid inherited feature learning network for facial expression recognition. IEEE Lett Comput Soc 2(4):36–39 Viola P, Jones MJ (2004) Robust real-time face detection. Int J Comput Vis 57(2):137–154 Vo DM, Le TH (2016) Deep generic features and SVM for facial expression recognition. In: 2016 3rd national foundation for science and technology development conference on information and computer science (NICS), pp 80–84. IEEE Wu BF, Lin CH (2018) Adaptive feature mapping for customizing deep learning based facial expression recognition model. IEEE Access 6:12451–12461 Xie S, Hu H, Wu Y (2019) Deep multi-path convolutional neural network joint with salient region attention for facial expression recognition. Pattern Recogn 92:177–191 Yan Y, Zhang Z, Chen S, Wang H (2020) Low-resolution facial expression recognition: a filter learning perspective. Signal Process 169:107370 Yang J, Wang X, Han S, Wang J, Park DS, Wang Y (2019) Improved real-time facial expression recognition based on a novel balanced and symmetric local gradient coding. Sensors 19(8):1899 Yang S, Bhanu B (2011) Facial expression recognition using emotion avatar image. In: Face and gesture 2011, pp 866–871. IEEE Zeng N, Li H, Peng Y (2021) A new deep belief network-based multi-task learning for diagnosis of Alzheimer’s disease. Neural Comput Appl 20:1–12 Zeng N, Wang Z, Zhang H, Kim KE, Li Y, Liu X (2019) An improved particle filter with a novel hybrid proposal distribution for quantitative analysis of gold immunochromatographic strips. IEEE Trans Nanotechnol 18:819–829 Zeng N, Zhang H, Song B, Liu W, Li Y, Dobaie AM (2018) Facial expression recognition via learning deep sparse autoencoders. Neurocomputing 273:643–649 Zhao G, Huang X, Taini M, Li SZ, PietikäInen M (2011) Facial expression recognition from near-infrared videos. Image Vis Comput 29(9):607–619 Zhao G, Pietikainen M (2007) Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Trans Pattern Anal Mach Intell 29(6):915–928 Zhao H, Liu Q, Yang Y (2018) Transfer learning with ensemble of multiple feature representations. In: 2018 IEEE 16th international conference on software engineering research, management and applications (SERA), pp 54–61. IEEE Zhou H, Wang R, Wang C (2008) A novel extended local-binary-pattern operator for texture analysis. Inf Sci 178(22):4314–4325