Modified chess patterns: handcrafted feature descriptors for facial expression recognition
Tóm tắt
Facial expressions are predominantly important in the social interaction as they convey the personal emotions of an individual. The main task in Facial Expression Recognition (FER) systems is to develop feature descriptors that could effectively classify the facial expressions into various categories. In this work, towards extracting distinctive features, Radial Cross Pattern (RCP), Chess Symmetric Pattern (CSP) and Radial Cross Symmetric Pattern (RCSP) feature descriptors have been proposed and are implemented in a 5
$$\times $$
5 overlapping neighborhood to overcome some of the limitations of the existing methods such as Chess Pattern (CP), Local Gradient Coding (LGC) and its variants. In a 5
$$\times $$
5 neighborhood, the 24 pixels surrounding the center pixel are arranged into two groups, namely Radial Cross Pattern (RCP), which extracts two feature values by comparing 16 pixels with the center pixel and Chess Symmetric Pattern (CSP) extracts one feature value from the remaining 8 pixels. The experiments are conducted using RCP and CSP independently and also with their fusion RCSP using different weights, on a variety of facial expression datasets to demonstrate the efficiency of the proposed methods. The results obtained from the experimental analysis demonstrate the efficiency of the proposed methods.
Tài liệu tham khảo
Aifanti N, Papachristou C, Delopoulos A (2010) The mug facial expression database. In: 11th international workshop on image analysis for multimedia interactive services WIAMIS 10, pp 1–4. IEEE
Alphonse AS, Shankar K, Rakkini MJ, Ananthakrishnan S, Athisayamani S, Singh AR, Gobi R (2020)A multi-scale and rotation-invariant phase pattern (MRIPP) and a stack of restricted Boltzmann machine (RBM) with preprocessing for facial expression classification. J Ambient Intell Human Comput 20: 1–17
Aneja D, Colburn A, Faigin G, Shapiro L, Mones B (2016) Modeling stylized character expressions via deep learning. In: Asian conference on computer vision. Springer, pp 136–153
Belhumeur PN, Hespanha JP, Kriegman DJ (1997) Eigenfaces vs fisherfaces: recognition using class specific linear projection. IEEE Trans Pattern Anal Mach Intell 19(7):711–720
Chan TH, Jia K, Gao S, Lu J, Zeng Z, Ma Y (2015) Pcanet: a simple deep learning baseline for image classification. IEEE Trans Image Process 24(12):5017–5032
Chen LF, Yen YS (2007) Taiwanese facial expression image database. Brain Mapping Laboratory, Institute of Brain Science, National Yang-Ming University, Taipei
Cootes TF, Taylor CJ, Cooper DH, Graham J (1995) Active shape models-their training and application. Comput Vis Image Underst 61(1):38–59
Donato G, Bartlett MS, Hager JC, Ekman P, Sejnowski TJ (1999) Classifying facial actions. IEEE Trans Pattern Anal Mach Intell 21(10):974–989
Ekmen B, Ekenel HK (2019) From 2d to 3d real-time expression transfer for facial animation. Multimed Tools Appl 78(9):12519–12535
Feutry C, Piantanida P, Bengio Y, Duhamel P (2018) Learning anonymized representations with adversarial neural networks. arXiv:1802.09386 (arXiv preprint)
Goeleven E, De Raedt R, Leyman L, Verschuere B (2008) The karolinska directed emotional faces: a validation study. Cogn Emot 22(6):1094–1118
He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778
Hong H, Neven H, Von der Malsburg C (1998) Online facial expression recognition based on personalized galleries. In: Proceedings third IEEE international conference on automatic face and gesture recognition, pp 354–359. IEEE
Howard AG, Zhu M, Chen B, Kalenichenko D, Wang W, Weyand T, Andreetto M, Adam H (2017) Mobilenets: efficient convolutional neural networks for mobile vision applications. arXiv:1704.04861 (arXiv preprint)
Huang Z, Song G, Zhao Y, Han J, Zhao X (2018) Smile recognition based on support vector machine and local binary pattern. In: 2018 IEEE 8th annual international conference on CYBER technology in automation, control, and intelligent systems (CYBER), pp 938–942. IEEE
Iqbal MTB, Abdullah-Al-Wadud M, Ryu B, Makhmudkhujaev F, Chae O (2018) Facial expression recognition with neighborhood-aware edge directional pattern (nedp). IEEE Trans Affect Comput 11(1):125–137
Jabid T, Kabir MH, Chae O (2010) Robust facial expression recognition based on local directional pattern. ETRI J 32(5):784–794
Ji Y, Hu Y, Yang Y, Shen F, Shen HT (2019) Cross-domain facial expression recognition via an intra-category common feature and inter-category distinction feature fusion network. Neurocomputing 333:231–239
Jung H, Lee S, Park S, Kim B, Kim J, Lee I, Ahn C (2015) Development of deep learning-based facial expression recognition system. In: 2015 21st Korea-Japan joint workshop on frontiers of computer vision (FCV), pp 1–4. IEEE
Jung H, Lee S, Park S, Lee I, Ahn C, Kim J (2015) Deep temporal appearance-geometry network for facial expression recognition. arXiv:1503.01532 (arXiv preprint)
Kartheek MN, Prasad MVNK, Bhukya R (2020) Local optimal oriented pattern for person independent facial expression recognition. In: Twelfth international conference on machine vision (ICMV 2019), vol. 11433. International Society for Optics and Photonics, p 114330R1–8
Kas M, Ruichek Y, Messoussi R et al (2020) Multi level directional cross binary patterns: new handcrafted descriptor for SVM-based texture classification. Eng Appl Artif Intell 94:103743
Kola DGR, Samayamantula SK (2020) A novel approach for facial expression recognition using local binary pattern with adaptive window. Multimed Tools Appl 20:1–20
Kola DGR, Samayamantula SK (2021) Facial expression recognition using singular values and wavelet-based LGC-HD operator. IET Biom 20:20
Kumar RJR, Sundaram M (2020) Arumugam N Facial emotion recognition using subband selective multilevel stationary wavelet gradient transform and fuzzy support vector machine. Vis Comput 20:1–15
Kumar YR, Narayanappa C, Dayananda P (2020) Weighted full binary tree-sliced binary pattern: an RGB-D image descriptor. Heliyon 6(5):e03751
Lai CC, Ko CH (2014) Facial expression recognition based on two-stage features extraction. Optik-Int J Light Electron Opt 125(22):6678–6680
Li H, Xu H (2020) Deep reinforcement learning for robust emotional classification in facial expression recognition. Knowl Based Syst 20:106172
Li S, Deng W (2018) Reliable crowdsourcing and deep locality-preserving learning for unconstrained facial expression recognition. IEEE Trans Image Process 28(1):356–370
Lucey P, Cohn JF, Kanade T, Saragih J, Ambadar Z, Matthews I (2010) The extended cohn-kanade dataset (ck+): a complete dataset for action unit and emotion-specified expression. In: 2010 IEEE computer society conference on computer vision and pattern recognition-workshops, pp 94–101. IEEE
Lyons M, Akamatsu S, Kamachi M, Gyoba J (1998) Coding facial expressions with gabor wavelets. In: Proceedings of third IEEE international conference on automatic face and gesture recognition, pp 200–205. IEEE
Maheswari VU, Varaprasad G, Raju SV (2020) Local directional maximum edge patterns for facial expression recognition. J Ambient Intell Human Comput 20:1–9
Makhmudkhujaev F, Abdullah-Al-Wadud M, Iqbal MTB, Ryu B, Chae O (2019) Facial expression recognition with local prominent directional pattern. Signal Process Image Commun 74:1–12
Makhmudkhujaev F, Iqbal MTB, Ryu B, Chae O (2019) Local directional-structural pattern for person-independent facial expression recognition. Turk J Electr Eng Comput Sci 27(1):516–531
Mandal M, Verma M, Mathur S, Vipparthi SK, Murala S, Kumar DK (2019) Regional adaptive affinitive patterns (RADAP) with logical operators for facial expression recognition. IET Image Proc 13(5):850–861
Minaee S, Abdolrashidi A (2019) Deep-emotion: facial expression recognition using attentional convolutional network. arXiv:1902.01019 (arXiv preprint)
Olszanowski M, Pochwatko G, Kuklinski K, Scibor-Rylski M, Lewinski P, Ohme RK (2015) Warsaw set of emotional facial expression pictures: a validation study of facial display photographs. Front Psychol 5:1–8
Reddy PCS, Rao PVP, Reddy PKK, Sridhar M (2019) Motif shape primitives on Fibonacci weighted neighborhood pattern for age classification. Soft computing and signal processing. Springer, Berlin, pp 273–280
Revina IM, Emmanuel WS (2019) MDTP: a novel multi-directional triangles pattern for face expression recognition. Multimed Tools Appl 78(18):26223–26238
Rivera AR, Castillo JR, Chae O (2015) Local directional texture pattern image descriptor. Pattern Recogn Lett 51:94–100
Rivera AR, Castillo JR, Chae OO (2012) Local directional number pattern for face analysis: face and expression recognition. IEEE Trans Image Process 22(5):1740–1752
Ryu B, Rivera AR, Kim J, Chae O (2017) Local directional ternary pattern for facial expression recognition. IEEE Trans Image Process 26(12):6006–6018
Sadeghi H, Raie AA (2019) Human vision inspired feature extraction for facial expression recognition. Multimed Tools Appl 78(21):30335–30353
Saurav S, Gidde P, Saini R, Singh S (2021) Dual integrated convolutional neural network for real-time facial expression recognition in the wild. Vis Comput 20:1–14
Shan C, Gong S, McOwan PW (2009) Facial expression recognition based on local binary patterns: a comprehensive study. Image Vis Comput 27(6):803–816
Shen S, Si S (2017) Facial expression recognition based on LGC in 5 \(\times \) 5 neighborhood. Intell Comput Appl 7:47–48
Shojaeilangari S, Yau WY, Nandakumar K, Li J, Teoh EK (2015) Robust representation and recognition of facial emotions using extreme sparse learning. IEEE Trans Image Process 24(7):2140–2152
Shojaeilangari S, Yau WY, Teoh EK (2016) Pose-invariant descriptor for facial emotion recognition. Mach Vis Appl 27(7):1063–1070
Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556 (arXiv preprint)
Subramanian K, Radhakrishnan VB, Ramasamy S (2014) Database independent human emotion recognition with meta-cognitive neuro-fuzzy inference system. In: 2014 IEEE ninth international conference on intelligent sensors, sensor networks and information processing (ISSNIP), pp 1–6. IEEE
Sun Z, Zp Hu, Wang M, Zhao S (2017) Individual-free representation-based classification for facial expression recognition. Signal Image Video Process 11(4):597–604
Sun Z, Hu Zp, Wang M, Zhao SH (2019) Dictionary learning feature space via sparse representation classification for facial expression recognition. Artif Intell Rev 51(1):1–18
Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D. Erhan D, Vanhoucke V, Rabinovich A (2015) Going deeper with convolutions. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1–9
Tong Y, Chen R (2019) Local dominant directional symmetrical coding patterns for facial expression recognition. Comput Intell Neurosci 2019:1–13
Tong Y, Chen R, Cheng Y (2014) Facial expression recognition algorithm using LGC based on horizontal and diagonal prior principle. Optik 125(16):4186–4189
Tuncer T, Dogan S, Abdar M, Plawiak P (2020) A novel facial image recognition method based on perceptual hash using quintet triple binary pattern. Multimed Tools Appl 1:1–21
Tuncer T, Dogan S, Ataman V (2019) A novel and accurate chess pattern for automated texture classification. Phys A Stat Mech Appl 536:122584
Turk M, Pentland A (1991) Face recognition using eigenfaces. In: Proceedings. 1991 IEEE computer society conference on computer vision and pattern recognition, pp 586–587
Van Der Schalk J, Hawk ST, Fischer AH, Doosje B (2011) Moving faces, looking places: validation of the Amsterdam dynamic facial expression set (ADFES). Emotion 11(4):907–920
Verma M, Vipparthi SK, Singh G (2019) Hinet: hybrid inherited feature learning network for facial expression recognition. IEEE Lett Comput Soc 2(4):36–39
Viola P, Jones MJ (2004) Robust real-time face detection. Int J Comput Vis 57(2):137–154
Vo DM, Le TH (2016) Deep generic features and SVM for facial expression recognition. In: 2016 3rd national foundation for science and technology development conference on information and computer science (NICS), pp 80–84. IEEE
Wu BF, Lin CH (2018) Adaptive feature mapping for customizing deep learning based facial expression recognition model. IEEE Access 6:12451–12461
Xie S, Hu H, Wu Y (2019) Deep multi-path convolutional neural network joint with salient region attention for facial expression recognition. Pattern Recogn 92:177–191
Yan Y, Zhang Z, Chen S, Wang H (2020) Low-resolution facial expression recognition: a filter learning perspective. Signal Process 169:107370
Yang J, Wang X, Han S, Wang J, Park DS, Wang Y (2019) Improved real-time facial expression recognition based on a novel balanced and symmetric local gradient coding. Sensors 19(8):1899
Yang S, Bhanu B (2011) Facial expression recognition using emotion avatar image. In: Face and gesture 2011, pp 866–871. IEEE
Zeng N, Li H, Peng Y (2021) A new deep belief network-based multi-task learning for diagnosis of Alzheimer’s disease. Neural Comput Appl 20:1–12
Zeng N, Wang Z, Zhang H, Kim KE, Li Y, Liu X (2019) An improved particle filter with a novel hybrid proposal distribution for quantitative analysis of gold immunochromatographic strips. IEEE Trans Nanotechnol 18:819–829
Zeng N, Zhang H, Song B, Liu W, Li Y, Dobaie AM (2018) Facial expression recognition via learning deep sparse autoencoders. Neurocomputing 273:643–649
Zhao G, Huang X, Taini M, Li SZ, PietikäInen M (2011) Facial expression recognition from near-infrared videos. Image Vis Comput 29(9):607–619
Zhao G, Pietikainen M (2007) Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Trans Pattern Anal Mach Intell 29(6):915–928
Zhao H, Liu Q, Yang Y (2018) Transfer learning with ensemble of multiple feature representations. In: 2018 IEEE 16th international conference on software engineering research, management and applications (SERA), pp 54–61. IEEE
Zhou H, Wang R, Wang C (2008) A novel extended local-binary-pattern operator for texture analysis. Inf Sci 178(22):4314–4325