Histogram-based local descriptors for facial expression recognition (FER): A comprehensive study

Liu, 2014, 143 S. Shojaeilangari, W.-Y. Yau, J. Li, and E.-K. Teoh, Feature extraction through binary pattern of phase congruency for facial expression recognition, Control Automation Robotics & Vision (ICARCV), 2012 12th International Conference on, IEEE, 2012, pp. 166–170. P. Liu, S. Han, Z. Meng, and Y. Tong, Facial expression recognition via a boosted deep belief network, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2014, pp. 1805–1812. M. Liu, S. Shan, R. Wang, and X. Chen, Learning expressionlets on spatio-temporal manifold for dynamic facial expression recognition, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2014, pp. 1749–1756. Ahmed, 2012, Gradient directional pattern: a robust feature descriptor for facial expression recognition, Electron. Lett., 48, 1203, 10.1049/el.2012.1841 W. Chu, Facial expression recognition based on local binary pattern and gradient directional pattern, Green Computing and Communications (GreenCom), 2013 IEEE and Internet of Things (iThings/CPSCom), IEEE International Conference on and IEEE Cyber, Physical and Social Computing, IEEE, 2013, pp. 1458–1462. Ekman, 1971, Constants across cultures in the face and emotion, J. Pers. Soc. Psychol., 17, 124, 10.1037/h0030377 Ekman, 2003, Darwin, deception, and facial expression, Ann. N. Y. Acad. Sci., 1000, 205, 10.1196/annals.1280.010 Islam, 2013, Gender classification using gradient direction pattern, Sci. Int., 25 M. Valstar, and M. Pantic, Fully automatic facial action unit detection and temporal analysis, Computer Vision and Pattern Recognition Workshop, 2006. CVPRW'06. Conference on, IEEE, 2006, pp. 149–150. R. Walecki, V. Pavlovic, B. Schuller, and M. Pantic, Deep Structured Learning for Facial Action Unit Intensity Estimation. arXiv preprint arXiv:1704.04481, 2017. H. Jung, S. Lee, S. Park, I. Lee, C. Ahn, and J. Kim, Deep temporal appearance-geometry network for facial expression recognition. arXiv preprint arXiv:1503.01532, 2015. Ahmed, 2013, Automated facial expression recognition using gradient-based ternary texture patterns, Chin. J. Eng., 2013, 10.1155/2013/831747 C. Turan, K.-M. Lam, and X. He, Facial expression recognition with emotion-based feature fusion, Signal and Information Processing Association Annual Summit and Conference (APSIPA), 2015 Asia-Pacific, IEEE, 2015, pp. 1–6. C. Turan, and K.-M. Lam, Region-based feature fusion for facial-expression recognition, Image Processing (ICIP), 2014 IEEE International Conference on, IEEE, 2014, pp. 5966–5970. Shan, 2009, Facial expression recognition based on local binary patterns: a comprehensive study, Image Vis. Comput., 27, 803, 10.1016/j.imavis.2008.08.005 A. Mollahosseini, D. Chan, and M.H. Mahoor, Going deeper in facial expression recognition using deep neural networks, Applications of Computer Vision (WACV), 2016 IEEE Winter Conference on, IEEE, 2016, pp. 1–10. Yang, 2016, A novel face recognition method based on iwld and iwbc, Multimedia Tools Appl., 75, 6979, 10.1007/s11042-015-2623-4 E. Vural, M. Cetin, A. Ercil, G. Littlewort, M. Bartlett, and J. Movellan, Drowsy driver detection through facial movement analysis. Human-computer interaction, 2007, pp. 6–18. Zhao, 2016, Peak-piloted deep network for facial expression recognition, 425 Islam, 2014, Facial expression recognition using local arc pattern, Trends Appl. Sci. Res., 9, 113, 10.3923/tasr.2014.113.120 Fong, 2003, A survey of socially interactive robots, Rob. Auton. Syst., 42, 143, 10.1016/S0921-8890(02)00372-X Benitez-Garcia, 2017, Facial expression recognition based on local fourier coefficients and facial fourier descriptors, J. Sign. Inform. Process., 8, 132, 10.4236/jsip.2017.83009 Ojala, 2002, Multiresolution gray-scale and rotation invariant texture classification with local binary patterns, IEEE Trans. Pattern Anal. Mach. Intell., 24, 971, 10.1109/TPAMI.2002.1017623 H. Ding, S.K. Zhou, and R. Chellappa, Facenet2expnet: Regularizing a deep face recognition net for expression recognition, in: Automatic Face & Gesture Recognition (FG 2017), 2017 12th IEEE International Conference on, IEEE, 2017, pp. 118–126. T. Jabid, M.H. Kabir, and O. Chae, Local directional pattern (LDP)–A robust image descriptor for object recognition, Advanced Video and Signal Based Surveillance (AVSS), in: 2010 Seventh IEEE International Conference on, IEEE, 2010, pp. 482–487. J.F. Cohn, T.S. Kruez, I. Matthews, Y. Yang, M.H. Nguyen, M.T. Padilla, F. Zhou, and F. De la Torre, Detecting depression from facial actions and vocal prosody, Affective Computing and Intelligent Interaction and Workshops, 2009, in: ACII 2009. 3rd International Conference on, IEEE, 2009, pp. 1–7. S. Jaiswal, M.F. Valstar, A. Gillott, and D. Daley, Automatic detection of ADHD and ASD from expressive behaviour in RGBD data, Automatic Face & Gesture Recognition (FG 2017), 2017 12th IEEE International Conference on, IEEE, 2017, pp. 762–769. Kirsh, 2007, Violent video game play impacts facial emotion recognition, Aggress. Behav., 33, 353, 10.1002/ab.20191 M.H. Kabir, T. Jabid, and O. Chae, A local directional pattern variance (LDPv) based face descriptor for human facial expression recognition, Advanced Video and Signal Based Surveillance (AVSS), 2010 Seventh IEEE International Conference on, IEEE, 2010, pp. 526–532. Fan, 2014, A novel local pattern descriptor—local vector pattern in high-order derivative space for face recognition, IEEE Trans. Image Process., 23, 2877, 10.1109/TIP.2014.2321495 Martinez, 2017, Automatic analysis of facial actions: a survey, IEEE Trans. Affective Comput. Rivera, 2013, Local directional number pattern for face analysis: Face and expression recognition, IEEE Trans. Image Process., 22, 1740, 10.1109/TIP.2012.2235848 Rivera, 2015, Local directional texture pattern image descriptor, Pattern Recogn. Lett., 51, 94, 10.1016/j.patrec.2014.08.012 S. Jaiswal, and M. Valstar, Deep learning the dynamic appearance and shape of facial action units, Applications of Computer Vision (WACV), 2016 IEEE Winter Conference on, IEEE, 2016, pp. 1–8. Tősér, 2016, Deep learning for facial action unit detection under large head poses, Computer Vision–ECCV, Workshops., Springer, 2016, 359 Z. Lei, T. Ahonen, M. Pietikäinen, and S.Z. Li, Local frequency descriptor for low-resolution face recognition, Automatic Face & Gesture Recognition and Workshops (FG 2011), 2011 IEEE International Conference on, IEEE, 2011, pp. 161–166. W. Zhang, S. Shan, W. Gao, X. Chen, and H. Zhang, Local gabor binary pattern histogram sequence (lgbphs): A novel non-statistical model for face representation and recognition, Computer Vision, 2005. ICCV 2005, in: Tenth IEEE International Conference on, IEEE, 2005, pp. 786–791. S.Z. Ishraque, A.H. Banna, and O. Chae, Local Gabor directional pattern for facial expression recognition, in: Computer and Information Technology (ICCIT), 2012 15th International Conference on, IEEE, 2012, pp. 164–167. Z. Lubing, and W. Han, Local gradient increasing pattern for facial expression recognition, in: Image Processing (ICIP), 2012 19th IEEE International Conference on, IEEE, 2012, pp. 2601–2604. Islam, 2014, Local gradient pattern-A novel feature representation for facial expression recognition, J. AI Data Min., 2, 33 Ahsan, 2013, Facial expression recognition using local transitional pattern on Gabor filtered facial images, IETE Tech. Rev., 30, 47, 10.4103/0256-4602.107339 Yang, 2012, Monogenic binary coding: an efficient local feature extraction approach to face recognition, IEEE Trans. Inf. Forensics Secur., 7, 1738, 10.1109/TIFS.2012.2217332 Li, 2016, Face recognition with Riesz binary pattern, Digital Signal Process., 51, 196, 10.1016/j.dsp.2016.02.003 T. Mohammad, and M.L. Ali, Robust facial expression recognition based on local monotonic pattern (LMP), in: Computer and Information Technology (ICCIT), 2011 14th International Conference on, IEEE, 2011, pp. 572–576. Khan, 2013, Framework for reliable, real-time facial expression recognition for low resolution images, Pattern Recogn. Lett., 34, 1159, 10.1016/j.patrec.2013.03.022 Song, 2017, LETRIST: locally encoded transform feature histogram for rotation-invariant texture classification, IEEE Trans. Circuits Syst. Video Technol. Ojansivu, 2008, Blur insensitive texture classification using local phase quantization, 236 A. Dhall, A. Asthana, R. Goecke, and T. Gedeon, Emotion recognition using PHOG and LPQ features, in: Automatic Face & Gesture Recognition and Workshops (FG 2011), 2011 IEEE International Conference on, IEEE, 2011, pp. 878–883. Zhao, 2007, 165 F. Bashar, A. Khan, F. Ahmed, and M.H. Kabir, Robust facial expression recognition based on median ternary pattern (MTP), in: Electrical Information and Communication Technology (EICT), 2013 International Conference on, IEEE, 2014, pp. 1–5. Nanni, 2010, Local binary patterns variants as texture descriptors for medical image analysis, Artif. Intell. Med., 49, 117, 10.1016/j.artmed.2010.02.006 N.P. Doshi, and G. Schaefer, A comprehensive benchmark of local binary pattern algorithms for texture retrieval, in: Pattern Recognition (ICPR), 2012 21st International Conference on, IEEE, 2012, pp. 2760–2763. Nanni, 2012, Survey on LBP based texture descriptors for image classification, Expert Syst. Appl., 39, 3634, 10.1016/j.eswa.2011.09.054 R.L. Kristensen, Z.-H. Tan, Z. Ma, and J. Guo, Binary pattern flavored feature extractors for Facial Expression Recognition: An overview, Information and Communication Technology, in: Electronics and Microelectronics (MIPRO), 2015 38th International Convention on, IEEE, 2015, pp. 1131–1137. Balntas, 2018, Binary online learned descriptors, IEEE Trans. Pattern Anal. Mach. Intell., 40, 555, 10.1109/TPAMI.2017.2679193 Lu, 2015, Learning compact binary face descriptor for face recognition, IEEE Trans. Pattern Anal. Mach. Intell., 37, 2041, 10.1109/TPAMI.2015.2408359 Duan, 2017, Learning rotation-invariant local binary descriptor, IEEE Trans. Image Process., 26, 3636 T. Jabid, and O. Chae, Local Transitional Pattern: A Robust Facial Image Descriptor for Automatic Facial Expression Recognition, in: Proc. International Conference on Computer Convergence Technology, Seoul, Korea, 2011, pp. 333–44. Jabid, 2012, Facial Expression Recognition Based on Local Transitional Pattern. International Information Institute (Tokyo), Information, 15, 2007 Xia, 2014, 437 Lu, 2015, Cost-sensitive local binary feature learning for facial age estimation, IEEE Trans. Image Process., 24, 5356, 10.1109/TIP.2015.2481327 Duan, 2017, Context-aware local binary feature learning for face recognition, IEEE Trans. Pattern Anal. Mach. Intell. Lu, 2017, Simultaneous local binary feature learning and encoding for homogeneous and heterogeneous face recognition, IEEE Trans. Pattern Anal. Mach. Intell. Liu, 2016, Median robust extended local binary pattern for texture classification, IEEE Trans. Image Process., 25, 1368, 10.1109/TIP.2016.2522378 A. Toshev and C. Szegedy, Deeppose: Human pose estimation via deep neural networks, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2014, pp. 1653–1660. F. Schroff, D. Kalenichenko, and J. Philbin, Facenet: A unified embedding for face recognition and clustering, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015, pp. 815–823. A. Bosch, A. Zisserman, and X. Munoz, Representing shape with a spatial pyramid kernel, in: Proceedings of the 6th ACM international conference on Image and video retrieval, ACM, 2007, pp. 401–408. Mansanet, 2016, Local deep neural networks for gender recognition, Pattern Recogn. Lett., 70, 80, 10.1016/j.patrec.2015.11.015 Li, 2013, Face recognition using Weber local descriptors, Neurocomputing, 122, 272, 10.1016/j.neucom.2013.05.038 S. Liu, Y. Zhang, and K. Liu, Facial expression recognition under partial occlusion based on Weber Local Descriptor histogram and decision fusion, in: Control Conference (CCC), 2014 33rd Chinese, IEEE, 2014, pp. 4664–4668. K. He, X. Zhang, S. Ren, and J. Sun, Deep residual learning for image recognition, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778. K. Simonyan, and A. Zisserman, Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014. Chakraborty, 2016, Local gradient hexa pattern: a descriptor for face recognition and retrieval, IEEE Trans. Circuits Syst. Video Technol. Zhang, 2007, Histogram of gabor phase patterns (hgpp): a novel object representation approach for face recognition, IEEE Trans. Image Process., 16, 57, 10.1109/TIP.2006.884956 Choi, 2008, Simplified Gabor wavelets for human face recognition, Pattern Recogn., 41, 1186, 10.1016/j.patcog.2007.07.025 Pong, 2014, Multi-resolution feature fusion for face recognition, Pattern Recogn., 47, 556, 10.1016/j.patcog.2013.08.023 Du, 2016, Local spiking pattern and its application to rotation-and illumination-invariant texture classification, Optik-Int. J. Light Electron Optics, 127, 6583, 10.1016/j.ijleo.2016.04.002 Verma, 2016, Local tri-directional patterns: a new texture feature descriptor for image retrieval, Digital Signal Process., 51, 62, 10.1016/j.dsp.2016.02.002 Fadaei, 2017, Local derivative radial patterns: a new texture descriptor for content-based image retrieval, Signal Process., 137, 274, 10.1016/j.sigpro.2017.02.013 Ahonen, 2006, Face description with local binary patterns: application to face recognition, IEEE Trans. Pattern Anal. Mach. Intell., 28, 2037, 10.1109/TPAMI.2006.244 Takala, 2005, Block-based methods for image retrieval using local binary patterns, Image Anal., 13 Liu, 2017, Local binary features for texture classification: taxonomy and experimental study, Pattern Recogn., 62, 135, 10.1016/j.patcog.2016.08.032 Zhang, 2010, Local derivative pattern versus local binary pattern: face recognition with high-order local pattern descriptor, IEEE Trans. Image Process., 19, 533, 10.1109/TIP.2009.2035882 N. Dalal, and B. Triggs, Histograms of oriented gradients for human detection, Computer Vision and Pattern Recognition, 2005. CVPR 2005. IEEE Computer Society Conference on, IEEE, 2005, pp. 886-893. Belhumeur, 1997, Eigenfaces vs. fisherfaces: Recognition using class specific linear projection, IEEE Trans. Pattern Anal. Mach. Intell., 19, 711, 10.1109/34.598228 Felsberg, 2001, The monogenic signal, IEEE Trans. Signal Process., 49, 3136, 10.1109/78.969520 L. Zhang, L. Zhang, Z. Guo, and D. Zhang, Monogenic-LBP: A new approach for rotation invariant texture classification, in: Image Processing (ICIP), 2010 17th IEEE International Conference on, IEEE, 2010, pp. 2677–2680. M. Yang, L. Zhang, L. Zhang, and D. Zhang, Monogenic binary pattern (MBP): A novel feature extraction and representation model for face recognition, Pattern Recognition (ICPR), 2010 20th International Conference on, IEEE, 2010, pp. 2680–2683. Y.-H. Oh, A.C. Le Ngo, J. See, S.-T. Liong, R.C.-W. Phan, and H.-C. Ling, Monogenic Riesz wavelet representation for micro-expression recognition, in: Digital Signal Processing (DSP), 2015 IEEE International Conference on, IEEE, 2015, pp. 1237–1241. Zeng, 2015, A new image retrieval model based on monogenic signal representation, J. Vis. Commun. Image Represent., 33, 85, 10.1016/j.jvcir.2015.08.014 Huang, 2012, Spatiotemporal local monogenic binary patterns for facial expression recognition, IEEE Signal Process Lett., 19, 243, 10.1109/LSP.2012.2188890 Erdem, 2015, BAUM-2: a multilingual audio-visual affective face database, Multimedia Tools Appl., 74, 7429, 10.1007/s11042-014-1986-2 T. Kanade, J.F. Cohn, and Y. Tian, Comprehensive database for facial expression analysis, Automatic Face and Gesture Recognition, 2000, in: Proceedings. Fourth IEEE International Conference on, IEEE, 2000, pp. 46–53. M. Lyons, S. Akamatsu, M. Kamachi, and J. Gyoba, Coding facial expressions with gabor wavelets, Automatic Face and Gesture Recognition, 1998. Proceedings. Third IEEE International Conference on, IEEE, 1998, pp. 200–205. Chen, 2007 C. Turan, K.-M. Lam, and X. He, Soft Locality Preserving Map (SLPM) for Facial Expression Recognition. arXiv preprint arXiv:1801.03754, 2018. D. McDuff, R. El Kaliouby, and R.W. Picard, Crowdsourcing facial responses to online videos, in: Affective Computing and Intelligent Interaction (ACII), 2015 International Conference on, IEEE, 2015, pp. 512–518. Douglas-Cowie, 2007, The HUMAINE database: addressing the collection and annotation of naturalistic and induced emotional data, Affective computing and intelligent interaction, 488, 10.1007/978-3-540-74889-2_43 A. Dhall, R. Goecke, S. Lucey, and T. Gedeon, Static facial expression analysis in tough conditions: Data, evaluation protocol and benchmark, in: Computer Vision Workshops (ICCV Workshops), 2011 IEEE International Conference on, IEEE, 2011, pp. 2106–2112. X. Peng, Z. Xia, L. Li, and X. Feng, Towards facial expression recognition in the wild: a new database and deep recognition system, Proceedings of the IEEE conference on computer vision and pattern recognition workshops, 2016, pp. 93–99. A. Mollahosseini, B. Hasani, and M.H. Mahoor, Affect Net: A Database for Facial Expression, Valence, and Arousal Computing in the Wild. arXiv preprint arXiv:1708.03985, 2017.