Multi-scale joint feature network for micro-expression recognition
Tóm tắt
Micro-expression recognition is a substantive cross-study of psychology and computer science, and it has a wide range of applications (e.g., psychological and clinical diagnosis, emotional analysis, criminal investigation, etc.). However, the subtle and diverse changes in facial muscles make it difficult for existing methods to extract effective features, which limits the improvement of micro-expression recognition accuracy. Therefore, we propose a multi-scale joint feature network based on optical flow images for micro-expression recognition. First, we generate an optical flow image that reflects subtle facial motion information. The optical flow image is then fed into the multi-scale joint network for feature extraction and classification. The proposed joint feature module (JFM) integrates features from different layers, which is beneficial for the capture of micro-expression features with different amplitudes. To improve the recognition ability of the model, we also adopt a strategy for fusing the feature prediction results of the three JFMs with the backbone network. Our experimental results show that our method is superior to state-of-the-art methods on three benchmark datasets (SMIC, CASME II, and SAMM) and a combined dataset (3DB).
Tài liệu tham khảo
Haggard, E. A.; Isaacs, K. S. Micromomentary facial expressions as indicators of ego mechanisms in psychotherapy. In: Methods of Research in Psychotherapy. The Century Psychology Series. Boston: Springer, 154–165, 1966.
Ekman, P.; Friesen, W. V. Nonverbal leakage and clues to deception. Psychiatry Vol. 32, No. 1, 88–106, 1969.
Ekman, P. METT: Micro expression training tool. CDROM. Oakland, 2003.
Huang, X. H.; Zhao, G. Y.; Hong, X. P.; Zheng, W. M.; Pietikäinen, M. Spontaneous facial microexpression analysis using Spatiotemporal Completed Local Quantized Patterns. Neurocomputing Vol. 175, 564–578, 2016.
Polikovsky, S.; Kameda, Y.; Ohta, Y. Facial microexpressions recognition using high speed camera and 3D-gradient descriptor. In: Proceedings of the 3rd International Conference on Imaging for Crime Detection and Prevention, 1–6, 2009.
Zhao, G. Y.; Pietikainen, M. Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 29, No. 6, 915–928, 2007.
Liong, S. T.; See, J.; Wong, K.; Phan, R. C. W. Less is more: Micro-expression recognition from video using apex frame. Signal Processing: Image Communication Vol. 62, 82–92, 2018.
Liu, Y.-J.; Zhang, J.-K.; Yan, W.-J.; Wang, S.-J.; Zhao, G.; Fu, X. A main directional mean optical ow feature for spontaneous microexpression recognition. IEEE Transactions on Affective Computing Vol. 7, No. 4, 299–310, 2015.
Xu, F.; Zhang, J. P.; Wang, J. Z. Microexpression identification and categorization using a facial dynamics map. IEEE Transactions on Affective Computing Vol. 8, No. 2, 254–267, 2017.
Khor, H. Q.; See, J.; Phan, R. C. W.; Lin, W. Y. Enriched long-term recurrent convolutional network for facial micro-expression recognition. In: Proceedings of the 13th IEEE International Conference on Automatic Face & Gesture Recognition, 667–674, 2018.
Xia, Z. Q.; Hong, X. P.; Gao, X. Y.; Feng, X. Y.; Zhao, G. Y. Spatiotemporal recurrent convolutional networks for recognizing spontaneous micro-expressions. IEEE Transactions on Multimedia Vol. 22, No. 3, 626–640, 2019.
Gan, Y. S.; Liong, S. T.; Yau, W. C.; Huang, Y. C.; Tan, L. K. OFF-ApexNet on micro-expression recognition system. Signal Processing: Image Communication Vol. 74, 129–139, 2019.
Khor, H.-Q.; See, J.; Liong, S.-T.; Phan, R. C.; Lin, W. Dual-stream shallow networks for facial microexpression recognition. In: Proceedings of the IEEE International Conference on Image Processing, 36–40, 2019.
Shreve, M.; Godavarthy, S.; Manohar, V.; Goldgof, D.; Sarkar, S. Towards macro- and micro-expression spotting in video using strain patterns. In: Proceedings of the Workshop on Applications of Computer Vision, 1–6, 2009.
Li, X.; Pfister, T.; Huang, X.; Zhao, G.; Pietikäinen, M. A spontaneous micro-expression database: Inducement, collection and baseline. In: Proceedings of the 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition, 1–6, 2013.
Yan, W. J.; Qi, W.; Liu, Y. J.; Wang, S. J.; Fu, X. sL. CASME database: A dataset of spontaneous micro-expressions collected from neutralized faces. In: Proceedings of the 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition, 1–7, 2013.
Yan, W. J.; Li, X.; Wang, S. J.; Zhao, G.; Liu, Y. J.; Chen, Y. H.; Fu, X. CASME II: An improved spontaneous micro-expression database and the baseline evaluation. PLoS One Vol. 9, No. 1, e86041, 2014.
Davison, A. K.; Lansley, C.; Costen, N.; Tan, K.; Yap, M. H. SAMM: A spontaneous micro-facial movement dataset. IEEE Transactions on Affective Computing Vol. 9, No. 1, 116–129, 2016.
Huang, X.; Zhao, G.; Hong, X.; Pietikäinen, M.; Zheng, W. Texture description with completed local quantized patterns. In: Image Analysis. Lecture Notes in Computer Science, Vol. 7944. Kämäräinen, J. K.; Koskela, M. Eds. Springer Berlin Heidelberg, 1–10, 2013.
Wang, Y.; See, J.; Phan, R. C. W.; Oh, Y. H. LBP with six intersection points: Reducing redundant information in LBP-TOP for microexpression recognition. In: Computer Vision — ACCV 2014. Lecture Notes in Computer Science, Vol. 9003. Cremers, D.; Reid, I.; Saito, H.; Yang, M. H. Eds. Springer Cham, 525–537, 2015.
Ben, X. Y.; Jia, X. T.; Yan, R.; Zhang, X.; Meng, W. X. Learning effective binary descriptors for micro-expression recognition transferred by macroinformation. Pattern Recognition Letters Vol. 107, 50–58, 2018.
Wang, S.-J.; Yan, W.-J.; Li, X.; Zhao, G.; Fu, X. Microexpression recognition using dynamic textures on tensor independent color space. In: Proceedings of the 22nd International Conference on Pattern Recognition, 4678–4683, 2014.
Huang, X.; Wang, S.-J.; Liu, X.; Zhao, G.; Feng, X.; Pietikäinen, M. Discriminative spatiotemporal local binary pattern with revisited integral projection for spontaneous facial micro-expression recognition. IEEE Transactions on Affective Computing Vol. 10, No. 1, 32–47, 2017.
Sun, D.; Roth, S.; Black, M. J. A quantitative analysis of current practices in optical ow estimation and the principles behind them. International Journal of Computer Vision Vol. 106, No. 2, 115–137, 2014.
Zach, C.; Pock, T.; Bischof, H. A duality based approach for realtime TV-L1 optical flow. In: Pattern Recognition. Lecture Notes in Computer Science, Vol. 4713. Hamprecht, F. A.; Schnörr, C.; Jähne, B. Eds. Springer Berlin Heidelberg, 214–223, 2007.
Liu, Y. J.; Li, B. J.; Lai, Y. K. Sparse MDMO: Learning a discriminative feature for micro-expression recognition. IEEE Transactions on Affective Computing Vol. 12, No. 1, 254–261, 2021.
Peng, S.; Huang, H. B.; Chen, W. J.; Zhang, L.; Fang, W. W. More trainable inception-ResNet for face recognition. Neurocomputing Vol. 411, 9–19, 2020.
Wang, S.; Cheng, Z.; Deng, X.; Chang, L.; Duan, F.; Lu, K. Leveraging 3D blendshape for facial expression recognition using CNN. Science China Information Sciences Vol. 63, No. 2, 120114, 2020.
Kim, D. H.; Baddar, W. J.; Ro, Y. M. Micro-expression recognition with expression-state constrained spatiotemporal feature representations. In: Proceedings of the 24th ACM international Conference on Multimedia, 382–386, 2016.
Peng, M.; Wu, Z.; Zhang, Z.; Chen, T. From macro to micro expression recognition: Deep learning on small datasets using transfer learning. In: Proceedings of the 13th IEEE International Conference on Automatic Face & Gesture Recognition, 657–661, 2018.
Wang, C. Y.; Peng, M.; Bi, T.; Chen, T. Micro-attention for micro-expression recognition. Neurocomputing Vol. 410, 354–362, 2020.
Wang, S. J.; Li, B. J.; Liu, Y. J.; Yan, W. J.; Ou, X. Y.; Huang, X. H.; Fu, X. Micro-expression recognition with small sample size by transferring longterm convolutional neural network. Neurocomputing Vol. 312, 251–262, 2018.
Wu, H. Y.; Rubinstein, M.; Shih, E.; Guttag, J.; Durand, F.; Freeman, W. Eulerian video magnification for revealing subtle changes in the world. ACM Transactions on Graphics Vol. 31, No. 4, Article No. 65, 2012.
Quang, N. V.; Chun, J.; Tokuyama, T. CapsuleNet for micro-expression recognition. In: Proceedings of the 14th IEEE International Conference on Automatic Face & Gesture Recognition, 1–7, 2019.
Peng, M.; Wang, C.; Bi, T.; Shi, Y.; Zhou, X.; Chen, T. A novel apex-time network for cross-dataset microexpression recognition. In: Proceedings of the 8th International Conference on Affective Computing and Intelligent Interaction, 1–6, 2019.
Liu, Y. C.; Du, H. M.; Zheng, L.; Gedeon, T. A neural micro-expression recognizer. In: Proceedings of the 14th IEEE International Conference on Automatic Face & Gesture Recognition, 1–4, 2019.
Zhou, L.; Mao, Q. R.; Xue, L. Y. Dual-inception network for cross-database micro-expression recognition. In: Proceedings of the 14th IEEE International Conference on Automatic Face & Gesture Recognition, 1–5, 2019
Liong, S. T.; Gan, Y. S.; See, J.; Khor, H. Q.; Huang, Y. C. Shallow triple stream three-dimensional CNN (STSTNet) for micro-expression recognition. In: Proceedings of the 14th IEEE International Conference on Automatic Face & Gesture Recognition, 1–5, 2019.
Li, Y. T.; Huang, X. H.; Zhao, G. Y. Can microexpression be recognized based on single apex frame? In: Proceedings of the 25th IEEE International Conference on Image Processing, 3094–3098, 2018.
See, J.; Yap, M. H.; Li, J.; Hong, X.; Wang, S.-J. MEGC 2019 — The second facial micro-expressions grand challenge.In: Proceedings of the 14th IEEE International Conference on Automatic Face & Gesture Recognition, 1–5, 2019.
Krizhevsky, A.; Sutskever, I.; Hinton, G. E. ImageNet classification with deep convolutional neural networks. Communications of the ACM Vol. 60, No. 6, 84–90, 2017.
Le Ngo, A. C.; Phan, R. C. W.; See, J. Spontaneous subtle expression recognition: Imbalanced databases and solutions. In: Computer Vision — ACCV 2014. Lecture Notes in Computer Science, Vol. 9006. Cremers, D.; Reid, I.; Saito, H.; Yang, M. H. Eds. Springer Cham, 33–48, 2015