Swapped face detection using deep learning and subjective assessment

EURASIP Journal on Information Security - 2020
Xinyi Ding1, Zohreh Raziei2, Eric C. Larson1, Eli V. Olinick2, Paul S. Krueger3, Michael Hahsler2
1Department of Computer Science, Southern Methodist University, Dallas, USA
2Department of Engineering Management, Information and Systems, Southern Methodist University, Dallas, USA
3Department of Mechanical Engineering, Southern Methodist University, Dallas, USA

Tóm tắt

AbstractThe tremendous success of deep learning for imaging applications has resulted in numerous beneficial advances. Unfortunately, this success has also been a catalyst for malicious uses such as photo-realistic face swapping of parties without consent. In this study, we use deep transfer learning for face swapping detection, showing true positive rates greater than 96% with very few false alarms. Distinguished from existing methods that only provide detection accuracy, we also provide uncertainty for each prediction, which is critical for trust in the deployment of such detection systems. Moreover, we provide a comparison to human subjects. To capture human recognition performance, we build a website to collect pairwise comparisons of images from human subjects. Based on these comparisons, we infer a consensus ranking from the image perceived as most real to the image perceived as most fake. Overall, the results show the effectiveness of our method. As part of this study, we create a novel dataset that is, to the best of our knowledge, the largest swapped face dataset created using still images. This dataset will be available for academic research use per request. Our goal of this study is to inspire more research in the field of image forensics through the creation of a dataset and initial analysis.

Từ khóa


Tài liệu tham khảo

I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, Y. Bengio, in Advances in Neural Information Processing Systems. Generative adversarial nets (Neural Information Processing SystemsMontreal, 2014), pp. 2672–2680.

Deepfakes porn has serious consequences - BBC News. https://www.bbc.com/news/technology-42912529. Accessed 28 May 2019.

J. Schmidhuber, Deep learning in neural networks: an overview. Neural Netw.61:, 85–117 (2015).

Y. Nirkin, I. Masi, A. T. Tuan, T. Hassner, G. Medioni, in 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018). On face segmentation, face swapping, and face perception (IEEE, 2018). https://doi.org/10.1109/fg.2018.00024.

GitHub - shaoanlu/faceswap-GAN: a denoising autoencoder + adversarial losses and attention mechanisms for face swapping. https://github.com/shaoanlu/faceswap-GAN. Accessed 11 May 2019.

G. Wang, W. Li, M. Aertsen, J. Deprest, S. Ourselin, T. Vercauteren, Aleatoric uncertainty estimation with test-time augmentation for medical image segmentation with convolutional neural networks. Neurocomputing. 338:, 34–45 (2019).

R. Heckel, M. Simchowitz, K. Ramchandran, M. J. Wainwright, Approximate ranking from pairwise comparisons. arXiv preprint arXiv:1801.01253 (2018).

V. Blanz, K. Scherbaum, T. Vetter, H. -P. Seidel, Exchanging faces in images. Comput. Graph. Forum. 23:, 669–676 (2004). https://doi.org/10.1111/j.1467-8659.2004.00799.x.

D. Bitouk, N. Kumar, S. Dhillon, P. Belhumeur, S. K. Nayar, in ACM Transactions on Graphics (TOG), vol. 27. Face swapping: automatically replacing faces in photographs (ACM, 2008), p. 39.

S. Mahajan, L. -J. Chen, T. -C. Tsai, in 2017 IEEE 31st International Conference on Advanced Information Networking and Applications (AINA). Swapitup: a face swap application for privacy protection (IEEE, 2017). https://doi.org/10.1109/aina.2017.53.

D. Chen, Q. Chen, J. Wu, X. Yu, T. Jia, Face swapping: realistic image synthesis based on facial landmarks alignment. Math. Probl. Eng.2019:, 1–1 (2019). https://doi.org/10.1155/2019/8902701.

P. Pérez, M. Gangnet, A. Blake, Poisson image editing. ACM Trans. Graph. (TOG). 22(3), 313–318 (2003).

I. Korshunova, W. Shi, J. Dambre, L. Theis, in 2017 IEEE International Conference on Computer Vision (ICCV). Fast face-swap using convolutional neural networks (IEEE, 2017). https://doi.org/10.1109/iccv.2017.397.

R. Natsume, T. Yatagawa, S. Morishima, Rsgan: face swapping and editing using face and hair representation in latent spaces. arXiv preprint arXiv:1804.03447 (2018).

K. Dale, K. Sunkavalli, M. K. Johnson, D. Vlasic, W. Matusik, H. Pfister, in ACM Transactions on Graphics (TOG), vol. 30. Video face replacement (ACM, 2011), p. 130.

Y. Zhang, L. Zheng, V. L. Thing, in 2017 IEEE 2nd International Conference on Signal and Image Processing (ICSIP). Automated face swapping and its detection (IEEE, 2017). https://doi.org/10.1109/siprocess.2017.8124497.

G. B. Huang, M. Ramesh, T. Berg, E. Learned-Miller, Labeled Faces in the Wild: a Database for Studying Face Recognition in Unconstrained Environments. Technical Report 07-49 (University of Massachusetts, Amherst, 2007).

H. Bay, T. Tuytelaars, L. Van Gool, in European Conference on Computer Vision. Surf: speeded up robust features (SpringerGraz,2006), pp. 404–417.

A. Khodabakhsh, R. Ramachandra, K. Raja, P. Wasnik, C. Busch, in 2018 International Conference of the Biometrics Special Interest Group (BIOSIG). Fake face detection methods: can they be generalized? (IEEE, 2018). https://doi.org/10.23919/biosig.2018.8553251.

A. Agarwal, R. Singh, M. Vatsa, A. Noore, in 2017 IEEE International Joint Conference on Biometrics (IJCB). Swapped! Digital face presentation attack detection via weighted local magnitude pattern (IEEE, 2017). https://doi.org/10.1109/btas.2017.8272754.

P. Korshunov, S. Marcel, Deepfakes: a new threat to face recognition? assessment and detection. arXiv preprint arXiv:1812.08685 (2018).

A. Rössler, D. Cozzolino, L. Verdoliva, C. Riess, J. Thies, M. Nießner, Faceforensics++: learning to detect manipulated facial images. arXiv preprint arXiv:1901.08971 (2019).

K. He, X. Zhang, S. Ren, J. Sun, in 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Deep residual learning for image recognition (IEEE, 2016). https://doi.org/10.1109/cvpr.2016.90.

J. Deng, W. Dong, R. Socher, L. -J. Li, K. Li, L. Fei-Fei, in 2009 IEEE Conference on Computer Vision and Pattern Recognition. Imagenet: a large-scale hierarchical image database (IEEE, 2009). https://doi.org/10.1109/cvpr.2009.5206848.

H. -C. Shin, H. R. Roth, M. Gao, L. Lu, Z. Xu, I. Nogues, J. Yao, D. Mollura, R. M. Summers, Deep convolutional neural networks for computer-aided detection: CNN architectures, dataset characteristics and transfer learning. IEEE Trans. Med. Imaging. 35(5), 1285–1298 (2016).

R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, D. Batra, in 2017 IEEE International Conference on Computer Vision (ICCV). Grad-cam: visual explanations from deep networks via gradient-based localization (IEEE, 2017). https://doi.org/10.1109/iccv.2017.74.

K. Guo, D. Tunnicliffe, H. Roebuck, Human spontaneous gaze patterns in viewing of faces of different species. Perception. 39(4), 533–542 (2010).

Thông tin
Thông tin xuất bản

EURASIP Journal on Information Security

Thông tin tác giả