Customizing blendshapes to capture facial details

Springer Science and Business Media LLC - Tập 79 - Trang 6347-6372 - 2022
Ju Hee Han1, Jee In Kim1, Jang Won Suh2, Hyungseok Kim1
1Graduate School of Computer Science, Konkuk University, Seoul, South Korea
2Embedded Intelligence Lab., Ellexi, Seoul, South Korea

Tóm tắt

Blendshape technique is an effective tool in the computer facial animation. Every character requires its own unique blendshapes to cover numerous facial expressions in the Visual Effects industry. Despite outstanding advances in this area, existing techniques still need a professional artist’s intuition and complex hardware. In this paper, we propose a framework for customizing blendshapes to capture facial details. The suggested method primarily consists of two stages: Blendshape generation and Blendshape augmentation. In the first stage, localized blendshapes are automatically generated from real-time captured faces with two methods: linear regression and an autoencoder Han (in: IEEE International Conference on Big Data and Smart Computing (BigComp) 2021) (2021). In our experiment, face construction with the former outperforms that of the later method. However, generated blendshapes are slightly missing the source features, especially mouth movements. To overcome this, in the last stage, we extend Han (in: IEEE International Conference on Big Data and Smart Computing (BigComp) 2021), (2021) by adding a blendshape incrementally to minimize erroneous expression transfer.

Tài liệu tham khảo

Anjyo K, Todo H, Lewis JP (2012) A practical approach to direct manipulation blendshapes. J Graph Tools 16(3):160–176 Berson E, Soladie C, Barrielle V, Stoiber N (2019) A robust interactive facial animation editing system. Motion Interaction Games, pp 1–10 Bouaziz S, Wang Y, Pauly M (2013) Online modeling for realtime facial animation. ACM Trans Graph (ToG) 32(4):1–10 Cao C, Weng Y, Lin S, Zhou K (2013) 3d shape regression for real-time facial animation. ACM Trans Graphics (TOG) 32(4):1–10 Casas D, Feng A, Alexander O, Fyffe G, Debevec P, Ichikari R, Li H, Olszewski K, Suma E, Shapiro A (2016) Rapid photorealistic blendshape modeling from rgb-d sensors. In: Proceedings of the 29th International Conference on Computer Animation and Social Agents, pp 121–129 Cetinaslan O, Orvalho V (2020) Sketching manipulators for localized blendshape editing. Graph Mod 108:101059 Cetinaslan O, Orvalho V (2020) Stabilized blendshape editing using localized Jacobian transpose descent. Graph Mod 112:101091 Chaudhuri B, Vesdapunt N, Shapiro L, Wang B (2020) Personalized face modeling for improved face reconstruction and motion retargeting. In: European Conference on Computer Vision, pp 142–160. Springer Chen K, Zheng J, Cai J, Zhang J (2020) Modeling caricature expressions by 3d blendshape and dynamic texture. arXiv preprint arXiv:2008.05714 Chheang V, Jeong S, Lee G, Ha JS, Yoo KH (2020) Natural embedding of live actors and entities into 360 virtual reality scenes. J Supercomput 76(7):5655–5677 Cong M, Fedkiw R (2019) Muscle-based facial retargeting with anatomical constraints. In: ACM SIGGRAPH 2019 Talks, pp 1–2 Costigan T, Gerdelan A, Carrigan E, McDonnell R (2016) Improving blendshape performance for crowds with gpu and gpgpu techniques. In: Proceedings of the 9th International Conference on Motion in Games, pp 73–78 Costigan T, Prasad M, McDonnell R (2014) Facial retargeting using neural networks. In: Proceedings of the Seventh International Conference on Motion in Games, pp 31–38 Ding C, Tao D (2015) Robust face recognition via multimodal deep face representation. IEEE Trans Multimedia 17(11):2049–2058 Han JH, Kim JI, Kim H, Suh JW (2021) Generate individually optimized blendshapes. In: 2021 IEEE International Conference on Big Data and Smart Computing (BigComp), pp 114–120. IEEE Hui Q (2019) Motion video tracking technology in sports training based on mean-shift algorithm. J Supercomput 75(9):6021–6037 Joshi P, Tien WC, Desbrun M, Pighin F (2006) Learning controls for blend shape based realistic facial animation. In: ACM Siggraph 2006 Courses, pp 17–es Kan M, Shan S, Chang H, Chen X (2014) Stacked progressive auto-encoders (spae) for face recognition across poses. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1883–1890 Kang J, Lee S (2020) A greedy pursuit approach for fitting 3d facial expression models. IEEE Access 8:192682–192692 Kim PH, Seol Y, Song J, Noh J (2011) Facial retargeting by adding supplemental blendshapes. In: PG (Short Papers) Kingma DP, Welling M (2013) Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 Kommineni J, Mandala S, Sunar MS, Chakravarthy PM (2021) Accurate computing of facial expression recognition using a hybrid feature extraction technique. J Supercomput 77(5):5019–5044 Lewis JP, Anjyo K, Rhee T, Zhang M, Pighin FH, Deng Z (2014) Practice and theory of blendshape facial models. Eurographics (State of the Art Reports) 1(8):2 Lewis JP, Anjyo KI (2010) Direct manipulation blendshapes. IEEE Comput Graph Appl 30(4):42–50 Li J, Kuang Z, Zhao Y, He M, Bladin K, Li H (2020) Dynamic facial asset and rig generation from a single scan. ACM Trans. Graph. 39(6):215–1 Li Q, Deng Z (2008) Orthogonal-blendshape-based editing system for facial motion capture data. IEEE Comput Graph Appl 28(6):76–82 Lombardi S, Saragih J, Simon T, Sheikh Y (2018) Deep appearance models for face rendering. ACM Transactions on Graphics (TOG) 37(4):1–13 Orvalho V, Bastos P, Parke FI, Oliveira B, Alvarez X (2012) A facial rigging survey. Eurographics (State of the Art Reports) pp 183–204 Parent R (2012) Computer animation: algorithms and techniques. Newnes Parke FI (1972) Computer generated animation of faces. In: Proceedings of the ACM annual conference-Volume 1, pp 451–457 Parke FI (1974) A parametric model for human faces. The University of Utah Parke FI, Waters K (2008) Computer facial animation. CRC press Pighin F, Hecker J, Lischinski D, Szeliski R, Salesin DH (2006) Synthesizing realistic facial expressions from photographs. In: ACM SIGGRAPH 2006 Courses, pp 19–es Pighin F, Lewis JP (2006) Facial motion retargeting. In: ACM SIGGRAPH 2006 Courses, pp 2–es Rao S, Ortiz-Cayon R, Munaro M, Liaudanskas A, Chande K, Bertel T, Richardt C, JB A, Holzer S, Kar A (2020) Free-viewpoint facial re-enactment from a casual capture. In: SIGGRAPH Asia 2020 Posters, pp 1–2 Ribera RBI, Zell E, Lewis JP, Noh J, Botsch M (2017) Facial retargeting with automatic range of motion alignment. ACM Trans Graph (TOG) 36(4):1–12 Seo J, Irving G, Lewis JP, Noh J (2011) Compression and direct manipulation of complex blendshape models. ACM Trans Graph (TOG) 30(6):1–10 Seol Y, Lewis JP, Seo J, Choi B, Anjyo K, Noh J (2012) Spacetime expression cloning for blendshapes. ACM Trans Graph (TOG) 31(2):1–12 Seol Y, Ma WC, Lewis J (2016) Creating an actor-specific facial rig from performance capture. In: Proceedings of the 2016 Symposium on Digital Production, pp 13–17 Sumner RW, Popović J (2004) Deformation transfer for triangle meshes. ACM Trans Graph (TOG) 23(3):399–405 Thomas D, Taniguchi RI (2016) Augmented blendshapes for real-time simultaneous 3d head modeling and facial motion capture. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 3299–3308 Toshpulatov M, Lee W, Lee S (2021) Generative adversarial networks and their application to 3d face generation: a survey. Image Vis Comput, p 104119 Wang S, Cheng Z, Deng X, Chang L, Duan F, Lu K (2020) Leveraging 3d blendshape for facial expression recognition using cnn. Sci China Inf Sci 63(120114):1–120114 Zhang J, Chen K, Zheng J (2020) Facial expression retargeting from human to avatar made easy. IEEE Trans Vis Comput Graph