A Survey on Human Performance Capture and Animation

Springer Science and Business Media LLC - Tập 32 - Trang 536-554 - 2017
Shihong Xia1, Lin Gao1, Yu-Kun Lai2, Ming-Ze Yuan1,3, Jinxiang Chai4
1Advanced Computing Research Laboratory, Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China
2School of Computer Science and Informatics, Cardiff University, Cardiff, U.K.
3University of Chinese Academy of Sciences, Beijing, China
4Department of Computer Science and Engineering, Texas A&M University, College Station, U.S.A.

Tóm tắt

With the rapid development of computing technology, three-dimensional (3D) human body models and their dynamic motions are widely used in the digital entertainment industry. Human performance mainly involves human body shapes and motions. Key research problems in human performance animation include how to capture and analyze static geometric appearance and dynamic movement of human bodies, and how to simulate human body motions with physical effects. In this survey, according to the main research directions of human body performance capture and animation, we summarize recent advances in key research topics, namely human body surface reconstruction, motion capture and synthesis, as well as physics-based motion simulation, and further discuss future research problems and directions. We hope this will be helpful for readers to have a comprehensive understanding of human performance capture and animation.

Tài liệu tham khảo

Robinette K M, Daanen H, Paquet E. The CAESAR project: A 3D surface anthropometry survey. In Proc. the 2nd International Conference on 3-D Digital Imaging and Modeling, Oct. 1999, pp.380-386.

Wang C C, Chang T K, Yuen M M. From laser-scanned data to feature human model: A system based on fuzzy logic concept. Computer-Aided Design, 2003, 35(3): 241-253.

Woodham R J. Shape from shading. In Shape from Shading, Horm B K P, Brooks M J (eds.), Cambridge, USA: MIT Press, 1989, pp.513-531.

Vlasic D, Peers P, Baran I, Debevec P, Popović J, Rusinkiewicz S, Matusik W. Dynamic shape capture using multi-view photometric stereo. ACM Transactions on Graphics, 2009, 28(5): Article No. 174.

Wu C, Varanasi K, Liu Y, Seidel H P, Theobalt C. Shading-based dynamic shape refinement from multi-view video under general illumination. In Proc. the IEEE International Conference on Computer Vision, Nov. 2011, pp.1108-1115.

Stoll C, Gall J, de Aguiar E, Thrun S, Theobalt C. Video-based reconstruction of animatable human characters. ACM Transactions on Graphics, 2010, 29(6): Article No. 139.

Zhu H, Liu Y, Fan J, Dai Q, Cao X. Video-based outdoor human reconstruction. IEEE Transactions on Circuits and Systems for Video Technology, 2016, 27(4): 760-770.

Newcombe R A, Izadi S, Hilliges O, Molyneaux D, Kim D, Davison A J, Kohi P, Shotton J, Hodges S, Fitzgibbon A. KinectFusion: Real-time dense surface mapping and tracking. In Proc. IEEE International Symposium on Mixed and Augmented Reality, Oct. 2011, pp.127-136.

Li H, Vouga E, Gudym A, Luo L, Barron J T, Gusev G. 3D self-portraits. ACM Transactions on Graphics, 2013, 32(6): Article No. 187.

Newcombe R A, Fox D, Seitz S M. DynamicFusion: Reconstruction and tracking of non-rigid scenes in real-time. In Proc. Conference on Computer Vision and Pattern Recognition, Jun. 2015, pp.343-352.

Dou M, Taylor J, Fuchs H, Fitzgibbon A, Izadi S. 3D scanning deformable objects with a single RGB-D sensor. In Proc. Conference on Computer Vision and Pattern Recognition, Jun. 2015, pp.493-501.

Butler D A, Izadi S, Hilliges O, Molyneaux D, Hodges S, Kim D. Shake‘n’sense: Reducing interference for overlapping structured light depth cameras. In Proc. the ACM Annual Conference on Human Factors in Computing Systems, May 2012, pp.1933-1936.

Tong J, Zhou J, Liu L, Pan Z, Yan H. Scanning 3D full human bodies using kinects. IEEE Transactions on Visualization and Computer Graphics, 2012, 18(4): 643-650.

Lin S, Chen Y, Lai Y K, Martin R R, Cheng Z Q. Fast capture of textured fullbody avatar with RGB-D cameras. The Visual Computer, 2016, 32(6/7/8): 681-691.

Ye G, Liu Y, Hasler N, Ji X, Dai Q, Theobalt C. Performance capture of interacting characters with handheld kinects. In Proc. the 12th European Conference on Computer Vision, Volume Part II, Oct. 2012, pp.828-841.

Wang C, Liu Y, Guo X, Zhong Z, Le B, Deng Z. Spectral animation compression. Journal of Computer Science and Technology, 2015, 30(3): 540-552.

Anguelov D, Srinivasan P, Koller D, Thrun S, Rodgers J, Davis J. SCAPE: Shape completion and animation of people. ACM Transactions on Graphics, 2005, 24(3): 408-416.

Weiss A, Hirshberg D, Black M J. Home 3D body scans from noisy image and range data. In Proc. the IEEE International Conference on Computer Vision, Nov. 2011, pp. 1951-1958.

Bogo F, Black M J, Loper M, Romero J. Detailed full-body reconstructions of moving people from monocular RGB-D sequences. In Proc. the IEEE International Conference on Computer Vision, Dec. 2015, pp.2300-2308.

Chen Y, Liu Z, Zhang Z. Tensor-based human body modeling. In Proc. Conference on Computer Vision and Pattern Recognition, Jun. 2013, pp.105-112.

Hasler N, Stoll C, Sunkel M, Rosenhahn B, Seidel H P. A statistical model of human pose and body shape. Computer Graphics Forum, 2009, 28(2): 337-346.

Cheng K L, Tong R F, Tang M, Qian J Y, Sarkis M. Parametric human body reconstruction based on sparse key points. IEEE Transactions on Visualization and Computer Graphics, 2016, 22(11): 2467-2479.

Zeng M, Zheng J, Cheng X, Liu X. Templateless quasi-rigid shape modeling with implicit loop-closure. In Proc. the Conference on Computer Vision and Pattern Recognition, Jun. 2013, pp.145-152.

Chen Y, Cheng Z Q, Lai C, Martin R R, Dang G. Real-time reconstruction of an animating human body from a single depth camera. IEEE Transactions on Visualization and Computer Graphics, 2016, 22(8): 2000-2011.

Guan P, Reiss L, Hirshberg D A, Weiss A, Black M J. DRAPE: DRessing any PErson. ACM Transactions on Graphics, 2012, 31(4): Article No. 35.

Tsoli A, Mahmood N, Black M J. Breathing life into shape: Capturing, modeling and animating 3D human breathing. ACM Transactions on Graphics, 2014, 33(4): Article No. 52.

Zheng J, Zeng M, Cheng X, Liu X. Scape-based human performance reconstruction. Computers & Graphics, 2014, 38: 191-198.

Ye M, Wang H, Deng N, Yang X, Yang R. Real-time human pose and shape estimation for virtual try-on using a single commodity depth camera. IEEE Transactions on Visualization and Computer Graphics, 2014, 20(4): 550-559.

Pons-Moll G, Romero J, Mahmood N, Black M J. Dyna: A model of dynamic human shape in motion. ACM Transactions on Graphics, 2015, 34(4): Article No. 120.

Zuffi S, Black M J. The stitched puppet: A graphical model of 3D human shape and pose. In Proc. Conference on Computer Vision and Pattern Recognition, Jun. 2015.

Bogo F, Romero J, Loper M, Black M J. FAUST: Dataset and evaluation for 3D mesh registration. In Proc. Conference on Computer Vision and Pattern Recognition, Jun. 2014, pp.3794-3801.

Bogo F, Romero J, Pons-Moll G, Black M. Dynamic faust: Registering human bodies in motion. In Proc. the Conference on Computer Vision and Pattern Recognition, July 2017.

Brigante C, Abbate N, Basile A, Faulisi A, Sessa S. Towards miniaturization of a mems-based wearable motion capture system. IEEE Transactions on Industrial Electronics, 2011, 58(8): 3234-3241.

Andrews S, Huerta I, Komura T, Sigal L, Mitchell K. Real-time physics-based motion capture with sparse sensors. In Proc. the 13th European Conference on Visual Media Production, Dec. 2016.

Hou J, Chau L P, Magnenat-Thalmann N, He Y. Human motion capture data tailored transform coding. IEEE Trans actions on Visualization and Computer Graphics, 2015, 21(7): 848-859.

Vlasic D, Baran I, Matusik W, Popović J. Articulated mesh animation from multiview silhouettes. ACM Transactions on Graphics, 2008, 27(3): Article No. 97.

Liu Y, Stoll C, Gall J, Seidel H P, Theobalt C. Marker-less motion capture of interacting characters using multi-view image segmentation. In Proc. the IEEE Conference on Computer Vision and Pattern Recognition, Jun. 2011, pp.1249-1256.

Hasler N, Rosenhahn B, Thormahlen T, Wand M, Gall J, Seidel H P. Markerless motion capture with unsynchronized moving cameras. In Proc. the IEEE Conference on Computer Vision and Pattern Recognition, Jun. 2009, pp.224-231.

Shiratori T, Park H S, Sigal L, Sheikh Y, Hodgins J K. Motion capture from bodymounted cameras. ACM Transactions on Graphics, 2011, 30(4): Article No. 31.

Elhayek A, Aguiar E, Jain A, Tompson J, Pishchulin L, Andriluka M, Bregler C, Schiele B, Theobalt C. Efficient convnetbased marker-less motion capture in general scenes with a low number of cameras. In Proc. the IEEE Conference on Computer Vision and Pattern Recognition, Jun. 2015, pp.3810-3818.

Ionescu C, Papava D, Olaru V, Sminchisescu C. Human3.6M: Large scale datasets and predictive methods for 3D human sensing in natural environments. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2014, 36(7): 1325-1339.

Sigal L, Balan A O, Black M J. HumanEva: Synchronized video and motion capture dataset and baseline algorithm for evaluation of articulated human motion. International Journal of Computer Vision, 2010, 87: 4-27.

Dantone M, Gall J, Leistner C, van Gool L. Body parts dependent joint regressors for human pose estimation in still images. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2014, 36(11): 2131-2143.

Wei S E, Ramakrishna V, Kanade T, Sheikh Y. Convolutional pose machines. In Proc. the IEEE Conference on Computer Vision and Pattern Recognition, Jun. 2016, pp. 4724-4732.

Wei X, Chai J. Videomocap: Modeling physically realistic human motion from monocular video sequences. ACM Transactions on Graphics, 2010, 29(4): Article No. 42.

Insafutdinov E, Pishchulin L, Andres B, Andriluka M, Schiele B. DeeperCut: A deeper, stronger, and faster multi-person pose estimation model. In Proc. the European Conference on Computer Vision, Oct. 2016, pp.34-50.

Andriluka M, Pishchulin L, Gehler P, Schiele B. 2D human pose estimation: New benchmark and state of the art analysis. In Proc. the IEEE Conference on Computer Vision and Pattern Recognition, Jun. 2014, pp.3686-3693.

Baak A, Müller M, Bharaj G, Seidel H P, Theobalt C. A data-driven approach for real-time full body pose reconstruction from a depth camera. In Consumer Depth Cameras for Computer Vision, Fossati A, Gall J, Grabrier H et al. (eds.), Springer, 2013, pp.71-98.

Ye M, Wang X, Yang R, Ren L, Pollefeys M. Accurate 3D pose estimation from a single depth image. In Proc. the International Conference on Computer Vision, Nov. 2011, pp. 731-738.

Liu Z, Zhou L, Leung H, Shum H P. Kinect posture reconstruction based on a local mixture of gaussian process models. IEEE Transactions on Visualization and Computer Graphics, 2016, 22(11): 2437-2450.

Wei X, Zhang P, Chai J. Accurate realtime full-body motion capture using a single depth camera. ACM Transactions on Graphics, 2012, 31(6): Article No. 188.

Zhang P, Siu K, Zhang J, Liu C K, Chai J. Leveraging depth cameras and wearable pressure sensors for fullbody kinematics and dynamics capture. ACM Transactions on Graphics, 2014, 33(6): Article No. 221.

von Marcard T, Ponsmoll G, Rosenhahn B. Human pose estimation from video and IMUs. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016, 38(8): 1533-1547.

Arikan O, Forsyth D. Interactive motion generation from examples. ACM Transactions on Graphics, 2002, 21(3): 483-490.

Kovar L, Gleicher M, Pighin F. Motion graphs. ACM Transactions on Graphics, 2002, 21(3): 473-482.

Ikemoto L, Forsyth D A. Enriching a motion collection by transplanting limbs. In Proc. the ACM SIGGRAPH/Eurographics Symposium on Computer Animation, Aug. 2004, pp. 99-108.

Gleicher M. Motion path editing. In Proc. the Symposium on Interactive 3D Graphics, Mar. 2001, pp.195-202.

Huang Y, Kallmann M. Planning motions and placements for virtual demonstrators. IEEE Transactions on Visualization and Computer Graphics, 2016, 22(5): 1568-1579.

Kim Y, Park H, Bang S, Lee S H. Retargeting human-object interaction to virtual avatars. IEEE Transactions on Visualization and Computer Graphics, 2016, 22(11): 2405-2412.

Mukai T, Kuriyama S. Geostatistical motion interpolation. ACM Transactions on Graphics, 2005, 24(3): 1062-1070.

Kovar L, Gleicher M. Automated extraction and parameterization of motions in large data sets. ACM Transactions on Graphics, 2004, 23(3): 559-568.

Wang H, Ho E S, Komura T. An energydriven motion planning method for two distant postures. IEEE Transactions on Visualization and Computer Graphics, 2015, 21(1): 18-30.

Tanco L M, Hilton A. Realistic synthesis of novel human movements from a database of motion capture examples. In Proc. Workshop on Human Motion, Dec. 2000, pp.137-142.

Pullen K, Bregler C. Animating by multilevel sampling. In Proc. Computer Animation, May 2000, pp.36-42.

Hsu E, Pulli K, Popović J. Style translation for human motion. ACM Transactions on Graphics, 2005, 24(3): 1082-1089.

Chai J, Hodgins J K. Constraint-based motion optimization using a statistical dynamic model. ACM Transactions on Graphics, 2007, 26(3): Article No. 8.

Lau M, Bar-Joseph Z, Kuffner J. Modeling spatial and temporal variation in motion data. ACM Transactions on Graphics, 2009, 28(5): Article No. 171.

Min J, Chai J. Motion graphs++: A compact generative model for semantic motion analysis and synthesis. ACM Transactions on Graphics, 2012, 31(6): Article No. 153.

Holden D, Saito J, Komura T, Joyce T. Learning motion manifolds with convolutional autoencoders. In Proc. SIG-GRAPH Asia 2015 Technical Briefs, Nov. 2015.

Holden D, Saito J, Komura T. A deep learning framework for character motion synthesis and editing. ACM Transactions on Graphics, 2016, 35(4): Article No. 138.

Ikemoto L, Arikan O, Forsyth D. Generalizing motion edits with Gaussian processes. ACM Transactions on Graphics, 2009, 28(1): Article No. 1.

Xia S, Wang C, Chai J, Hodgins J K. Realtime style transfer for unlabeled heterogeneous human motion. ACM Transactions on Graphics, 2015, 34(4): Article No. 119.

Brand M, Hertzmann A. Style machines. In Proc. the 27th Annual Conference on Computer Graphics and Interactive Techniques, Jul. 2000, pp.183-192.

Wang J M, Fleet D J, Hertzmann A. Multifactor Gaussian process models for style-content separation. In Proc. the 24th International Conference on Machine Learning, Jun. 2007, pp.975-982.

Min J, Liu H, Chai J. Synthesis and editing of personalized stylistic human motion. In Proc. the ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games, Feb. 2010, pp.39-46.

Ma W, Xia S, Hodgins J K, Yang X, Li C, Wang Z. Modeling style and variation in human motion. In Proc. the ACM SIGGRAPH/Eurographics Symposium on Computer Animation, Jul. 2010, pp.21-30.

Drumwright E. A fast and stable penalty method for rigid body simulation. IEEE Transactions on Visualization and Computer Graphics, 2008, 14(1): 231-240.

Wieber P B. On the stability of walking systems. In Proc. the International Workshop on Humanoid and Human Friendly Robotics, Dec. 2002.

Yin K, Loken K, Panne M. SIMBICON: Simple biped locomotion control. ACM Transactions on Graphics, 2007, 26(3): Article No. 105.

Coros S, Beaudoin P, Panne M. Generalized biped walking control. ACM Transactions on Graphics, 2010, 29(4): Article No. 130.

Wang J M, Fleet D J, Hertzmann A. Optimizing walking controllers. ACM Transactions on Graphics, 2009, 28(5): Article No. 168.

Wang J, Hamner S R, Delp S L, Koltun V. Optimizing locomotion controllers using biologically-based actuators and objectives. ACM Transactions on Graphics, 2012, 31(4): Article No. 25.

Liu L, Yin K, de Panne M V, Guo B. Terrain runner: Control, parameterization, composition, and planning for highly dynamic motions. ACM Transactions on Graphics, 2012, 31(6): Article No. 154.

Liu L, de Panne M V, Yin K. Guided learning of control graphs for physics-based characters. ACM Transactions on Graphics, 2016, 35(3): Article No. 29.

Zordan V B, Hodgins J K. Motion capture-driven simulations that hit and react. In Proc. the ACM SIGGRAPH/Eurographics Symposium on Computer Animation, Jul. 2002, pp.89-96.

Zordan V B, Majkowska A, Chiu B, Fast M. Dynamic response for motion capture animation. ACM Transactions on Graphics, 2005, 24(3): 697-701.

Sharon D, Panne M. Synthesis of controllers for stylized planar bipedal walking. In Proc. the IEEE International Conference on Robotics and Automation, Apr. 2005, pp. 2387-2392.

Sok K W, Kim M, Lee J. Simulating biped behaviors from human motion data. ACM Transactions on Graphics, 2007, 26(3): Article No. 107.

Silva M, Abe Y, Popović J. Interactive simulation of stylized human locomotion. ACM Transactions on Graphics, 2008, 27(3): Article No. 82.

Geijtenbeek T, Pronost N, Stappen A F. Simple data-driven control for simulated bipeds. In Proc. the ACM SIGGRAPH/Eurographics Symposium on Computer Animation, Jul. 2012, pp.211-219.

Hamalainen P, Rajamaki J, Liu C K. Online control of simulated humanoids using particle belief propagation. ACM Transactions on Graphics, 2015, 34(4): Article No. 81.

Agrawal S, de Panne M V. Taskbased locomotion. ACM Transactions on Graphics, 2016, 35(4): Article No. 82.

Yeadon M R. The simulation of aerial movement-II. A mathematical inertia model of the human body. Journal of Biomechanics, 1990, 23(1): 67-74.

Sheets A, Abrams G D, Corazza S, Safran M R, Andriacchi T P. Kinematics differences between the flat, kick, and slice serves measured using a markerless motion capture method. Annals of Biomedical Engineering, 2011, 39(12): 3011-3020.

Lv X, Chai J, Xia S. Data-driven inverse dynamics for human motion. ACM Transactions on Graphics, 2016, 35(6): Article No. 163.

Witkin A, Kass M. Spacetime constraints. ACM SIGGRAPH Computer Graphics, 1988, 22(4): 159-168.

Liu C K, Hertzmann A, Popović Z. Composition of complex optimal multicharacter motions. In Proc. the 2006 ACM SIGGRAPH/Eurographics Symposium on Computer Animation, Sept. 2006, pp.215-222.

Borno M A, de Lasa M, Hertzmann A. Trajectory optimization for full-body movements with complex contacts. IEEE Transactions on Visualization and Computer Graphics, 2013, 19(8): 1405-1414.

Park C, Park J S, Tonneau S, Mansard N, Multon F, Pettre J, Manocha D. Dynamically balanced and plausible trajectory planning for human-like characters. In Proc. the 20th ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games, Feb., 2016, pp.39-48.

Geijtenbeek T, Pronost N. Interactive character animation using simulated physics: A state-of-the-art review. Computer Graphics Forum, 2012, 31(8): 2492-2515.

Abe Y, da Silva M, Popović J. Multiobjective control with frictional contacts. In Proc. the ACM SIGGRAPH/Eurographics Symposium on Computer Animation, Aug. 2007, pp.249-258.

Wu C C, Zordan V. Goal-directed stepping with momentum control. In Proc. the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation, Jul. 2010, pp.113-118.

de Lasa M, Mordatch I, Hertzmann A. Feature-based locomotion controllers. ACM Transactions on Graphics, 2010, 29(4): Article No. 131.

Muico U, Lee Y, Popović J, Popović Z. Contact-aware nonlinear control of dynamic characters. ACM Transactions on Graphics, 2009, 28(3): Article No. 81.

Muico U, Popović J, Popović Z. Composite control of physically simulated characters. ACM Transactions on Graphics, 2011, 30(3): Article No. 16.

Wu J C, Popović Z. Terrain-adaptive bipedal locomotion control. ACM Transactions on Graphics, 2010, 29(4): Article No. 72.

Kwon T, Hodgins J. Control systems for human running using an inverted pendulum model and a reference motion capture sequence. In Proc. the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation, Jul. 2010, pp.129-138.

Mordatch I, de Lasa M, Hertzmann A. Robust physics-based locomotion using low-dimensional planning. ACM Transactions on Graphics, 2010, 29(4): Article No. 71.

Han D, Noh J, Jin X, Shin J S, Shin S Y. On-line real-time physics-based predictive motion control with balance recovery. Computer Graphics Forum, 2014, 33(2): 245-254.

Thông tin
Thông tin xuất bản

Springer Science and Business Media LLC

Tập 32

536-554

Thông tin tác giả