Online optical marker-based hand tracking with deep labels

ACM Transactions on Graphics - Tập 37 Số 4 - Trang 1-10 - 2018
Shangchen Han1, Beibei Liu1, Robert Wang1, Yuting Ye1, Christopher D. Twigg1, Kenrick Kin1
1Facebook Reality Labs

Tóm tắt

Optical marker-based motion capture is the dominant way for obtaining high-fidelity human body animation for special effects, movies, and video games. However, motion capture has seen limited application to the human hand due to the difficulty of automatically identifying (or labeling) identical markers on self-similar fingers. We propose a technique that frames the labeling problem as a keypoint regression problem conducive to a solution using convolutional neural networks. We demonstrate robustness of our labeling solution to occlusion, ghost markers, hand shape, and even motions involving two hands or handheld objects. Our technique is equally applicable to sparse or dense marker sets and can run in real-time to support interaction prototyping with high-fidelity hand tracking and hand presence in virtual reality.

Từ khóa


Tài liệu tham khảo

2018. OptiTrack Motion Capture Systems. (2018). https://www.optitrack.com 2018. OptiTrack Motion Capture Systems. (2018). https://www.optitrack.com

2018. Vicon Motion Systems. (2018). https://www.vicon.com/ 2018. Vicon Motion Systems. (2018). https://www.vicon.com/

10.1145/2994258.2994264

10.1016/j.cag.2017.10.001

10.1145/2933540.2933551

A. Aristidou and J. Lasenby . 2010. Motion capture with constrained inverse kinematics for real-time hand tracking . In 2010 4th International Symposium on Communications, Control and Signal Processing (ISCCSP). A. Aristidou and J. Lasenby. 2010. Motion capture with constrained inverse kinematics for real-time hand tracking. In 2010 4th International Symposium on Communications, Control and Signal Processing (ISCCSP).

10.1007/978-3-319-46478-7_44

Thomas H. Cormen , Charles E. Leiserson , Ronald L. Rivest , and Clifford Stein . 2009. Introduction to Algorithms , Third Edition (3 rd ed.). The MIT Press . Thomas H. Cormen, Charles E. Leiserson, Ronald L. Rivest, and Clifford Stein. 2009. Introduction to Algorithms, Third Edition (3rd ed.). The MIT Press.

Kaiming He , Georgia Gkioxari , Piotr Dollar , and Ross Girshick . 2017 . Mask R-CNN. In The IEEE International Conference on Computer Vision (ICCV). Kaiming He, Georgia Gkioxari, Piotr Dollar, and Ross Girshick. 2017. Mask R-CNN. In The IEEE International Conference on Computer Vision (ICCV).

10.1007/978-3-642-34710-8_23

M. Kitagawa and B. Windsor. 2008. MoCap for Artists: Workflow and Techniques for Motion Capture. Focal Press. M. Kitagawa and B. Windsor. 2008. MoCap for Artists: Workflow and Techniques for Motion Capture. Focal Press.

Alex Krizhevsky Ilya Sutskever and Geoffrey E Hinton. 2012. ImageNet Classification with Deep Convolutional Neural Networks. In Advances in Neural Information Processing Systems (NIPS). Alex Krizhevsky Ilya Sutskever and Geoffrey E Hinton. 2012. ImageNet Classification with Deep Convolutional Neural Networks. In Advances in Neural Information Processing Systems (NIPS).

Jonathan Maycock , Tobias Rohlig , Matthias SchrÃűder , Mario Botsch , and Helge J. Ritter . 2015. Fully automatic optical motion tracking using an inverse kinematics approach .. In 15th IEEE-RAS International Conference on Humanoid Robots. 461--466 . Jonathan Maycock, Tobias Rohlig, Matthias SchrÃűder, Mario Botsch, and Helge J. Ritter. 2015. Fully automatic optical motion tracking using an inverse kinematics approach.. In 15th IEEE-RAS International Conference on Humanoid Robots. 461--466.

J. Meyer , M. Kuderer , J. Müller , and W. Burgard . 2014. Online marker labeling for fully automatic skeleton tracking in optical motion capture . In 2014 IEEE International Conference on Robotics and Automation (ICRA). J. Meyer, M. Kuderer, J. Müller, and W. Burgard. 2014. Online marker labeling for fully automatic skeleton tracking in optical motion capture. In 2014 IEEE International Conference on Robotics and Automation (ICRA).

Iason Oikonomidis , Nikolaos Kyriazis , and Antonis A. Argyros . 2012 . Tracking the Articulated Motion of Two Strongly Interacting Hands. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Iason Oikonomidis, Nikolaos Kyriazis, and Antonis A. Argyros. 2012. Tracking the Articulated Motion of Two Strongly Interacting Hands. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

Shaoqing Ren Kaiming He Ross Girshick and Jian Sun. 2015. Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. In Advances in Neural Information Processing Systems (NIPS). Shaoqing Ren Kaiming He Ross Girshick and Jian Sun. 2015. Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. In Advances in Neural Information Processing Systems (NIPS).

10.1109/CVPR.2017.701

Maurice Ringer and Joan Lasenby . 2002 a. Multiple Hypothesis Tracking for Automatic Optical Motion Capture. In The 7th European Conference on Computer Vision (ECCV). Maurice Ringer and Joan Lasenby. 2002a. Multiple Hypothesis Tracking for Automatic Optical Motion Capture. In The 7th European Conference on Computer Vision (ECCV).

10.5244/C.16.73

10.1145/2822013.2822026

Matthias Schröder , Thomas Waltemate , Jonathan Maycock , Tobias RÃűhlig , Helge Ritter , and Mario Botsch . 2017. Design and evaluation of reduced marker layouts for hand motion capture. Computer Animation and Virtual Worlds ( 2017 ), e1751. Matthias Schröder, Thomas Waltemate, Jonathan Maycock, Tobias RÃűhlig, Helge Ritter, and Mario Botsch. 2017. Design and evaluation of reduced marker layouts for hand motion capture. Computer Animation and Virtual Worlds (2017), e1751.

10.1109/ICRA.2016.7487771

T. Schubert , A. Gkogkidis , T. Ball , and W. Burgard . 2015. Automatic initialization for skeleton tracking in optical motion capture . In 2015 IEEE International Conference on Robotics and Automation (ICRA). T. Schubert, A. Gkogkidis, T. Ball, and W. Burgard. 2015. Automatic initialization for skeleton tracking in optical motion capture. In 2015 IEEE International Conference on Robotics and Automation (ICRA).

10.5244/C.29.128

Tomas Simon , Hanbyul Joo , Iain Matthews , and Yaser Sheikh . 2017 . Hand Keypoint Detection in Single Images Using Multiview Bootstrapping. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Tomas Simon, Hanbyul Joo, Iain Matthews, and Yaser Sheikh. 2017. Hand Keypoint Detection in Single Images Using Multiview Bootstrapping. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

K. Simonyan and A. Zisserman . 2015. Very Deep Convolutional Networks for Large-Scale Image Recognition . In Proceedings of the International Conference on Learning Representations (ICLR). K. Simonyan and A. Zisserman. 2015. Very Deep Convolutional Networks for Large-Scale Image Recognition. In Proceedings of the International Conference on Learning Representations (ICLR).

10.1145/2897824.2925965

10.1145/3130800.3130853

10.1145/2980179.2980226

10.1145/2629500

10.1145/2522628.2522656

10.1111/cgf.12595

Yonghui Wu , Mike Schuster , Zhifeng Chen , Quoc V. Le , Mohammad Norouzi , Wolfgang Macherey , Maxim Krikun , Yuan Cao , Qin Gao , Klaus Macherey , Jeff Klingner , Apurva Shah , Melvin Johnson , Xiaobing Liu , Lukasz Kaiser , Stephan Gouws , Yoshikiyo Kato , Taku Kudo , Hideto Kazawa , Keith Stevens , George Kurian , Nishant Patil , Wei Wang , Cliff Young , Jason Smith , Jason Riesa , Alex Rudnick , Oriol Vinyals , Gregory S. Corrado , Macduff Hughes , and Jeffrey Dean . 2016. Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation. CoRR abs/1609.08144 ( 2016 ). Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Gregory S. Corrado, Macduff Hughes, and Jeffrey Dean. 2016. Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation. CoRR abs/1609.08144 (2016).

Thông tin
Thông tin xuất bản

ACM Transactions on Graphics

Tập 37 Số 4

1-10

Thông tin tác giả