Hand Gesture Control for Human–Computer Interaction with Deep Learning
Tóm tắt
The use of gesture control has numerous advantages compared to the use of physical hardware. However, it has yet to gain popularity as most gesture control systems require extra sensors or depth cameras to detect or capture the movement of gestures before a meaningful signal can be triggered for corresponding course of action. This research proposes a method for a hand gesture control system with the use of an object detection algorithm, YOLOv3, combined with handcrafted rules to achieve dynamic gesture control on the computer. This project utilizes a single RGB camera for hand gesture recognition and localization. The dataset of all gestures used for training and its corresponding commands, are custom designed by the authors due to the lack of standard gestures specifically for human–computer interaction. Algorithms to integrate gesture commands with virtual mouse and keyboard input through the Pynput library in Python, were developed to handle commands such as mouse control, media control, and others. The mAP result of the YOLOv3 model obtained 96.68% accuracy based on testing result. The use of rule-based algorithms for gesture interpretation was successfully implemented to transform static gesture recognition into dynamic gesture.
Tài liệu tham khảo
Al-Shamayleh AS, Ahmad R, Abushariah MAM, Alam KA, Jomhari N (2018) A systematic literature review on vision based gesture recognition techniques. Multimed Tools Appl. https://doi.org/10.1007/s11042-018-5971-z
Anwar S, Sinha SK, Vivek S, Ashank V (2019). Hand gesture recognition: a survey. Lecture notes in electrical engineering. https://doi.org/10.1007/978-981-13-0776-8_33
Ayooshkathuria (2018) pytorch-yolo-v3. Github
Bai Y, Zhang L, Wang T, Zhou X (2019) A skeleton object detection-based dynamic gesture recognition method. In: Proceedings of the 2019 IEEE 16th international conference on networking, sensing and control, ICNSC 2019. https://doi.org/10.1109/ICNSC.2019.8743166
Beyer G, Meier M (2011) Music interfaces for novice users: composing music on a public display with hand gestures. In: Proceedings of the international conference on new interfaces for musical expression
Bochkovskiy A, Wang CY, Liao M (2020) YOLOv4: optimal speed and accuracy of object detection. https://arxiv.org/pdf/2004.10934v1.pdf
Bush IJ, Abiyev R, Arslan M (2019) Impact of machine learning techniques on hand gesture recognition. J Intell Fuzzy Syst. https://doi.org/10.3233/JIFS-190353
Camgoz NC, Hadfield S, Bowden R (2017) Particle filter based probabilistic forced alignment for continuous gesture recognition. In: Proceedings—2017 IEEE international conference on computer vision workshops, ICCVW 2017. https://doi.org/10.1109/ICCVW.2017.364
Chandrasekaran G, Periyasamy S, Panjappagounder Rajamanickam K (2020) Minimization of test time in system on chip using artificial intelligence-based test scheduling techniques. Neural Comput Appl. https://doi.org/10.1007/s00521-019-04039-6
Chai X, Liu Z, Yin F, Liu Z, Chen X (2016) Two streams recurrent neural networks for large-scale continuous gesture recognition. In: Proceedings—international conference on pattern recognition. https://doi.org/10.1109/ICPR.2016.7899603
Chen D, Li G, Sun Y, Kong J, Jiang G, Tang H, Ju Z, Yu H, Liu H (2017) An interactive image segmentation method in hand gesture recognition. Sensors (Switzerland). https://doi.org/10.3390/s17020253
Chua SND, Lim SF, Lai SN et al (2019) Development of a child detection system with artificial intelligence using object detection method. J Electr Eng Technol 14:2523–2529. https://doi.org/10.1007/s42835-019-00255-1
Everingham M, Van Gool L, Williams CKI, Winn J, Zisserman A (2010) The pascal visual object classes (VOC) challenge. Int J Comput Vis. https://doi.org/10.1007/s11263-009-0275-4
Flores CJL, Cutipa AEG, Enciso RL (2017) Application of convolutional neural networks for static hand gestures recognition under different invariant features. In: Proceedings of the 2017 IEEE 24th international congress on electronics, electrical engineering and computing, INTERCON 2017. https://doi.org/10.1109/INTERCON.2017.8079727
Geirhos R, Schütt HH, Medina Temme CR, Bethge M, Rauber J, Wichmann FA (2018) Generalisation in humans and deep neural networks. In: Advances in neural information processing systems
Huang H, Chong Y, Nie C, Pan S (2019) Hand gesture recognition with skin detection and deep learning method. J Phys Conf Ser. https://doi.org/10.1088/1742-6596/1213/2/022001
Islam MZ, Hossain MS, Ul Islam R, Andersson K (2019) Static hand gesture recognition using convolutional neural network with data augmentation. In: 2019 Joint 8th international conference on informatics, electronics and vision, ICIEV 2019 and 3rd international conference on imaging, vision and pattern recognition, IcIVPR 2019 with international conference on activity and behavior computing, ABC 2019. https://doi.org/10.1109/ICIEV.2019.8858563
Ji S, Xu W, Yang M, Yu K (2013) 3D Convolutional neural networks for human action recognition. IEEE Trans Pattern Anal Mach Intell. https://doi.org/10.1109/TPAMI.2012.59
Kim H, Albuquerque G, Havemann S, Fellner DW (2005) Tangible 3D: hand gesture interaction for immersive 3D modeling. In: 9th international workshop on immersive projection technology—11th Eurographics symposium on virtual environments, IPT/EGVE 2005
Kim S, Ji Y, Lee KB (2018) An effective sign language learning with object detection based ROI segmentation. In: Proceedings—2nd IEEE international conference on robotic computing, IRC 2018. https://doi.org/10.1109/IRC.2018.00069
Köpüklü O, Gunduz A, Kose N, Rigoll G (2019) Real-time hand gesture detection and classification using convolutional neural networks. In: Proceedings—14th IEEE international conference on automatic face and gesture recognition, FG 2019. https://doi.org/10.1109/FG.2019.8756576
Maqueda AI, Del-Blanco CR, Jaureguizar F, García N (2015) Human-computer interaction based on visual hand-gesture recognition using volumetric spatiograms of local binary patterns. Comput Vis Image Underst. https://doi.org/10.1016/j.cviu.2015.07.009
Molchanov P, Yang X, Gupta S, Kim K, Tyree S, Kautz J (2016) Online detection and classification of dynamic hand gestures with recurrent 3D convolutional neural networks. In: Proceedings of the IEEE computer society conference on computer vision and pattern recognition. https://doi.org/10.1109/CVPR.2016.456
Ni Z, Chen J, Sang N, Gao C, Liu L (2018) Light YOLO for high-speed gesture recognition. In: Proceedings—international conference on image processing, ICIP. https://doi.org/10.1109/ICIP.2018.8451766
Oyedotun OK, Khashman A (2017) Deep learning in vision-based static hand gesture recognition. Neural Comput Appl. https://doi.org/10.1007/s00521-016-2294-8
Rahmat RF, Chairunnisa T, Gunawan D, Pasha MF, Budiarto R (2019) Hand gestures recognition with improved skin color segmentation in human-computer interaction applications. J Theor Appl Inf Technol 97(3):727–739
Redmon J, Farhadi A (2018) YOLO v.3. Tech Report
Tzutalin (2015) LabelImg. LabelImg. https://github.com/tzutalin/labelImg
Walker A (2013) Voice commands or gesture recognition: how will we control the computers of the future? https://www.independent.co.uk/life-style/gadgets-and-tech/voice-commands-or-gesture-recognition-how-will-we-control-the-computers-of-the-future-8899614.html
Wan J, Li SZ, Zhao Y, Zhou S, Guyon I, Escalera S (2016) ChaLearn looking at people RGB-D isolated and continuous datasets for gesture recognition. In: IEEE computer society conference on computer vision and pattern recognition workshops. https://doi.org/10.1109/CVPRW.2016.100
Yang X, Tian Y (2014) Super normal vector for activity recognition using depth sequences. In: Proceedings of the IEEE computer society conference on computer vision and pattern recognition. https://doi.org/10.1109/CVPR.2014.108
Zhang Y, Cao C, Cheng J, Lu H (2018) EgoGesture: a new dataset and benchmark for egocentric hand gesture recognition. IEEE Trans Multimed. https://doi.org/10.1109/TMM.2018.2808769