Hand Gesture Control for Human–Computer Interaction with Deep Learning

Journal of Electrical Engineering & Technology - Tập 17 - Trang 1961-1970 - 2022
S. N. David Chua1, K. Y. Richard Chin1, S. F. Lim1, Pushpdant Jain2
1Faculty of Engineering, Universiti Malaysia Sarawak, Kota Samarahan, Malaysia
2School of Mechanical Engineering, VIT Bhopal University, Sehore, India

Tóm tắt

The use of gesture control has numerous advantages compared to the use of physical hardware. However, it has yet to gain popularity as most gesture control systems require extra sensors or depth cameras to detect or capture the movement of gestures before a meaningful signal can be triggered for corresponding course of action. This research proposes a method for a hand gesture control system with the use of an object detection algorithm, YOLOv3, combined with handcrafted rules to achieve dynamic gesture control on the computer. This project utilizes a single RGB camera for hand gesture recognition and localization. The dataset of all gestures used for training and its corresponding commands, are custom designed by the authors due to the lack of standard gestures specifically for human–computer interaction. Algorithms to integrate gesture commands with virtual mouse and keyboard input through the Pynput library in Python, were developed to handle commands such as mouse control, media control, and others. The mAP result of the YOLOv3 model obtained 96.68% accuracy based on testing result. The use of rule-based algorithms for gesture interpretation was successfully implemented to transform static gesture recognition into dynamic gesture.

Tài liệu tham khảo

Al-Shamayleh AS, Ahmad R, Abushariah MAM, Alam KA, Jomhari N (2018) A systematic literature review on vision based gesture recognition techniques. Multimed Tools Appl. https://doi.org/10.1007/s11042-018-5971-z Anwar S, Sinha SK, Vivek S, Ashank V (2019). Hand gesture recognition: a survey. Lecture notes in electrical engineering. https://doi.org/10.1007/978-981-13-0776-8_33 Ayooshkathuria (2018) pytorch-yolo-v3. Github Bai Y, Zhang L, Wang T, Zhou X (2019) A skeleton object detection-based dynamic gesture recognition method. In: Proceedings of the 2019 IEEE 16th international conference on networking, sensing and control, ICNSC 2019. https://doi.org/10.1109/ICNSC.2019.8743166 Beyer G, Meier M (2011) Music interfaces for novice users: composing music on a public display with hand gestures. In: Proceedings of the international conference on new interfaces for musical expression Bochkovskiy A, Wang CY, Liao M (2020) YOLOv4: optimal speed and accuracy of object detection. https://arxiv.org/pdf/2004.10934v1.pdf Bush IJ, Abiyev R, Arslan M (2019) Impact of machine learning techniques on hand gesture recognition. J Intell Fuzzy Syst. https://doi.org/10.3233/JIFS-190353 Camgoz NC, Hadfield S, Bowden R (2017) Particle filter based probabilistic forced alignment for continuous gesture recognition. In: Proceedings—2017 IEEE international conference on computer vision workshops, ICCVW 2017. https://doi.org/10.1109/ICCVW.2017.364 Chandrasekaran G, Periyasamy S, Panjappagounder Rajamanickam K (2020) Minimization of test time in system on chip using artificial intelligence-based test scheduling techniques. Neural Comput Appl. https://doi.org/10.1007/s00521-019-04039-6 Chai X, Liu Z, Yin F, Liu Z, Chen X (2016) Two streams recurrent neural networks for large-scale continuous gesture recognition. In: Proceedings—international conference on pattern recognition. https://doi.org/10.1109/ICPR.2016.7899603 Chen D, Li G, Sun Y, Kong J, Jiang G, Tang H, Ju Z, Yu H, Liu H (2017) An interactive image segmentation method in hand gesture recognition. Sensors (Switzerland). https://doi.org/10.3390/s17020253 Chua SND, Lim SF, Lai SN et al (2019) Development of a child detection system with artificial intelligence using object detection method. J Electr Eng Technol 14:2523–2529. https://doi.org/10.1007/s42835-019-00255-1 Everingham M, Van Gool L, Williams CKI, Winn J, Zisserman A (2010) The pascal visual object classes (VOC) challenge. Int J Comput Vis. https://doi.org/10.1007/s11263-009-0275-4 Flores CJL, Cutipa AEG, Enciso RL (2017) Application of convolutional neural networks for static hand gestures recognition under different invariant features. In: Proceedings of the 2017 IEEE 24th international congress on electronics, electrical engineering and computing, INTERCON 2017. https://doi.org/10.1109/INTERCON.2017.8079727 Geirhos R, Schütt HH, Medina Temme CR, Bethge M, Rauber J, Wichmann FA (2018) Generalisation in humans and deep neural networks. In: Advances in neural information processing systems Huang H, Chong Y, Nie C, Pan S (2019) Hand gesture recognition with skin detection and deep learning method. J Phys Conf Ser. https://doi.org/10.1088/1742-6596/1213/2/022001 Islam MZ, Hossain MS, Ul Islam R, Andersson K (2019) Static hand gesture recognition using convolutional neural network with data augmentation. In: 2019 Joint 8th international conference on informatics, electronics and vision, ICIEV 2019 and 3rd international conference on imaging, vision and pattern recognition, IcIVPR 2019 with international conference on activity and behavior computing, ABC 2019. https://doi.org/10.1109/ICIEV.2019.8858563 Ji S, Xu W, Yang M, Yu K (2013) 3D Convolutional neural networks for human action recognition. IEEE Trans Pattern Anal Mach Intell. https://doi.org/10.1109/TPAMI.2012.59 Kim H, Albuquerque G, Havemann S, Fellner DW (2005) Tangible 3D: hand gesture interaction for immersive 3D modeling. In: 9th international workshop on immersive projection technology—11th Eurographics symposium on virtual environments, IPT/EGVE 2005 Kim S, Ji Y, Lee KB (2018) An effective sign language learning with object detection based ROI segmentation. In: Proceedings—2nd IEEE international conference on robotic computing, IRC 2018. https://doi.org/10.1109/IRC.2018.00069 Köpüklü O, Gunduz A, Kose N, Rigoll G (2019) Real-time hand gesture detection and classification using convolutional neural networks. In: Proceedings—14th IEEE international conference on automatic face and gesture recognition, FG 2019. https://doi.org/10.1109/FG.2019.8756576 Maqueda AI, Del-Blanco CR, Jaureguizar F, García N (2015) Human-computer interaction based on visual hand-gesture recognition using volumetric spatiograms of local binary patterns. Comput Vis Image Underst. https://doi.org/10.1016/j.cviu.2015.07.009 Molchanov P, Yang X, Gupta S, Kim K, Tyree S, Kautz J (2016) Online detection and classification of dynamic hand gestures with recurrent 3D convolutional neural networks. In: Proceedings of the IEEE computer society conference on computer vision and pattern recognition. https://doi.org/10.1109/CVPR.2016.456 Ni Z, Chen J, Sang N, Gao C, Liu L (2018) Light YOLO for high-speed gesture recognition. In: Proceedings—international conference on image processing, ICIP. https://doi.org/10.1109/ICIP.2018.8451766 Oyedotun OK, Khashman A (2017) Deep learning in vision-based static hand gesture recognition. Neural Comput Appl. https://doi.org/10.1007/s00521-016-2294-8 Rahmat RF, Chairunnisa T, Gunawan D, Pasha MF, Budiarto R (2019) Hand gestures recognition with improved skin color segmentation in human-computer interaction applications. J Theor Appl Inf Technol 97(3):727–739 Redmon J, Farhadi A (2018) YOLO v.3. Tech Report Tzutalin (2015) LabelImg. LabelImg. https://github.com/tzutalin/labelImg Walker A (2013) Voice commands or gesture recognition: how will we control the computers of the future? https://www.independent.co.uk/life-style/gadgets-and-tech/voice-commands-or-gesture-recognition-how-will-we-control-the-computers-of-the-future-8899614.html Wan J, Li SZ, Zhao Y, Zhou S, Guyon I, Escalera S (2016) ChaLearn looking at people RGB-D isolated and continuous datasets for gesture recognition. In: IEEE computer society conference on computer vision and pattern recognition workshops. https://doi.org/10.1109/CVPRW.2016.100 Yang X, Tian Y (2014) Super normal vector for activity recognition using depth sequences. In: Proceedings of the IEEE computer society conference on computer vision and pattern recognition. https://doi.org/10.1109/CVPR.2014.108 Zhang Y, Cao C, Cheng J, Lu H (2018) EgoGesture: a new dataset and benchmark for egocentric hand gesture recognition. IEEE Trans Multimed. https://doi.org/10.1109/TMM.2018.2808769