Pose-guided matching based on deep learning for assessing quality of action on rehabilitation training

Cheng, 2018, The traditional Chinese Baduanjin exercise for the enhancements in health of older adults: a narrative review, Geriatric Care, 4, 41, 10.4081/gc.2018.7449 Liu, 2016, Efficacy of Ba Duan Jin in improving balance: a study in Chinese community-dwelling older adults, J. Gerontol. Nurs., 42, 38, 10.3928/00989134-20160201-03 B. Liu, S. Chen, Y. Li, et al., The effect of the modified eighth section of eight-section brocade on osteoporosis in postmenopausal women: a prospective randomized trial, Medicine 94 (25), 2015. Rossol, 2016, A multisensor technique for gesture recognition through intelligent skeletal pose analysis, IEEE Trans. Hum.-Mach. Syst., 46, 350, 10.1109/THMS.2015.2467212 T. Lisini Baldi, F. Farina, A. Garulli, A. Giannitrapani, D. Prattichizzo, Upper body pose estimation using wearable inertial sensors and multiplicative kalman filter, IEEE Sensors J., 20 (1), 492–500, Jan2020. S. Huang, L. Fu, P. Hsiao, Silhouette-based human pose estimation using reversible jump Markov chain Monte Carlo, Electr. Lett., 42 (10), 575–577, 2006. Dantone, 2014, Body parts dependent joint regressors for human pose estimation in still images, IEEE Trans. Pattern Anal. Mach. Intell., 36, 2131, 10.1109/TPAMI.2014.2318702 Z. Cao, G. Hidalgo Martinez, T. Simon, S. -E. Wei, Y. A. Sheikh, OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields, IEEE Trans. Pattern Anal. Mach. Intellig., 2019 (early access). Kidziński, 2020, Deep neural networks enable quantitative movement analysis using single-camera videos, Nat. Commun., 11, 10.1038/s41467-020-17807-z Baptista, 2019, Home self-training: visual feedback for assisting physical activity for stroke survivors, Comput. Methods Programs Biomed., 176, 111, 10.1016/j.cmpb.2019.04.019 Tang, 2020, Hybridized hierarchical deep convolutional neural network for sports rehabilitation exercises, IEEE Access, 8, 118969, 10.1109/ACCESS.2020.3005189 Liao, 2020, A deep learning framework for assessing physical rehabilitation exercises, IEEE Trans. Neural Syst. Rehabil. Eng., 28, 468, 10.1109/TNSRE.2020.2966249 Yu, 2021, Skeleton-based human action evaluation using graph convolutional network for monitoring Alzheimer’s progression, Pattern Recogn., vol. 119, 10.1016/j.patcog.2021.108095 Pirsiavash, Hamed, A. Torralba, et al, Assessing the quality of actions, in European Conference on Computer Vision, Springer, Cham, 2014, pp. 556–571. Parmar, 2019, Action quality assessment across multiple actions, 1468 Y. Hou, H. Yao, H. Li, X. Sun, Dancing like a superstar: Action guidance based on pose estimation and conditional pose alignment, in 2017 IEEE International Conference on Image Processing (ICIP), Beijing, 2017, pp. 1312–1316. Elkholy, 2020, Efficient and robust skeleton-based quality assessment and abnormality detection in human action performance, IEEE J. Biomed. Health. Inf., 24, 280, 10.1109/JBHI.2019.2904321 P. Parmar, B.T. Morris, Measuring the quality of exercises, in: 2016 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Orlando, FL, 2016, pp. 2241–2244. P. Parmar, B.T. Morris, Learning to Score Olympic Events, in 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Honolulu, HI, 2017, pp. 76–84. Zia, 2016, Automated video-based assessment of surgical skills for training and evaluation in medical schools, Int. J. Comput. Assist. Radiol. Surg., 11, 1623, 10.1007/s11548-016-1468-2 R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, D. Batra, Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization, in 2017 IEEE International Conference on Computer Vision (ICCV), Venice, 2017, pp. 618–626. Lv, 2019, Eight-section brocade exercises improve the sleep quality and memory consolidation and cardiopulmonary function of older adults with atrial fibrillation-associated stroke, Front. Psychol., 10, 10.3389/fpsyg.2019.02348 Wang, 2019, The health effects of Baduanjin exercise (a type of Qigong exercise) in breast cancer survivors: a randomized, controlled, single-blinded trial, Eur. J. Oncol. Nurs., 39, 90, 10.1016/j.ejon.2019.01.007 Chen, 2020, Intensity level and cardiorespiratory responses to Baduanjin exercise in patients with chronic heart failure, ESC Heart Failure, 7, 3782, 10.1002/ehf2.12959 G. Chen, et al. Effects of Baduanjin on postoperative rehabilitation of patients with breast cancer: a protocol for systematic review and meta-analysis. Medicine 100 (17), 2021. Feng, 2020, Qigong for the prevention, treatment, and rehabilitation of COVID-19 infection in older adults, Am. J. Geriatric Psychiatry, 28, 812, 10.1016/j.jagp.2020.05.012 Zha, 2020, Modified rehabilitation exercises for mild cases of COVID-19, Ann. Palliative Med., 9, 3100, 10.21037/apm-20-753 Yun, 2006, Design, Implementation, and experimental results of a quaternion-based kalman filter for human body motion tracking, IEEE Trans. Rob., 22, 1216, 10.1109/TRO.2006.886270 Panahandeh, 2013, Continuous hidden Markov model for pedestrian activity classification and gait analysis, IEEE Trans. Instrument. Measure., 62, 1073, 10.1109/TIM.2012.2236792 Y. Liao, et al., A review of computational approaches for evaluation of rehabilitation exercises, Comp. Biol. Med., 119, 2020. Sardari, 2020, VI-Net—View-invariant quality of human movement assessment, Sensors, 20, 5258, 10.3390/s20185258 Y. Li, et al. ScoringNet: learning key fragment for action quality assessment with ranking loss in skilled sports. Asian Conference on Computer Vision. Springer, Cham, pp. 149–164, 2018. Venkataraman, 2015, Dynamical regularity for action analysis S. Chopra, R. Hadsell, Y. LeCun. Learning a similarity metric discriminatively, with application to face verification, in 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05), San Diego, CA, USA, 2005, pp. 539–546. Lowe, 2004, Distinctive image features from scale-invariant keypoints, Int. J. Comput. Vision, 60, 91, 10.1023/B:VISI.0000029664.99615.94 A. Chowdhury, S. Kirchgasser, A. Uhl, A. Ross. Can a CNN Automatically Learn the Significance of Minutiae Points for Fingerprint Matching? In: 2020 IEEE Winter Conference on Applications of Computer Vision (WACV), Snowmass Village, CO, USA, 2020, pp. 340–348. M. Jaderberg, K. Simonyan, A. Zisserman, et al. Spatial Transformer Networks, in Advances in neural information processing systems, 2015, pp. 2017–2025. C. Szegedy et al., Going deeper with convolutions, in 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, 2015, pp. 1–9. Y. Lecun, L. Bottou, Y. Bengio, P. Haffner, Gradient-based learning applied to document recognition, in Proceedings of the IEEE, vol. 86, no. 11, pp. 2278–2324, Nov, 1998. Ren, 2017, Faster R-CNN: towards real-time object detection with region proposal networks, IEEE Trans. Pattern Anal. Mach. Intell., 39, 1137, 10.1109/TPAMI.2016.2577031 Hu, 2020, Squeeze-and-excitation networks, IEEE Trans. Pattern Anal. Mach. Intell., 42, 2011, 10.1109/TPAMI.2019.2913372 K. Simonyan, A. Zisserman, Very Deep Convolutional Networks for Large-Scale Image Recognition, arXiv preprint arXiv:1409.1556, 2014. He, 2016, Deep residual learning for image recognition, 770 Wang, 2018, Understanding convolution for semantic segmentation, 1451