Machine learning techniques for robotic and autonomous inspection of mechanical systems and civil infrastructure

H.M. La, N. Gucunski, K. Dana, S.-H. Kee, Development of an autonomous bridge deck inspection robotic system. J. Field Robot. 2017, 1489 (2017) L. Wang, Z. Zhang, Automatic detection of wind turbine blade surface cracks based on UAV-taken images. IEEE Trans. Ind. Electron. 64(9), 7293–7303 (2017) S. Bernardini, F. Jovan, Z. Jiang, S. Watson, A. Weightman, P. Moradi, T. Richardson, R. Sadeghian, S. Sareh, A multi-robot platform for the autonomous operation and maintenance of offshore wind farms blue sky ideas track, in Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems, AAMAS 2020, May 9–13, 2020, Auckland, New Zealand (2020) C. Stout, D. Thompson, UAV Approaches to Wind Turbine Inspection: Reducing Reliance on Rope-Access. Offshore Renewable Energy Catapult. (2019) D. Schmidt et al., Climbing robots for maintenance and inspections of vertical structures—A survey of design aspects and technologies. Robot. Auton. Syst. (2013). https://doi.org/10.1016/j.robot.2013.09.002 D. Lattanzi et al., Review of Robotic Infrastructure Inspection Systems. J. Infrastruct. Syst. (2017). https://doi.org/10.1061/(ASCE)IS.1943-555X.0000353 M.A.M. Yusoff et al., Development of a Remotely Operated Vehicle (ROV) for underwater inspection. Jurutera (2013) A.L. Meyrowitz et al., Autonomous vehicles, in Proceedings of the IEEE 1996 (1996). https://doi.org/10.1109/5.533960 F. Rubio et al., A review of mobile robots: Concepts, methods, theoretical framework, and applications. Int. J. Adv. Robot. Syst. 2019 (2019). https://doi.org/10.1177/1729881419839596 D.W. Gage, A Brief History of Unmanned Ground Vehicle (UGV) Development Efforts (1995) W. Shen et al., Proposed wall climbing robot with permanent magnetic tracks for inspecting oil tanks, in IEEE International Conference Mechatronics and Automation (2005). https://doi.org/10.1109/ICMA.2005.1626882 L.P. Kalra et al., A wall climbing robot for oil tank inspection, in 2006 IEEE International Conference on Robotics and Biomimetics (2006). https://doi.org/10.1109/ROBIO.2006.340155 S. Campbell et al., Sensor technology in autonomous vehicles: a review, in 2018 29th Irish Signals and Systems Conference, ISSC, 2018 (2018). https://doi.org/10.1109/ISSC.2018.8585340 J. Seo et al., Drone-enabled bridge inspection methodology and application. Autom. Constr. (2018). https://doi.org/10.1016/j.autcon.2018.06.006. https://www.sciencedirect.com/science/article/pii/S0926580517309755 DOI M. Shafiee et al., Unmanned Aerial Drones for Inspection of Offshore Wind Turbines: A Mission-Critical Failure Analysis. Robotics J. (2021). https://doi.org/10.3390/robotics10010026 M.H. Frederiksen et al., Drones for inspection of infrastructure: Barriers, opportunities and successful uses. Center for Integrative Innovation Management (2019) M. Drones Lt, Best Commercial Drones for Beginners, Sep. 02, 2019, 2018. https://www.coptrz.com/best-commercial-drones-for-beginners/ C. Eschmann et al., High-resolution multisensor infrastructure inspection with unmanned aircraft systems, in ISPRS - International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences 2013 (2013). https://doi.org/10.5194/isprsarchives-XL-1-W2-125-2013https://ui.adsabs.harvard.edu/abs/2013ISPAr.XL1b.125E X.L. Ding et al., A review of structures, verification, and calibration technologies of space robotic systems for on-orbit servicing (2020). https://doi.org/10.1007/s11431-020-1737-4 A. Flores-Abad et al., A Review of Space Robotics Technologies for on-Orbit Servicing (Elsevier, Amsterdam, 2014). https://doi.org/10.1016/j.paerosci.2014.03.002 P.J. Staritz et al., Skyworker: A Robot for Assembly, Inspection and Maintenance of Large-Scale Orbital Facilities. IEEE (2001). https://doi.org/10.1109/ROBOT.2001.933271 H. Choset, D. Kortenkamp, Path planning and control for free-flying inspection robot in space. J. Aerosp. Eng. (1999). https://doi.org/10.1061/(ASCE)0893-1321(1999)12:2(74) J.S. Mehling et al., A minimally invasive tendril robot for in-space inspection, in The First IEEE/RAS-EMBS International Conference on Biomedical Robotics and Biomechatronics, BioRob 2006 (2006), pp. 690–695. https://doi.org/10.1109/BIOROB.2006.1639170 S.-I. Nishida et al., Prototype of an end-effector for a space inspection robot. Adv. Robot. (2012). https://doi.org/10.1163/156855301300235788 L. Pedersen et al., A survey of space robotics, in ISAIRAS (2003) J. Redmon et al., You only look once: unified, real-time object detection, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016) W. Fan et al., Mining big data: current status, and forecast to the future, in 2013 Association for Computing Machinery (2013). https://doi.org/10.1145/2481244.2481246 B. Matturdi et al., Big data security and privacy: a review. China Commun. 11(14), 135–145 (2014). https://doi.org/10.1109/CC.2014.7085614 D. Laney, 3-D Data Management: Controlling Data Volume, Velocity and Variety. META Group Research Note, February, vol. 6 (2001) B.T. Bastian, J. N, S.K. Kumar, C.V. Jiji, Visual inspection and characterization of external corrosion in pipelines using deep neural network. NDT & E International Journal 107, 102134 (2019) A. Shihavuddin et al., Wind turbine surface damage detection by deep learning aided drone inspection analysis. Energies 12(4), 676 (2019). https://doi.org/10.3390/en12040676 M. Hassanalian et al., Classifications, applications, and design challenges of drones: a review. Prog. Aerosp. Sci. (2017). https://doi.org/10.1016/j.paerosci.2017.04.003. https://www.sciencedirect.com/science/article/pii/S0376042116301348 A. Alharam et al., Real time AI-based pipeline inspection using drone for oil and gas industries in Bahrain, in 2020 International Conference on Innovation and Intelligence for Informatics, Computing and Technologies (3ICT) (2020) V. Nasteski, An overview of the supervised machine learning methods. Horizons B 4 (2017). https://doi.org/10.20544/HORIZONS.B.04.1.17.P05 B. Mahesh, Machine Learning Algorithms – a Review (2019). https://doi.org/10.21275/ART20203995 A. Carrio et al., A review of deep learning methods and applications for unmanned aerial vehicles. Hindawi J. Sens. (2017). https://doi.org/10.1155/2017/3296874 M.N. Mohammed et al., Design and Development of Pipeline Inspection Robot for Crack and Corrosion Detection (2018) https://www.analyticssteps.com/blogs/how-does-k-nearest-neighbor-works-machine-learning-classification-problem F. Hoffmann et al., Benchmarking in classification and regression. WIREs Data Mining Knowl. Discov. 9, e1318 (2019). https://doi.org/10.1002/widm.1318 A. Geron, Hands-On Machine Learning with Scikit-Learn, Keras &TensorFlow, 2nd edn. (2019). 2019 I. Goodfellow et al., Deep Learning (MIT Press, Cambridge, 2016) F.Y. Osisanwo et al., Supervised machine learning algorithms: classification and comparison. Int. J. Comput. Trends. Technol. (IJCTT) 48(3) 128–138 (2017) C.-F. Tsai et al., Intrusion detection by machine learning: a review. Expert Syst. Appl. 36(10), 11994–12000 (2009) A. Matsunaga et al., On the use of machine learning to predict the time and resources consumed by applications, in 2010 10th IEEE/ACM International Conference on Cluster, Cloud and Grid Computing (2010), pp. 495–504. https://doi.org/10.1109/CCGRID.2010.98 M. Jogin et al., Feature extraction using Convolution Neural Networks (CNN) and deep learning, in 2018 3rd IEEE International Conference on Recent Trends in Electronics, Information & Communication Technology (RTEICT) (2018), pp. 2319–2323. https://doi.org/10.1109/RTEICT42901.2018.9012507 https://ujjwalkarn.me/2016/08/09/quick-intro-neural-networks/ https://morioh.com/p/73fce91e9846 Y. Guo et al., Deep learning for visual understanding: a review. Neurocomputing 187, 27–48 (2016). https://doi.org/10.1016/j.neucom.2015.09.116 K. Gopalakrishnan et al., Crack damage detection in unmanned aerial vehicle images of civil infrastructure using pre-trained deep learning model. Int. J. Traffic Transp. Eng. (IJTTE) (2017) H. Larochelle et al., Exploring strategies for training deep neural networks. J. Mach. Learn. Res. 1, 1–40 (2009). https://doi.org/10.1145/1577069.1577070 A. Fischer, C. Igel, An Introduction to Restricted Boltzmann Machines. Iberoamerican Congress on Pattern Recognition (Springer, Berlin, 2012) N. Agarwalla et al., Deep learning using restricted Boltzmann machines. Int. J. Comput. Sci. Inf. Secur. 7(3), 1552–1556 (2016) Y. Hua et al., Deep belief networks and deep learning, in Proceedings of 2015 International Conference on Intelligent Computing and Internet of Things (2015), pp. 1–4. https://doi.org/10.1109/ICAIOT.2015.7111524 https://blog.paperspace.com/faster-r-cnn-explained-object-detection/ https://neurohive.io/en/popular-networks/r-cnn/ Z.-Q. Zhao et al., Object detection with deep learning: a review. IEEE Trans. Neural Netw. Learn. Syst. 30(11), 3212–3232 (2019). https://doi.org/10.1109/TNNLS.2018.2876865 https://blog.athelas.com/a-brief-history-of-cnns-in-image-segmentation-from-r-cnn-to-mask-r-cnn-34ea83205de4 P.S. Bithas et al., A Survey on Machine-Learning Techniques for UAV-Based Communications. Sensors (Basel, Switzerland) 26 November 2019 (2019). https://europepmc.org/articles/PMC6929112. Accessed September 2020 R. Girshick et al., Rich feature hierarchies for accurate object detection and semantic segmentation, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2014) R. Girshick, Fast r-cnn, in Proceedings of the IEEE International Conference on Computer Vision (2015) K. He et al. Mask R-CNN. In ICCV, 2017 J. Dai et al., R-FCN: Object Detection via Region-based Fully Convolutional Networks (2016). arXiv:1605.06409 W. Liu et al., Ssd: Single shot multibox detector (2015). Preprint arXiv:1512.02325 S. Ren et al., Faster r-cnn: Towards real-time object detection with region proposal networks. Adv. Neural Inf. Process. Syst. 2015 (2015) S. Grossberg, Recurrent neural networks. Scholarpedia 8(2), 1888 (2013) J.A. Bullinaria, Recurrent neural networks. Neural Computation: Lecture 12 (2013) S. Hochreiter, J. Schmidhuber, Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997). https://doi.org/10.1162/neco.1997.9.8.1735 I.J. Goodfellow et al., Generative Adversarial Networks (2014). arXiv, stat.ML M. Mirza, S. Osindero, Conditional Generative Adversarial Nets (2014) L. Noriega, Multilayer perceptron tutorial. School of Computing. Staffordshire University (2005) H. Taud, J. Mas, Multilayer perceptron (MLP), in Geomatic Approaches for Modeling Land Change Scenarios. Lecture Notes in Geoinformation and Cartography. (Springer, Cham, 2018). https://doi.org/10.1007/978-3-319-60801-3_27 G. Alain, Y. Bengio, What Regularized Auto-Encoders Learn from the Data Generating Distribution (2014) F.Q. Lauzon, An introduction to deep learning, in 2012 11th International Conference on Information Science, Signal Processing and Their Applications (ISSPA), (2012), pp. 1438–1439. https://doi.org/10.1109/ISSPA.2012.6310529 P. Baldi, Autoencoders, unsupervised learning, and deep architectures, in Proceedings of ICML Workshop on Unsupervised and Transfer Learning (2012) Q.V. Le, A tutorial on deep learning part 2: autoencoders, convolutional neural networks and recurrent neural networks. Google Brain 20, 1–20 (2015) A. Agarwal, A. Motwani, An Overview of Convolutional and AutoEncoder Deep Learning Algorithm (2016) Y. Coadou, Boosted decision trees and applications. EPJ Web Conf. 55, 02004 (2013). https://doi.org/10.1051/epjconf/20135502004 E. Beauxis-Aussalet et al., Visualization of confusion matrix for non-expert users, in IEEE Conference on Visual Analytics Science and Technology (VAST) - Poster Proceedings (2014) G. Shobha et al., Machine learning, in Handbook of Statistics, vol. 38 (Elsevier, Amsterdam, 2018), pp. 197–228. https://doi.org/10.1016/bs.host.2018.07.004. https://www.sciencedirect.com/science/article/pii/S0169716118300191. ISSN 0169-7161. ISBN 9780444640420 A. Kulkarni et al., Foundations of data imbalance and solutions for a data democracy, in Data Democracy (Academic Press, San Diego, 2020), pp. 83–106. https://doi.org/10.1016/B978-0-12-818366-3.00005-8. ISBN 9780128183663 https://towardsdatascience.com/map-mean-average-precision-might-confuse-you-5956f1bfa9e2 N. Mohamed et al., Real-time big data analytics: applications and challenges, in Proc. Int. Conf. High Perform. Comput. Simulation (2014), pp. 305–310 J. Franko et al., Design of a multi-robot system for wind turbine maintenance. Energies (2020) B. Brandoli et al., Aircraft fuselage corrosion detection using artificial intelligence. Sensors 2021(21), 4026 (2021). https://doi.org/10.3390/s21124026 T. Malekzadeh et al., Aircraft Fuselage Defect Detection using Deep Neural Networks (2017). arXiv:1712.09213 J. Miranda et al., Machine learning approaches for defect classification on aircraft fuselage images aquired by an UAV, in Fourteenth International Conference on Quality Control by Artificial Vision. 16 July 2019, Proc. SPIE, vol. 11172 (2019), p. 1117208. https://doi.org/10.1117/12.2520567. B. Jalil et al., Fault detection in power equipment via an unmanned aerial system using multi modal data. Sensors 2019(19), 3014 (2019). https://doi.org/10.3390/s19133014 E. Titov et al., The deep learning based power line defect detection system built on data collected by the cablewalker drone, in 2019 International Multi-Conference on Engineering, Computer and Information Sciences (SIBIRCON) (2019), pp. 0700–0704. https://doi.org/10.1109/SIBIRCON48586.2019.8958397 A. Ortiz et al., First steps towards a roboticized visual inspection system for vessels, in 2010 IEEE 15th Conference on Emerging Technologies & Factory Automation (ETFA 2010) (2010), pp. 1–6. https://doi.org/10.1109/ETFA.2010.5641246 F. Bonnin-Pascual et al., Semi-autonomous visual inspection of vessels assisted by an unmanned micro aerial vehicle, in 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems (2012), pp. 3955–3961. https://doi.org/10.1109/IROS.2012.6385891 S. Kawabata et al., Autonomous flight drone with depth camera for inspection task of infra structure, in Proceedings of the International MultiConference of Engineers and Computer Scientists, vol. 2 (2018) Y.-J. Cha et al., Deep learning-based crack damage detection using convolutional neural networks. Comput.-Aided Civ. Infrastruct. Eng. 32, 361–378 (2017) M.M. Karim et al., Modeling and simulation of a robotic bridge inspection system, in Procedia Computer Science (2020), pp. 177–185. https://doi.org/10.1016/j.procs.2020.02.276. https://www.sciencedirect.com/science/article/pii/S1877050920304154. ISSN 1877-0509 M.A. Manzoor et al., Vehicle make and model classification system using bag of SIFT features, in 2017 IEEE 7th Annual Computing and Communication Workshop and Conference (CCWC), (2017), pp. 1–5. https://doi.org/10.1109/CCWC.2017.7868475 P. Rakshata et al., Car damage detection and analysis using deep learning algorithm for automotive. Int. J. Sci. Technol. Res. 5(6) (2019). Nov-Dec-2019, ISSN (Online): 2395-566X Q. Zhang et al., Vehicle-damage-detection segmentation algorithm based on improved mask RCNN. IEEE Access 8, 6997–7004 (2020). https://doi.org/10.1109/ACCESS.2020.2964055 H. Bandi et al., Assessing car damage with convolutional neural networks, in 2021 International Conference on Communication Information and Computing Technology (ICCICT) (2021), pp. 1–5. https://doi.org/10.1109/ICCICT50803.2021.9510069 C. Giovany Pachón-Suescún et al., Scratch Detection in Cars Using a Convolutional Neural Network by Means of Transfer Learning. IJAER (2018) 16 Nov 2018 K. Patil et al., Deep learning based car damage classification, in 2017 16th IEEE International Conference on Machine Learning and Applications (ICMLA) (2017), pp. 50–54. https://doi.org/10.1109/ICMLA.2017.0-179 R. Ali et al., Subsurface damage detection of a steel bridge using deep learning and uncooled micro-bolometer. Constr. Build. Mater. 226 (2019). https://doi.org/10.1016/j.conbuildmat.2019.07.293. 2019, 376-387, ISSN 0950-0618. https://www.sciencedirect.com/science/article/pii/S0950061819319671 Y. Liu et al., The method of insulator recognition based on deep learning, in Proceedings of the 2016 4th International Conference on Applied Robotics for the Power Industry (CARPI), Jinan, China, 11–13 October, 2016 (2016), pp. 1–5 Z. Zhao et al., Multi-patch deep features for power line insulator status classification from aerial images, in 2016 International Joint Conference on Neural Networks (IJCNN), 2016 (2016), pp. 3187–3194. https://doi.org/10.1109/IJCNN.2016.7727606 A. Ortiz et al., Vision-based corrosion detection assisted by a micro-aerial vehicle in a vessel inspection application. Sensors 2016(16), 2118 (2016). https://doi.org/10.3390/s16122118 V.N. Nguyen et al., Automatic autonomous vision-based power line inspection: A review of current status and the potential role of deep learning. Int. J. Electr. Power Energy Syst. 2018 (2018) S. Faghih-Roohi et al., Deep convolutional neural networks for detection of rail surface defects, in Neural Networks (IJCNN), 2016 International Joint Conference on, 2016 (2016), pp. 2584–2589 Z.-Q. Tong et al., Recent advances in small object detection based on deep learning: a review. Image Vis. Comput. 97, 103910 (2020) T.-Y. Lin et al., Microsoft Coco: Common Objects in Context. European Conference on Computer Vision (Springer, Cham, 2014) Y. Liu et al., A survey and performance evaluation of deep learning methods for small object detection. Expert Syst. Appl. 172, 114602 (2021) N.-D. Nguyen et al., An evaluation of deep learning methods for small object detection. J. Electr. Comput. Eng. 2020, 3189691 (2020) C. Chenyi et al., R-CNN for small object detection, in Asian Conference on Computer Vision (Springer, Cham, 2016) Z.-Q. Zhao et al., Object detection with deep learning: a review. IEEE Trans. Neural Netw. Learn. Syst. 30(11), 3212–3232 (2019) Y. Shen et al., Design alternative representations of confusion matrices to support non-expert public understanding of algorithm performance. Proc. ACM Hum. Comput. Interact. 4(CSCW2), 153 (2020) R.K. Rai et al., Intricacies of Unstructured Data. EAI Endorsed Transactions on Scalable Information Systems 4(14) (2017). https://doi.org/10.4108/eai.25-9-2017.153151 A. Gandomi et al., Beyond the hype: big data concepts, methods, and analytics. Int. J. Inf. Manag. 35(2), 137–144 (2015) D.P. Acharjya et al., A survey on big data analytics: challenges, open research issues and tools. Int. J. Adv. Comput. Sci. Appl. 7, 511–518 (2016) A. Oussous et al., Big data technologies: a survey. J. King Saud Univ, Comput. Inf. Sci. 30, 431–448 (2018) X. Jin et al., Significance andchallenges of big data research. Big Data Res. 2(2), 59–64 (2015) M.M. Najafabadi et al., Deep learning applications and challenges in big data analytics. Big Data 2(1), 1–21 (2015) I. Panella, Artificial intelligence methodologies applicable to support the decision-making capability on board unmanned aerial vehicles, in ECSIS Symposium on Bio-Inspired Learning and Intelligent Systems for Security, Edinburgh (2008), pp. 111–118. https://doi.org/10.1109/BLISS.2008.14 M. Ono et al., MAARS: machine learning-based analytics for automated rover systems, in Proc. IEEE Aerosp. Conf (2020), pp. 1–17 M. Hillebrand et al., A design methodology for deep reinforcement learning in autonomous systems. Procedia Manufacturing 52 (2020). https://doi.org/10.1016/j.promfg.2020.11.044. https://www.sciencedirect.com/science/article/pii/S2351978920321879 S. Contreras et al., Using deep learning for exploration and recognition of objects based on images, in 2016 XIII Latin American Robotics Symposium and IV Brazilian Robotics Symposium (LARS/SBR) (2016), pp. 1–6. https://doi.org/10.1109/LARS-SBR.2016.8 W. Chen et al., Door recognition and deep learning algorithm for visual based robot navigation, in 2014 IEEE International Conference on Robotics and Biomimetics (ROBIO 2014) (2014), pp. 1793–1798. https://doi.org/10.1109/ROBIO.2014.7090595