A Framework for Vision-Based Building Detection and Entering for Autonomous Delivery Drones
Tóm tắt
Autonomous delivery by aerial robots inside urban environments is getting closer to be operational everyday, but several concerns about proper navigation sources are to be addressed. As lot of GPS glitches are observed in urban environments, hereon a fully vision-based navigation framework is developed. The platform is assumed to have the ability to get into a vicinity of its destination using available “imperfect” navigation methods like GPS. Hereupon, the autonomous execution of mission is divided into four consecutive phases, all relying on a forward-looking camera: I) A novel vision-based method for detecting and confirming the target facade in the batch of neighboring buildings is introduced that performs matching using neural networks exploiting the texture features of facade segments; II) The intended window among the facade windows array is detected in real-time; III) The robot is guided to autonomously approach the intended window by a proper visual tracking algorithm; IV) Finally, a collision-free passage through the portal section of window based on visual Time-To-Contact estimation is commanded for the safe entrance. The network is trained using approximately 34,000 feature samples from a set of real-world building facades. As a result, a 95.4% accuracy along with 81.4% classification precision and 87.6% recall are achieved for the trained network in correct facade detection and confirmation. Also, the success rate of overall entrance mission is found to be 13 out of 15 in real world experiments, provided the initial distance being less than 20 meters.
Tài liệu tham khảo
Flores, G., Zhou, S., Lozano, R., Castillo, P.: A vision and GPS-based real-time trajectory planning for MAV in unknown urban environments. In: 2013 International Conference on Unmanned Aircraft Systems (ICUAS), pp 1150–1155. IEEE (2013)
Popp, M., Prophet, S., Scholz, G., Trommer, G.: A novel guidance and navigation system for MAVs capable of autonomous collision-free entering of buildings. Gyroscopy Navigation 6(3), 157–165 (2015)
Falanga, D., Mueggler, E., Faessler, M., Scaramuzza, D.: Aggressive quadrotor flight through narrow gaps with onboard sensing and computing using active vision. In: 2017 IEEE International Conference on Robotics and Automation (ICRA), pp 5774–5781. IEEE (2017)
Wu, C., Frahm, J.-M., Pollefeys, M.: Detecting large repetitive structures with salient boundaries. In: European Conference on Computer Vision, pp 142–155. Springer (2010)
Wendel, A., Donoser, M., Bischof, H.: Unsupervised facade segmentation using repetitive patterns. In: Joint Pattern Recognition Symposium, pp 51–60. Springer (2010)
Recky, M., Leberl, F.: Window detection in complex facades. In: 2010 2nd European Workshop on Visual Information Processing (EUVIP), pp 220–225. IEEE (2010)
Kulkarni, V., Nagesh, R., Wu, H.: Window detection in frontal façades. Project work at CS294-69 Image Manipulation and Computational Photography University of Barkley (2011)
Lee, S.C., Nevatia, R.: Extraction and integration of window in a 3D building model from ground view images. In: Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004, vol. 2. IEEE (2004)
Neuhausen, M., König, M.: Improved window detection in facade images. In: Mutis, I., Hartmann, T. (eds.) Advances in Informatics and Computing in Civil and Construction Engineering, pp 537–543. Springer (2019)
Ali, H., Seifert, C., Jindal, N., Paletta, L., Paar, G.: Window detection in facades. In: 14th International Conference on Image Analysis and Processing (ICIAP 2007), pp 837–842. IEEE (2007)
Zhuo, X., Mönks, M., Esch, T., Reinartz, P.: Facade segmentation from oblique UAV imagery. In: 2019 Joint Urban Remote Sensing Event (JURSE), pp 1–4. IEEE (2019)
Schmitz, M., Mayer, H.: A convolutional network for semantic facade segmentation and interpretation. Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci. 41 (2016)
Huang, S., Nex, F., Lin, Y., Yang, M.: Semantic segmentation of building in airborne images. International Archives of the Photogrammetry, Remote Sensing & Spatial Information Sciences (2019)
Droeschel, D., Stückler, J., Behnke, S.: Local multi-resolution representation for 6D motion estimation and mapping with a continuously rotating 3D laser scanner. In: 2014 IEEE International Conference on Robotics and Automation (ICRA), pp 5221–5226. IEEE (2014)
Fahimi, H., hojat Mirtajadini, S., Shahbazi, M.: A vision-based guidance algorithm for entering buildings through windows for delivery drones. IEEE Aerospace and Electronic Systems Magazine (2022)
Achanta, R., Shaji, A., Smith, K., Lucchi, A., Fua, P., Süsstrunk, S.: SLIC superpixels compared to state-of-the-art superpixel methods. IEEE Trans. Pattern Anal. Mach. Intell. 34(11), 2274–2282 (2012)
Trémeau, A., Colantoni, P.: Regions adjacency graph applied to color image segmentation. IEEE Trans. Image Process. 9(4), 735–744 (2000)
Haralick, R.M., Shanmugam, K., Dinstein, I.H.: Textural features for image classification. IEEE Trans. Syst. Man Cybern (6), 610–621 (1973)
Hall-Beyer, M.: GLCM Texture: A Tutorial v. 3.0 March 2017 PRISM. https://doi.org/10.11575/PRISM/10182, https://prism.ucalgary.ca/handle/1880/51900 (2017)
Li, Y., Hao, P., Zhang, S., Li, Y.: Feature-matching pattern-based support vector machines for robust peptide mass fingerprinting. Mol. Cell. Proteome. 10(12) (2011)
Nguyen, M.H., Perez, J., De la Torre, F.: Facial feature detection with optimal pixel reduction SVM. In: 2008 8th IEEE International Conference on Automatic Face & Gesture Recognition, pp 1–6. IEEE (2008)
Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. In: 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), vol. 1, pp 886–893. IEEE (2005)
Zhang, Z., Lee, W.S.: Deep graphical feature learning for the feature matching problem. In: Proceedings of the IEEE International Conference on Computer Vision, pp 5087–5096 (2019)
Han, X., Leung, T., Jia, Y., Sukthankar, R., Berg, A.C.: MatchNet: Unifying feature and metric learning for patch-based matching. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 3279–3286 (2015)
Altwaijry, H., Veit, A., Belongie, S.J., Tech, C.: Learning to detect and match keypoints with deep architectures. In: BMVC (2016)
Zagoruyko, S., Komodakis, N.: Learning to compare image patches via convolutional neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 4353–4361 (2015)
Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res. 15(1), 1929–1958 (2014)
Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv:1412.6980(2014)
Ishii, S., Ljunggren D.: A comparative analysis of robustness to noise in machine learning classifiers (2021)
Du, P., Kibbe, W.A., Lin, S.M.: Improved peak detection in mass spectrum by incorporating continuous wavelet transform-based pattern matching. Bioinformatics 22(17), 2059–2065 (2006)
Suzuki, S., et al.: Topological structural analysis of digitized binary images by border following. Comput. Vis. Graph. Image Process. 30(1), 32–46 (1985)
Sazbon, D., Rotstein, H., Rivlin, E.: Finding the focus of expansion and estimating range using optical flow images and a matched filter. Mach. Vis. Appl. 15(4), 229–236 (2004)
Wang, C., Liu, W., Meng, M.Q.-H.: Obstacle avoidance for quadrotor using improved method based on optical flow. In: 2015 IEEE International Conference on Information and Automation, pp 1674–1679. IEEE (2015)
Souhila, K., Karim, A.: Optical flow based robot obstacle avoidance. Int. J. Adv. Robot. Syst. 4(1), 2 (2007)
Cho, G., Kim, J., Oh, H.: Vision-based obstacle avoidance strategies for MAVs using optical flows in 3-D textured environments. Sensors 19(11), 2523 (2019)
Chang, R., Ding, R., Lin, M., Meng, D., Wu, Z., Hang, M.: An experimental evaluation of balance strategy based obstacle avoidance. In: 2016 14th International Conference on Control, Automation, Robotics and Vision (ICARCV), pp 1–6. IEEE (2016)
Lucas, B.D., Kanade, T., et al: An iterative image registration technique with an application to stereo vision. Vancouver, British Columbia (1981)
Gil-Jiménez, P., Gómez-Moreno, H., López-Sastre, R. J., Bermejillo-Martín-Romo, A.: Estimating the focus of expansion in a video sequence using the trajectories of interest points. Image Vis. Comput. 50, 14–26 (2016)
Falanga, D., Kleber, K., Mintchev, S., Floreano, D., Scaramuzza, D.: The foldable drone: A morphing quadrotor that can squeeze and fly. IEEE Robot. Autom. Lett. 4(2), 209–216 (2018)
Brunner, G., Szebedy, B., Tanner, S., Wattenhofer, R.: The urban last mile problem: Autonomous drone delivery to your balcony. In: 2019 International Conference on Unmanned Aircraft Systems (ICUAS), pp 1005–1012. IEEE (2019)
Mueller, K., Fennel, M., Trommer, G.F.: Model predictive control for vision-based quadrotor guidance. In: 2020 IEEE/ION Position, Location and Navigation Symposium (PLANS), pp 50–61. IEEE (2020)
Reardon, C., Fink, J.: Air-ground robot team surveillance of complex 3D environments. In: 2016 IEEE International Symposium on Safety, Security, and Rescue Robotics (SSRR), pp 320–327. IEEE (2016)
Sanket, N.J., Singh, C.D., Ganguly, K., Fermüller, C., Aloimonos, Y.: Gapflyt: Active vision based minimalist structure-less gap detection for quadrotor flight. IEEE Robot. Autom. Lett. 3(4), 2799–2806 (2018)
Zhou, S., Flores, G., Bazan, E., Lozano, R., Rodriguez, A.: Real-time object detection and pose estimation using stereo vision. An application for a quadrotor MAV. In: 2015 Workshop on Research, Education and Development of Unmanned Aerial Systems (RED-UAS), pp 72–77. IEEE (2015)
Tang, Z., Cunha, R., Cabecinhas, D., Hamel, T., Silvestre, C.: Quadrotor going through a window and landing: An image-based visual servo control approach. Control. Eng. Pract. 112, 104827 (2021)
Veerawal, S., Bhushan, S., Mansharamani, M.R., Sharma, B.: Vision based autonomous drone navigation through enclosed spaces. In: International Conference on Computer Vision and Image Processing, pp 104–115. Springer (2020)
Finlayson, G.D., Drew, M.S., Lu, C.: Intrinsic images by entropy minimization. In: European Conference on Computer Vision, pp 582–595. Springer (2004)