Road obstacles positional and dynamic features extraction combining object detection, stereo disparity maps and optical flow data
Tóm tắt
One of the most relevant tasks in an intelligent vehicle navigation system is the detection of obstacles. It is important that a visual perception system for navigation purposes identifies obstacles, and it is also important that this system can extract essential information that may influence the vehicle’s behavior, whether it will be generating an alert for a human driver or guide an autonomous vehicle in order to be able to make its driving decisions. In this paper we present an approach for the identification of obstacles and extraction of class, position, depth and motion information from these objects that employs data gained exclusively from passive vision. We use a convolutional neural network for the obstacles detection, optical flow for the analysis of movement of the detected obstacles, both in relation to the direction and in relation to the intensity of the movement, and also stereo vision for the analysis of distance of obstacles in relation to the vehicle. We performed our experiments on two different datasets, and the results obtained showed a good efficacy from the use of depth and motion patterns to assess the obstacles’ potential threat status.
Tài liệu tham khảo
Bleyer, M.: VU Stereo Vision. Karlsplatz, Vienna, Austria (2013)
Bouchafa, S., Zavidovique, B.: Obstacle detection ”for free”; in the c-velocity space. In: 2011 14th International IEEE Conference on Intelligent Transportation Systems (ITSC), pp. 308–313 (2011). https://doi.org/10.1109/ITSC.2011.6082872
Chanawangsa, P., Chen, C.W.: A novel video analysis approach for overtaking vehicle detection. In: 2013 International Conference on Connected Vehicles and Expo (ICCVE), pp. 802–807 (2013). https://doi.org/10.1109/ICCVE.2013.6799901
Chen, Y.S., Tsai, A.C., Lin, T.T.: Road environment recognition method in complex traffic situations based on stereo vision. In: 2012 12th International Conference on ITS Telecommunications, pp. 180–184 (2012). https://doi.org/10.1109/ITST.2012.6425161
Commission, I.E.: Safety of laser products. Part 1: equipment classification, requirements and user’s guide (2001)
Deo, N., Rangesh, A., Trivedi, M.M.: How would surround vehicles move? A unified framework for maneuver classification and motion prediction. CoRR abs/1801.06523 (2018). http://arxiv.org/abs/1801.06523
Dosovitskiy, A., Fischer, P., Ilg, E., Husser, P., Hazirbas, C., Golkov, V., v d Smagt, P., Cremers, D., Brox, T.: Flownet: learning optical flow with convolutional networks. In: 2015 IEEE International Conference on Computer Vision (ICCV), pp. 2758–2766 (2015). https://doi.org/10.1109/ICCV.2015.316
Farnebäck, G.: Two-frame motion estimation based on polynomial expansion. In: Bigun, J., Gustavsson, T. (eds.) Image Analysis: 13th Scandinavian Conference, SCIA 2003 Halmstad, Sweden, June 29–July 2, 2003 Proceedings. Springer Berlin Heidelberg, Berlin, Heidelberg, pp. 363–370 (2003). https://doi.org/10.1007/3-540-45103-X_50
Fernandes, L.C., Souza, J.R., Pessin, G., Shinzato, P.Y., Sales, D., Mendes, C., Prado, M., Klaser, R., Magalhes, A.C., Hata, A., Pigatto, D., Branco, K.C., Osorio, F.S., Wolf, D.F.: Carina intelligent robotic car: architectural design and applications. J. Syst. Archit. 60(4), 372–392 (2014). https://doi.org/10.1016/j.sysarc.2013.12.003
Fleet, D.J., Weiss, Y.: Optical flow estimation (2005)
Geiger, A., Lenz, P., Stiller, C., Urtasun, R.: Vision meets robotics: the kitti dataset. Int. J. Rob. Res. 32(11), 1231–1237 (2013). https://doi.org/10.1177/0278364913491297
Giosan, I., Nedevschi, S.: Superpixel-based obstacle segmentation from dense stereo urban traffic scenarios using intensity, depth and optical flow information. In: 17th International IEEE Conference on Intelligent Transportation Systems (ITSC), pp. 1662–1668 (2014). https://doi.org/10.1109/ITSC.2014.6957932
Gupta, K., Upadhyay, S., Gandhi, V., Krishna, K.M.: Small obstacle detection using stereo vision for autonomous ground vehicle. In: Proceedings of the Advances in Robotics, ACM, New York, NY, USA, AIR ’17, pp. 25:1–25:6 (2017). https://doi.org/10.1145/3132446.3134889
He, K., Gkioxari, G., Dollr, P., Girshick, R.: Mask r-cnn. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 2980–2988 (2017). https://doi.org/10.1109/ICCV.2017.322
Huang, Y., Liu, S.: Multi-class obstacle detection and classification using stereovision and improved active contour models. IET Intell. Transp. Syst. 10(3), 197–205 (2016). https://doi.org/10.1049/iet-its.2014.0308
Hne, C., Heng, L., Lee, G.H., Fraundorfer, F., Furgale, P., Sattler, T., Pollefeys, M.: 3d visual perception for self-driving cars using a multi-camera system: calibration, mapping, localization, and obstacle detection, automotive vision: challenges, trends, technologies and systems for vision-based intelligent vehicles. Image Vis. Comput. 68, 14–27 (2017). https://doi.org/10.1016/j.imavis.2017.07.003
Ilg, E., Mayer, N., Saikia, T., Keuper, M., Dosovitskiy, A., Brox, T.: Flownet 2.0: evolution of optical flow estimation with deep networks. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1647–1655 (2017). https://doi.org/10.1109/CVPR.2017.179
Jawed, S., Boumaiza, E., Grabocka. J., Schmidt-Thieme, L.: Data-driven vehicle trajectory forecasting. CoRR abs/1902.05400 (2019). http://arxiv.org/abs/1902.05400
Kitchenham, B., Charters, S.: Guidelines for performing systematic literature reviews in software engineering (2007)
Li, J., Chen, M.: On-road multiple obstacles detection in dynamical background. In: 2014 Sixth International Conference on Intelligent Human-Machine Systems and Cybernetics, vol. 1, pp. 102–105 (2014), https://doi.org/10.1109/IHMSC.2014.33
LiDAR-UKcom.: LIDAR. UK (2015). http://www.lidar-uk.com/
Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft coco: common objects in context. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) Computer Vision - ECCV 2014, pp. 740–755. Springer International Publishing, Cham (2014)
Liu, S., Huang ,Y., Zhang, R.: Obstacle recognition for adas using stereovision and snake models. In: 17th International IEEE Conference on Intelligent Transportation Systems (ITSC), pp. 99–104 (2014). https://doi.org/10.1109/ITSC.2014.6957673
Lucas, B.D., Kanade, T.: An iterative image registration technique with an application to stereo vision. In: Proceedings of the 7th International Joint Conference on Artificial Intelligence - Volume 2, Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, IJCAI’81, pp. 674–679 (1981). http://dl.acm.org/citation.cfm?id=1623264.1623280
Mitzel, D., Floros, G., Sudowe, P., van der Zander, B., Leibe, B.: Real time vision based multi-person tracking for mobile robotics and intelligent vehicles. In: Jeschke, S., Liu, H., Schilberg, D. (eds.) Intelligent Robotics and Applications, pp. 105–115. Springer, Berlin Heidelberg, Berlin, Heidelberg (2011)
Poddar, A., Ahmed, S.T., Puhan, N.B.: (2015) Adaptive saliency-weighted obstacle detection for the visually challenged. In: 2015 2nd International Conference on Signal Processing and Integrated Networks (SPIN), pp. 477–482. https://doi.org/10.1109/SPIN.2015.7095312
Prabhakar, G., Kailath, B., Natarajan, S., Kumar, R.: (2017) Obstacle detection and classification using deep learning for tracking in high-speed autonomous driving. In: 2017 IEEE Region 10 Symposium (TENSYMP), pp. 1–6. https://doi.org/10.1109/TENCONSpring.2017.8069972
Rateke, T., von Wangenheim, A.: Systematic literature review for passive vision road obstacle detection. Tech. rep., Brazilian Institute for Digital Convergence - INCoD (2018). https://doi.org/10.13140/RG.2.2.10198.14408
Rateke, T., von Wangenheim, A.: Passive vision road obstacle detection: a literature mapping. Int. J. Comput. Appl. (2020). https://doi.org/10.1080/1206212X.2020.1758877
Ren, S., He, K., Girshick , R., Sun, J.: Faster r-cnn: towards real-time object detection with region proposal networks. In: Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 1. MIT Press, Cambridge, MA, USA, NIPS’15, pp. 91–99 (2015). http://dl.acm.org/citation.cfm?id=2969239.2969250
Sengar, S.S., Mukhopadhyay, S.: Motion detection using block based bi-directional optical flow method. J. Vis. Commun. Image Rep. 49(C), 89–103 (2017a)
Sengar, S.S., Mukhopadhyay, S.: Moving object detection based on frame difference and w4. Sig. Image Video Process. 11(7), 1357–1364 (2017b). https://doi.org/10.1007/s11760-017-1093-8
Sengar, S.S., Mukhopadhyay, S.: Motion segmentation-based surveillance video compression using adaptive particle swarm optimization. Neural Comput. Appl. 32(15), 11443–11457 (2020a). https://doi.org/10.1007/s00521-019-04635-6
Sengar, S.S., Mukhopadhyay, S.: Moving object detection using statistical background subtraction in wavelet compressed domain. Multimed. Tools Appl. 79(9), 5919–5940 (2020). https://doi.org/10.1007/s11042-019-08506-z
Shinzato, P.Y., dos Santos, T.C., Rosero, L.A., Ridel, D.A., Massera, C.M., Alencar, F., Batista, M.P., Hata, A.Y., Orio F.S., Wolf, D.F.: Carina dataset: an emerging-country urban scenario benchmark for road detection systems. In: 2016 IEEE 19th International Conference on Intelligent Transportation Systems (ITSC), pp. 41–46 (2016). https://doi.org/10.1109/ITSC.2016.7795529
STANDARD AN: American National Standard for Safe use of Lasers Outdoors. Orlando, FL (2005)
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z.: Rethinking the inception architecture for computer vision. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2818–2826 (2016). https://doi.org/10.1109/CVPR.2016.308
Urmson, C., Anhalt, J., Bae, H., Bagnell, J.A.D, Baker, C.R., Bittner, R.E., Brown, T., Clark, M.N., Darms, M., Demitrish, D., Dolan, J.M., Duggins, D., Ferguson, D., Galatali, T., Geyer, C.M., Gittleman, M., Harbaugh, S., Hebert, M., Howard, T., Kolski, S., Likhachev, M., Litkouhi, B., Kelly, A., McNaughton, M., Miller, N., Nickolaou, J., Peterson, K., Pilnick, B., Rajkumar, R., Rybski, P., Sadekar, V., Salesky, B., Seo, Y.W., Singh, S., Snider, J.M., Struble, J.C., Stentz, A.T., Taylor, M., Whittaker, W.R.L., Wolkowicki, Z., Zhang, W., Ziglar, J.: Autonomous driving in urban environments: Boss and the urban challenge. Journal of Field Robotics - 2007 DARPA Urban Challenge 25(8), 425–466 (2008)
Yao., Y., Xu, M., Choi, C., Crandall, DJ., Atkins, E.M., Dariush, B.: Egocentric vision-based future vehicle localization for intelligent driving assistance systems. CoRR abs/1809.07408 (2018). http://arxiv.org/abs/1809.07408