Deep Learning Based Data Fusion for Sensor Fault Diagnosis and Tolerance in Autonomous Vehicles
Tóm tắt
Environmental perception is one of the key technologies to realize autonomous vehicles. Autonomous vehicles are often equipped with multiple sensors to form a multi-source environmental perception system. Those sensors are very sensitive to light or background conditions, which will introduce a variety of global and local fault signals that bring great safety risks to autonomous driving system during long-term running. In this paper, a real-time data fusion network with fault diagnosis and fault tolerance mechanism is designed. By introducing prior features to realize the lightweight network, the features of the input data can be extracted in real time. A new sensor reliability evaluation method is proposed by calculating the global and local confidence of sensors. Through the temporal and spatial correlation between sensor data, the sensor redundancy is utilized to diagnose the local and global confidence level of sensor data in real time, eliminate the fault data, and ensure the accuracy and reliability of data fusion. Experiments show that the network achieves state-of-the-art results in speed and accuracy, and can accurately detect the location of the target when some sensors are out of focus or out of order. The fusion framework proposed in this paper is proved to be effective for intelligent vehicles in terms of real-time performance and reliability.
Tài liệu tham khảo
G Xiang, X Wang, D Liang. Background reconstruction and object extraction based on color and object tracking. Chinese Journal of Mechanical Engineering, 2006, 19(3): 471-474 .
A Krizhevsky, I Sutskever, G E Hinton. Imagenet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems, 2012: 1097-1105.
K He, X Zhang, S Ren, et al. Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016: 770-778.
R Girshick, J Donahue, T Darrell, et al. Rich feature hierarchies for accurate object detection and semantic segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2014: 580-587.
R Girshick. Fast r-cnn. Proceedings of the IEEE International Conference on Computer Vision, 2015: 1440-1448.
S Ren, K He, R Girshick, et al. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in Neural Information Processing Systems, 2015: 91-99.
V Murugan, V Vijaykumar, A Nidhila. A deep learning rcnn approach for vehicle recognition in traffic surveillance system. 2019 International Conference on Communication and Signal Processing (ICCSP), 2019: 0157-0160.
X Liu, W Liu, T Mei, et al. A deep learning-based approach to progressive vehicle re-identification for urban surveillance. European Conference on Computer Vision, 2016: 869-884.
K Shi, H Bao, N Ma. Forward vehicle detection based on incremental learning and fast r-cnn. 2017 13th International Conference on Computational Intelligence and Security (CIS), 2017: 73-76.
Y Qu, L Jiang, X Guo. Moving vehicle detection with convolutional networks in uav videos. 2016 2nd International Conference on Control, Automation and Robotics (ICCAR), 2016: 225-229.
J Redmon, S Divvala, R Girshick, et al. You only look once: Unified, real-time object detection. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016: 779-788.
J Redmon, A Farhadi. Yolo9000: better, faster, stronger. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017: 7263-7271.
J Redmon, A Farhadi. Yolov3: An incremental improvement. arXiv preprint arXiv:1804.02767 (2018).
W Liu, D Anguelov, D Erhan, et al. Ssd: Single shot multibox detector. European Conference on Computer Vision, 2016: 21-37.
N Chodosh, C Wang, S Lucey. Deep convolutional compressed sensing for lidar depth completion. Asian Conference on Computer Vision, 2018: 499-513.
F Mal, S Karaman. Sparse-to-dense: Depth prediction from sparse depth samples and a single image. 2018 IEEE International Conference on Robotics and Automation (ICRA), 2018: 1-8.
A Eldesokey, F Felsberg, F S Khan. Propagating confidences through cnns for sparse data regression. arXiv preprint arXiv:1805.11913 (2018).
S Shao, W Sun, R Yan, et al. A deep learning approach for fault diagnosis of induction motors in manufacturing. Chinese Journal of Mechanical Engineering, 2017, 30(6): 1347-1356.
H Cho, T W Seo, B V Kumar, et al. A multi-sensor fusion system for moving object detection and tracking in urban driving environments. 2014 IEEE International Conference on Robotics and Automation (ICRA), 2014: 1836-1843.
X Chen, H Ma, J Wan, et al. Multi-view 3D object detection network for autonomous driving. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017: 1907-1915.
J Ku, M Mozifian, J Lee, et al. Joint 3D proposal generation and object detection from view aggregation. 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2018: 1-8.
Y He, M Ji, C Zhang, et al. Global exponential stability of neural networks with time-varying delay based on free-matrix-based integral inequality. Neural Networks, 2016, 77: 80-86.
F Shen, Y Hui, R Yan, et al. A new penalty domain selection machine enabled transfer learning for gearbox fault recognition. IEEE Transactions on Industrial Electronics, 2020, 67: 8743-8754.
S Shao, S McAleer, R Yan, et al. Highly accurate machine fault diagnosis using deep transfer learning. IEEE Transactions on Industrial Informatics, 2018, 15(4): 2446-2455.
T Lin, P Dollár, R Girshick, et al. Feature pyramid networks for object detection. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017: 2117-2125.
J Fritsch, T Kuehnl, A Geiger. A new performance measure and evaluation benchmark for road detection algorithms. International Conference on Intelligent Transportation Systems (ITSC), 2013.
T Y Lin, M Maire, S Belongie, et al. Microsoft COCO: Common objects in context. European Conference on Computer Vision, Springer International Publishing, 2014: 740-755.
T Lin, P Goyal, R Girshick, et al. Focal loss for dense object detection. Proceedings of the IEEE International Conference on Computer Vision, 2017: 2980-2988.
R N Rajaram, E Ohn-Bar, M M Trivedi. Refinenet: Refining object detectors for autonomous driving. IEEE Transactions on Intelligent Vehicles, 2016, 1(4): 358-368.
W Bao, B Xu, Z Chen. MonoFENet: Monocular 3D object detection with feature enhancement networks. IEEE Transactions on Image Processing, 2020, 29: 2753-2765.