DUDES: Deep Uncertainty Distillation using Ensembles for Semantic Segmentation
Tóm tắt
The intersection of deep learning and photogrammetry unveils a critical need for balancing the power of deep neural networks with interpretability and trustworthiness, especially for safety-critical application like autonomous driving, medical imaging, or machine vision tasks with high demands on reliability. Quantifying the predictive uncertainty is a promising endeavour to open up the use of deep neural networks for such applications. Unfortunately, most current available methods are computationally expensive. In this work, we present a novel approach for efficient and reliable uncertainty estimation for semantic segmentation, which we call Deep Uncertainty Distillation using Ensembles for Segmentation (DUDES). DUDES applies student-teacher distillation with a Deep Ensemble to accurately approximate predictive uncertainties with a single forward pass while maintaining simplicity and adaptability. Experimentally, DUDES accurately captures predictive uncertainties without sacrificing performance on the segmentation task and indicates impressive capabilities of highlighting wrongly classified pixels and out-of-domain samples through high uncertainties on the Cityscapes and Pascal VOC 2012 dataset. With DUDES, we manage to simultaneously simplify and outperform previous work on Deep-Ensemble-based Uncertainty Distillation.
Từ khóa
Tài liệu tham khảo
Besnier V, Picard D, Briot A (2021) Learning Uncertainty for Safety-Oriented Semantic Segmentation in Autonomous Driving. In: 2021 IEEE International Conference on Image Processing (ICIP), IEEE, Anchorage, AK, USA, pp 3353–3357, https://doi.org/10.1109/ICIP42928.2021.9506719
Bianco S, Cadene R, Celona L, Napoletano P (2018) Benchmark Analysis of Representative Deep Neural Network Architectures. In: IEEE Access, vol 6, pp 64270–64277, https://doi.org/10.1109/ACCESS.2018.2877890
Blundell C, Cornebise J, Kavukcuoglu K, Wierstra D (2015) Weight uncertainty in neural network. In: Bach F, Blei D (eds) Proceedings of the 32nd International Conference on Machine Learning, PMLR, Lille, France, vol 37, pp 1613–1622
Chen LC, Zhu Y, Papandreou G, Schroff F, Adam H (2018) Encoder-decoder with atrous separable convolution for semantic image segmentation. In: Proceedings of the European Conference on Computer Vision (ECCV)
Cordts M, Omran M, Ramos S, Rehfeld T, Enzweiler M, Benenson R, Franke U, Roth S, Schiele B (2016) The cityscapes dataset for semantic urban scene understanding. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
Deng J, Dong W, Socher R, Li LJ, Kai Li, Li Fei-Fei (2009) ImageNet: A large-scale hierarchical image database. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition, IEEE, Miami, FL, pp 248–255, https://doi.org/10.1109/CVPR.2009.5206848
Everingham M, Van Gool L, Williams CK, Winn J, Zisserman A (2010) The PASCAL visual object classes (VOC) challenge. International journal of computer vision 88:303–338
Fort S, Hu H, Lakshminarayanan B (2020) Deep Ensembles: A Loss Landscape Perspective. arXiv:191202757 1912.02757
Gal Y, Ghahramani Z (2016) Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. In: Balcan MF, Weinberger KQ (eds) Proceedings of The 33rd International Conference on Machine Learning, PMLR, New York, New York, USA, Proceedings of Machine Learning Research, vol 48, pp 1050–1059, https://proceedings.mlr.press/v48/gal16.html
Gawlikowski J, Tassi CRN, Ali M, Lee J, Humt M, Feng J, Kruspe A, Triebel R, Jung P, Roscher R, Shahzad M, Yang W, Bamler R, Zhu XX (2022) A Survey of Uncertainty in Deep Neural Networks. arXiv:210703342 2107.03342
Guo C, Pleiss G, Sun Y, Weinberger KQ (2017) On calibration of modern neural networks. In: Precup D, Teh YW (eds) Proceedings of the 34th International Conference on Machine Learning, PMLR, Proceedings of Machine Learning Research, vol 70, pp 1321–1330, https://proceedings.mlr.press/v70/guo17a.html
Gustafsson FK, Danelljan M, Schon TB (2020) Evaluating scalable bayesian deep learning methods for robust computer vision. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pp 318–319
He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
He K, Girshick R, Dollar P (2019) Rethinking ImageNet pre-training. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)
Hinton G, Vinyals O, Dean J (2015) Distilling the knowledge in a neural network. In: NIPS Deep Learning and Representation Learning Workshop, http://arxiv.org/abs/1503.02531
Holder CJ, Shafique M (2021) Efficient Uncertainty Estimation in Semantic Segmentation via Distillation. In: 2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW), IEEE, Montreal, BC, Canada, pp 3080–3087, https://doi.org/10.1109/ICCVW54120.2021.00343
Howard AG, Zhu M, Chen B, Kalenichenko D, Wang W, Weyand T, Andreetto M, Adam H (2017) MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications. arXiv:170404861 1704.04861
Hu X, Fu CW, Zhu L, Heng PA (2019) Depth-attentional features for single-image rain removal. In: Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pp 8022–8031
Jospin LV, Laga H, Boussaid F, Buntine W, Bennamoun M (2022) Hands-on Bayesian neural networks—a tutorial for deep learning users. IEEE Computational Intelligence Magazine 17(2):29–48, https://doi.org/10.1109/MCI.2022.3155327
Kang J, Gwak J (2019) Ensemble of instance segmentation models for polyp segmentation in colonoscopy images. IEEE Access 7:26440–26447
Lakshminarayanan B, Pritzel A, Blundell C (2017) Simple and scalable predictive uncertainty estimation using deep ensembles. In: Guyon I, Luxburg UV, Bengio S, Wallach H, Fergus R, Vishwanathan S, Garnett R (eds) Advances in Neural Information Processing Systems, Curran Associates, Inc., vol 30
Lee K, Lee H, Lee K, Shin J (2018) Training Confidence-calibrated Classifiers for Detecting Out-of-Distribution Samples. arXiv:171109325 1711.09325
Leibig C, Allken V, Ayhan MS, Berens P, Wahl S (2017) Leveraging uncertainty information from deep neural networks for disease detection. Scientific Reports 7(1):17816, https://doi.org/10.1038/s41598-017-17876-z
Liu J, Lin Z, Padhy S, Tran D, Bedrax Weiss T, Lakshminarayanan B (2020) Simple and principled uncertainty estimation with deterministic deep learning via distance awareness. Advances in Neural Information Processing Systems 33:7498–7512
Loquercio A, Segu M, Scaramuzza D (2020) A General Framework for Uncertainty Estimation in Deep Learning. IEEE Robotics and Automation Letters 5(2):3153–3160
Lumini A, Nanni L, Maguolo G (2021) Deep ensembles based on stochastic activations for semantic segmentation. Signals 2(4):820–833
MacKay DJC (1992) A Practical Bayesian Framework for Backpropagation Networks. Neural Computation 4(3):448–472, https://doi.org/10.1162/neco.1992.4.3.448
Malinin A, Mlodozeniec B, Gales M (2019) Ensemble Distribution Distillation. arXiv:190500076 1905.00076
Marmanis D, Wegner JD, Galliani S, Schindler K, Datcu M, Stilla U (2016) Semantic segmentation of aerial images with an ensemble of cnns. ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences 3:473–480
McAllister R, Gal Y, Kendall A, van der Wilk M, Shah A, Cipolla R, Weller A (2017) Concrete Problems for Autonomous Vehicle Safety: Advantages of Bayesian Deep Learning. In: Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, International Joint Conferences on Artificial Intelligence Organization, Melbourne, Australia, pp 4745–4753, https://doi.org/10.24963/ijcai.2017/661
Minaee S, Boykov Y, Porikli F, Plaza A, Kehtarnavaz N, Terzopoulos D (2022) Image segmentation using deep learning: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence 44(7):3523–3542, https://doi.org/10.1109/TPAMI.2021.3059968
Mukhoti J, Kirsch A, van Amersfoort J, Torr PH, Gal Y (2023) Deep deterministic uncertainty: A new simple baseline. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 24384–24394
Naeini MP, Cooper G, Hauskrecht M (2015) Obtaining well calibrated probabilities using bayesian binning. In: Proceedings of the AAAI conference on artificial intelligence, vol 29
Nanni L, Fusaro D, Fantozzi C, Pretto A (2023) Improving existing segmentators performance with zero-shot segmentators. Entropy 25(11):1502
Nigam I, Huang C, Ramanan D (2018) Ensemble knowledge transfer for semantic segmentation. In: 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), IEEE, pp 1499–1508
Ovadia Y, Fertig E, Ren J, Nado Z, Sculley D, Nowozin S, Dillon J, Lakshminarayanan B, Snoek J (2019) Can you trust your model’s uncertainty? Evaluating predictive uncertainty under dataset shift. In: Wallach H, Larochelle H, Beygelzimer A, d’Alché-Buc F, Fox E, Garnett R (eds) Advances in Neural Information Processing Systems, Curran Associates, Inc., vol 32
Robbins H, Monro S (1951) A Stochastic Approximation Method. The Annals of Mathematical Statistics 22(3):400–407, https://doi.org/10.1214/aoms/1177729586
Romero A, Ballas N, Kahou SE, Chassang A, Gatta C, Bengio Y (2015) FitNets: Hints for Thin Deep Nets. arXiv:14126550 1412.6550
Ronneberger O, Fischer P, Brox T (2015) U‑net: Convolutional networks for biomedical image segmentation. In: Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5–9, 2015, Proceedings, Part III 18, Springer, pp 234–241
Sakaridis C, Dai D, Van Gool L (2018) Semantic foggy scene understanding with synthetic data. International Journal of Computer Vision 126:973–992
Shen Y, Zhang Z, Sabuncu MR, Sun L (2021) Real-time uncertainty estimation in computer vision via uncertainty-aware distribution distillation. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp 707–716
Simpson IJA, Vicente S, Campbell NDF (2022) Learning structured gaussians to approximate deep ensembles. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp 366–374
Srivastava N, Hinton G, Krizhevsky A, Sutskever I, Salakhutdinov R (2014) Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research 15(56):1929–1958
Steger C, Ulrich M, Wiedemann C (2018) Machine Vision Algorithms and Applications. John Wiley & Sons, 2nd Edition
Thanh NC, Long TQ, et al. (2020) Polyp segmentation in colonoscopy images using ensembles of u‑nets with efficientnet and asymmetric similarity loss function. In: 2020 RIVF International Conference on Computing and Communication Technologies (RIVF), IEEE, pp 1–6
Ulrich M, Hillemann M (2024) Uncertainty-aware hand–eye calibration. IEEE Transactions on Robotics 40:573–591, https://doi.org/10.1109/TRO.2023.3330609
Van Amersfoort J, Smith L, Teh YW, Gal Y (2020) Uncertainty estimation using a single deep deterministic neural network. In: International conference on machine learning, PMLR, pp 9690–9700
Wursthorn K, Hillemann M, Ulrich M (2022) Comparison of uncertainty quantification methods for CNN-based regression. The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences XLIII-B2-2022:721–728
Xie E, Wang W, Yu Z, Anandkumar A, Alvarez JM, Luo P (2021) Segformer: Simple and efficient design for semantic segmentation with transformers. Advances in Neural Information Processing Systems 34:12077–12090