Paced-curriculum distillation with prediction and label uncertainty for image segmentation
Tóm tắt
In curriculum learning, the idea is to train on easier samples first and gradually increase the difficulty, while in self-paced learning, a pacing function defines the speed to adapt the training progress. While both methods heavily rely on the ability to score the difficulty of data samples, an optimal scoring function is still under exploration. Distillation is a knowledge transfer approach where a teacher network guides a student network by feeding a sequence of random samples. We argue that guiding student networks with an efficient curriculum strategy can improve model generalization and robustness. For this purpose, we design an uncertainty-based paced curriculum learning in self-distillation for medical image segmentation. We fuse the prediction uncertainty and annotation boundary uncertainty to develop a novel paced-curriculum distillation (P-CD). We utilize the teacher model to obtain prediction uncertainty and spatially varying label smoothing with Gaussian kernel to generate segmentation boundary uncertainty from the annotation. We also investigate the robustness of our method by applying various types and severity of image perturbation and corruption. The proposed technique is validated on two medical datasets of breast ultrasound image segmentation and robot-assisted surgical scene segmentation and achieved significantly better performance in terms of segmentation and robustness. P-CD improves the performance and obtains better generalization and robustness over the dataset shift. While curriculum learning requires extensive tuning of hyper-parameters for pacing function, the level of performance improvement suppresses this limitation.
Tài liệu tham khảo
Bengio Y, Louradour J, Collobert R, Weston J (2009) Curriculum learning. In: Proceedings of the 26th annual international conference on machine learning, pp 41– 48
Sinha S, Garg A, Larochelle H (2020) Curriculum by smoothing. Adv Neural Inf Process Syst 33:21653–21664
Kumar M, Packer B, Koller D (2010) Self-paced learning for latent variable models. Adv Neural Inf Process Syst 23:1189–1197
Jiang L, Meng D, Zhao Q, Shan S, Hauptmann AG ( 2015) Self-paced curriculum learning. In: Twenty-Ninth AAAI Conference on Artificial Intelligence
Zhao H, Sun X, Dong J, Dong Z, Li Q (2021) Knowledge distillation via instance-level sequence learning. Knowl-Based Syst 233:107519
Wei J, Suriawinata A, Ren B, Liu X, Lisovsky M, Vaickus L, Brown C, Baker M, Nasir-Moin M, Tomita N, Torresani L, Wei J, Hassanpour S ( 2021) Learn like a pathologist: Curriculum learning by annotator agreement for histopathology image classification. In: Proceedings of the IEEE/cvf winter conference on applications of computer vision (WACV), pp 2473– 2483
Shi W, Song Y, Zhou H, Li B, Li L (2021) Follow your path: a progressive method for knowledge distillation. Joint European conference on machine learning and knowledge discovery in databases. Springer, Berlin, pp 596–611
Ye D, Chen G, Zhang W, Chen S, Yuan B, Liu B, Chen J, Liu Z, Qiu F, Yu H et al (2020) Towards playing full MOBA games with deep reinforcement learning. Adv Neural Inf Process Syst 33:621–632
Zhu Q, Chen X, Wu P, Liu J, Zhao D (2021) Combining curriculum learning and knowledge distillation for dialogue generation. In: Findings of the Association for Computational Linguistics: EMNLP, pp 1284– 1295
Xiang L, Ding G, Han J (2020) Learning from multiple experts: self-paced knowledge distillation for long-tailed classification. European conference on computer vision. Springer, Berlin, pp 247–263
Wang Y, Gan W, Yang J, Wu W, Yan J ( 2019) Dynamic curriculum learning for imbalanced data classification. In: Proceedings of the IEEE/CVF international conference on computer Vision, pp 5017– 5026
Chang H-S, Learned-Miller E, McCallum A (2017) Active bias: training more accurate neural networks by emphasizing high variance samples. Adv Neu Inform Proc Syst. 30
Zhou Y, Yang B, Wong DF, Wan Y, Chao LS ( 2020) Uncertainty-aware curriculum learning for neural machine translation. In: Proceedings of the 58th Annual Meeting of the association for computational linguistics, pp 6934– 6944
Platanios EA, Stretcu O, Neubig G, Poczos B, Mitchell TM ( 2019) Competence-based curriculum learning for neural machine translation. In: International conference of the north american chapter of the association for computational linguistics: Human Language Technologies, pp 1162– 1172
Zhang Y, Abbeel P, Pinto L (2020) Automatic curriculum learning through value disagreement. Adv Neural Inf Process Syst 33:7648–7659
Lee HJ, Kim JU, Lee S, Kim HG, Ro YM ( 2020) Structure boundary preserving segmentation for medical image with ambiguous boundary. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 4817– 4826
Islam M, Glocker B ( 2021) Spatially varying label smoothing: Capturing uncertainty from expert annotations. In: international conference on information processing in medical imaging, pp 677– 688 . Springer
Tang P, Yang P, Nie D, Wu X, Zhou J, Wang Y (2022) Unified medical image segmentation by learning from uncertainty in an end-to-end manner. Knowl-Based Syst 241:108215
Allan M, Kondo S, Bodenstedt S, Leger S, Kadkhodamohammadi R, Luengo I, Fuentes F, Flouty E, Mohammed AK, Pedersen M, Kori AK, Varghese A, Krishnamurthi G, Rauber D, Mendel R, Palm C, Bano S, Saibro G, Shih C, Chiang H, Zhuang J, Yang J, Iglovikov V, Dobrenkii A, Reddiboina M, Reddy A, Liu X, Gao C, Unberath M, Azizian M, Stoyanov D, Maier-Hein L, Speidel S (2020) 2018 robotic scene segmentation challenge. arXiv preprint arXiv:2001.11190
Al-Dhabyani W, Gomaa M, Khaled H, Fahmy A (2020) Dataset of breast ultrasound images. Data Brief 28:104863
Hinton G, Vinyals O, Dean J (2015) Distilling the knowledge in a neural network. arXiv preprint. 2(7)
Kim K, Ji B, Yoon D, Hwang S ( 2021) Self-knowledge distillation with progressive refinement of targets. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 6567– 6576
Guo C, Pleiss G, Sun Y, Weinberger KQ ( 2017) On calibration of modern neural networks. In: International conference on machine learning, pp 1321– 1330 . PMLR
Ronneberger O, Fischer P, Brox T (2015) U-net: Convolutional networks for biomedical image segmentation. International conference on medical image computing and computer-assisted intervention. Springer, Berlin, pp 234–241
Chaurasia A, Culurciello E (2017) Linknet: Exploiting encoder representations for efficient semantic segmentation. In: 2017 IEEE visual communications and image processing (VCIP), pp 1– 4 . IEEE
Chen L-C, Zhu Y, Papandreou G, Schroff F, Adam H ( 2018) Encoder-decoder with atrous separable convolution for semantic image segmentation. In: Proceedings of the european conference on computer vision (ECCV), pp 801– 818
Gong T, Zhao Q, Meng D, Xu Z (2016) Why curriculum learning & self-paced learning work in big/noisy data: a theoretical perspective. Big Data Inform Anal 1(1):111
Wu X, Dyer E, Neyshabur B (2020) When do curricula work? arXiv preprint arXiv:2012.03107
Hendrycks D, Dietterich T (2019) Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint arXiv:1903.12261