Perception consistency ultrasound image super-resolution via self-supervised CycleGAN
Tóm tắt
Due to the limitations of sensors, the transmission medium, and the intrinsic properties of ultrasound, the quality of ultrasound imaging is always not ideal, especially its low spatial resolution.
To remedy this situation, deep learning networks have been recently developed for ultrasound image super-resolution (SR) because of the powerful approximation capability.
However, most current supervised SR methods are not suitable for ultrasound medical images because the medical image samples are always rare, and usually, there are no low-resolution (LR) and high-resolution (HR) training pairs in reality. In this work, based on self-supervision and cycle generative adversarial network, we propose a new perception consistency ultrasound image SR method, which only requires the LR ultrasound data and can ensure the re-degenerated image of the generated SR one to be consistent with the original LR image, and vice versa. We first generate the HR fathers and the LR sons of the test ultrasound LR image through image enhancement, and then make full use of the cycle loss of LR–SR–LR and HR–LR–SR and the adversarial characteristics of the discriminator to promote the generator to produce better perceptually consistent SR results. The evaluation of PSNR/IFC/SSIM, inference efficiency and visual effects under the benchmark CCA-US and CCA-US datasets illustrate our proposed approach is effective and superior to other state-of-the-art methods.
Tài liệu tham khảo
Blau Y, Michaeli T (2018) The perception–distortion tradeoff. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 6228–6237
Bulat A, Yang J, Tzimiropoulos G (2018) To learn image super-resolution, use a gan to learn how to do image degradation first. In: Proceedings of the European conference on computer vision (ECCV), pp 185–200
Choi W, Kim M, HakLee J, Kim J, BeomRa J (2018) Deep CNN-based ultrasound super-resolution for high-speed high-resolution b-mode imaging. In: Proceedings of the IEEE international ultrasonics symposium, pp 1–4. https://doi.org/10.1109/ULTSYM.2018.8580032
Diamantis K, Greenaway AH, Anderson T, Jensen JA, Dalgarno PA, Sboros V (2018) Super-resolution axial localization of ultrasound scatter using multi-focal imaging. IEEE Trans Biomed Eng 65(8):1840–1851
Dong C, Loy CC, He K, Tang X (2016) Image super-resolution using deep convolutional networks. IEEE Trans Pattern Anal Mach Intell 38(2):295–307
Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2014) Generative adversarial nets. In: Advances in neural information processing systems, pp 2672–2680
Hudson JM, Williams R, Tremblay-Darveau C, Sheeran PS, Milot L, Bjarnason GA, Burns PN (2015) Dynamic contrast enhanced ultrasound for therapy monitoring. Eur J Radiol 84(9):1650–1657
Isola P, Zhu JY, Zhou T, Efros AA (2017) Image-to-image translation with conditional adversarial networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1125–1134
Kim J, Kwon LJ, Mu LK (2016) Accurate image super-resolution using very deep convolutional networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1646–1654
Ledig C, Theis L, Huszár F, Caballero J, Cunningham A, Acosta A, Aitken A, Tejani A, Totz J, Wang Z et al (2017) Photo-realistic single image super-resolution using a generative adversarial network. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4681–4690
Liang Y, Wang J, Zhou S, Gong Y, Zheng N (2016) Incorporating image priors with deep convolutional neural networks for image super-resolution. Neurocomputing 194:340–347
Lim B, Son S, Kim H, Nah S, Mu LK (2017) Enhanced deep residual networks for single image super-resolution. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp 136–144
Lim Y, Bliesener Y, Narayanan S, Nayak KS (2020) Deblurring for spiral real-time MRI using convolutional neural networks. Magn Reson Med 84:3438–3452
Liu H, Huang D, Hou S, Yue R (2017) Large size single image fast defogging and the real time video defogging FPGA architecture. Neurocomputing 269:97–107
Liu H, Fu Z, Han J, Shao L, Hou S, Chu Y (2019a) Single image super-resolution using multi-scale deep encoder–decoder with phase congruency edge map guidance. Inf Sci 473:44–58
Liu H, Qin J, Fu Z, Li X, Han J (2020a) Fast simultaneous image super-resolution and motion deblurring with decoupled cooperative learning. J Real-time Image Process. https://doi.org/10.1007/s11554-020-00976-x
Liu J, Liu H, Zheng X, Han J (2020) Exploring multi-scale deep encoder–decoder and patchgan for perceptual ultrasound image super-resolution. In: International conference on neural computing for advanced applications. Springer, pp 47–59
Liu K, Ma Y, Xiong H, Yan Z, Zhou ZJ, Fang P, Liu C (2019b) Medical image super-resolution method based on dense blended attention network. arXiv:1905.05084
Lu J, Liu W (2018) Unsupervised super-resolution framework for medical ultrasound images using dilated convolutional neural networks. In: Proceedings of the IEEE 3rd international conference on image, vision and computing. IEEE, pp 739–744
Ma J, Wang X, Jiang J (2020) Image superresolution via dense discriminative network. IEEE Trans Ind Electron 67(7):5687–5695
Mallat S (1999) A wavelet tour of signal processing. Academic press, New York
Morin R, Bidon S, Basarab A, Kouamé D (2013) Semi-blind deconvolution for resolution enhancement in ultrasound imaging. In: Proceedings of the IEEE international conference on image processing. IEEE, pp 1413–1417
Park SJ, Son H, Cho S, Hong KS, Lee S (2018) Srfeat: single image super-resolution with feature discrimination. In: Proceedings of the European conference on computer vision (ECCV), pp 439–455
Ronneberger O, Fischer P, Brox T (2015) U-net: convolutional networks for biomedical image segmentation. In: Proceedings of the international conference on medical image computing and computer-assisted intervention. Springer, pp 234–241
Sakkos D, Liu H, Han J, Shao L (2018) End-to-end video background subtraction with 3D convolutional neural networks. Multimed Tools Appl 77(17):23023–23041
Sheikh HR, Bovik AC, De Veciana G (2005) An information fidelity criterion for image quality assessment using natural scene statistics. IEEE Trans Image Process 14(12):2117–2128
Shocher A, Cohen N, Irani M (2018) “Zero-shot” super-resolution using deep internal learning. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3118–3126
Shrivastava A, Pfister T, Tuzel O, Susskind J, Wang W, Webb R (2017) Learning from simulated and unsupervised images through adversarial training. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2107–2116
Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556
Skourt BA, El Hassani A, Majda A (2018) Lung CT image segmentation using deep neural networks. Procedia Comput Sci 127:109–113
Umehara K, Ota J, Ishida T (2018) Application of super-resolution convolutional neural network for enhancing image resolution in chest CT. J Digit Imaging 31(4):441–450
van Sloun RJ, Solomon O, Bruce M, Khaing ZZ, Eldar YC, Mischi M (2019) Deep learning for super-resolution vascular ultrasound imaging. In: ICASSP 2019—2019 IEEE international conference on acoustics. Speech and signal processing (ICASSP). IEEE, pp 1055–1059
Wang Z, Bovik AC, Sheikh HR, Simoncelli EP (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 13(4):600–612
Wang Z, Yi P, Jiang K, Jiang J, Han Z, Lu T, Ma J (2019) Multi-memory convolutional neural network for video super-resolution. IEEE Trans Image Process 28(5):2530–2544
Zhao N, Wei Q, Basarab A, Kouamé D, Tourneret JY (2016) Single image super-resolution of medical ultrasound images using a fast algorithm. In: Proceedings of the IEEE 13th international symposium on biomedical imaging. IEEE, pp 473–476
Zhu J, Yang G, Lio P (2019) How can we make gan perform better in single medical image super-resolution? A lesion focused multi-scale approach. In: Proceedings of the 16th IEEE international symposium on biomedical imaging (ISBI), pp 1669–1673
Zhu JY, Park T, Isola P, Efros AA (2017) Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Proceedings of the IEEE international conference on computer vision, pp 2223–2232