Panoramic image generation using deep neural networks
Tóm tắt
A traditional approach for panoramic image generation consists of a random sample consensus (RANSAC) algorithm on a set of scale-invariant feature transform (SIFT) correspondences to generate a homography matrix between two images. Although producing adequate results for some type of images, hand-crafted SIFT features are not robust enough for highly varying natural images and the iterative RANSAC algorithm with its randomness does not always find the desired homography matrix. Recently, deep neural networks have been producing significant results in many challenging computer vision problems by learning features from large amounts of data. However, only very few recent works have been applied deep learning to panoramic image generation with the objective of finding feature correspondences and estimating homography matrix. Moreover, the absence of a proper dataset for the image stitching task hinders the standardization of models and comparison of their results. This paper attempts to generate panoramic images by extensively experimenting with various approaches using deep neural networks. The best proposed deep learning model achieved 7.31 and 1.07 pixels of the average absolute value loss for corner difference in X and Y directions, respectively. At the same time, qualitative results demonstrate superiority in comparison with the state-of-the-art SIFT+RANSAC algorithm. Specifically, in 72% of time the proposed framework either produced better results than SIFT+RANSAC or results of the proposed approach and SIFT+RANSAC were indistinguishable. Although SIFT + RANSAC produces better results in 28% of the time with respect to the loss function, our results are still visually comparable in many of these cases. Finally, a novel panoramic image generation dataset is introduced in this paper.
Tài liệu tham khảo
Adel E, Elmogy M (2014) Image stitching based on feature extraction techniques: a survey
Adel E, Elmogy M, Elbakry H (2014) Image stitching based on feature extraction techniques: a survey. Int J Comput Appl (0975-8887)
Baker S, Matthews IA (2004) Lucas-kanade 20 years on: a unifying framework. Int J Comput Vis 56(3):221–255. https://doi.org/10.1023/B:VISI.0000011205.11775.fd
Barakbayeva T, Demirci MF (2022) Fully automatic CNN design with inception and resnet blocks. Neural Comput Appl. https://doi.org/10.1007/s00521-022-07700-9
Bay H, Ess A, Tuytelaars T, Van Gool L (2008) Speeded-up robust features (surf). Comput Vis Image Underst 110(3):346–359
Boykov Y, Kolmogorov V (2004) An experimental comparison of min-cut/max-flow algorithms for energy minimization in vision. IEEE Trans Pattern Anal Mach Intell 26(9):1124–1137
Brown, Lowe (2003) Recognising panoramas. In: Proceedings 9th IEEE international conference on computer vision, vol 2, pp 1218–1225
Brown M, Lowe DG (2007) Automatic panoramic image stitching using invariant features. Int J Comput Vision 74(1):59–73
Bujnak M, Sara R (2007) A robust graph-based method for the general correspondence problem demonstrated on image stitching. In: 2007 IEEE 11th international conference on computer vision, pp 1–8
Chang C-H, Chou C-N, Chang EY (2017) Clkn: Cascaded lucas-kanade networks for image alignment. In: 2017 IEEE conference on computer vision and pattern recognition (CVPR), pp 3777–3785
Chang C-H, Sato Y, Chuang Y-Y (2014) Shape-preserving half-projective warps for image stitching. In: IEEE conference on computer vision and pattern recognition 2014, pp 3254–3261
Collins R (1996) A space-sweep approach to true multi-image matching. In: Proceedings CVPR IEEE computer society conference on computer vision and pattern recognition, pp 358–363
Dai Q, Fang F, Li J, Zhang G, Zhou A (2021) Edge-guided composition network for image stitching. Pattern Recogn 118:108019
Dalal N, Triggs B (2005) Histograms of oriented gradients for human detection. In: IEEE computer society conference on computer vision and pattern recognition (CVPR-05), 1. IEEE, pp 886–893
DeTone D, Malisiewicz T, Rabinovich A (2016) Deep image homography estimation, CoRR, vol. abs/1606.03798, [Online]. arxiv:1606.03798
Duan Y, Han C, Tao X, Geng B, Du Y, Lu J (2020) Panoramic image generation: from 2-d sketch to spherical image. IEEE J Sel Top Signal Process 14(1):194–208
Erlik Nowruzi F, Laganiere R, Japkowicz N (2017) Homography estimation from image pairs with hierarchical convolutional networks. In: Proceedings of the IEEE international conference on computer vision, pp 913–920
Fernandez-Labrador C, Perez-Yus A, Lopez-Nicolas G, Guerrero JJ (2018) Layouts from panoramic images with geometry and deep learning. IEEE Robot Autom Lett 3(4):3153–3160
Fischler MA, Bolles RC (1981) Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun ACM 24(6):381–395
Gao J, Kim SJ, Brown MS (2011) Constructing image panoramas using dual-homography warping. In: CVPR 2011, pp 49–56
Goforth H, Lucey S (2019) Gps-denied UAV localization using pre-existing satellite imagery. In: 2019 International conference on robotics and automation (ICRA), pp 2974–2980
Hoang V-D, Tran D-P, Nhu NG, Pham V-H et al. (2020) Deep feature extraction for panoramic image stitching. In: Asian conference on intelligent information and database systems. Springer, Cham, pp 141–151
Hou J, Lin W, Zhao B (2020) Content-dependency reduction with multi-task learning in blind stitched panoramic image quality assessment. In: 2020 IEEE international conference on image processing (ICIP). IEEE, pp 3463–3467
Jia J, Tang C-K (2005) Eliminating structure and intensity misalignment in image stitching. In: 10th IEEE international conference on computer vision (ICCV–05), vol 1, 2, pp 1651–1658
Kang L, Wei Y, Jiang J, Xie Y (2019) Robust cylindrical panorama stitching for low-texture scenes based on image alignment using deep learning and iterative optimization. Sensors 19(23):5310
Kingma DP, Ba J (2014) Adam: a method for stochastic optimization, arXiv preprint arXiv:1412.6980
Lee K-Y, Sim J-Y (2020) Warping residual based image stitching for large parallax. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 8198–8206
Lee H, Lee S, Choi O (2020) Improved method on image stitching based on optical flow algorithm. Int J Eng Bus Manag 12:1847979020980928
Le H, Liu F, Zhang S, Agarwala A (2020) Deep homography estimation for dynamic scenes. In: 2020 IEEE/CVF conference on computer vision and pattern recognition, CVPR 2020, Seattle, WA, USA, June 13–19. Computer Vision Foundation/IEEE, pp 7649–7658
Levin A, Zomet A, Peleg S, Weiss Y (2004) Seamless image stitching in the gradient domain. In: European conference on computer vision. Springer, pp 377–389
Lin W-Y, Liu S, Matsushita Y, Ng T-T, Cheong L-F (2011) Smoothly varying affine stitching. In: CVPR. IEEE, pp 345–352
Liu S, Yuan L, Tan P, Sun J (2013) Bundled camera paths for video stabilization. ACM Trans Graph. https://doi.org/10.1145/2461912.bib51
Lowe DG (2004) Distinctive image features from scale-invariant keypoints. Int J Comput Vision 60(2):91–110
Mur-Artal R, Montiel JMM, Tardós JD (2015) Orb-slam: a versatile and accurate monocular slam system. IEEE Trans Rob 31(5):1147–1163
Nie L, Lin C, Liao K, Liu M, Zhao Y (2020) A view-free image stitching network based on global homography. J Vis Commun Image Represent 73:102950
Nie L, Lin C, Liao K, Liu S, Zhao Y (2021) Unsupervised deep image stitching: reconstructing stitched features to images. IEEE Trans Image Process
Nocedal J, Wright SJ (2006) Numerical optimization, 2nd edn. Springer, New York
Peleg S, Rousso B, Rav-Acha A, Zomet A (2000) Mosaicing on adaptive manifolds. IEEE Trans Pattern Anal Mach Intell 22(10):1144–1154
Philbin J, Chum O, Isard M, Sivic J, Zisserman A (2007) Object retrieval with large vocabularies and fast spatial matching. In: 2007 IEEE conference on computer vision and pattern recognition. IEEE, pp 1–8
Simon G, Fitzgibbon A, Zisserman A (2000) Markerless tracking using planar structures in the scene. In: Proceedings IEEE and ACM international symposium on augmented reality (ISAR 2000), pp 120–128
Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition, arXiv preprint arXiv:1409.1556
Ullah H, Irfan M, Han K, Lee JW (2020) Dlnr-siqa: deep learning-based no-reference stitched image quality assessment. Sensors 20(22):6457
Uyttendaele M, Eden A, Skeliski R (2001) Eliminating ghosting and exposure artifacts in image mosaics. In: Proceedings of the 2001 IEEE computer society conference on computer vision and pattern recognition. CVPR 2001, vol 2, pp II–II
Wang Z, Chen Y, Zhu Z, Zhao W (2016) An automatic panoramic image mosaic method based on graph model. Multimedia Tools Appl 75(5):2725–2740
Wei L, Zhong Z, Lang C, Yi Z (2019) A survey on image and video stitching. Virt Real Intell Hardware 1(1):55–83
Wyawahare MV, Patil P, Abhyankar HK (2009) Image registration techniques: an overview. Int J Signal Process Image Process Pattern Recogn 2:09
Yi K Moo, Trulls E, Ono Y, Lepetit V, Salzmann M, Fua P, (2018) Learning to find good correspondences. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2666–2674
Ying J, Shen H-L, Cao S-Y (2022) Unaligned hyperspectral image fusion via registration and interpolation modeling. IEEE Trans Geosci Remote Sens 60:1–14
Zaragoza J, Chin T-J, Brown MS, Suter D (2013) As-projective-as-possible image stitching with moving dlt. In: IEEE conference on computer vision and pattern recognition 2013, pp 2339–2346
Zhang H, Zhao M (2021) Panoramic image stitching using double encoderdecoders. SN Comput Sci 2(2):1–12
Zhang G, He Y, Chen W, Jia J, Bao H (2016) Multi-viewpoint panorama construction with wide-baseline images. IEEE Trans Image Process 25(7):3099–3111
Zhang F, Liu F (2014) Parallax-tolerant image stitching. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3262–3269
Zhao Q, Ma Y, Zhu C, Yao C, Feng B, Dai F (2021) Image stitching via deep homography estimation. Neurocomputing 450:219–229
Zhao Y, Huang X, Zhang Z (2021) Deep lucas-kanade homography for multimodal image alignment. In: IEEE conference on computer vision and pattern recognition, CVPR 2021, virtual, June 19–25, 2021. Computer Vision Foundation/IEEE, pp 15950–15959
Zhi Q, Cooperstock JR (2012) Toward dynamic image mosaic generation with robustness to parallax. IEEE Trans Image Process 21(1):366–378