ZEPI-Net: Light Field Super Resolution via Internal Cross-Scale Epipolar Plane Image Zero-Shot Learning
Tóm tắt
Many applications of light field (LF) imaging have been limited by the spatial-angular resolution problem, hence the need for efficient super-resolution techniques. Recently, learning-based solutions have achieved remarkably better performances than traditional super-resolution (SR) techniques. Unfortunately, the learning or training process relies heavily on the training dataset, which could be limited for most LF imaging applications. In this paper, we propose a novel LF spatial-angular SR algorithm based on zero-shot learning. We suggest learning cross-scale reusable features in the epipolar plane image (EPI) space, and avoiding explicitly modeling scene priors or implicitly learning that from a large number of LFs. Most importantly, without using any external LFs, the proposed algorithm can simultaneously super-resolve a LF in both spatial and angular domains. Moreover, the proposed solution is free of depth or disparity estimation, which is usually employed by existing LF spatial and angular SR. By using a simple 8-layers fully convolutional network, we show that the proposed algorithm can generate comparable results to the state-of-the-art spatial SR. Our algorithm outperforms the existing methods in terms of angular SR on multiple groups of public LF datasets. The experiment results indicate that the cross-scale features can be well learned and be reused for LF SR in the EPI space.
Tài liệu tham khảo
Wu G, Masia B, Jarabo A, Zhang Y, Wang L, Dai Q, Chai T, Liu Y (2017) Light field image processing: an overview. IEEE J Sel Top Signal Process 11(7):926–954
Ng R, Levoy M, Bredif M, Duval G, Horowitz ME, Hanrahan P (2005) Light field photography with a hand-held plenoptic camera. Stanford University CSTR. http://graphics.stanford.edu/papers/lfcamera/
Jin J, Hou J, Yuan H, Kwong S (2020) Learning light field angular super-resolution via a geometry-aware network. California, AAAI, pp 11141–11148
Wu G, Liu Y, Dai Q, Chai T (2019) Learning sheared epi structure for light field reconstruction. IEEE Trans Image Process 28(7):3261–3273
Shocher A, Cohen N, Irani M (2018) Zero-shot super resolution using deep internal learning. In: IEEE conference on computer vision and pattern recognition. pp. 3118–3126
Cheng Z, Xiong Z, Chen C, et al. (2021) Light field super-resolution with zero-shot learning. In: IEEE conference on computer vision and pattern recognition. pp. 10010–10019
Tian J, Ma K-K (2011) A survey on super-resolution imaging. SIViP 5(3):329–342
Bishop TE, Zanetti S, Favaro P (2009) Light field super-resolution. In: IEEE international conference on computational photography. pp. 1–9
Mitra K, Veeraraghavan A (2012) Light field denoising, super-resolution and stereo camera based refocussing using a GMM light field patch prior. IEEE conference on computer vision and pattern recognition. pp. 22–28
Wanner S, Goldluecke B (2014) Variational light field analysis for disparity estimation and super resolution. IEEE Trans Pattern Anal Mach Intell 36(3):606–619
Rossi M, Frossard P (2018) Geometry-consistent light field super-resolution via graph-based regularization. IEEE Trans Image Process 27(9):4207–4218
Meng N, So HK, Sun X, Lam E (2021) High-dimensional dense residual convolutional neural network for light field reconstruction. IEEE Trans Pattern Anal Mach Intell 43(3):873–886
Yoon Y, Jeon H, Yoo D, Lee J, Kweon IS (2015) Learning a deep convolutional network for light-field image super resolution. In: IEEE international conference on computer vision workshop. pp. 57–65
Wu G, Zhao M, Wang L, Dai Q, Chai T, Liu Y (2017) Light field reconstruction using deep convolutional network on EPI. In: IEEE conference on computer vision and pattern recognition. pp. 1638–1646
Zhang S, Lin Y, Sheng H (2019) Residual networks for light field image super-resolution. In: IEEE international conference on computer vision. pp. 11046–11055
Wang Y, Liu F, Zhang K, Hou G, Sun Z, Tan T (2018) Lfnet: a novel bidirectional recurrent convolutional neural network for light field image super-resolution. IEEE Trans Image Process 27(9):4274–4286
Yeung HWF, Hou J, Chen X, Chen J, Chen Z, Chung YY (2018) Light field spatial super-resolution using deep efficient spatial-angular separable convolution. IEEE Trans Image Process 28(5):2319–2330
Jin J, Hou J, Chen J, Kwong S (2020) Light field spatial super-resolution via deep combinatorial geometry embedding and structural consistency regularization. In: IEEE conference on computer vision and pattern recognition. pp. 2260–2269
Farrugia C (2020) Guillemot light field super-resolution using a low-rank prior and deep convolutional neural networks. IEEE TPAMI 42(9):1162–1175
Mildenhall B, Srinivasan PP, Tancik M, Barron JT, Ng R (2022) Nerf: representing scenes as neural radiance fields for view synthesis. Commun ACM 65(1):99–106
Lee S, Rao R (2004) Scale-based formulations of statistical self-similarity in images. IEEE Int Conf Image Process 4:2323–2326
Zontak M, Irani M (2011) Internal statistics of a single natural image. In: IEEE conference on computer vision and pattern recognition. pp. 977–984
Glasner D, Bagon S, Irani M (2009) Super resolution from a single image. In: IEEE international conference on computer vision
Peng J, Xiong Z, Wang Y, Zhang Y, Liu D (2020) Zero-shot depth estimation from light field using a convolutional neural network. IEEE Trans Comput Imaging 6:682–696
Cheng Z, Xiong Z, Liu D (2020) Light field super-resolution by jointly exploiting internal and external similarities. IEEE Trans Circuits Syst Video Technol 30(8):2604–2616
Wanner S, Meister S, Goldlücke B (2013) Datasets and benchmarks for densely sampled 4D light fields. Annual workshop on vision, modeling and visualization: VMV, pp 225–226
Shi J, Jiang X, Guillemot C (2019) A framework for learning depth from a flexible subset of dense and sparse light field views. IEEE Trans Image Process 28(12):5867–5880
Dong C, Loy CC, He K, Tang X (2014) Learning a deep convolutional network for image super resolution. ECCV 8692:184–199
Kim J, Lee JK, Lee KM (2016) Accurate image super resolution using very deep convolutional networks. In: IEEE conference on computer vision and pattern recognition. pp. 1646–1654