Learning conditional photometric stereo with high-resolution features

Springer Science and Business Media LLC - Tập 8 Số 1 - Trang 105-118 - 2022
Yakun Ju1, Yuxin Peng2, Muwei Jian3, Feng Gao1, Junyu Dong1
1Department of Computer Science and Technology, Ocean University of China, Qingdao 266100, China
2Wangxuan Institute of Computer Technology, Peking University, Beijing, 100871, China
3School of Computer Science and Technology, Shandong University of Finance and Economics, Jinan, 250002, China

Tóm tắt

AbstractPhotometric stereo aims to reconstruct 3D geometry by recovering the dense surface orientation of a 3D object from multiple images under differing illumination. Traditional methods normally adopt simplified reflectance models to make the surface orientation computable. However, the real reflectances of surfaces greatly limit applicability of such methods to real-world objects. While deep neural networks have been employed to handle non-Lambertian surfaces, these methods are subject to blurring and errors, especially in high-frequency regions (such as crinkles and edges), caused by spectral bias: neural networks favor low-frequency representations so exhibit a bias towards smooth functions. In this paper, therefore, we propose a self-learning conditional network with multi-scale features for photometric stereo, avoiding blurred reconstruction in such regions. Our explorations include: (i) a multi-scale feature fusion architecture, which keeps high-resolution representations and deep feature extraction, simultaneously, and (ii) an improved gradient-motivated conditionally parameterized convolution (GM-CondConv) in our photometric stereo network, with different combinations of convolution kernels for varying surfaces. Extensive experiments on public benchmark datasets show that our calibrated photometric stereo method outperforms the state-of-the-art.

Từ khóa


Tài liệu tham khảo

Jian, M. W.; Dong, J. Y.; Gong, M. G.; Yu, H.; Nie, L. Q.; Yin, Y. L.; Lam, K.-M. Learning the traditional art of Chinese calligraphy via three-dimensional reconstruction and assessment. IEEE Transactions on Multimedia Vol. 22, No. 4, 970–979, 2020.

Woodham, R. J. Photometric method for determining surface orientation from multiple images. Optical Engineering Vol. 19, No. 1, 191139, 1980.

Khanian, M.; Boroujerdi, A. S.; Breuß, M. Photometric stereo for strong specular highlights. Computational Visual Media Vol. 4, No. 1, 83–102, 2018.

Wu, L.; Ganesh, A.; Shi, B.; Matsushita, Y.; Wang, Y.; Ma, Y. Robust photometric stereo via low-rank matrix completion and recovery. In: Proceedings of the Asian Conference on Computer Vision, 703–717, 2010.

Ikehata, S.; Wipf, D.; Matsushita, Y.; Aizawa, K. Robust photometric stereo using sparse regression. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 318–325, 2012.

Ikehata, S.; Aizawa, K. Photometric stereo using constrained bivariate regression for general isotropic surfaces. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2187–2194, 2014.

Higo, T.; Matsushita, Y.; Ikeuchi, K. Consensus photometric stereo. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 1157–1164, 2010.

Wei, K.; Yang, M. L.; Wang, H.; Deng, C.; Liu, X. L. Adversarial fine-grained composition learning for unseen attribute-object recognition. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, 3740–3748, 2019.

Yang, X.; Deng, C.; Liu, T. L.; Tao, D. C. Heterogeneous graph attention network for unsupervised multiple-target domain adaptation. IEEE Transactions on Pattern Analysis and Machine Intelligence doi: https://doi.org/10.1109/TPAMI.2020.3026079, 2020.

Wei, K.; Deng, C.; Yang, X. Lifelong zero-shot learning. In: Proceedings of the 29th International Joint Conference on Artificial Intelligence, 551–557, 2020.

Santo, H.; Samejima, M.; Sugano, Y.; Shi, B. X.; Matsushita, Y. Deep photometric stereo network. In: Proceedings of the IEEE International Conference on Computer Vision Workshops, 501–509, 2017.

Chen, G. Y.; Han, K.; Wong, K. Y. K. PS-FCN: A flexible learning framework for photometric stereo. In: Computer Vision — ECCV 2018. Lecture Notes in Computer Science, Vol. 11213. Ferrari, V.; Hebert, M.; Sminchisescu, C.; Weiss, Y. Eds. Springer Cham, 3–19, 2018.

Chen, G.; Han, K.; Shi, B.; Matsushita, Y.; Wong, K.-Y. K. Self-calibrating deep photometric stereo networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 8739–8747, 2019.

Ju, Y. K.; Jian, M. W.; Dong, J. Y.; Lam, K. M. Learning photometric stereo via manifold-based mapping. In: Proceedings of the IEEE International Conference on Visual Communications and Image Processing, 411–414, 2020.

Taniai, T.; Maehara, T. Neural inverse rendering for general reflectance photometric stereo. In: Proceedings of the 35th International Conference on Machine Learning, 4857–4866, 2018.

Rahaman, N.; Baratin, A.; Arpit, D.; Draxler, F.; Lin, M.; Hamprecht, F.; Bengio, Y.; Courville, A. On the spectral bias of neural networks. In: Proceedings of the International Conference on Machine Learning, 5301–5310, 2019.

Sun, K.; Xiao, B.; Liu, D.; Wang, J. D. Deep high-resolution representation learning for human pose estimation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 5686–5696, 2019.

Yang, B.; Bender, G.; Le, Q. V.; Ngiam, J. Condconv: Conditionally parameterized convolutions for efficient inference. In: Proceedings of the Advances in Neural Information Processing Systems, 1307–1318, 2019.

Shi, B.; Mo, Z.; Wu, Z.; Duan, D.; Yeung, S.; Tan, P. A benchmark dataset and evaluation for non-Lambertian and uncalibrated photometric stereo. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 41, No. 2, 271–284, 2019.

Herbort, S.; Wöhler, C. An introduction to image-based 3D surface reconstruction and a survey of photometric stereo methods. 3D Research Vol. 2, No. 3, 4, 2011.

Alldrin, N.; Zickler, T.; Kriegman, D. Photometric stereo with non-parametric and spatially-varying reflectance. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1–8, 2008.

Shi, B. X.; Tan, P.; Matsushita, Y.; Ikeuchi, K. Bi-polynomial modeling of low-frequency reflectances. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 36, No. 6, 1078–1091, 2014.

Goldman, D. B.; Curless, B.; Hertzmann, A.; Seitz, S. M. Shape and spatially-varying BRDFs from photometric stereo. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 32, No. 6, 1060–1071, 2010.

Chung, H.-S.; Jia, J. Y. Efficient photometric stereo on glossy surfaces with wide specular lobes. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1–8, 2008.

Yeung, S. K.; Wu, T. P.; Tang, C. K.; Chan, T. F.; Osher, S. J. Normal estimation of a transparent object using a video. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 37, No. 4, 890–897, 2015.

Chen, T. B.; Goesele, M.; Seidel, H. P. Mesostructure from specularity. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 1825–1832, 2006.

Tozza, S.; Mecca, R.; Duocastella, M.; Del Bue, A. Direct differential photometric stereo shape recovery of diffuse and specular surfaces. Journal of Mathematical Imaging and Vision Vol. 56, No. 1, 57–76, 2016.

Georghiades, A. S. Incorporating the Torrance and sparrow model of reflectance in uncalibrated photometric stereo. In: Proceedings of the 9th IEEE International Conference on Computer Vision, 816–823, 2003.

Verbiest, F.; van Gool, L. Photometric stereo with coherent outlier handling and confidence estimation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1–8, 2008.

Sunkavalli, K.; Zickler, T.; Pfister, H. Visibility subspaces: Uncalibrated photometric stereo with shadows. In: Computer Vision — ECCV 2010. Lecture Notes in Computer Science, Vol. 6312. Daniilidis, K.; Maragos, P.; Paragios, N. Eds. Springer Berlin Heidelberg, 251–264, 2010.

Yu, C.; Seo, Y.; Lee, S. W. Photometric stereo from maximum feasible Lambertian reflections. In: Computer Vision — ECCV 2010. Lecture Notes in Computer Science, Vol. 6314. Daniilidis, K.; Maragos, P.; Paragios, N. Eds. Springer Berlin Heidelberg, 115–126, 20

Ju, Y. K.; Dong, X. H.; Wang, Y. Y.; Qi, L.; Dong, J. Y. A dual-cue network for multispectral photometric stereo. Pattern Recognition Vol. 100, 107162, 2020.

Wang, C.; Wu, Y. T.; Su, Z. X.; Chen, J. Y. Joint self-attention and scale-aggregation for self-calibrated deraining network. In: Proceedings of the 28th ACM International Conference on Multimedia, 2517–2525, 2020.

Ikehata, S. CNN-PS: CNN-based photometric stereo for general non-convex surfaces. In: Computer Vision — ECCV 2018. Lecture Notes in Computer Science, Vol. 11219. Ferrari, V.; Hebert, M.; Sminchisescu, C.; Weiss, Y. Eds. Springer Cham, 3–19, 2018.

Li, J. X.; Robles-Kelly, A.; You, S. D.; Matsushita, Y. Learning to minify photometric stereo. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 7560–7568, 2019.

Zheng, Q.; Jia, Y.; Shi, B.; Jiang, X.; Duan, L.-Y.; Kot, A. C. SPLINE-Net: Sparse photometric stereo through lighting interpolation and normal estimation networks. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, 8549–8558, 2019.

Ju, Y. K.; Lam, K. M.; Chen, Y.; Qi, L.; Dong, J. Y. Pay attention to Devils: A photometric stereo network for better details. In: Proceedings of the 29th International Joint Conference on Artificial Intelligence, 694–700, 2020.

Mildenhall, B.; Srinivasan, P. P.; Tancik, M.; Barron, J. T.; Ramamoorthi, R.; Ng, R. NeRF: Representing scenes as neural radiance fields for view synthesis. In: Computer Vision — ECCV 2020. Lecture Notes in Computer Science, Vol. 12346. Vedaldi, A.; Bischof, H.; Brox, T.; Frahm, J. M. Eds. Springer Cham, 405–421, 2020.

Liu, L.; Liu, J. Z.; Yuan, S. X.; Slabaugh, G., Leonardis, A., Zhou, W. G.; Tian, Q. Wavelet-based dual-branch network for image demoiréing. In: Computer Vision — ECCV 2020. Lecture Notes in Computer Science, Vol. 12358. Vedaldi, A.; Bischof, H.; Brox, T.; Frahm, J. M. Eds. Springer Cham, 86–102, 2020.

Paszke, A.; Gross, S.; Massa, F.; Lerer, A.; Bradbury, J.; Chanan, G.; Killeen, T.; Lin, Z.; Gimelshein, N.; Antiga, L. et al. PyTorch: An imperative style, high-performance deep learning library. In: Proceedings of the Advances in Neural Information Processing Systems, 8026–8037, 2019.

Kingma, D. P.; Ba, J. Adam: A method for stochastic optimization. In: Proceedings of the 3rd International Conference on Learning Representations, 2015.

Johnson, M. K.; Adelson, E. H. Shape estimation in natural illumination. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2553–2560, 2011.

Wiles, O.; Zisserman, A. SilNet: Single- and multi-view reconstruction by learning from silhouettes. In: Proceedings of the British Machine Vision Conference, 2017.

Matusik, W.; Pfister, H.; Brand, M.; McMillan, L. A data-driven reflectance model. ACM Transactions on Graphics Vol. 22, No. 3, 759–769, 2003.

Jakob, W. Mitsuba renderer. 2010. Available at https://www.mitsuba-renderer.org/.

Einarsson, P.; Chabert, C.-F.; Jones, A.; Ma, W.-C.; Lamond, B.; Hawkins, T.; Bolas, M.; Sylwan, S.; Debevec, P. Relighting human locomotion with owed reflectance fields. In: Proceedings of the 17th Eurographics Conference on Rendering Techniques, 183–194, 2006.

Shi, B. X.; Tan, P.; Matsushita, Y.; Ikeuchi, K. Bi-polynomial modeling of low-frequency reflectances. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 36, No. 6, 1078–1091, 2014.

Alldrin, N. G.; Mallick, S. P.; Kriegman, D. J. Resolving the generalized bas-relief ambiguity by entropy minimization. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1–7, 2007.

Shi, B. X.; Matsushita, Y.; Wei, Y. C.; Xu, C.; Tan, P. Self-calibrating photometric stereo. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 1118–1125, 2010.

Wu, Z.; Tan, P. Calibrating photometric stereo by holistic reflectance symmetry analysis. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1498–1505, 2013.

Papadhimitri, T.; Favaro, P. A closed-form, consistent and robust solution to uncalibrated photometric stereo via local diffuse reflectance maxima. International Journal of Computer Vision Vol. 107, No. 2, 139–154, 2014.

Simchony, T.; Chellappa, R.; Shao, M. Direct analytical methods for solving Poisson equations in computer vision problems. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 12, No. 5, 435–446, 1990.

Thông tin
Thông tin xuất bản

Springer Science and Business Media LLC

Tập 8 Số 1

105-118

Thông tin tác giả