Super-resolution of three-dimensional temperature and velocity for building-resolving urban micrometeorology using physics-guided convolutional neural networks with image inpainting techniques

U.N.H.S. Programme, World Cities Report 2020, United Nations, 2020, http://dx.doi.org/10.18356/27bc31a5-en, URL. Mauree, 2019, A review of assessment methods for the urban environment and its energy sustainability to guarantee climate adaptation of future cities, Renew. Sustain. Energy Rev., 112, 733, 10.1016/j.rser.2019.06.005 Lai, 2019, A review of mitigating strategies to improve the thermal environment and thermal comfort in urban outdoor spaces, Sci. Total Environ., 661, 337, 10.1016/j.scitotenv.2019.01.062 Naboni, 2013, Environmental simulation tools in architectural practice: the impact on processes, methods and design Toparlar, 2017, A review on the CFD analysis of urban microclimate, Renew. Sustain. Energy Rev., 80, 1613, 10.1016/j.rser.2017.05.248 Lam, 2021, A review on the significance and perspective of the numerical simulations of outdoor thermal environment, Sustainable Cities Soc., 71, 10.1016/j.scs.2021.102971 Onishi, 2019, Super-resolution simulation for real-time prediction of urban micrometeorology, SOLA, 15, 178, 10.2151/sola.2019-032 Kodi Ramanah, 2020, Super-resolution emulator of cosmological simulations using deep physical models, Mon. Not. R. Astron. Soc., 495, 4227, 10.1093/mnras/staa1428 Shiina, 2021, Inverse renormalization group based on image super-resolution using deep convolutional networks, Sci. Rep., 11, 10.1038/s41598-021-88605-w Dong, 2014, Learning a deep convolutional network for image super-resolution, 184 Ha, 2019, Deep learning based single image super-resolution: A survey, Int. J. Autom. Comput., 10.1007/s11633-019-1183-x Anwar, 2020, A deep journey into super-resolution: A survey, ACM Comput. Surv., 53, 10.1145/3390462 Ducournau, 2016, 1 Xie, 2018, TempoGAN: A temporally coherent, volumetric GAN for super-resolution fluid flow, ACM Trans. Graph., 37, 10.1145/3197517.3201304 Deng, 2019, Super-resolution reconstruction of turbulent velocity fields using a generative adversarial network-based artificial intelligence framework, Phys. Fluids, 31 Fukami, 2019, Super-resolution reconstruction of turbulent flows with machine learning, J. Fluid Mech., 870, 106, 10.1017/jfm.2019.238 Fukami, 2021, Machine-learning-based spatio-temporal super resolution reconstruction of turbulent flows, J. Fluid Mech., 909, A9, 10.1017/jfm.2020.948 Wang, 2021, A novel framework for cost-effectively reconstructing the global flow field by super-resolution, Phys. Fluids, 33, 10.1063/5.0062775 Obiols-Sales, 2021, SURFNet: Super-resolution of turbulent flows with transfer learning using small datasets, 331 Obiols-Sales, 2022 Jiang, 2020, MESHFREEFLOWNET: A physics-constrained deep continuous space-time super-resolution framework, 1 C. Wang, E. Bentivegna, W. Zhou, L. Klein, B. Elmegreen, Physics-Informed Neural Network Super Resolution for Advection-Diffusion Models, in: Third Workshop on Machine Learning and the Physical Sciences (NeurIPS 2020), 2020, URL. Bode, 2021, Using physics-informed enhanced super-resolution generative adversarial networks for subfilter modeling in turbulent reactive flows, Proc. Combust. Inst., 38, 2617, 10.1016/j.proci.2020.06.022 T. Bao, S. Chen, T.T. Johnson, P. Givi, S. Sammak, X. Jia, Physics Guided Neural Networks for Spatio-temporal Super-resolution of Turbulent Flows, in: The 38th Conference on Uncertainty in Artificial Intelligence, 2022, URL. Wu, 2021, Deep learning-based super-resolution climate simulator-emulator framework for urban heat studies, Geophys. Res. Lett., 48, 10.1029/2021GL094737 Wang, 2021, Fast and accurate learned multiresolution dynamical downscaling for precipitation, Geosci. Model Dev., 14, 6355, 10.5194/gmd-14-6355-2021 Vandal, 2017, 1663 Rodrigues, 2018, 415 Stengel, 2020, Adversarial super-resolution of climatological wind and solar data, Proc. Natl. Acad. Sci., 117, 16805, 10.1073/pnas.1918964117 Yasuda, 2022, Super-resolution of near-surface temperature utilizing physical quantities for real-time prediction of urban micrometeorology, Build. Environ., 209, 10.1016/j.buildenv.2021.108597 Fukami, 2023 Elharrouss, 2020, Image inpainting: A review, Neural Process. Lett., 51, 2007, 10.1007/s11063-019-10163-0 Qin, 2021, Image inpainting based on deep learning: A review, Displays, 69, 10.1016/j.displa.2021.102028 Li, 2019, Thin cloud removal with residual symmetrical concatenation network, ISPRS J. Photogramm. Remote Sens., 153, 137, 10.1016/j.isprsjprs.2019.05.003 Kadow, 2020, Artificial intelligence reconstructs missing climate information, Nat. Geosci., 13, 408, 10.1038/s41561-020-0582-5 Meraner, 2020, Cloud removal in sentinel-2 imagery using a deep residual neural network and SAR-optical data fusion, ISPRS J. Photogramm. Remote Sens., 166, 333, 10.1016/j.isprsjprs.2020.05.013 Geiss, 2021, Inpainting radar missing data regions with deep learning, Atmos. Meas. Tech., 14, 7729, 10.5194/amt-14-7729-2021 Maduskar, 2021, Navier–Stokes-based image inpainting for restoration of missing data due to clouds, 497 Schweri, 2021, A physics-aware neural network approach for flow data reconstruction from satellite observations, Front. Clim., 3, 10.3389/fclim.2021.656505 Czerkawski, 2022, Deep internal learning for inpainting of cloud-affected regions in satellite imagery, Remote Sens., 14, 10.3390/rs14061342 Tung, 2017, Adversarial inverse graphics networks: Learning 2D-to-3D lifting and image-to-image translation from unpaired supervision, 4364 R. Rombach, A. Blattmann, D. Lorenz, P. Esser, B. Ommer, High-Resolution Image Synthesis With Latent Diffusion Models, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR, 2022, pp. 10684–10695. G. Liu, F.A. Reda, K.J. Shih, T.-C. Wang, A. Tao, B. Catanzaro, Image Inpainting for Irregular Holes Using Partial Convolutions, in: Proceedings of the European Conference on Computer Vision, ECCV, 2018. Le Meur, 2012, Super-resolution-based inpainting, 554 Sharma, 2018, An end-to-end deep learning framework for super-resolution based inpainting, 198 Li, 2022, Srinpaintor: When super-resolution meets transformer for image inpainting, IEEE Trans. Comput. Imaging, 8, 743, 10.1109/TCI.2022.3190142 Y. Huang, F. Zheng, D. Wang, J. Jiang, X. Wang, L. Shao, Super-Resolution and Inpainting with Degraded and Upgraded Generative Adversarial Networks, in: Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI ’20, ISBN: 9780999241165, 2021. Lluís, 2020, Sound field reconstruction in rooms: Inpainting meets super-resolution, J. Acoust. Soc. Am., 148, 649, 10.1121/10.0001687 Ronneberger, 2015, U-net: Convolutional networks for biomedical image segmentation, 234 J. Yu, Z. Lin, J. Yang, X. Shen, X. Lu, T.S. Huang, Free-Form Image Inpainting With Gated Convolution, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, ICCV, 2019. A.L. Maas, A.Y. Hannun, A.Y. Ng, et al., Rectifier nonlinearities improve neural network acoustic models, in: Proc. ICML, Vol. 30, Atlanta, Georgia, USA, 2013, p. 3. K. He, X. Zhang, S. Ren, J. Sun, Deep Residual Learning for Image Recognition, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR, 2016. Shi, 2016, Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network, 1874 Takahashi, 2013, Challenge toward the prediction of typhoon behaviour and down pour, J. Phys. Conf. Ser., 454, 10.1088/1742-6596/454/1/012072 Onishi, 2012, A warm-bin–cold-bulk hybrid cloud microphysical model*, J. Atmos. Sci., 69, 1474, 10.1175/JAS-D-11-0166.1 Sasaki, 2016, MJO simulation in a cloud-system-resolving global ocean-atmosphere coupled model, Geophys. Res. Lett., 43, 9352, 10.1002/2016GL070550 Matsuda, 2018, Tree-crown-resolving large-eddy simulation coupled with three-dimensional radiative transfer model, J. Wind Eng. Ind. Aerodyn., 173, 53, 10.1016/j.jweia.2017.11.015 Giorgi, 2015, Regional dynamical downscaling and the CORDEX initiative, Annu. Rev. Environ. Resour., 40, 467, 10.1146/annurev-environ-102014-021217 Japan meteorological business support center. URL http://www.jmbsc.or.jp/en/index-e.html. MapBox. URL https://www.mapbox.com/about/maps/. OpenStreetMap. URL https://www.openstreetmap.org/about/. Kamiya, 2019, Estimation of time-course core temperature and water loss in realistic adult and child models with urban micrometeorology prediction, Int. J. Environ. Res. Public Health, 16, 10.3390/ijerph16245097 Kawahara, 2015, Realistic representation of clouds in google earth Lu, 2022, Single image super-resolution based on a modified U-net with mixed gradient loss, Signal Image Video Process., 16, 1143, 10.1007/s11760-021-02063-5 Kingma, 2015 Paszke, 2019, Pytorch: An imperative style, high-performance deep learning library, 8024 Wang, 2004, Image quality assessment: from error visibility to structural similarity, IEEE Trans. Image Process., 13, 600, 10.1109/TIP.2003.819861 Brunton, 2020, Machine learning for fluid mechanics, Annu. Rev. Fluid Mech., 52, 477, 10.1146/annurev-fluid-010719-060214 Vinuesa, 2022, Enhancing computational fluid dynamics with machine learning, Nature Comput. Sci., 2, 358, 10.1038/s43588-022-00264-7 Kim, 2019, Deep fluids: A generative network for parameterized fluid simulations, Comput. Graph. Forum, 38, 59, 10.1111/cgf.13619 Nakamura, 2021, Convolutional neural network and long short-term memory based reduced order surrogate for minimal turbulent channel flow, Phys. Fluids, 33, 10.1063/5.0039845 Wu, 2021, Reduced order model using convolutional auto-encoder with self-attention, Phys. Fluids, 33, 10.1063/5.0051155 Song, 2020