A spatiotemporal fusion method based on interpretable deep networks
Tóm tắt
Remote sensing spatiotemporal fusion is currently a popular research field in remote sensing that can cost-effectively generate remote sensing images with high spatiotemporal resolution. The deep neural network-based approach has strong feature extraction capability and has achieved great success in the fields of signal and image processing, but the deep network lacks interpretability. In this paper, a new interpretable deep network-based spatiotemporal fusion (UNSTF) method is proposed. The method proposes a new network model by first establishing a concise a priori formulation using sparse representation, constructing the proposed network by unfolding the proximal gradient algorithm for solving the model, and carefully designing each basic network module in the model to have a reasonable physical meaning, making the whole network interpretable. In addition, the UNSTF network method proposes a new network iteration loss function, where the predicted images of each iteration stage of the network are constrained by the real images, and the final stage and the intermediate network stages conform well to their inherent prior structures, effectively improving the accuracy and reliability of model prediction. Through extensive experiments, it is shown that the proposed method outperforms existing fusion methods in terms of both subjective and objective metrics.
Tài liệu tham khảo
Chen Y, Shi K, Ge Y, Zhou Y (2021) Spatiotemporal remote sensing image fusion using multiscale two-stream convolutional neural networks. IEEE Transactions on Geoscience and Remote Sensing 60:1–12. https://doi.org/10.1109/TGRS.2021.3069116
Zhukov B, Oertel D, Lanzl F, Reinhackel G (1999) Unmixing-based multisensor multiresolution image fusion. IEEE Transactions on Geoscience and Remote Sensing 37(3):1212–1226. https://doi.org/10.1109/36.763276
Shen H, Meng X, Zhang L (2016) An integrated framework for the spatio-temporal-spectral fusion of remote sensing images. IEEE Transactions on Geoscience and Remote Sensing 54(12):7135–7148. https://doi.org/10.1109/TGRS.2016.2596290
Xue J, Leung Y, Fung T (2017) A bayesian data fusion approach to spatio-temporal fusion of remotely sensed images. Remote Sensing 9(12):1310. https://doi.org/10.3390/rs9121310
Li A, Bo Y, Zhu Y, Guo P, Bi J, He Y (2013) Blending multi-resolution satellite sea surface temperature (sst) products using bayesian maximum entropy method. Remote sensing of environment 135:52–63. https://doi.org/10.1016/j.rse.2013.03.021
Gao F, Masek J, Schwaller M, Hall F (2006) On the blending of the landsat and modis surface reflectance: Predicting daily landsat surface reflectance. IEEE Transactions on Geoscience and Remote sensing 44(8):2207–2218. https://doi.org/10.1109/TGRS.2006.872081
Zhu X, Helmer EH, Gao F, Liu D, Chen J, Lefsky MA (2016) A flexible spatiotemporal method for fusing satellite images with different resolutions. Remote Sensing of Environment 172:165–177. https://doi.org/10.1016/j.rse.2015.11.016
Li X, Foody GM, Boyd DS, Ge Y, Zhang Y, Du Y, Ling F (2020) Sfsdaf: An enhanced fsdaf that incorporates sub-pixel class fraction change information for spatio-temporal image fusion. Remote Sensing of Environment 237:111537. https://doi.org/10.1016/j.rse.2019.111537
Gevaert CM, García-Haro FJ (2015) A comparison of starfm and an unmixing-based algorithm for landsat and modis data fusion. Remote sensing of Environment 156:34–44. https://doi.org/10.1016/j.rse.2014.09.012
Quan J, Zhan W, Ma T, Du Y, Guo Z, Qin B (2018) An integrated model for generating hourly landsat-like land surface temperatures over heterogeneous landscapes. Remote Sensing of Environment 206:403–423. https://doi.org/10.1016/j.rse.2017.12.003
Li X, Ling F, Foody GM, Ge Y, Zhang Y, Du Y (2017) Generating a series of fine spatial and temporal resolution land cover maps by fusing coarse spatial resolution remotely sensed images and fine spatial resolution land cover maps. Remote Sensing of Environment 196:293–311. https://doi.org/10.1016/j.rse.2017.05.011
Huang B, Song H (2012) Spatiotemporal reflectance fusion via sparse representation. IEEE Transactions on Geoscience and Remote Sensing 50(10):3707–3716. https://doi.org/10.1109/TGRS.2012.2186638
Wei J, Wang L, Liu P, Chen X, Li W, Zomaya AY (2017) Spatiotemporal fusion of modis and landsat-7 reflectance images via compressed sensing. IEEE Transactions on Geoscience and Remote Sensing 55(12):7126–7139. https://doi.org/10.1109/TGRS.2017.2742529
Peng Y, Li W, Luo X, Du J, Zhang X, Gan Y, Gao X (2021) Spatiotemporal reflectance fusion via tensor sparse representation. IEEE Transactions on Geoscience and Remote Sensing 60:1–18. https://doi.org/10.1109/TGRS.2021.3091157
Tan Z, Di L, Zhang M, Guo L, Gao M (2019) An enhanced deep convolutional model for spatiotemporal image fusion. Remote Sensing 11(24):2898. https://doi.org/10.3390/rs11242898
Liu Q, Meng X, Shao F, Li S (2022) Pstaf-gan: Progressive spatio-temporal attention fusion method based on generative adversarial network. IEEE Transactions on Geoscience and Remote Sensing 60:1–13. https://doi.org/10.1109/TGRS.2022.3161563
Yang G, Qian Y, Liu H, Tang B, Qi R, Lu Y, Geng J (2022) Msfusion: Multistage for remote sensing image spatiotemporal fusion based on texture transformer and convolutional neural network. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing 15:4653–4666. https://doi.org/10.1109/JSTARS.2022.3179415
Meng X, Liu Q, Shao F, Li S (2022) Spatio-temporal-spectral collaborative learning for spatio-temporal fusion with land cover changes. IEEE Transactions on Geoscience and Remote Sensing 60:1–16. https://doi.org/10.1109/TGRS.2022.3185459
Xie Q, Zhou M, Zhao Q, Xu Z, Meng D (2020) Mhf-net: An interpretable deep network for multispectral and hyperspectral image fusion. IEEE Transactions on Pattern Analysis and Machine Intelligence. https://doi.org/10.1109/TPAMI.2020.3015691
Rubinstein R, Bruckstein AM, Elad M (2010) Dictionaries for sparse representation modeling. Proceedings of the IEEE 98(6):1045–1057. https://doi.org/10.1109/JPROC.2010.2040551
Monga V, Li Y, Eldar YC (2021) Algorithm unrolling: Interpretable, efficient deep learning for signal and image processing. IEEE Signal Processing Magazine 38(2):18–44. https://doi.org/10.1109/MSP.2020.3016905
Ma, J., Liu, X.-Y., Shou, Z., Yuan, X.: Deep tensor admm-net for snapshot compressive imaging. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10223–10232 (2019)
Meinhardt, T., Moller, M., Hazirbas, C., Cremers, D.: Learning proximal operators: Using denoising networks for regularizing inverse imaging problems. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1781–1790 (2017)
Vu, H., Cheung, G., Eldar, Y.C.: Unrolling of deep graph total variation for image denoising. In: ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 2050–2054 (2021). https://doi.org/10.1109/ICASSP39728.2021.9414453. IEEE
Yang, D., Sun, J.: Proximal dehaze-net: A prior learning-based deep network for single image dehazing. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 702–717 (2018)
Xie, Q., Zhou, M., Zhao, Q., Meng, D., Zuo, W., Xu, Z.: Multispectral and hyperspectral image fusion by ms/hs fusion net. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1585–1594 (2019)
Zhang, K., Gool, L.V., Timofte, R.: Deep unfolding network for image super-resolution. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3217–3226 (2020)
Chen, S., Eldar, Y.C.: Time-varying graph signal inpainting via unrolling networks. In: ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 8092–8097 (2021). https://doi.org/10.1109/ICASSP39728.2021.9413406. IEEE
Shao Z, Wang L, Wang Z, Deng J (2019) Remote sensing image super-resolution using sparse representation and coupled sparse autoencoder. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing 12(8):2663–2674. https://doi.org/10.1109/JSTARS.2019.2925456
Khader A, Yang J, Xiao L (2022) Nmf-dunet: Nonnegative matrix factorization inspired deep unrolling networks for hyperspectral and multispectral image fusion. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing 15:5704–5720. https://doi.org/10.1109/JSTARS.2022.3189551
Molina R, Vega M, Mateos J, Katsaggelos AK (2008) Variational posterior distribution approximation in bayesian super resolution reconstruction of multispectral images. Applied and Computational Harmonic Analysis 24(2):251–267. https://doi.org/10.1016/j.acha.2007.03.006
Wang, X., Zhu, Q., Qi, N.: Ista-net model-driven deep unfolding network for hyperspectral and multispectral image fusion. In: 2022 IEEE 10th Joint International Information Technology and Artificial Intelligence Conference (ITAIC), vol. 10, pp. 1208–1212 (2022). https://doi.org/10.1109/ITAIC54216.2022.9836742. IEEE
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)
Liu X, Deng C, Chanussot J, Hong D, Zhao B (2019) Stfnet: A two-stream convolutional neural network for spatiotemporal image fusion. IEEE Transactions on Geoscience and Remote Sensing 57(9):6552–6564. https://doi.org/10.1109/TGRS.2019.2907310
Lei D, Ran G, Zhang L, Li W (2022) A spatiotemporal fusion method based on multiscale feature extraction and spatial channel attention mechanism. Remote Sensing 14(3):461. https://doi.org/10.3390/rs14030461