Image Captioning using Hybrid LSTM-RNN with Deep Features
Tóm tắt
Automated image captioning is the process of creating textual, human-like subtitles or explanations for photos based on their content. Throughout the image captioning problem, getting good results and consistency on par with humans always has been difficult for machines. The captioning framework used in this research is an efficient hybrid deep learning framework. Deep feature extraction and automatic image captioning are the two main parts of the proposed method. The inceptionv3 model first extracts the deep characteristics from the gathered photos. The automatic Image captioning step is then modeled using a hybrid classifier, which blends LSTM and RNN from two deep learning models. The captured deep features obtained during the feature extraction stage are used to train these two deep learning models. Additionally, we will fine-tune the weight of RNN using a novel self-improved Rock Hyraxes Swarm Optimization (SI-RHSO), which is a conceptual enhancement of conventional RHSO, to increase the Auto Image Captioning precision. As a result, the final result was obtained with a precise Image caption. A comparison analysis has been conducted to confirm the effectiveness of the suggested work.
Tài liệu tham khảo
Shen, X., Liu, B., Zhou, Y., Zhao, J., & Liu, M. (2020). Remote sensing image captioning via Variational Autoencoder and Reinforcement Learning. Knowledge-Based Systems, 203, 105920.
Zeng, X., Wen, L., Liu, B., & Qi, X. (2019). Deep learning for ultrasound image caption generation based on object detection. Neurocomputing, 392, 132–141.
Zhang, J., Li, K., & Wang, Z. (2021). Parallel-fusion LSTM with synchronous semantic and visual information for image captioning. Journal of Visual Communication and Image Representation, 75(2021), 103044.
Yan, S., Xie, Y., Wu, F., Smith, J. S., Lu, W., & Zhang, B. (2019). Image captioning via hierarchical attention mechanism and policy gradient optimization. Signal Processing, 167(2020), 107329.
Wang, S., Lan, L., Zhang, X., & Luo, Z. (2020). GateCap Gated spatial and semantic attention model for image captioning. Multimedia Tools and Applications, 79(17), 11531–11549.
Chen, C., Mu, S., Xiao, W., Ye, Z., Wu, L., & Ju, Q. (2019). Improving image captioning with conditional generative adversarial nets. In The 33rd AAAI conference on artificial intelligence (AAAI-19).
Zhao, W., Wu, X., & Luo, J. (2021). Cross-domain image captioning via cross-modal retrieval and model adaptation. IEEE Transactions on Image Processing, 30, 1180–1192. https://doi.org/10.1109/TIP.2020.3042086
Yang, M., Zhao, W., Xu, W., Feng, Y., Zhao, Z., Chen, X., & Lei, K. (2019). Multitask learning for cross-domain image captioning. IEEE Transactions on Multimedia, 21(4), 1047–1061. https://doi.org/10.1109/TMM.2018.2869276
Hoxha, G., Melgani, F., & Demir, B. (2020). Toward remote sensing image retrieval under a deep image captioning perspective. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 13, 4462–4475. https://doi.org/10.1109/JSTARS.2020.3013818
Yadav, A. K. (2021). Image captioning using R-CNN & LSTM deep learning model. Image, 5, 8.
Liu, S., Bai, L., Hu, Y., & Wang, H. (2018). Image captioning based on deep neural networks. In MATEC web of conferences (Vol. 232). EDP Sciences.
Huang, W., Wang, Q., & Li, X. (2021). Denoising-based multiscale feature fusion for remote sensing image captioning. IEEE Geoscience and Remote Sensing Letters, 18(3), 436–440. https://doi.org/10.1109/LGRS.2020.2980933
Yu, J., Li, J., Yu, Z., & Huang, Q. (2020). Multimodal transformer with multi-view visual representation for image captioning. IEEE Transactions on Circuits and Systems for Video Technology, 30(12), 4467–4480. https://doi.org/10.1109/TCSVT.2019.2947482
Amirian, S., Rasheed, K., Taha, T. R., & Arabnia, H. R. (2020). Automatic image and video caption generation with deep learning a concise review and algorithmic overlap. IEEE Access, 8, 218386–218400. https://doi.org/10.1109/ACCESS.2020.3042484
Yu, N., Hu, X., Song, B., Yang, J., & Zhang, J. (2019). Topic-oriented image captioning based on order-embedding. IEEE Transactions on Image Processing, 28(6), 2743–2754. https://doi.org/10.1109/TIP.2018.2889922
Wang, B., Zheng, X., Qu, B., & Lu, X. (2020). Retrieval topic recurrent memory network for remote sensing image captioning. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 13, 256–270. https://doi.org/10.1109/JSTARS.2019.2959208
Zhang, M., Yang, Y., Zhang, H., Ji, Y., Shen, H. T., & Chua, T. (2019). More is better precise and detailed image captioning using online positive recall and missing concepts mining. IEEE Transactions on Image Processing, 28(1), 32–44. https://doi.org/10.1109/TIP.2018.2855415
Zhang, Z., Zhang, W., Diao, W., Yan, M., Gao, X., & Sun, X. (2019). VAA visual aligning attention model for remote sensing image captioning. IEEE Access, 7, 137355–137364. https://doi.org/10.1109/ACCESS.2019.2942154
Zhou, L., Zhang, Y., Jiang, Y.-G., Zhang, T., & Fan, W. (2020). Re-caption saliency-enhanced image captioning through two-phase learning. IEEE Transactions on Image Processing, 29, 694–709. https://doi.org/10.1109/TIP.2019.2928144
Rampal, H., & Mohanty, A. (2020). Efficient CNN-LSTM based image captioning using neural network compression. arXiv preprint arXiv:2012.09708.
Wu, L., Xu, M., Wang, J., & Perry, S. (2020). Recall what you see continually using GridLSTM in image captioning. IEEE Transactions on Multimedia, 22(3), 808–818. https://doi.org/10.1109/TMM.2019.2931815
Li, X., Xu, C., Wang, X., Lan, W., Jia, Z., Yang, G., & Xu, J. (2019). COCO-CN for cross-lingual image tagging, captioning, and retrieval. IEEE Transactions on Multimedia, 21(9), 2347–2360. https://doi.org/10.1109/TMM.2019.2896494
Yang, Z., & Liu, Q. (2020). ATT-BM-SOM a framework of effectively choosing image information and optimizing syntax for image captioning. IEEE Access, 8, 50565–50573. https://doi.org/10.1109/ACCESS.2020.2980578
Cheng, L., Wei, W., Mao, X., Liu, Y., & Miao, C. (2020). Stack-VS stacked visual-semantic attention for image caption generation. IEEE Access, 8, 154953–154965. https://doi.org/10.1109/ACCESS.2020.3018752
Lu, X., Wang, B., Zheng, X., & Li, X. (2018). Exploring models and data for remote sensing image caption generation. IEEE Transactions on Geoscience and Remote Sensing, 56(4), 2183–2195. https://doi.org/10.1109/TGRS.2017.2776321
Wang, B., Wang, C., Zhang, Q., Su, Y., Wang, Y., & Xu, Y. (2020). Cross-lingual image caption generation based on visual attention model. IEEE Access, 8, 104543–104554. https://doi.org/10.1109/ACCESS.2020.2999568
Mohana, S., Sahaaya, S. A., & Mary, A. (2016). A comparitive framework for feature selction in privacy preserving data mining techniques using pso and k-anonumization. Iioab Journal, 7(9), 804–811.
Yang, M., Liu, J., Shen, Y., Zhao, Z., Chen, X., Wu, Q., & Li, C. (2020). An ensemble of generation- and retrieval-based image captioning with dual generator generative adversarial network. IEEE Transactions on Image Processing, 29, 9627–9640. https://doi.org/10.1109/TIP.2020.3028651
Huang, Y., Chen, J., Ouyang, W., Wan, W., & Xue, Y. (2020). Image captioning with end-to-end attribute detection and subsequent attributes prediction. IEEE Transactions on Image Processing, 29, 4013–4026. https://doi.org/10.1109/TIP.2020.2969330
Yuan, Z., Li, X., & Wang, Q. (2020). Exploring multi-level attention and semantic relationship for remote sensing image captioning. IEEE Access, 8, 2608–2620. https://doi.org/10.1109/ACCESS.2019.2962195
Tan, J. H., Chan, C. S., & Chuah, J. H. (2019). COMIC toward a compact image captioning model with attention. IEEE Transactions on Multimedia, 21(10), 2686–2696. https://doi.org/10.1109/TMM.2019.2904878
Xian, Y., & Tian, Y. (2019). Self-guiding multimodal LSTM—when we do not have a perfect training dataset for image captioning. IEEE Transactions on Image Processing, 28(11), 5241–5252. https://doi.org/10.1109/TIP.2019.2917229
Fu, K., Li, J., Jin, J., & Zhang, C. (2018). Image-text surgery efficient concept learning in image captioning by generating pseudopairs. IEEE Transactions on Neural Networks and Learning Systems, 29(12), 5910–5921. https://doi.org/10.1109/TNNLS.2018.2813306
Jing, Y., Zhiwei, X., & Guanglai, G. (2020). Context-driven image caption with global semantic relations of the named entities. IEEE Access, 8, 143584–143594. https://doi.org/10.1109/ACCESS.2020.3013321
Lu, X., Wang, B., & Zheng, X. (2020). Sound active attention framework for remote sensing image captioning. IEEE Transactions on Geoscience and Remote Sensing, 58(3), 1985–2000. https://doi.org/10.1109/TGRS.2019.2951636
Kapse, S. S. S., Daigavane, D. M. B., & Daigavane, D. P. M. (2019). Localization and sizing of UPFC: a hybrid optimization algorithm for solving reactive power dispatch. Journal of Computational Mechanics, Power System and Control, 2(1), 19–30.
Quazi, M. H., & Kahalekar, S. G. (2019). Artifacts removal using dragonfly Levenberg Marquardt-based learning algorithm from electroencephalogram signal. Multimedia Research, 2(2), 1–9.
Vinolin, V. (2019). Breast cancer detection by optimal classification using GWO algorithm. Multimedia Research, 2(2), 10–18.
Jadhav, A. N., & Gomathi, N. (2019). DIGWO: hybridization of dragonfly algorithm with improved grey wolf optimization algorithm for data clustering. Multimedia Research, 2(3), 1–11.
Gangappa, M., Mai, C., & Sammulal, P. (2019). enhanced crow search optimization algorithm and hybrid NN-CNN classifiers for classification of land cover images. Multimedia Research, 2(3), 12–22.
Nipanikar, S. I., & Hima Deepthi, V. (2019). Enhanced whale optimization algorithm and wavelet transform for image steganography. Multimedia Research, 2(3), 23–32.
Kulkarni, Y. R., & Murugan, T. S. (2019). Hybrid weed-particle swarm optimization algorithm and C-mixture for data publishing. Multimedia Research, 2(3), 33–42.
Chithra, R. S., & Jagatheeswari, P. (2019). Enhanced WOA and modular neural network for severity analysis of tuberculosis. Multimedia Research, 2(3), 43–55.
Santosh, K. B., & P, Venkata Ramanaiah K.,. (2019). An efficient hybrid optimization algorithm for image compression. Multimedia Research, 2(4), 1–11.
Yang, Q., Ni, Z., & Ren, P. (2022). Meta captioning: a meta learning based remote sensing image captioning framework. ISPRS Journal of Photogrammetry and Remote Sensing, 186, 190–200.
Bagherzadeh, S. Z., & Toosizadeh, S. (2022). Eye tracking algorithm based on multi model Kalman filter. HighTech and Innovation Journal, 3(1), 15–27.
Cham, D. D., Son, N. T., Minh, N. Q., Thanh, N. T., & Dung, T. T. (2020). An analysis of shoreline changes using combined multitemporal remote sensing and digital evaluation model. Civil Engineering Journal, 6(1), 1–10.
Su, Y., Li, Y., Xu, N., & Liu, A. A. (2020). Hierarchical deep neural network for image captioning. Neural Processing Letters, 52(2), 1057–1067.
Huang, G., & Hu, H. (2019). c-RNN a fine-grained language model for image captioning. Neural Processing Letters, 49(2), 683–691.
Al-Khateeb, B., Ahmed, K., Mahmood, M., & Le, D. N. (2021). Rock hyraxes swarm optimization a new nature-inspired metaheuristic optimization algorithm. Computers, Materials & Continua, 68(1), 643–654.
Rajeyyagari, S. (2020). Automatic speaker diarization using deep LSTM in audio lecturing of e-Khool Platform. Journal of Networking and Communication Systems, 3(4), 17–25.
Swamy, S. M., Rajakumar, B. R., & Valarmathi, I. R. (2013). Design of hybrid wind and photovoltaic power system using opposition-based genetic algorithm with cauchy mutation. In IET Chennai Fourth International Conference on Sustainable Energy and Intelligent Systems (SEISCON 2013), Chennai, India. pp. 504–510. https://doi.org/10.1049/ic.2013.0361