Joint regression and learning from pairwise rankings for personalized image aesthetic assessment
Tóm tắt
Recent image aesthetic assessment methods have achieved remarkable progress due to the emergence of deep convolutional neural networks (CNNs). However, these methods focus primarily on predicting generally perceived preference of an image, making them usually have limited practicability, since each user may have completely different preferences for the same image. To address this problem, this paper presents a novel approach for predicting personalized image aesthetics that fit an individual user’s personal taste. We achieve this in a coarse to fine manner, by joint regression and learning from pairwise rankings. Specifically, we first collect a small subset of personal images from a user and invite him/her to rank the preference of some randomly sampled image pairs. We then search for the K-nearest neighbors of the personal images within a large-scale dataset labeled with average human aesthetic scores, and use these images as well as the associated scores to train a generic aesthetic assessment model by CNN-based regression. Next, we fine-tune the generic model to accommodate the personal preference by training over the rankings with a pairwise hinge loss. Experiments demonstrate that our method can effectively learn personalized image aesthetic preferences, clearly outperforming state-of-the-art methods. Moreover, we show that the learned personalized image aesthetic benefits a wide variety of applications.
Tài liệu tham khảo
Zhang, F.-L.; Wang, M.; Hu, S.-M. Aesthetic image enhancement by dependence-aware object recomposition. IEEE Transactions on Multimedia Vol. 15, No. 7, 1480–1490, 2013.
Zhang, Q.; Nie, Y. W.; Zhang, L.; Xiao, C. X. Underexposed video enhancement via perception-driven progressive fusion. IEEE Transactions on Visualization and Computer Graphics Vol. 22, No. 6, 1773–1785, 2016.
Zhang, Q.; Yuan, G. Z.; Xiao, C. X.; Zhu, L.; Zheng, W. S. High-quality exposure correction of underexposed photos. In: Proceedings of the 26th ACM international conference on Multimedia, 582–590, 2018.
Zhang, F. L.; Wu, X.; Li, R. L.; Wang, J.; Zheng, Z. H.; Hu, S. M. Detecting and removing visual distractors for video aesthetic enhancement. IEEE Transactions on Multimedia Vol. 20, No. 8, 1987–1999, 2018.
Zhang, Q.; Nie, Y.; Zheng, W.-S. Dual illumination estimation for robust exposure correction. Computer Graphics Forum Vol. 38, 243–252, 2019.
Zhang, Q.; Yin, G. L.; Nie, Y. W.; Zheng, W. S. Deep camouflage images. Proceedings of the AAAI Conference on Artificial Intelligence Vol. 34, No. 7, 12845–12852, 2020.
Zhang, Q.; Nie, Y.; Zhu, L.; Xiao, C.; Zheng, W.-S. Enhancing underexposed photos using perceptually bidirectional similarity. IEEE Transactions on Multimedia Vol. 23, 189–202, 2021.
Murray, N.; Marchesotti, L.; Perronnin, F. AVA: A large-scale database for aesthetic visual analysis. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2408–2415, 2012.
Kong, S.; Shen, X. H.; Lin, Z.; Mech, R.; Fowlkes, C. Photo aesthetics ranking network with attributes and content adaptation. In: Proceedings of the European Conference on Computer Vision, 662–679, 2016.
Ren, J.; Shen, X.; Lin, Z.; Mech, R.; Foran, D. J. Personalized image aesthetics. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 638–647, 2017.
Park, K.; Hong, S.; Baek, M.; Han, B. Personalized image aesthetic quality assessment by joint regression and ranking. In: Proceedings of the IEEE Winter Conference on Applications of Computer Vision, 1206–1214, 2017.
Sarwar, B.; Karypis, G.; Konstan, J.; Reidl, J. Item-based collaborative filtering recommendation algorithms. In: Proceedings of the 10th international Conference on World Wide Web, 285–295, 2001.
Breese, J. S.; Heckerman, D.; Kadie, C. Empirical analysis of predictive algorithms for collaborative filtering. arXiv preprint arXiv:1301.7363, 2013.
Wang, G.; Yan, J.; Qin, Z. Collaborative and attentive learning for personalized image aesthetic assessment. In: Proceedings of the International Joint Conference on Artificial Intelligence, 957–963, 2018.
Korhonen, J. Assessing personally perceived image quality via image features and collaborative filtering. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 8169–8177, 2019.
Luo, W.; Wang, X.; Tang, X. Content-based photo quality assessment. In: Proceedings of the IEEE International Conference on Computer Vision, 2206–2213, 2011.
Dhar, S.; Ordonez, V.; Berg, T. L. High level describable attributes for predicting aesthetics and interestingness. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1657–1664, 2011.
Marchesotti, L.; Perronnin, F.; Larlus, D.; Csurka, G. Assessing the aesthetic quality of photographs using generic image descriptors. In: Proceedings of the IEEE International Conference on Computer Vision, 1784–1791, 2011.
Lu, X.; Lin, Z.; Shen, X.; Mech, R.; Wang. J. Z. Deep multi-patch aggregation network for image style, aesthetics, and quality estimation. In: Proceedings of the IEEE International Conference on Computer Vision, 990–998, 2015.
Sheng, K. K.; Dong, W. M.; Ma, C. Y.; Mei, X.; Huang, F. Y.; Hu, B. G. Attention-based multipatch aggregation for image aesthetic assessment. In: Proceedings of the ACM International Conference on Multimedia, 879–886, 2018.
Mai, L.; Jin, H.; Liu, F. Composition-preserving deep photo aesthetics assessment. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 497–506, 2016.
Talebi, H.; Milanfar, P. NIMA: Neural image assessment. IEEE Transactions on Image Processing Vol. 27, No. 8, 3998–4011, 2018.
Zeng, H.; Cao, Z.; Zhang, L.; Bovik, A. C. A unified probabilistic formulation of image aesthetic assessment. IEEE Transactions on Image Processing Vol. 29, 1548–1561, 2019.
Zhang, X. D.; Gao, X. B.; Lu, W.; He, L. H. A gated peripheral-foveal convolutional neural network for unified image aesthetic prediction. IEEE Transactions on Multimedia Vol. 21, No. 11, 2815–2826, 2019.
Pan, B. W.; Wang, S. F.; Jiang, Q. S. Image aesthetic assessment assisted by attributes through adversarial learning. Proceedings of the AAAI Conference on Artificial Intelligence Vol. 33, 679–686, 2019.
Wang, X. C.; Liang, X. H.; Yang, B. L.; Li, F. W. B. No-reference synthetic image quality assessment with convolutional neural network and local image saliency. Computational Visual Media Vol. 5, No. 2, 193–208, 2019.
Sheng, K. K.; Dong, W. M.; Chai, M. L.; Wang, G. H.; Zhou, P.; Huang, F. Y.; Hu, B.; Ji, R.; Ma, C. Revisiting image aesthetic assessment via self-supervised feature learning. Proceedings of the AAAI Conference on Artificial Intelligence Vol. 34, No. 4, 5709–5716, 2020.
Deng, Y. B.; Loy, C. C.; Tang, X. O. Image aesthetic assessment: An experimental survey. IEEE Signal Processing Magazine Vol. 34, No. 4, 80–106, 2017.
Li, L. D.; Zhu, H. C.; Zhao, S. C.; Ding, G. G.; Jiang, H. Y.; Tan, A. Personality driven multi-task learning for image aesthetic assessment. In: Proceedings of the International Conference on Multimedia and Expo, 430–435, 2019.
Lee, J. T.; Kim, C. S. Image aesthetic assessment based on pairwise comparison: A unified approach to score regression, binary classification, and personalization. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, 1191–1200, 2019.
Zhu, H.; Li, L.; Wu, J.; Zhao, S.; Ding, G.; Shi, G. Personalized image aesthetics assessment via meta-learning with bilevel gradient optimization. IEEE Transactions on Cyberneticshttps://doi.org/10.1109/TCYB.2020.2984670, 2020.
Cui, C. R.; Yang, W. Y.; Shi, C.; Wang, M.; Nie, X. S.; Yin, Y. L. Personalized image quality assessment with social-sensed aesthetic preference. Information Sciences Vol. 512, 780–794, 2020.
Yan, J.; Lin, S.; Kang, S. B.; Tang, X. A learning-to-rank approach for image color enhancement. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2987–2994, 2014.
Paisitkriangkrai, S.; Shen, C. H.; van den Hengel, A. Learning to rank in person re-identification with metric ensembles. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1846–1855, 2015.
Liu, X. L.; van de Weijer, J.; Bagdanov, A. D. RankIQA: Learning from rankings for no-reference image quality assessment. In: Proceedings of the IEEE International Conference on Computer Vision, 1040–1049, 2017.
Liu, X. L.; van de Weijer, J.; Bagdanov, A. D. Leveraging unlabeled data for crowd counting by learning to rank. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 7661–7669, 2018.
Gong, Y. C.; Jia, Y. Q.; Leung, T.; Toshev, A.; Ioffe, S. Deep convolutional ranking for multilabel image annotation. arXiv preprint arXiv:1312.4894, 2013.
Wang, Y. L.; Wang, S. H.; Tang, J. L.; Liu, H.; Li, B. X. PPP: Joint pointwise and pairwise image label prediction. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 6005–6013, 2016.
Simonyan, K.; Zisserman, A. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.
Deng, J.; Dong, W.; Socher, R.; Li, L. J.; Li, K.; Li, F. F. ImageNet: A large-scale hierarchical image database. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 248–255, 2009.
Myers, J. L.; Well, A.; Lorch, R. F. Research Design and Statistical Analysis. Routledge, 2010.
Talebi, H.; Milanfar, P. Learned perceptual image enhancement. In: Proceedings of the IEEE International Conference on Computational Photography, 1–13, 2018.
Hosu, V.; Goldlücke, B.; Saupe, D. Effective aesthetics prediction with multi-level spatially pooled features. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 9375–9383, 2019.
O’Donovan, P.; Agarwala, A.; Hertzmann, A. Collaborative filtering of color aesthetics. In: Proceedings of the Workshop on Computational Aesthetics, 33–40, 2014.
Joachims, T. Optimizing search engines using click-through data. In: Proceedings of the ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 133–142, 2002.
Burges, C.; Shaked, T.; Renshaw, E.; Lazier, A.; Deeds, M.; Hamilton, N.; Hullender, G. Learning to rank using gradient descent. In: Proceedings of the 22nd International Conference on Machine Learning, 89–96, 2005.
Wang, R.; Zhang, Q.; Fu, C.-W.; Shen, X.; Zheng, W.-S.; Jia, J. Underexposed photo enhancement using deep illumination estimation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 6849–6857, 2019.
Park, J.; Lee, J. Y.; Yoo, D.; Kweon, I. S. Distort-and-recover: Color enhancement using deep reinforcement learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 5928–5936, 2018.
Deng, Z.; Zhu, L.; Hu, X.; Fu, C.-W.; Xu, X.; Zhang, Q.; Qin, J.; Heng, P.-A. Deep multi-model fusion for single-image dehazing. In: Proceedings of the IEEE International Conference on Computer Vision, 2453–2462, 2019.