Learning to detect soft shadow from limited data
Tóm tắt
Soft shadow is more challenging to detect than hard shadow due to its ambiguous boundary. Existing shadow detection methods pay more attention to hard shadow scene since collecting and annotating hard shadow images is more effortless. Motivated by that soft shadow has similar characteristics with hard shadow, and many traditional hard shadow datasets are publicly available, we propose a novel soft shadow detection method (namely Soft-DA) based on adversarial learning and domain adaptation scheme. Specifically, we create a limited soft shadow dataset, containing 1K soft shadow images with various scenes and shapes. Note that we just only need to annotate 0.4K shadow masks for semi-supervised learning. Besides, to tackle obvious domain discrepancy and potential intention difference between different datasets and similar tasks, we first align data distributions between domains by feature adversarial adaptation. And then, we introduce a novel detector separation strategy to tackle the intention difference issue. In this way, Soft-DA can effectively detect soft shadow with only a small number of soft shadow annotations. Extensive experiments demonstrate that our method can achieve superior performance to state of the arts.
Tài liệu tham khảo
citation_journal_title=IEEE Trans. Pattern Anal. Mach. Intell.; citation_title=Shadow removal using intensity surfaces and texture anchor points; citation_author=E Arbel, H Hel-Or; citation_volume=33; citation_issue=6; citation_publication_date=2010; citation_pages=1202-1216; citation_doi=10.1109/TPAMI.2010.157; citation_id=CR1
citation_journal_title=Vis. Comput.; citation_title=Deep learned compact binary descriptor with a lightweight network-in-network architecture for visual description; citation_author=R Bandara, L Ranathunga, NA Abdullah; citation_publication_date=2020; citation_doi=10.1007/s00371-020-01798-5; citation_id=CR2
Ganin, Y., Lempitsky, V.: Unsupervised domain adaptation by backpropagation. In: International Conference on Machine Learning, pp. 1180–1189 (2015)
Gong, H., Cosker, D.: Interactive shadow removal and ground truth for variable scene categories. In: BMVC, pp. 1–11 (2014)
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial nets. In: Advances in Neural Information Processing Systems, pp. 2672–2680 (2014)
citation_journal_title=ACM Trans. Graph.; citation_title=Learning to remove soft shadows; citation_author=M Gryka, M Terry, GJ Brostow; citation_volume=34; citation_issue=5; citation_publication_date=2015; citation_pages=153; citation_doi=10.1145/2732407; citation_id=CR6
Guo, R., Dai, Q., Hoiem, D.: Single-image shadow detection and removal using paired regions. In: CVPR 2011, pp. 2033–2040. IEEE (2011)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)
Hsu, H.K., Yao, C.H., Tsai, Y.H., Hung, W.C., Tseng, H.Y., Singh, M., Yang, M.H.: Progressive domain adaptation for object detection. In: The IEEE Winter Conference on Applications of Computer Vision, pp. 749–757 (2020)
Hu, X., Zhu, L., Fu, C.W., Qin, J., Heng, P.A.: Direction-aware spatial context features for shadow detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7454–7462 (2018)
citation_journal_title=The Vis. Comput.; citation_title=Deeplight: light source estimation for augmented reality using deep learning; citation_author=P Kán, H Kafumann; citation_volume=35; citation_issue=6–8; citation_publication_date=2019; citation_pages=873-883; citation_doi=10.1007/s00371-019-01666-x; citation_id=CR11
citation_journal_title=ACM Trans. Graph. (TOG); citation_title=Rendering synthetic objects into legacy photographs; citation_author=K Karsch, V Hedau, D Forsyth, D Hoiem; citation_volume=30; citation_issue=6; citation_publication_date=2011; citation_pages=1-12; citation_doi=10.1145/2070781.2024191; citation_id=CR12
citation_journal_title=IEEE Trans. Pattern Anal. Mach. Intell.; citation_title=Automatic shadow detection and removal from a single image; citation_author=SH Khan, M Bennamoun, F Sohel, R Togneri; citation_volume=38; citation_issue=3; citation_publication_date=2015; citation_pages=431-446; citation_doi=10.1109/TPAMI.2015.2462355; citation_id=CR13
Khodabandeh, M., Vahdat, A., Ranjbar, M., Macready, W.G.: A robust learning approach to domain adaptive object detection. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 480–490 (2019)
Lafarge, M.W., Pluim, J.P., Eppenhof, K.A., Moeskops, P., Veta, M.: Domain-adversarial neural networks to address the appearance variability of histopathology images. In: Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support, pp. 83–91. Springer (2017)
Lalonde, J.F., Efros, A.A., Narasimhan, S.G.: Estimating natural illumination from a single outdoor image. In: 2009 IEEE 12th International Conference on Computer Vision, pp. 183–190. IEEE (2009)
Le, H., Vicente, T.F.Y., Nguyen, V., Hoai, M., Samaras, D.: A+ D net: training a shadow detector with adversarial shadow attenuation. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 680–696 (2018)
Lin, T.Y., Goyal, P., Girshick, R., He, K., Dollár, P.: Focal loss for dense object detection. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2980–2988 (2017)
Long, M., Zhu, H., Wang, J., Jordan, M.I.: Deep transfer learning with joint adaptation networks. In: International Conference on Machine Learning, pp. 2208–2217 (2017)
Luo, Y., Zheng, L., Guan, T., Yu, J., Yang, Y.: Taking a closer look at domain shift: category-level adversaries for semantics consistent domain adaptation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2507–2516 (2019)
Luo, Z., Zou, Y., Hoffman, J., Fei-Fei, L.F.: Label efficient learning of transferable representations acrosss domains and tasks. In: Advances in Neural Information Processing Systems, pp. 165–177 (2017)
Mao, X., Li, Q., Xie, H., Lau, R.Y., Wang, Z., Paul Smolley, S.: Least squares generative adversarial networks. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2794–2802 (2017)
Mohajerani, S., Saeedi, P.: Cpnet: a context preserver convolutional neural network for detecting shadows in single rgb images. In: 2018 IEEE 20th International Workshop on Multimedia Signal Processing (MMSP), pp. 1–5 (2018)
Motiian, S., Piccirilli, M., Adjeroh, D.A., Doretto, G.: Unified deep supervised domain adaptation and generalization. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 5715–5725 (2017)
Nguyen, V., Yago Vicente, T.F., Zhao, M., Hoai, M., Samaras, D.: Shadow detection with conditional generative adversarial networks. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 4510–4518 (2017)
Nielsen, M., Madsen, C.B.: Graph cut based segmentation of soft shadows for seamless removal and augmentation. In: SCIA’07 Proceedings of the 15th Scandinavian Conference on Image Analysis, pp. 918–927 (2007)
Okabe, T., Sato, I., Sato, Y.: Attached shadow coding: estimating surface normals from shadows under unknown reflectance and lighting conditions. In: 2009 IEEE 12th International Conference on Computer Vision, pp. 1693–1700. IEEE (2009)
Ronneberger, O., Fischer, P., Brox, T.: U-net: convolutional networks for biomedical image segmentation. In: International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 234–241. Springer (2015)
Saito, K., Watanabe, K., Ushiku, Y., Harada, T.: Maximum classifier discrepancy for unsupervised domain adaptation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3723–3732 (2018)
Sanchez-Matilla, R., Poiesi, F., Cavallaro, A.: Online multi-target tracking with strong and weak detections. In: European Conference on Computer Vision, pp. 84–99. Springer (2016)
Savarese, S., Rushmeier, H., Bernardini, F., Perona, P.: Shadow carving. In: Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001, vol. 1, pp. 190–197. IEEE (2001)
Shen, L., Wee Chua, T., Leman, K.: Shadow optimization from structured deep edge detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2067–2074 (2015)
Tzeng, E., Hoffman, J., Saenko, K., Darrell, T.: Adversarial discriminative domain adaptation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7167–7176 (2017)
Tzeng, E., Hoffman, J., Zhang, N., Saenko, K., Darrell, T.: Deep domain confusion: maximizing for domain invariance. arXiv preprint
arXiv:1412.3474
(2014)
Vicente, T.F.Y., Hou, L., Yu, C.P., Hoai, M., Samaras, D.: Large-scale training of shadow detectors with noisily-annotated shadow examples. In: European Conference on Computer Vision, pp. 816–832. Springer (2016)
Wang, J., Li, X., Yang, J.: Stacked conditional generative adversarial networks for jointly learning shadow detection and shadow removal. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1788–1797 (2018)
Wang, S., Yu, L., Li, K., Yang, X., Fu, C.W., Heng, P.A.: Boundary and entropy-driven adversarial learning for fundus image segmentation. In: International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 102–110. Springer (2019)
Wang, T., Hu, X., Wang, Q., Heng, P.A., Fu, C.W.: Instance shadow detection. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1880–1889 (2020)
citation_journal_title=ACM Trans. Graph. (TOG); citation_title=Natural shadow matting; citation_author=TP Wu, CK Tang, MS Brown, HY Shum; citation_volume=26; citation_issue=2; citation_publication_date=2007; citation_pages=8-es; citation_doi=10.1145/1243980.1243982; citation_id=CR39
Zhang, Y., Chen, H., Wei, Y., Zhao, P., Cao, J., Fan, X., Lou, X., Liu, H., Hou, J., Han, X., et al.: From whole slide imaging to microscopy: deep microscopy adaptation network for histopathology cancer image classification. In: International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 360–368. Springer (2019)
citation_journal_title=Image Vis. Comput.; citation_title=Intelligent detection of building cracks based on deep learning; citation_author=M Zheng, Z Lei, K Zhang; citation_volume=103; citation_publication_date=2020; citation_pages=103987; citation_doi=10.1016/j.imavis.2020.103987; citation_id=CR41
Zheng, Q., Qiao, X., Cao, Y., Lau, R.W.: Distraction-aware shadow detection. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5167–5176 (2019)
Zhu, L., Deng, Z., Hu, X., Fu, C.W., Xu, X., Qin, J., Heng, P.A.: Bidirectional feature pyramid network with recurrent attention residual modules for shadow detection. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 122–137 (2018)
citation_journal_title=Vis. Comput.; citation_title=Dual-modality spatiotemporal feature learning for spontaneous facial expression recognition in e-learning using hybrid deep neural network; citation_author=X Zhu, Z Chen; citation_volume=36; citation_publication_date=2019; citation_pages=1-13; citation_id=CR44