Efficient 3D object recognition in mobile edge environment

Springer Science and Business Media LLC - Tập 11 - Trang 1-16 - 2022
Mofei Song1,2, Qi Guo1,2
1The School of Computer Science and Engineering, Southeast University, Nanjing, China
2The Key Lab of Computer Network and Information Integration (Ministry of Education), Southeast University, Nanjing, China

Tóm tắt

3D object recognition has great research and application value in the fields of automatic drive, virtual reality, and commercial manufacturing. Although various deep models have been exploited and achieved remarkable results for 3D object recognition, their computational cost is too high for most mobile applications. This paper combines edge computing and 3D object recognition into a powerful and efficient framework. It consists of a cloud-based rendering stage and a terminal-based recognition stage. In the first stage, inspired by the cloud-based rendering technique, we upload the 3D object data from the mobile device to the edge cloud server for multi-view rendering. The rendering stage utilizes the powerful computing resource in the edge cloud server to generate multiple view images of the given 3D object from different views by parallel high-quality rendering. During the terminal-based recognition stage, we integrate a lightweight CNN architecture and a neural network quantization technique into a 3D object recognition model based on the multiple images rendered in the edge cloud server, which can be executed fast in the mobile device. To reduce the cost of network training, we propose a novel semi-supervised 3D deep learning method with fewer labeled samples. Experiments demonstrate that our method achieves competitive performance compared to the state-of-the-art methods with low latency running in the mobile edge environment.

Tài liệu tham khảo

Xu X, Li H, Xu W, Liu Z, Yao L, Dai F (2021) Artificial intelligence for edge service optimization in internet of vehicles: A survey. Tsinghua Sci Technol 27(2):270–287 Qi L, Hu C, Zhang X, Khosravi MR, Sharma S, Pang S, Wang T (2020) Privacy-aware data fusion and prediction with spatial-temporal context for smart city industrial environment. IEEE Trans Industr Inform 17(6):4159–4167 Tong Z, Ye F, Yan M, Liu H, Basodi S (2021) A survey on algorithms for intelligent computing and smart city applications. Big Data Min Analytics 4(3):155–172 Chen Y, Gu W, Xu J, et al (2022) Dynamic task offloading for digital twin-empowered mobile edge computing via deep reinforcement learning. China Commun Huang J, Zhang C, Zhang J (2020) A multi-queue approach of energy efficient task scheduling for sensor hubs. Chin J Electron 29(2):242–247 Zhang W, Chen X, Jiang J (2020) A multi-objective optimization method of initial virtual machine fault-tolerant placement for star topological data centers of cloud systems. Tsinghua Sci Technol 26(1):95–111 Chen Y, Zhao F, Chen X, Wu Y (2022) Efficient multi-vehicle task offloading for mobile edge computing in 6g networks. IEEE Trans Veh Technol 71(5):4584–4595 Hou C, Wu J, Cao B, Fan J (2021) A deep-learning prediction model for imbalanced time series data forecasting. Big Data Min Analytics 4(4):266–278 Ioannidou A, Chatzilari E, Nikolopoulos S, Kompatsiaris I (2017) Deep learning advances in computer vision with 3d data: A survey. Acm Comput Surv 50(2):1–38 Mirbauer M, Krabec M, Krivanek J, Sikudova E (2022) Survey and evaluation of neural 3d shape classification approaches. IEEE Trans Pattern Anal Mach Intell 44(11):8635–8656 Xiao YP, Lai YK, Zhang FL, Li C, Gao L (2020) A survey on deep geometry learning: From a representation perspective. Comput Vis Media 6(2):113–133 Su H, Maji S, Kalogerakis E, Learned-Miller EG (2015) Multi-view convolutional neural networks for 3d shape recognition. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, Santiago, 7-13 December 2015. IEEE, New York Chen SL, Zheng L, Zhang Y, Sun Z, Xu K (2018) Veram: View-enhanced recurrent attention model for 3d shape classification. IEEE Trans Vis Comput Graph 25:3244–3257 Chen Y, Xing H, Ma Z, et al (2022) Cost-efficient edge caching for noma-enabled iot services. China Commun Zhi S, Liu Y, Li X, Guo Y (2018) Toward real-time 3d object recognition: A lightweight volumetric cnn framework using multitask learning. Comput Graph 71:199–207 Wu X, Chang J, Lai YK, Yang J, Tian Q (2021) Bispl: Bidirectional self-paced learning for recognition from web data. IEEE Trans Image Process 30:6512–6527 Chang AX, Funkhouser T, Guibas L, Hanrahan P, Huang Q, Li Z, Savarese S, Savva M, Song S, Su H, et al (2015) Shapenet: An information-rich 3d model repository. arXiv preprint arXiv:1502.03167 Mo K, Zhu S, Chang AX, Yi L, Tripathi S, Guibas LJ, Su H (2019) Partnet: A large-scale benchmark for fine-grained and hierarchical part-level 3d object understanding. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, Long Beach, CA, 16-20 June 2019. IEEE, New York Yu F, Liu K, Zhang Y, Zhu C, Xu K (2019) Partnet: A recursive part decomposition network for fine-grained and hierarchical shape segmentation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, Long Beach, CA, 16-20 June 2019. IEEE, New York Fu H, Jia R, Gao L, Gong M, Zhao B, Maybank S, Tao D (2021) 3d-future: 3d furniture shape with texture. Int J Comput Vision 129(12):3313–3337 Cheraghian A, Rahman S, Campbell D, Petersson L (2020) Transductive zero-shot learning for 3d point cloud classification. In: Proceedings of the IEEE/CVF winter conference on applications of computer vision, Colorado, 1-5 March 2020. IEEE, New York Wu Z, Zhang Y, Zeng M, Qin F, Wang Y (2018) Joint analysis of shapes and images via deep domain adaptation. Comput Graph 70:140–147 Han Z, Shang M, Liu YS, Zwicker M (2019) View inter-prediction gan: Unsupervised representation learning for 3d shapes by learning global shape memories to support local view predictions. In: Proceedings of the AAAI Conference on artificial intelligence, Hawaii, 27 January–1 February 2019. AAAI, Menlo Park Zhang X, Zhou X, Lin M, Sun J (2018) Shufflenet: An extremely efficient convolutional neural network for mobile devices. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, Salt Lake City, UT, 18-22 June 2018. IEEE, New York Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: Proceedings of advances in neural information processing systems, Nevada, 3-8 December 2012. MIT Press, Cambridge Sohn K, Berthelot D, Carlini N, Zhang Z, Zhang H, Raffel CA, Cubuk ED, Kurakin A, Li CL (2020) Fixmatch: Simplifying semi-supervised learning with consistency and confidence. In: Proceedings of advances in neural information processing systems, virtual, 6-10 December 2020. MIT Press, Cambridge Sharma A, Grau O, Fritz M (2016) Vconv-dae: Deep volumetric shape learning without object labels. In: Proceedings of geometry meets deep learning workshop at european conference on computer vision, Amsterdam, 9 October 2016. Springer, Berlin Wu J, Zhang C, Xue T, Freeman B, Tenenbaum JB (2016) Learning a probabilistic latent space of object shapes via 3d generative-adversarial modeling. In: Proceedings of advances in neural information processing systems, Barcelona, 5-10 December 2016. MIT Press, Cambridge Yang Y, Feng C, Shen Y, Tian D (2018) Foldingnet: Point cloud auto-encoder via deep grid deformation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, Salt Lake City, UT, 18-22 June 2018. IEEE, New York Achlioptas P, Diamanti O, Mitliagkas I, Guibas L (2018) Learning representations and generative models for 3d point clouds. In: Proceedings of international conference on machine learning, Stockholm Sweden, 10-15 July 2018. ACM, New York Eckart B, Yuan W, Liu C, Kautz J (2021) Self-supervised learning on 3d point clouds by learning discrete generative models. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, virtual, 19-25 June 2021. IEEE, New York Afham M, Dissanayake I, Dissanayake D, Dharmasiri A, Thilakarathna K, Rodrigo R (2022) Crosspoint: Self-supervised cross-modal contrastive learning for 3d point cloud understanding. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, Louisiana, 19-24 June 2022. IEEE, New York Qi CR, Su H, Mo K, Guibas LJ (2017a) Pointnet: Deep learning on point sets for 3d classification and segmentation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, Honolulu, HI, 22-25 July 2017. IEEE, New York Qi CR, Yi L, Su H, Guibas LJ (2017b) Pointnet++: Deep hierarchical feature learning on point sets in a metric space. In: Proceedings of advances in neural information processing systems, Long Beach, 4 December 2017. MIT Press, Cambridge Liu Y, Fan B, Xiang S, Pan C (2019) Relation-shape convolutional neural network for point cloud analysis. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, Long Beach, 16-20 June 2019. IEEE, New York Berg A, Oskarsson M, O’Connor M (2022) Points to patches: Enabling the use of self-attention for 3d shape recognition. arXiv preprint arXiv:2204.03957 Wijaya KT, Paek DH, Kong SH (2022) Advanced feature learning on point clouds using multi-resolution features and learnable pooling. arXiv preprint arXiv:2205.09962 Song M, Liu Y, Liu XF (2020) Semi-supervised 3d shape recognition via multimodal deep co-training. Comput Graph Forum 39(7):279–289 Chen L, Zhang Y, Lin Y, Jiang M, Huang Y, Lei Y (2021) Consistency-based semi-supervised learning for point cloud classification. In: Proceedings of international conference on pattern recognition and artificial intelligence, virtual, 20-22 August 2021. Springer, Berlin Shi X, Xu X, Zhang W, Zhu X, Foo CS, Jia K (2022) Open-set semi-supervised learning for 3d point cloud understanding. arXiv preprint arXiv:2205.01006 Bader C, Dingler S, Schwieger V (2021) Pvenet: Point voxel encoder network for real-time classification of lidar point cloud segments. In: Proceedings of international conference on advanced robotics, virtual, 6-10 December 2021. IEEE, New York Li F, Yu X, Ge R, Wang Y, Cui Y, Zhou H (2021) Bcse: Blockchain-based trusted service evaluation model over big data. Big Data Min Analytics 5(1):1–14 Sandhu AK (2021) Big data with cloud computing: Discussions and challenges. Big Data Min Analytics 5(1):32–40 Huang J, Lv B, Wu Y, Chen Y, Shen X (2022) Dynamic admission control and resource allocation for mobile edge computing enabled small cell network. IEEE Trans Veh Technol 71(2):1964–1973 Qi L, Lin W, Zhang X, Dou W, Xu X, Chen J (2022) A correlation graph based approach for personalized and compatible web apis recommendation in mobile app development. IEEE Trans Knowl Data Eng. https://doi.org/10.1109/TKDE.2022.3168611 Xu J, Li D, Gu W et al (2022) Uav-assisted task offloading for iot in smart buildings and environment via deep reinforcement learning. Build Environ 222. https://doi.org/10.1016/j.buildenv.2022.109218 Li K, Zhao J, Hu J, Chen Y (2022) Dynamic energy efficient task offloading and resource allocation for noma-enabled iot in smart buildings and environment. Build Environ. https://doi.org/10.1016/j.buildenv.2022.109513 Wang K (2020) Migration strategy of cloud collaborative computing for delay-sensitive industrial iot applications in the context of intelligent manufacturing. Comput Commu 150:413–420 Huang J, Tong Z, Feng Z (2022) Geographical poi recommendation for internet of things: A federated learning approach using matrix factorization. Int J Commun Syst. https://doi.org/10.1002/dac.5161 Huang J, Gao H, Wan S et al (2023) Aoi-aware energy control and computation offloading for industrial iot. Future Gener Comput Syst 139:29–37 Chen Y, Gu W, Li K (2022) Dynamic task offloading for internet of things in mobile edge computing via deep reinforcement learning. Int J Commun Syst. https://doi.org/10.1002/dac.5154 Chen Y, Liu Z, Zhang Y et al (2021) Deep reinforcement learning-based dynamic resource management for mobile edge computing in industrial internet of things. IEEE Trans Industr Inform 17(7):4925–4934 Bi R, Liu Q, Ren J, Tan G (2020) Utility aware offloading for mobile-edge computing. Tsinghua Sci Technol 26(2):239–250 Wu Y, Zhang L, Berretti S, Wan S (2022) Medical image encryption by content-aware dna computing for secure healthcare. IEEE Trans Industr Inform. https://doi.org/10.1109/TII.2022.3194590 Wu Y, Guo H, Chakraborty C, Khosravi M, Berretti S, Wan S (2022) Edge computing driven low-light image dynamic enhancement for object detection. IEEE Trans Netw Sci Eng. https://doi.org/10.1109/TNSE.2022.3151502 Shi G, Wu Y, Liu J, Wan S, Wang W, Lu T (2022) Incremental few-shot semantic segmentation via embedding adaptive-update and hyper-class representation. arXiv preprint arXiv:2207.12964 Kim D, Lee S, Lee H, Cho S (2008) A distance-based compression of 3d meshes for mobile devices. IEEE Trans Consum Electron 54(3):1398–1405 Deng Y, Ni Y, Li Z, Mu S, Zhang W (2017) Toward real-time ray tracing: A survey on hardware acceleration and microarchitecture techniques. ACM Comput Surv 50(4):1–41 Su JC, Gadelha M, Wang R, Maji S (2018) A deeper look at 3d shape classifiers. In: Proceedings of the european conference on computer vision, Munich, 8–14 September 2018. Springer, Berlin Fu H, Cohen-Or D, Dror G, Sheffer A (2008) Upright orientation of man-made objects. In: Proceedings of special interest group on computer graphics and interactive techniques conference, Los Angeles, 11-15 August 2008. ACM, New York Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. Ithaca, NY. arXiv preprint https://arxiv.org/abs/1409.1556 He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, Las Vegas, 27-30 June 2016 Iandola FN, Han S, Moskewicz MW, Ashraf K, Dally WJ, Keutzer K (2016) Squeezenet: Alexnet-level accuracy with 50x fewer parameters and 0.5 mb model size. arXiv preprint arXiv:1602.07360 Chollet F (2017) Xception: Deep learning with depthwise separable convolutions. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, Honolulu, HI, 22-25 July 2017. IEEE, New York Howard AG, Zhu M, Chen B, Kalenichenko D, Wang W, Weyand T, Andreetto M, Adam H (2017) Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861 Cubuk ED, Zoph B, Shlens J, Le QV (2020) Randaugment: Practical automated data augmentation with a reduced search space. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, virtual, 14-19 June 2020. IEEE, New York Kurakin A, Raffel C, Berthelot D, Cubuk ED, Zhang H, Sohn K, Carlini N (2020) Remixmatch: Semi-supervised learning with distribution matching and augmentation anchoring. In: Proceedings of international conference on learning representations, virtual, 26 April-1 May 2020. Ithaca, NY Johnson R, Zhang T (2013) Accelerating stochastic gradient descent using predictive variance reduction. In: Proceedings of advances in neural information processing systems, Nevada, 5-10 December 2013. MIT Press, Cambridge Yang J, Shen X, Xing J, Tian X, Li H, Deng B, Huang J, Hua Xs (2019) Quantization networks. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, Long Beach, 16-20 June 2019. IEEE, New York Tailor SA, Fernandez-Marques J, Lane ND (2020) Degree-quant: Quantization-aware training for graph neural networks. arXiv preprint arXiv:2008.05000 Wang Y, Sun Y, Liu Z, Sarma SE, Bronstein MM, Solomon JM (2019) Dynamic graph cnn for learning on point clouds. ACM Trans Graph 38(5):1–12 Khan SH, Guo Y, Hayat M, Barnes N (2019) Unsupervised primitive discovery for improved 3d generative modeling. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, Long Beach, 16-20 June 2019. IEEE, New York