An efficient pruning scheme of deep neural networks for Internet of Things applications
Tóm tắt
Nowadays, deep neural networks (DNNs) have been rapidly deployed to realize a number of functionalities like sensing, imaging, classification, recognition, etc. However, the computational-intensive requirement of DNNs makes it difficult to be applicable for resource-limited Internet of Things (IoT) devices. In this paper, we propose a novel pruning-based paradigm that aims to reduce the computational cost of DNNs, by uncovering a more compact structure and learning the effective weights therein, on the basis of not compromising the expressive capability of DNNs. In particular, our algorithm can achieve efficient end-to-end training that transfers a redundant neural network to a compact one with a specifically targeted compression rate directly. We comprehensively evaluate our approach on various representative benchmark datasets and compared with typical advanced convolutional neural network (CNN) architectures. The experimental results verify the superior performance and robust effectiveness of our scheme. For example, when pruning VGG on CIFAR-10, our proposed scheme is able to significantly reduce its FLOPs (floating-point operations) and number of parameters with a proportion of 76.2% and 94.1%, respectively, while still maintaining a satisfactory accuracy. To sum up, our scheme could facilitate the integration of DNNs into the common machine-learning-based IoT framework and establish distributed training of neural networks in both cloud and edge.
Tài liệu tham khảo
J. Lin, W. Yu, N. Zhang, X. Yang, H. Zhang, W. Zhao, A survey on Internet of Things: Architecture, enabling technologies, security and privacy, and applications. IEEE Internet Things J.4(5), 1125–1142 (2017).
M. A. Al-Garadi, A. Mohamed, A. Al-Ali, X. Du, M. Guizani, A survey of machine and deep learning methods for Internet of Things (IoT) security. arXiv preprint arXiv:1807.11023 (2018).
M. Mohammadi, A. Al-Fuqaha, S. Sorour, M. Guizani, Deep learning for IoT big data and streaming analytics: a survey. IEEE Commun. Surv. Tutorials. 20(4), 2923–2960 (2018).
E. Park, Y. Cho, J. Han, S. J. Kwon, Comprehensive approaches to user acceptance of Internet of Things in a smart home environment. IEEE Internet Things J.4(6), 2342–2350 (2017).
O. Elijah, T. A. Rahman, I. Orikumhi, C. Y. Leow, M. N. Hindia, An overview of Internet of Things (IoT) and data analytics in agriculture: Benefits and challenges. IEEE Internet Things J.5(5), 3758–3773 (2018).
H. Li, K. Ota, M. Dong, Learning IoT in edge: Deep learning for the Internet of Things with edge computing. IEEE Netw.32(1), 96–101 (2018).
X. Ma, T. Yao, M. Hu, Y. Dong, W. Liu, F. Wang, J. Liu, A survey on deep learning empowered IoT applications. IEEE Access. 7:, 181721–181732 (2019).
X. Xie, K. -H. Kim, in The 25th Annual International Conference on Mobile Computing and Networking. Source compression with bounded DNN perception loss for IoT edge computer vision (ACMLos Cabos, 2019), pp. 1–16.
D. Jia, D. Wei, S. Richard, L. Li-Jia, L. Kai, L. Fei-Fei, in The IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Imagenet: A large-scale hierarchical image database (IEEEMiami, 2009).
A. Krizhevsky, I. Sutskever, G. E. Hinton, in Advances in Neural Information Processing Systems. Imagenet classification with deep convolutional neural networks (Curran Associates, Inc.Harrahs and Harveys, Lake Tahoe, 2012), pp. 1097–1105.
K. Simonyan, A. Zisserman, in International Conference on Learning Representations (ICLR). Very deep convolutional networks for large-scale image recognition (OpenReview.netSan Diego, 2015).
C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, A. Rabinovich, in The IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Going deeper with convolutions (IEEEBoston, 2015).
K. He, X. Zhang, S. Ren, J. Sun, in The IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Deep residual learning for image recognition (IEEELas Vegas Nevada, 2016).
A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, H. Adam, Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861 (2017).
S. Han, J. Pool, J. Tran, W. Dally, in Advances in Neural Information Processing Systems 28, ed. by C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett. Learning both weights and connections for efficient neural network (Curran Associates, Inc.Montreal, 2015), pp. 1135–1143.
S. Han, H. Mao, W. Dally, in International Conference on Learning Representations (ICLR). Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding (OpenReview.netCaribe Hilton, San Juan, 2016).
P. Molchanov, S. Tyree, T. Karras, T. Aila, J. Kautz, in International Conference on Learning Representations (ICLR). Pruning convolutional neural networks for resource efficient inference (OpenReview.netPalais des Congreptune, Toulon, 2017).
H. Li, A. Kadav, I. Durdanovic, H. Samet, H. P. Graf, in International Conference on Learning Representations (ICLR). Pruning filters for efficient convnets (OpenReview.netPalais des Congreptune, Toulon, 2017).
Z. Liu, J. Li, Z. Shen, G. Huang, S. Yan, C. Zhang, in The IEEE International Conference on Computer Vision (ICCV). Learning efficient convolutional networks through network slimming (IEEEVenice, 2017).
J. Luo, J. Wu, W. Lin, in The IEEE International Conference on Computer Vision (ICCV). Thinet: A filter level pruning method for deep neural network compression (IEEEVenice, 2017).
T. Dettmers, L. Zettlemoyer, Sparse networks from scratch: Faster training without losing performance. arXiv preprint arXiv:1907.04840 (2019).
H. Mostafa, X. Wang, in Proceedings of the 36th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol.97, ed. by K. Chaudhuri, R. Salakhutdinov. Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization (Long Beach, 2019), pp. 4646–4655.
E. De Coninck, T. Verbelen, B. Vankeirsbilck, S. Bohez, P. Simoens, P. Demeester, B. Dhoedt, in International Internet of Things Summit. Distributed neural networks for Internet of Things: The big-little approach (Springer, 2015), pp. 484–492.
R. Hu, Y. Guo, E. P. Ratazzi, Y. Gong, Differentially private federated learning for resource-constrained Internet of Things. arXiv preprint arXiv:2003.12705 (2020).
A. Krizhevsky, G. Hinton, Learning multiple layers of features from tiny images. Technical Report (2009).
H. Amroun, M. H. Temkit, M. Ammi, in 2017 IEEE International Conference on Internet of Things (iThings) and IEEE Green Computing and Communications (GreenCom) and IEEE Cyber, Physical and Social Computing (CPSCom) and IEEE Smart Data (SmartData). Best feature for CNN classification of human activity using IoT network (IEEEExeter, 2017), pp. 943–950.
X. Ding, G. Ding, X. Zhou, Y. Guo, J. Han, J. Liu, in Advances in Neural Information Processing Systems 32. Global sparse momentum SGD for pruning very deep neural networks (Curran Associates, Inc.Vancouver, 2019).
W. Wen, C. Wu, Y. Wang, Y. Chen, H. Li, in Advances in Neural Information Processing Systems 29. Learning structured sparsity in deep neural networks (Curran Associates, Inc.Vancouver, 2016), pp. 2074–2082.
S. Ioffe, C. Szegedy, in Proceedings of the 32nd International Conference on Machine Learning (ICML). Batch normalization: Accelerating deep network training by reducing internal covariate shift (ACMLille, 2015).
Z. Zhuang, M. Tan, B. Zhuang, J. Liu, Y. Guo, Q. Wu, J. Huang, J. Zhu, in Advances in Neural Information Processing Systems 31. Discrimination-aware channel pruning for deep neural networks (Curran Associates, Inc.Montreal, 2018), pp. 875–886.
Y. He, X. Zhang, J. Sun, in The IEEE International Conference on Computer Vision (ICCV). Channel pruning for accelerating very deep neural networks (IEEEVenice, 2017).
P. Singh, V. K. Verma, P. Rai, V. P. Namboodiri, Leveraging filter correlations for deep model compression. arXiv e-prints, 1811–10559 (2018).
W. Wang, C. Fu, J. Guo, D. Cai, X. He, in Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19. Cop: Customized deep model compression via regularized correlation-based filter-level pruning (Morgan KaufmannMacao, 2019).
Z. Liu, H. Mu, X. Zhang, Z. Guo, X. Yang, K. -T. Cheng, J. Sun, in Proceedings of the IEEE International Conference on Computer Vision (ICCV). Metapruning: Meta learning for automatic neural network channel pruning (IEEESeoul, 2019), pp. 3296–3305.
S. Lin, R. Ji, C. Yan, B. Zhang, L. Cao, Q. Ye, F. Huang, D. Doermann, in The IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Towards optimal structured CNN pruning via generative adversarial learning (IEEELong Beach, 2019).
Z. Liu, M. Sun, T. Zhou, G. Huang, T. Darrell, in International Conference on Learning Representations (ICLR). Rethinking the value of network pruning (OpenReview.netNew Orleans, 2019).
Y. He, G. Kang, X. Dong, Y. Fu, Y. Yang, in IJCAI International Joint Conference on Artificial Intelligence. Soft filter pruning for accelerating deep convolutional neural networks (Morgan KaufmannStockholm, 2018).
Y. Wang, X. Zhang, L. Xie, J. Zhou, H. Su, B. Zhang, X. Hu, Pruning from scratch. arXiv e-prints, 1909–12579 (2019).
M. Lin, R. Ji, Y. Zhang, B. Zhang, Y. Wu, Y. Tian, Channel pruning via automatic structure search. arXiv e-prints, 2001–08565 (2020).
T. Elsken, J. H. Metzen, F. Hutter, Neural architecture search: A survey. J. Mach. Learn. Res.20(55), 1–21 (2019).
M. Tan, B. Chen, R. Pang, V. Vasudevan, M. Sandler, A. Howard, Q. V. Le, in 2019 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Mnasnet: Platform-aware neural architecture search for mobile (IEEELong Beach, 2019).
H. Liu, K. Simonyan, Y. Yang, in International Conference on Learning Representations(ICLR). DARTS: Differentiable architecture search (OpenReview.netNew Orleans, 2019).
Y. Xu, L. Xie, X. Zhang, X. Chen, G. -J. Qi, Q. Tian, H. Xiong, in International Conference on Learning Representations(ICLR). Pc-darts: Partial channel connections for memory-efficient architecture search (OpenReview.netVirtual Conference, Formerly Addis Ababa ETHIOPIA, 2020).
A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, A. Lerer, in NIPS-W. Automatic differentiation in pytorch (Curran Associates, Inc.Long Beach, 2017).
Y. He, P. Liu, Z. Wang, Z. Hu, Y. Yang, in The IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Filter pruning via geometric median for deep convolutional neural networks acceleration (IEEELong Beach, 2019).