Trends in AI inference energy consumption: Beyond the performance-vs-parameter laws of deep learning

McDonald, 2022 Freund, 2019 Hernandez, 2020 Barr, 2019 Leopold, 2019 Amodei, 2018 Gholami, 2021 Canziani, 2017 Li, 2016, Evaluating the energy efficiency of deep convolutional neural networks on cpus and gpus, 477 Anthony, 2020 Thompson, 2020 Rodrigues, 2018, Synergy: An energy measurement and prediction framework for convolutional neural networks on jetson tx1, 375 Cai, 2017, Neuralpower: Predict and deploy energy-efficient convolutional neural networks, 622 Balas, 2019 Park, 2018 Xu, 2018, Scaling for edge inference of deep neural networks, Nat. Electron., 1, 216, 10.1038/s41928-018-0059-3 Bianco, 2018, Benchmark analysis of representative deep neural network architectures, IEEE Access, 6, 64270, 10.1109/ACCESS.2018.2877890 European Environment Information, 2021 Russakovsky, 2015 A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, S.R. Bowman, GLUE: A multi-task benchmark and analysis platform for natural language understanding, in: The Proceedings of ICLR, 2019. Hernandez, 2020 Schwartz, 2019 Strubell, 2019 Henderson, 2020, Towards the systematic reporting of the energy and carbon footprints of machine learning, J. Mach. Learn. Res., 21, 1 Bommasani, 2021 Tan, 2020 Tan, 2021 Molom-Ochir, 2021, Energy and cost considerations for GPU accelerated AI inference workloads, 1 Paszke, 2016 Cadene, 2016 Sémery, 2019 Albanie, 2016 Coleman, 2017, Dawnbench: An end-to-end deep learning benchmark and competition, Training, 100, 102 Mattson, 2020, Mlperf: An industry standard benchmark suite for machine learning performance, IEEE Micro, 40, 8, 10.1109/MM.2020.2974843 Stojnic, 2021 Sovrasov, 2020 Krizhevsky, 2012, Imagenet classification with deep convolutional neural networks, Adv. Neural Inf. Process. Syst., 25, 1097 Simonyan, 2015 Zhai, 2021 Clark, 2020 Batra, 2018 HajiRassouliha, 2018, Suitability of recent hardware accelerators (dsps, fpgas, and gpus) for computer vision and image processing algorithms, Signal Process., Image Commun., 68, 101, 10.1016/j.image.2018.07.007 Asano, 2009, Performance comparison of fpga, gpu and cpu in image processing, 126 Deng, 2009, Imagenet: A large-scale hierarchical image database, 248 Vaswani, 2017, Attention is all you need, 5998 Hollemans, 2018 Zeiler, 2013 Szegedy, 2014 Howard, 2017 Sandler, 2019 Xie, 2020 Sun, 2020 Iandola, 2020 Ritchie, 2020 Hestness, 2017 Kaplan, 2020 Henighan, 2020 NVIDIA, 2015 Ioffe, 2015 Szegedy, 2015 He, 2015 He, 2015 Chollet, 2015 Szegedy, 2016 Huang, 2018 Chollet, 2017 Xie, 2017 Zhang, 2017 Chen, 2017 Zoph, 2018 Hu, 2019 Liu, 2018 Real, 2019 Mahajan, 2018 Ma, 2018 Yalniz, 2019 Yalniz, 2019 Touvron, 2020 Pham, 2021 Zhang, 2020 Dosovitskiy, 2021 Touvron, 2021 Touvron, 2021 Yuan, 2021 Srinivas, 2021 Brock, 2021 Liu, 2021 Chen, 2021 Touvron, 2021 NVIDIA, 2017 Li, 2020 NVIDIA, 2018 Peters, 2018 Radford, 2018 Devlin, 2019 Radford, 2019 Shoeybi, 2020 Lan, 2020 Xu, 2020 Rosset, 2020 Brown, 2020 Cao, 2021 Z. Tang, Y. Wang, Q. Wang, X. Chu, The impact of gpu dvfs on the energy and performance of deep learning: An empirical study, in: Proceedings of the Tenth ACM International Conference on Future Energy Systems, 2019, pp. 315–325. NVIDIA, 2022 Harmon, 2019 Lin, 2021, Low-power ultra-small edge ai accelerators for image recognition with convolution neural networks: Analysis and future directions, Electronics, 10, 2048, 10.3390/electronics10172048 NVIDIA, 2019 NVIDIA, 2019 NVIDIA, 2022 PyTorch, 2022 PyTorch, 2022 Kim, 2021, I-bert: Integer-only bert quantization, 5506