Resource efficient activation functions for neural network accelerators

Amin, 1997, Piecewise linear approximation applied to nonlinear function of a neural network, IEE Proc.-Circuits Devices Syst., 144, 313, 10.1049/ip-cds:19971587 D. Aymeric, Tensorflow examples (2017). URL: https://bit.ly/327SNUN (accessed 3 June 2019). Azari, 2019, An energy-efficient reconfigurable lstm accelerator for natural language processing, 4450 Basterretxea, 2007, An experimental study on nonlinear function computation for neural/fuzzy hardware design, IEEE Trans. Neural Networks, 18, 266, 10.1109/TNN.2006.884680 J. Brownlee, Sequence classification with LSTM recurrent neural networks in python with keras (2019). URL: https://bit.ly/3iNfhB5 (accessed 3 March 2019). Cao, 2019, Efficient and effective sparse lstm on FPGA with bank-balanced sparsity, 63 A.X.M. Chang, B. Martini, E. Culurciello, Recurrent neural networks hardware implementation on FPGA (2015). arXiv preprint arXiv:1511.05552. Chen, 2014, Diannao: A small-footprint high-throughput accelerator for ubiquitous machine-learning, ACM SIGARCH Computer Architecture News, 42, 269, 10.1145/2654822.2541967 D.-A. Clevert, T. Unterthiner, S. Hochreiter, Fast and accurate deep network learning by exponential linear units (ELUs), ICLR, 2015. Courbariaux, 2014, Training deep neural networks with low precision multiplications, International Conference on Learning Representation Deng, 2018, GXNOR-Net: Training deep neural networks with ternary weights and activations without full-precision memory under a unified discretization framework, Neural Networks, 100, 49, 10.1016/j.neunet.2018.01.010 Desjardins, 2015, Natural neural networks, Advances in Neural Information Processing Systems, 2071 Elfwing, 2018, Sigmoid-weighted linear units for neural network function approximation in reinforcement learning, Neural Networks, 107, 3, 10.1016/j.neunet.2017.12.012 Fan, 2013, Efficient VLSI architecture for training radial basis function networks, Sensors, 13, 3848, 10.3390/s130303848 Fayek, 2017, Matdl: A lightweight deep learning library in matlab, J. Open Source Software, 2, 413, 10.21105/joss.00413 Fernando, 2018, Soft+ hardwired attention: An lstm framework for human trajectory prediction and abnormal event detection, Neural Networks, 108, 466, 10.1016/j.neunet.2018.09.002 Ferreira, 2016, An FPGA implementation of a long short-term memory neural network, 1 Gomar, 2016, Precise digital implementations of hyperbolic tanh and sigmoid function, 1586 Guan, 2017, FPGA-based accelerator for long short-term memory recurrent neural networks, 629 Hajduk, 2017, High accuracy FPGA activation function implementation for neural networks, Neurocomputing, 247, 59, 10.1016/j.neucom.2017.03.044 Han, 2017, Ese: Efficient speech recognition engine with sparse lstm on FPGA, 75 K. Harrison, RNN w/ LSTM cell example in tensorflow and python (2016). URL: https://bit.ly/31Z4fC1 (accessed 3 June 2019). He, 2015, Delving deep into rectifiers: Surpassing human-level performance on imagenet classification, 1026 He, 2016, Deep residual learning for image recognition, 770 D. Hendrycks, K. Gimpel, Gaussian error linear units (gelus) (2016). arXiv preprint arXiv:1606.08415. Hochreiter, 1997, Long short-term memory, Neural Comput., 9, 1735, 10.1162/neco.1997.9.8.1735 K. Hornik, M. Stinchcombe, H. White, et al., Multilayer feedforward networks are universal approximators, Neural Networks 2(5) (1989) 359–366. Hubara, 2016, Binarized neural networks, Advances in Neural Information Processing Systems, 4107 Intel, Cyclone V device overview (2018). URL: https://intel.ly/2UbUwVY. Kaggle. US baby names: Explore naming trends from babies born in the US (2016). Karevan, 2020, Transductive LSTM for time-series prediction: An application to weather forecasting, Neural Networks, 125, 1, 10.1016/j.neunet.2019.12.030 Klambauer, 2017, Self-normalizing neural networks, Advances in Neural Information Processing Systems, 971 Kouretas, 2018, Hardware aspects of long short term memory, 525 Larkin, 2006, An efficient hardware architecture for a neural network activation function generator, 1319 Lee, 2016, FPGA-based low-power speech recognition with recurrent neural networks, 230 M. Leshno, V.Y. Lin, A. Pinkus, S. Schocken, Multilayer feedforward networks with a nonpolynomial activation function can approximate any function, Neural Networks 6(6) (1993) 861–867. Liao, 2020, An intelligent low-power low-cost mobile lab-on-chip yeast cell culture platform, IEEE Access, 8, 70733, 10.1109/ACCESS.2020.2987206 M. Lin, Q. Chen, S. Yan, Network in network (2013). arXiv preprint arXiv:1312.4400. Maas, 2013, Rectifier nonlinearities improve neural network acoustic models, Proc. ICML, 30, 1 M. Marcus, B. Santorini, M.A. Marcinkiewicz, Building a large annotated corpus of english: The penn treebank (1993). Merity, 2017, Pointer sentinel mixture models, International Conference on Learning Representation Moss, 2018, A two-speed, radix-4, serial–parallel multiplier, IEEE Trans. Very Large Scale Integr. VLSI Syst., 27, 769, 10.1109/TVLSI.2018.2883645 Pushpa, 2014, Implementation of hyperbolic tangent activation function in vlsi, Int. J. Adv. Res. Comput. Sci. Technol., 2, 225 Ramachandran, 2017, Searching for activation functions, ICLR Workshop Track G. Roopal, LSTM tensorflow (2017). URL: https://bit.ly/3hhNc4n (accessed 3 June 2019). Rybalkin, 2018, Finn-l: Library extensions and design trade-off analysis for variable precision lstm networks on FPGAs, 89 Saichand, 2008, FPGA realization of activation function for artificial neural networks, vol. 3, 159 Scardapane, 2019, Kafnets: Kernel-based non-parametric activation functions for neural networks, Neural Networks, 110, 19, 10.1016/j.neunet.2018.11.002 Sidek, 2020, Booth algorithm with implementation of UART module using FPGA, Int. J. Integr. Eng., 12, 151 Siegel, 2020, Approximation rates for neural networks with general activation functions, Neural Networks, 10.1016/j.neunet.2020.05.019 Sim, 2017, A new stochastic computing multiplier with application to deep convolutional neural networks, 1 Sim, 2019, Cost-effective stochastic MAC circuits for deep neural networks, Neural Networks, 117, 152, 10.1016/j.neunet.2019.04.017 Simonyan, 2015, Very deep convolutional networks for large-scale image recognition, Internation Conference on Learning Sze, 2017, Efficient processing of deep neural networks: A tutorial and survey, Proc. IEEE, 105, 2295, 10.1109/JPROC.2017.2761740 Tan, 2007 Terada, 2020, Fast generalization error bound of deep learning without scale invariance of activation functions, Neural Networks, 10.1016/j.neunet.2020.05.033 Tiwari, 2015, Hardware implementation of neural network with sigmoidal activation functions using cordic, Microprocess. Microsyst., 39, 373, 10.1016/j.micpro.2015.05.012 Tommiska, 2003, Efficient digital implementation of the sigmoid function for reprogrammable logic, IEE Proc.-Comput. Digital Tech., 150, 403, 10.1049/ip-cdt:20030965 Tsmots, 2019, Hardware implementation of sigmoid activation functions using fpga, 34 N. Virdee, Lstm neural network from scratch (2018). Wang, 2019, Deep neural network approximation for custom hardware: Where we’ve been, where we’re going, ACM Computing Surveys (CSUR), 52, 1, 10.1145/3214306 Wang, 2018, C-lstm: Enabling efficient LSTM using structured compression techniques on FPGAs, 11 L. Wei, Convolutional neural networks for cifar-10 (2018). URL: https://github.com/BIGBALLON/cifar-10-cnn (accessed 25 May 2019). Wu, 2020, Accuracy tolerant neural networks under aggressive power optimization, 774 Wuraola, 2021, Efficient activation functions for embedded inference engines, Neurocomputing, 442, 73, 10.1016/j.neucom.2021.02.030 H. Xiao, K. Rasul, R. Vollgraf, Fashion-MNIST: a novel image dataset for benchmarking machine learning algorithms (2017). arXiv preprint arXiv:1708.07747. Yang, 2018, Design space exploration of neural network activation function circuits, IEEE Trans. Comput. Aided Des. Integr. Circuits Syst., 38, 1974, 10.1109/TCAD.2018.2871198 W. Zaremba, I. Sutskever, O. Vinyals, Recurrent neural network regularization (2014). arXiv preprint arXiv:1409.2329.