FPGA Implementation of a Pipelined On-Line Backpropagation
Journal of VLSI signal processing systems for signal, image and video technology - Tập 40 - Trang 189-213 - 2005
Tóm tắt
The paper describes the implementation of a systolic array for a multilayer perceptron with a hardware-friendly learning algorithm. A pipelined modification of the on-line backpropagation algorithm is shown and explained. It better exploits the parallelism because both the forward and backward phases can be performed simultaneously. The neural network performance for the proposed modification is discussed and compared with the standard so-called on-line backpropagation algorithm in typical databases and with the various precisions required. Although the preliminary results are positive, subsequent theoretical analysis and further experiments with different training sets will be necessary. For this reason our VLSI systolic architecture—together with the combination of FPGA reconfiguration properties and a design flow based on generic VHDL—can create a reusable, flexible, and fast method of designing a complete ANN on a single FPGA and can permit very fast hardware verifications for our trials of the Pipeline On-line Backpropagation algorithm and the standard algorithms.
Tài liệu tham khảo
S. Hauck, “The Roles of FPGAs in Reprogrammable Systems,” Proceedings of the IEEE, vol. 86, no. 4, 1998, pp. 615–638.
C.E. Cox and W.E. Blanz, “GANGLION—A Fast Field-Programmable Gate Array Implementation of a Connectionist Classifier,” Journal of Solid State Circuits, vol. 27, no. 3, 1992, pp. 288–299.
V. Jean, B. Patrice, R. Didier, S. Mark, T. Hervé, and B. Philippe, “Programmable Active Memories: Reconfigurable Systems Come of Age,” IEEE Transactions on VLSI Systems, vol. 4, no. 1, 1996, pp. 56–69.
P. Lysaght, J. Stockwood, J. Law, and D. Girma, “Artificial Neural Network Implementation on a Fine Grained FPGA,” in Proc. of FPL 94, pp. 421–432.
V. Salapura, M. Gschwind, and O. Maischberger, “A Fast FPGA Implementation of a General Purpose Neuron,” in Proc. of the Fourth International Workshop on Field Programmable Logic and Applications, Sept. 1994.
S.L. Bade and B.L. Hutchings, “FPGA-Based Stochastic Neural Network Implementation,” IEEE Workshop on FPGAs for Custom Computing Machines, 1994, pp. 189–198.
K. Kollmann, K. Riemschneider, and H.C. Zeider, “On-Chip Backpropagation Training Using Parallel Stochastic Bit Streams,” in Proceedings of the IEEE International Conference on Microelectronics for Neural Networks and Fuzzy Systems MicroNeuro’96, 1996, pp. 149–156.
J.G. Elredge and B.L. Hutchings, “RRANN: A hardware Implementation of the Backpropagation Algorithm Using Reconfigurable FPGAs,” IEEE World Conference on Computational Intelligence, June 1994, pp. 77–80.
J.-L. Beuchat, J.-O. Haenni, and E. Sanchez, “Hardware Reconfigurable Neural Networks,” Parallel and Distributed Processing, Lecture Notes in Computer Science, Springer-Verlag, vol. 1388, 1998, pp. 91–98.
N. Izeboudjen, A. Farah, S. Titri, and H. Boumeridja, “Digital Implementation of Artificial Neural Networks: From VHDL Description to FPGA Implementation,” Lecture Notes in Computer Science, vol. 1607, June 1999, pp. 139–148.
S. Titri, H. Boumeridja, D. Lazib, and N. Izeboudjen, “A Reuse Oriented Design Methodology for Artificial Neural Network Implementation,” ASIC/SOC Conference, 1999. Proceedings. 1999, pp. 409–413.
R. Gadea and A. Mochol’, “Forward-Backward Parallelism in On-Line Backpropagation,” Lecture Notes in Computer Science, vol. 1607, 1999, pp. 157–165.
C.R. Rosemberg and G. Belloch, “An Implementation of Network Learning on the Connection Machine,” in Connectionist Models and their Implications, D. Waltz and J Feldman, eds., Ablex, Norwood, NJ, 1988.
A. Petrowski, G. Dreyfus, and C. Girault, “Performance Analysis of a Pipelined Backpropagation Parallel Algorithm,” IEEE Transaction on Neural Networks, vol. 4, no. 6, 1993, pp. 970–981.
A. Singer, “Implementations of Artificial Neural Networks on the Connection Machine,” Parallel Computing, vol. 14, 1990, pp. 305–315.
S. Shams and J.L. Gaudiot, “Implementing Regularly Structured Neural Networks on the Dream Machine,” IEEE Transactions on Neural networks, vol. 6, no. 2, 1995, pp. 408–421.
W.-M. lin, V.K. Prasanna, and K.W. Przytula, “Algorithmic Mapping of Neural Network Models onto Parallel SIMD Machines,” IEEE Transactions on Computers, vol. 40, no. 12, 1991, pp. 1390–1401.
S.R. Jones, K.M. Sammut, and J. Hunter, “Learning in Linear Systolic Neural Network Engines: Analysis and Implementation,” Transactions on Neural Netwoks, vol. 5, no. 4, 1994, pp. 584–593.
D. Naylor, S. Jones, and D. Myers, “Backpropagation in Linear Arrays—A Performance Analysis and Optimization,” IEEE Transactions on Neural Networks, vol. 6, no. 3, 1995, pp. 583–595.
P. Murtagh, A.C. Tsoi, and N. Bergmann, “Bit-Serial Array Implementation af a Multilayer Perceptron,” IEEE Proceedings-E, vol. 140, no. 5, 1993, pp. 277–288.
B. Burton, R.G. Harley, G. Diana, and J.R. Rodgerson, “Reducing the Computational Demands of Continually Online-Trained Artificial Neural Networks for System Identification and Control of Fast Processes,” IEEE Transaction on Industry Applications, vol. 34. no. 3, 1998, pp. 589–596.
D.E. Rumelhart, G.E. Hinton, and R.J. Williams, Learning Internal Representations by Error Backpropagation, Parallel Distributed Processing, vol. 1, MIT Press, Cambridge, MA, 1986, pp. 318–362.
S.E. Falhman, “Faster Learning Variations on Backpropagation: An Empirical Study,” in Proc. 1988 Connectionist Models Summer School, 1988, pp. 38–50.
M.R. Zargham, Computer Architecture: Single and Parallel Systems, Prentice Hall International Inc. 1996.
F. Bayo, Y. Cheneval, A. Guérin-Dugué, R. Chentouf, C. Aviles-Cruz, J. Madrenas, M. Moreno, and J.L. Voz, ESPRIT Basic Research Project Number 6891: ELENA, Project Coordinator C. Jutten, Task B4: Benchmarks, 1995.
J. Qualy and G. Saucier, “Fast Generation of Neuro-ASICs,” in Proc. Int. Neural Networks Conf, vol. 2, 1990, pp. 563–567.
D.J. Myers and R.A. Hutchinson, “Efficient Implementation of Piecewise Linear Activation Function for Digital VLSI Neural Networks,” Elec. Lett., vol. 25, no. 24, 1989, pp. 1662–1663.
H. Hassler and N. Takagi “Function Evaluation by Table Look-up and Addition,” in Procedings of the 12th Symposium on Computer Arithmetic, 1995, pp. 10–16.
D.J. Myers and R.A. Hutchinson, “Efficient Implementation of Piecewise Linear Activation Function for Digital VLSI Neural Networks,” Elec. Lett., vol. 25, no. 24, 1989, pp. 1662–1663.
C. Alippi and G. Storti-Gajani, “Simple Approximation of Sigmoidal Functions: Realistic Design of Digital Neural Networks Capable of Learning,” in Proc. IEEE Int. Symp. Circuits and Syst., 1991, pp. 1505–1508.
P. Murtagh and A.C. Tsoi, “ Implementation Issues of Sigmoid Function and its Derivative for VLSI Digital Neural Networks,” IEE Proceedings-E, vol. 139, no. 3, 1992, pp. 201–214.
H. Hikawa, “Improvement of the Learning Performance of Multiplierless Multiplayer Neural Network,” in IEEE International Synposium on Circuits and Systems, 1997, pp. 641–644.
B. Girau and A. Tisserand, “ MLP Computing and Learning on FPGA Using On-line Arithmetic,” Int. Journal on System Research and Information Science, special issue on Parallel and Distributed Systems for Neural Computing, 2000.
B. Girau, “Building a 2D-Compatible Multilayer Neural Network,” in Proc. IJCNN. IEEE, 2000, pp. 59–64.
S. Marusic and G. Deng, “A Neural Network Based Adaptive Non-Linear Lossless Predictive Coding Technique,” Signal Processing and Its Applications, ISSPA ‘99, 1999.