Modified Kalman filter based method for training state-recurrent multilayer perceptrons
Tóm tắt
Kalman filter based training algorithms for recurrent neural networks provide a clever alternative to the standard backpropagation in time. However, these algorithms do not take into account the optimization of the hidden state variables of the recurrent network. In addition, their formulation requires Jacobian evaluations over the entire network, adding to their computational complexity. We propose a spatial-temporal extended Kalman filter algorithm for training recurrent neural network weights and internal states. This new formulation also reduces the computational complexity of Jacobian evaluations drastically by decoupling the gradients of each layer. Monte Carlo comparisons with backpropagation through time point out the robust and fast convergence of the algorithm.
Từ khóa
#Multilayer perceptrons #Backpropagation algorithms #Recurrent neural networks #Signal processing algorithms #Computational complexity #Jacobian matrices #Kalman filters #Neural networks #Convergence #Neural engineeringTài liệu tham khảo
bishop, 1995, NeuralNetworls for pattern recognition
10.1038/323533a0
julier, 1997, A New Extension of the Kalman Filter to Nonlinear Systems, Proc of AeroSense The 11th Int Symp on Aerospace/Defense Sensing Simulations and Controls
10.1109/72.279181
10.1109/5.58337
10.1109/IJCNN.1991.155276
singhal, 1988, Training Multilayer Perceptrons with the Extended Kalman Algorithm, Advances inNenral Information Processing Systems NIPS'91, 133
miller, 1990, NeuralNetworks for Control
10.1109/72.279191
haykin, 1999, NeuralNetworks AComprehensiveFoundation