Neural Associative Memory with Optimal Bayesian Learning
Tóm tắt
Neural associative memories are perceptron-like single-layer networks with fast synaptic learning typically storing discrete associations between pairs of neural activity patterns. Previous work optimized the memory capacity for various models of synaptic learning: linear Hopfield-type rules, the Willshaw model employing binary synapses, or the BCPNN rule of Lansner and Ekeberg, for example. Here I show that all of these previous models are limit cases of a general optimal model where synaptic learning is determined by probabilistic Bayesian considerations. Asymptotically, for large networks and very sparse neuron activity, the Bayesian model becomes identical to an inhibitory implementation of the Willshaw and BCPNN-type models. For less sparse patterns, the Bayesian model becomes identical to Hopfield-type networks employing the covariance rule. For intermediate sparseness or finite networks, the optimal Bayesian learning rule differs from the previous models and can significantly improve memory performance. I also provide a unified analytical framework to determine memory capacity at a given output noise level that links approaches based on mutual information, Hamming distance, and signal-to-noise ratio.
Từ khóa
Tài liệu tham khảo
Abramowitz M., 1972, Handbook of mathematical functions with formulas, graphs, and mathematical tables.
Borwein J., 2003, Mathematics by experiment: Plausible reasoning in the 21st century.
Hebb D., 1949, The organization of behavior: A neuropsychological theory.
Hertz J., 1991, Introduction to the theory of neural computation.
Johansson C., 2002, Proceedings of the International Conference on Artificial Neural Networks (ICANN), 192
Kanerva P., 1988, Sparse distributed memory.
Knoblauch, A. (2007). On the computational benefits of inhibitory neural associative networks (HRI-EU Rep. 07-05). Offenbach/Main, Germany: Honda Research Institute Europe.
Knoblauch A., 2009, Neural associative networks with optimal Bayesian learning
Knoblauch A., 2009, Zip nets: Neural associative networks with non-linear learning
Knoblauch A., 2010, Comparison of the Lansner/Ekeberg rule to optimal Bayesian learning in neural associative memory
Kononenko I., 1994, Informatica (Slovenia), 18, 183
Lansner A., 1987, Proceedings of the IEEE First International Conference on Neural Networks, II
Palm G., 1988, Neural computers, 271
Palm G., 1990, Concepts in Neuroscience, 1, 133
Palm G., 1991, Concepts in Neuroscience, 2, 97
Papoulis A., 1991, Probability, random variables, and stochastic processes, 3
Shannon C., 1949, The mathematical theory of communication.
Zhang H., 2004, Proceedings of the 17th Florida Artificial Intelligence Research Society Conference, 562