Neural Associative Memory with Optimal Bayesian Learning

Neural Computation - Tập 23 Số 6 - Trang 1393-1451 - 2011
Andreas Knoblauch1
1Honda Research Institute Europe GmbH, D-63073 Offenbach, Germany

Tóm tắt

Neural associative memories are perceptron-like single-layer networks with fast synaptic learning typically storing discrete associations between pairs of neural activity patterns. Previous work optimized the memory capacity for various models of synaptic learning: linear Hopfield-type rules, the Willshaw model employing binary synapses, or the BCPNN rule of Lansner and Ekeberg, for example. Here I show that all of these previous models are limit cases of a general optimal model where synaptic learning is determined by probabilistic Bayesian considerations. Asymptotically, for large networks and very sparse neuron activity, the Bayesian model becomes identical to an inhibitory implementation of the Willshaw and BCPNN-type models. For less sparse patterns, the Bayesian model becomes identical to Hopfield-type networks employing the covariance rule. For intermediate sparseness or finite networks, the optimal Bayesian learning rule differs from the previous models and can significantly improve memory performance. I also provide a unified analytical framework to determine memory capacity at a given output noise level that links approaches based on mutual information, Hamming distance, and signal-to-noise ratio.

Từ khóa


Tài liệu tham khảo

Abramowitz M., 1972, Handbook of mathematical functions with formulas, graphs, and mathematical tables.

10.1016/0025-5564(71)90051-4

10.1007/BF00365229

10.1016/0893-6080(89)90043-9

10.1007/BF00288902

10.1037/0033-295X.84.5.413

10.1016/0893-6080(89)90038-5

10.1023/A:1008925909305

Borwein J., 2003, Mathematics by experiment: Plausible reasoning in the 21st century.

10.1007/978-3-642-93083-6_9

10.1088/0954-898X_3_4_005

10.1088/0954-898X_4_4_003

10.1162/089976601300014367

10.1002/0471200611

10.1162/neco.1993.5.2.205

10.1007/BF00206223

10.1023/A:1007413511361

10.1088/0954-898X/9/2/006

10.1016/j.neuron.2005.02.001

10.1088/0305-4470/21/1/030

10.1088/0305-4470/21/1/031

10.1098/rspb.1976.0084

10.1103/PhysRevA.41.1843

10.1007/BF00202789

10.1109/SFCS.1994.365720

Hebb D., 1949, The organization of behavior: A neuropsychological theory.

10.1209/0295-5075/11/5/003

Hertz J., 1991, Introduction to the theory of neural computation.

10.1073/pnas.79.8.2554

10.1007/s00521-004-0464-6

10.1162/neco.2007.19.7.1871

Johansson C., 2002, Proceedings of the International Conference on Artificial Neural Networks (ICANN), 192

Kanerva P., 1988, Sparse distributed memory.

10.1007/3-540-44989-2_39

10.1016/j.ipl.2005.05.021

Knoblauch, A. (2007). On the computational benefits of inhibitory neural associative networks (HRI-EU Rep. 07-05). Offenbach/Main, Germany: Honda Research Institute Europe.

10.1137/070700012

Knoblauch A., 2009, Neural associative networks with optimal Bayesian learning

10.1142/9789812834232_0007

Knoblauch A., 2009, Zip nets: Neural associative networks with non-linear learning

Knoblauch A., 2010, Comparison of the Lansner/Ekeberg rule to optimal Bayesian learning in neural associative memory

10.1109/IJCNN.2010.5596589

10.1016/S0893-6080(01)00084-3

10.1162/neco.2009.08-07-588

10.1109/TC.1972.5008975

10.1007/978-3-642-96384-1

10.1007/BF01259390

10.1007/BF00200801

10.1007/BFb0017015

Kononenko I., 1994, Informatica (Slovenia), 18, 183

10.1016/j.tins.2008.12.002

Lansner A., 1987, Proceedings of the IEEE First International Conference on Neural Networks, II

10.1142/S0129065789000499

10.1142/S0129065796000816

10.1038/nrn964

10.1016/S0960-9822(03)00135-0

10.1016/0025-5564(74)90031-5

10.1007/978-94-011-3460-6_22

10.1038/nrn1519

10.1113/jphysiol.1969.sp008820

10.1098/rstb.1971.0078

10.1007/s11063-006-9001-1

10.1088/0305-4470/24/5/023

10.1007/BF00337019

10.1007/978-3-642-81792-2

10.1007/978-3-642-74119-7_7

Palm G., 1988, Neural computers, 271

Palm G., 1990, Concepts in Neuroscience, 1, 133

Palm G., 1991, Concepts in Neuroscience, 2, 97

10.1088/0954-898X_3_2_006

10.1007/978-1-4612-0723-8_3

Papoulis A., 1991, Probability, random variables, and stochastic processes, 3

10.1016/S0896-6273(01)00252-5

10.1016/0885-2308(89)90015-6

10.1017/CBO9780511615528

10.1016/j.neucom.2005.12.080

10.1002/(SICI)1098-1063(1996)6:6<601::AID-HIPO5>3.0.CO;2-J

10.1016/S0925-2312(00)00270-8

10.1016/0022-5193(77)90146-1

10.1007/BF00275079

Shannon C., 1949, The mathematical theory of communication.

10.1109/72.701183

10.1016/S0893-6080(98)00125-7

10.1007/BF00293853

10.1016/S0896-6273(02)00652-9

10.1162/neco.2007.08-06-301

10.1209/0295-5075/6/2/002

10.1038/36103

10.1073/pnas.0307711101

10.1523/JNEUROSCI.2101-06.2006

10.1016/j.neucom.2005.04.008

10.1038/222960a0

10.1162/neco.1990.2.1.85

10.1093/acprof:oso/9780195159561.003.0009

Zhang H., 2004, Proceedings of the 17th Florida Artificial Intelligence Research Society Conference, 562

Thông tin
Thông tin xuất bản

Neural Computation

Tập 23 Số 6

1393-1451

Thông tin tác giả