Inhomogeneous Sparseness Leads to Dynamic Instability During Sequence Memory Recall in a Recurrent Neural Network Model
Tóm tắt
Theoretical models of associative memory generally assume most of their parameters to be homogeneous across the network. Conversely, biological neural networks exhibit high variability of structural as well as activity parameters. In this paper, we extend the classical clipped learning rule by Willshaw to networks with inhomogeneous sparseness, i.e., the number of active neurons may vary across memory items. We evaluate this learning rule for sequence memory networks with instantaneous feedback inhibition and show that little surprisingly, memory capacity degrades with increased variability in sparseness. The loss of capacity, however, is very small for short sequences of less than about 10 associations. Most interestingly, we further show that, due to feedback inhibition, too large patterns are much less detrimental for memory capacity than too small patterns.
Tài liệu tham khảo
Little WA: The existence of persistent states in the brain. Math Biosci 1974, 19: 101–120. 10.1016/0025-5564(74)90031-5
Hopfield JJ: Neural networks and physical systems with emergent collective computational abilities. Proc Natl Acad Sci USA 1982, 79(8):2554–2558. 10.1073/pnas.79.8.2554
Wennekers T, Palm G: Modelling generic cognitive functions with operational Hebbian cell assemblies. In Neural Network Research Horizons. Edited by: Weiss M. Nova Science Publishers, New York; 2007:225–294.
Lee AK, Wilson MA: Memory of sequential experience in the hippocampus during slow wave sleep. Neuron 2002, 36(6):1183–1194. 10.1016/S0896-6273(02)01096-6
Diba K, Buzsaki G: Forward and reverse hippocampal place-cell sequences during ripples. Nat Neurosci 2007, 10(10):1241–1242. 10.1038/nn1961
Maier N, Tejero-Cantero Á, Dorrn AL, Winterer J, Beed PS, Morris G, Kempter R, Poulet JF, Leibold C, Schmitz D: Coherent phasic excitation during hippocampal ripples. Neuron 2011, 72: 137–152. 10.1016/j.neuron.2011.08.016
Kammerer A, Tejero-Cantero Á, Leibold C: Inhibition enhances memory capacity: optimal feedback, transient replay and oscillations. J Comput Neurosci 2013, 34: 125–136. 10.1007/s10827-012-0410-z
Nadal JP: Associative memory: on the (puzzling) sparse coding limit. J Phys A 1991, 24: 1093–1101. 10.1088/0305-4470/24/5/023
Gibson WG, Robinson J: Statistical analysis of the dynamics of a sparse associative memory. Neural Netw 1992, 5: 645–661. 10.1016/S0893-6080(05)80042-5
Willshaw DJ, Buneman OP, Longuet-Higgins HC: Non-holographic associative memory. Nature 1969, 222(5197):960–962. 10.1038/222960a0
Leibold C, Kempter R: Memory capacity for sequences in a recurrent network with biological constraints. Neural Comput 2006, 18(4):904–941. 10.1162/neco.2006.18.4.904
Hirase H, Recce M: A search for the optimal thresholding sequence in an associative memory. Network 1996, 4: 741–756.
Kapfer C, Glickfeld L, Atallah B, Scanziani M: Supralinear increase of recurrent inhibition during sparse activity in the somatosensory cortex. Nat Neurosci 2007, 10: 743–753. 10.1038/nn1909
Silberberg G, Markram H: Disynaptic inhibition between neocortical pyramidal cells mediated by Martinotti cells. Neuron 2007, 53: 735–746. 10.1016/j.neuron.2007.02.012
Amit Y, Huang Y: Precise capacity analysis in binary networks with multiple coding level inputs. Neural Comput 2010, 22(3):660–688. 10.1162/neco.2009.02-09-967
Huang Y, Amit Y: Capacity analysis in multi-state synaptic models: a retrieval probability perspective. J Comput Neurosci 2011, 30(3):699–720. 10.1007/s10827-010-0287-7
Amit DJ, Fusi S: Learning in neural networks with material synapses. Neural Comput 1994, 6: 957–982. 10.1162/neco.1994.6.5.957
Fusi S, Drew PJ, Abbott LF: Cascade models of synaptically stored memories. Neuron 2005, 45(4):599–611. 10.1016/j.neuron.2005.02.001
Leibold C, Kempter R: Sparseness constrains the prolongation of memory lifetime via synaptic metaplasticity. Cereb Cortex 2008, 18: 67–77. 10.1093/cercor/bhm037
Barrett AB, van Rossum MC: Optimal learning rules for discrete synapses. PLoS Comput Biol 2008., 4(11): Article ID e1000230 Article ID e1000230
Päpper M, Kempter R, Leibold C: Synaptic tagging, evaluation of memories, and the distal reward problem. Learn Mem 2011, 18: 58–70.
van Rossum MC, Shippi M, Barrett AB: Soft-bound synaptic plasticity increases storage capacity. PLoS Comput Biol 2012., 8(12): Article ID e1002836 Article ID e1002836
Milekic MH, Alberini CM: Temporally graded requirement for protein synthesis following memory reactivation. Neuron 2002, 36(3):521–525. 10.1016/S0896-6273(02)00976-5