Computational models of reinforcement learning: the role of dopamine as a reward signal

Abercrombie ED, Keefe KA et al (1989) Differential effect of stress on in vivo dopamine release in striatum, nucleus accumbens, and medial frontal cortex. J Neurochem 52:1655–1658

Arbuthnott GW, Wickens J (2007) Space, time and dopamine. Trends Neurosci 30:62–69

Baras D, Meir R (2007) Reinforcement learning, spike-time-dependent plasticity, and the bcm rule. Neural Comput 19:2245–2279

Barto AG (1995) Adaptive critics and the basal ganglia. Models of information processing in the basal ganglia. 215–232

Barto AG, Sutton RS, Anderson C (1983) Neuron-like adaptive elements that can solve difficult learning control problems, IEEE transactions on systems, man, and cybernetics. SMC 13:834–846

Bauer EP, LeDoux JE (2004) Heterosynaptic long-term potentiation of inhibitory interneurons in the lateral amygdala. J Neurosci 24:9507–9512

Bauer EP, Schafe GE, LeDoux JE (2002) NMDA receptors and L-type voltage-gated calcium channels contribute to long-term potentiation and different components of fear memory formation in the lateral amygdala. J Neurosci 22:5239–5249

Bayer HM, Glimcher PW (2005) Midbrain dopamine neurons encode a quantitative reward prediction error signal. Neuron 47:129–141

Bayer HM, Lau B, Glimcher PW (2007) Statistics of midbrain dopamine neuron spike trains in the awake primate. J Neurophysiol 98:1428–1439

Bergstrom BP, Garris PA (2003) Passive stabilization of striatal extracellular dopamine across the lesion spectrum encompassing the presymptomatic phase of Parkinson’s disease: a voltammetric study in the 6-OHDA-lesioned rat. J Neurochem 87:1224–1236

Berke JD, Hyman SE (2000) Addiction, dopamine, and the molecular mechanisms of memory. Neuron 25:515–532

Berridge KC (2007) The debate over dopamine’s role in reward: the case for incentive salience. Psychopharmacology (Berl) 191:391–431

Berridge KC, Robinson TE (1998) What is the role of dopamine in reward: hedonic impact, reward learning, or incentive salience? Brain Res Brain Res Rev 28:309–369

Bertin M, Schweighofer N, Doya K (2007) Multiple model-based reinforcement learning explains dopamine neuronal activity. Neural Netw 20:668–675

Bi G, Poo M (1998) Synaptic modifications in cultured hippocampal neurons: dependence on spike timing, synaptic strength, and postsynaptic cell type. J Neurosci 18:10464–10472

Bissière S, Humeau Y, Luthi A (2003) Dopamine gates ltp induction in lateral amygdala by suppressing feedforward inhibition. Nature Neurosci 6:587–592

Bouton ME (2004) Context and behavioral processes in extinction. Learn Mem 11:485–494

Calabresi P, Picconi B, Tozzi A, Di Filippo M (2007) Dopamine-mediated regulation of corticostriatal synaptic plasticity. Trends Neurosci 30:211–219

Camerer CF (2008) Neuroeconomics: opening the gray box. Neuron 60:416–419

Canavan AG, Passingham RE et al (1989) The performance on learning tasks of patients in the early stages of Parkinson’s disease. Neuropsychologia 27:141–156

Cardinal RN, Parkinson JA et al (2002) Emotion and motivation: the role of the amygdala, ventral striatum, and prefrontal cortex. Neurosci Biobehav Rev 26:321–352

Centonze D, Grande C et al (2003) Distinct roles of D1 and D5 dopamine receptors in motor activity and striatal synaptic plasticity. J Neurosci 23:8506–8512

Cepeda C, Levine MS (1998) Dopamine and N-methyl-d-aspartate receptor interactions in the neostriatum. Dev Neurosci 20:1–18

Chuhma N, Choi WY et al (2009) Dopamine neuron glutamate cotransmission: frequency-dependent modulation in the mesoventromedial projection. Neuroscience 164:1068–1083

Cohen MX, Frank MJ (2009) Neurocomputational models of basal ganglia function in learning, memory and choice. Behav Brain Res 199:141–156

Dalley JW, Laane K et al (2005) Time-limited modulation of appetitive Pavlovian memory by D1 and NMDA receptors in the nucleus accumbens. Proc Natl Acad Sci USA 102:6189–6194

Daniel JA, Galbraith S et al (2009) Functional heterogeneity at dopamine release sites. J Neurosci 29:14670–14680

Daw ND, Niv Y et al (2005) Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nat Neurosci 8:1704–1711

Daw ND, Courville AC, Touretzky DS (2006) Representation and timing in theories of the dopamine system. Neural Comput 18:1637–1677

Dayan P, Abbott LF (2001) Theoretical neuroscience. Computational and mathematical modeling of neural systems. The MIT Press, Cambridge

Dayan P, Balleine BW (2002) Reward, motivation, and reinforcement learning. Neuron 36:285–298

Dayan P, Daw ND (2008) Decision theory, reinforcement learning, and the brain. Cogn Affect Behav Neurosci 8:429–453

Di Chiara G (2002) Nucleus accumbens shell and core dopamine: differential role in behavior and addiction. Behav Brain Res 137:75–114

Doya K, Samejima K et al (2002) Multiple model-based reinforcement learning. Neural Comput 14:1347–1369

Ehrlich I, Humeau Y et al (2009) Amygdala inhibitory circuits and the control of fear memory. Neuron 62:757–771

Everitt BJ, Parkinson JA et al (1999) Associative processes in addiction and reward. The role of amygdala-ventral striatal subsystems. Ann N Y Acad Sci 877:412–438

Everitt BJ, Robbins TW (2005) Neural systems of reinforcement for drug addiction: from actions to habits to compulsion. Nat Neurosci 8:1481–1489

Fellous J-M, Suri RE (2003) The roles of dopamine. The handbook of brain theory and neural networks. MIT Press, Cambridge, pp 361–365

Fenu S, Di Chiara G (2003) Facilitation of conditioned taste aversion learning by systemic amphetamine: role of nucleus accumbens shell dopamine D1 receptors. Eur J Neurosci 18:2025–2030

Fibiger HC, Phillips AG (1986) Reward, motivation, cognition, psychobiology of mesotelencephalic dopamine systems. Handbook of physiology—The nervous system IV. F E Bloom Baltimore, Williams and Wilkins, 647–675

Fields HL, Hjelmstad GO et al (2007) Ventral tegmental area neurons in learned appetitive behavior and positive reinforcement. Annu Rev Neurosci 30:289–316

Floresco SB, West AR et al (2003) Afferent modulation of dopamine neuron firing differentially regulates tonic and phasic dopamine transmission. Nature Neurosci 6:968–973

Florian RV (2007) Reinforcement learning through modulation of spike-timing-dependent synaptic plasticity. Neural Comput 19:1468–1502

Fourcaudot E, Gambino F et al (2009) L-type voltage-dependent Ca(2+) channels mediate expression of presynaptic LTP in amygdala. Nat Neurosci 12:1093–1095

Frank MJ (2005) Dynamic dopamine modulation in the basal ganglia: a neurocomputational account of cognitive deficits in medicated and nonmedicated Parkinsonism. J Cogn Neurosci 17:51–72

Frank MJ, Seeberger LC, O’reilly RC (2004) By carrot or by stick: cognitive reinforcement learning in Parkinsonism. Science 306:1940–1943

Frank MJ, Moustafa AA et al (2007) Genetic triple dissociation reveals multiple roles for dopamine in reinforcement learning. Proc Natl Acad Sci USA 104:16311–16316

Frank MJ, Doll BB et al (2009) Prefrontal and striatal dopaminergic genes predict individual differences in exploration and exploitation. Nature Neurosci 12:1062–1068

Freeman WJ (2007) Definitions of state variables and state space for brain-computer interface: part 1. Multiple hierarchical levels of brain function. Cogn Neurodyn 1:3–14

Frey U, Hartmann S et al (1989) Domperidone, an inhibitor of the D2-receptor, blocks a late phase of an electrically induced long-term potentiation in the CA1-region in rats. Biomed Biochim Acta 48:473–476

Frey U, Schroeder H et al (1990) Dopaminergic antagonists prevent long-term maintenance of posttetanic LTP in the CA1 region of rat hippocampal slices. Brain Res 522:69–75

Frey U, Matthies H et al (1991) The effect of dopaminergic D1 receptor blockade during tetanization on the expression of long-term potentiation in the rat CA1 region in vitro. Neurosci Lett 129:111–114

Furuyashiki T, Gallagher M (2007) Neural encoding in the orbitofrontal cortex related to goal-directed behavior. Ann N Y Acad Sci 1121:193–215

Furuyashiki T, Holland PC et al (2008) Rat orbitofrontal cortex separately encodes response and outcome information during performance of goal-directed behavior. J Neurosci 28:5127–5138

Gall CM, Hendry SH et al (1987) Evidence for coexistence of GABA and dopamine in neurons of the rat olfactory bulb. J Comp Neurol 266:307–318

Gallagher M, McMahan RW et al (1999) Orbitofrontal cortex and representation of incentive value in associative learning. J Neurosci 19:6610–6614

Glimcher PW, Rustichini A (2004) Neuroeconomics: the consilience of brain and decision. Science 306:447–452

Goto Y, Grace AA (2005) Dopaminergic modulation of limbic and cortical drive of nucleus accumbens in goal-directed behavior. Nat Neurosci 8:805–812

Grace AA (1991) Phasic versus tonic dopamine release and the modulation of dopamine system responsivity: a hypothesis for the etiology of schizophrenia. Neuroscience 41:1–24

Gurden H, Tassin JP et al (1999) Integrity of the mesocortical dopaminergic system is necessary for complete expression of in vivo hippocampal–prefrontal cortex long-term potentiation. Neuroscience 94:1019–1027

Hatfield T, Han JS et al (1996) Neurotoxic lesions of basolateral, but not central, amygdala interfere with pavlovian second-order conditioning and reinforcer devaluation effects. J Neurosci 16:5256–5265

Hernandez PJ, Andrzejewski ME et al (2005) AMPA/kainate, NMDA, and dopamine D1 receptor function in the nucleus accumbens core: a context-limited role in the encoding and consolidation of instrumental memory. Learn Mem 12:285–295

Hikosaka O, Bromberg-Martin E et al (2008) New insights on the subcortical representation of reward. Curr Opin Neurobiol 18:203–208

Hiraoka K, Yoshida M, Mishima T (2009) Parallel reinforcement learning for weighted multi-criteria model with adaptive margin. Cogn Neurodyn 3:17–24

Holland PC, Gallagher M (1999) Amygdala circuitry in attentional and representational processes. Trends Cogn Sci 3:65–73

Huang Y-Y, Kandel ER (1998) Postsynaptic induction and PKA-dependent expression of LTP in the lateral amygdala. Neuron 21:169–178

Huang Y-Y, Kandel ER (2007) Low-frequency stimulation induces a pathway-specific late phase of LTP in the amygdala that is mediated by PKA and dependent on protein synthesis. Learn Mem 14:497–503

Huang CC, Lin HJ et al (2007) Repeated cocaine administration promotes long-term potentiation induction in rat medial prefrontal cortex. Cereb Cortex 17:1877–1888

Humeau Y, Shaban H et al (2003) Presynaptic induction of heterosynaptic associative plasticity in the mammalian brain. Nature 426:841–845

Ikemoto S, Panksepp J (1999) The role of nucleus accumbens dopamine in motivated behavior: a unifying interpretation with special reference to reward-seeking. Brain Res Brain Res Rev 31:6–41

Izhikevich EM (2007) Solving the distal reward problem through linkage of stdp and dopamine signaling. Cereb Cortex 17:2443–2452

Joel D, Niv Y et al (2002) Actor-critic models of the basal ganglia: new anatomical and computational perspectives. Neural Netw 15:535–547

Kaelbling LP, Littman ML et al (1996) Reinforcement learning: a survey. JAIR 4:237–285

Kelley AE, Berridge KC (2002) The neuroscience of natural rewards: relevance to addictive drugs. J Neurosci 22:3306–3311

Kerr JN, Wickens JR (2001) Dopamine D-1/D-5 receptor activation is required for long-term potentiation in the rat neostriatum in vitro. J Neurophysiol 85:117–124

Kilpatrick MR, Rooney MB et al (2000) Extracellular dopamine dynamics in rat caudate-putamen during experimenter-delivered and intracranial self-stimulation. Neuroscience 96:697–706

Knowlton BJ, Mangels JA et al (1996) A neostriatal habit learning system in humans. Science 273:1399–1402

Kobayashi S, Schultz W (2008) Influence of reward delays on responses of dopamine neurons. J Neurosci 28:7837–7846

Kolomiets B, Marzo A et al (2009) Background dopamine concentration dependently facilitates long-term potentiation in rat prefrontal cortex through postsynaptic activation of extracellular signal-regulated kinases. Cereb Cortex 19:2708–2718

Kombian SB, Malenka RC (1994) Simultaneous LTP of non-NMDA- and LTD of NMDA-receptor-mediated responses in the nucleus accumbens. Nature 368:242–246

Konda VR, Borkar VS (1999) Actor-critic—type learning algorithms for markov decision processes. SIAM J Control Optim 38:94–123

Koob GF (1992) Drugs of abuse: anatomy, pharmacology and function of reward pathways. Trends Pharmacol Sci 13:177–184

Kötter R, Wickens J (1995) Interactions of glutamate and dopamine in a computational model of the striatum. J Comput Neurosci 2:195–214

Kroener S, Chandler LJ et al (2009) Dopamine modulates persistent synaptic activity and enhances the signal-to-noise ratio in the prefrontal cortex. PLoS One 4:e6507

Lapish CC, Kroener S et al (2007) The ability of the mesocortical dopamine system to operate distinct temporal modes. Psychopharmacology (Berl) 191:609–626

Legenstein R, Pecevski D et al (2008) A learning theory for reward-modulated spike-timing-dependent plasticity with application to biofeedback. PLoS Comput Biol 4:e1000180

Li Y, Kauer JA (2004) Repeated exposure to amphetamine disrupts dopaminergic modulation of excitatory synaptic plasticity and neurotransmission in nucleus accumbens. Synapse 51:1–10

Lindvall O, Bjorklund A (1978) Anatomy of the dopaminergic neuron systems in the rat brain. Adv Biochem Psychopharmacol 19:1–23

Ljungberg T, Apicella P et al (1992) Responses of monkey dopamine neurons during learning of behavioral reactions. J Neurophys 67:145–163

Louilot A, Le Moal M et al (1986) Differential reactivity of dopaminergic neurons in the nucleus accumbens in response to different behavioral situations. An in vivo voltammetric study in free moving rats. Brain Res 397(2):395–400

Mackintosh NJ (1983) Conditioning and associative learning. Oxford University Press, New York

Maher BJ, Westbrook GL (2008) Co-transmission of dopamine and GABA in periglomerular cells. J Neurophysiol 99:1559–1564

Markram H, Lübke J et al (1997) Regulation of synaptic effocacy by coincidence of postsynaptic aps and epsps. Science 275:213–215

Matsumoto M, Hikosaka O (2009) Two types of dopamine neuron distinctly convey positive and negative motivational signals. Nature 459:837–841

McGaugh JL (2002) Memory consolidation and the amygdala: a systems perspective. Trends Neurosci 25:456

Mirenowicz J, Schultz W (1994) Importance of unpredictability for reward responses in primate dopamine neurons. J Neurophys 72:1024–1027

Mirenowicz J, Schultz W (1996) Preferential activation of midbrain dopamine neurons by appetitive rather than aversive stimuli. Nature 379:449–451

Montague PR, Dayan P et al (1996) A framework for mesencephalic dopamine systems based on predictive hebbian learning. J Neurosci 16:1936–1947

Montague PR, Hyman SE et al (2004a) Computational roles for dopamine in behavioural control. Nature 431:760–767

Montague PR, McClure SM et al (2004b) Dynamic gain control of dopamine delivery in freely moving animals. J Neurosci 24:1754–1759

Morris G, Arkadir D et al (2004) Coincident but distinct messages of midbrain dopamine and striatal tonically active neurons. Neuron 43:133–143

Morris G, Nevet A et al (2006) Midbrain dopamine neurons encode decisions for future action. Nature Neurosci 9:1057–1063

Moustafa AA, Cohen MX et al (2008) A role for dopamine in temporal decision making and reward maximization in parkinsonism. J Neurosci 28:12294–12304

Nakahara H, Doya K et al (2001) Parallel cortico-basal ganglia mechanisms for acquisition and execution of visuomotor sequences—a computational approach. J Cogn Neurosci 13:626–647

Nakahara H, Itoh H et al (2004) Dopamine neurons can represent context-dependent prediction error. Neuron 41:269–280

Nicola SM, Surmeier J et al (2000) Dopaminergic modulation of neuronal excitability in the striatum and nucleus accumbens. Annu Rev Neurosci 23:185–215

O’Doherty JP, Deichmann R et al (2002) Neural responses during anticipation of a primary taste reward. Neuron 33:815–826

O’Doherty J, Dayan P et al (2003) Temporal difference models and reward-related learning in the human brain. Neuron 38:329–337

O’Doherty J, Dayan P et al (2004) Dissociable roles of ventral and dorsal striatum in instrumental conditioning. Science 304:452–454

Otani S, Daniel H et al (2003) Dopaminergic modulation of long-term synaptic plasticity in rat prefrontal neurons. Cereb Cortex 13:1251–1256

Otmakhova NA, Lisman JE (1996) D1/d5 dopamine receptor activation increases the magnitude of early long-term potentiation at ca1 hippocampal synapses. J Neurosci 16:7478–7486

Otmakhova NA, Lisman JE (1998) D1/d5 dopamine receptors inhibit depotentiation at ca1 synapses via camp-dependent mechanism. J Neurosci 18:1270–1279

Pan WX, Schmidt R et al (2005) Dopamine cells respond to predicted events during classical conditioning: Evidence for eligibility traces in the reward-learning network. J Neurosci 25:6235–6242

Pawlak V, Kerr JN (2008) Dopamine receptor activation is required for corticostriatal spike-timing-dependent plasticity. J Neurosci 28:2435–2446

Pennartz CM, Ameerun RF et al (1993) Synaptic plasticity in an in vitro slice preparation of the rat nucleus accumbens. Eur J Neurosci 5:107–117

Phillips PE, Stuber GD et al (2003) Subsecond dopamine release promotes cocaine seeking. Nature 422:614–618

Potjans W, Morrison A, Diesmann M (2009) A spiking neural network model of an actor-critic learning agent. Neural Comput 21:301–339

Quirk GJ, Mueller D (2007) Neural mechanisms of extinction learning and retrieval. Neuropsychopharmacology 33:56–72

Redish AD, Jensen S et al (2007) Reconciling reinforcement learning models with behavioral extinction and renewal: Implications for addiction, relapse, and problem gambling. Psychol Rev 114:784–805

Reynolds JN, Wickens JR (2002) Dopamine-dependent plasticity of corticostriatal synapses. Neural Netw 15:507–521

Reynolds JN, Hyland BI et al (2001) A cellular mechanism of reward-related learning. Nature 413:67–70

Robbe D, Alonso G et al (2002) Role of p/q-Ca2+ channels in metabotropic glutamate receptor 2/3-dependent presynaptic long-term depression at nucleus accumbens synapses. J Neurosci 22:4346–4356

Robbins TW, Everitt BJ (1996) Neurobehavioural mechanisms of reward and motivation. Curr Opin Neurobiol 6:228–236

Roberts PD, Santiago RA et al (2008) An implementation of reinforcement learning based on spike timing dependent plasticity. Biol Cybern 99:517–523

Robertson EM, Cohen DA (2006) Understanding consolidation through the architecture of memories. Neuroscientist 12:261–271

Robinson TE, Berridge KC (2008) Review. The incentive sensitization theory of addiction: some current issues. Philos Trans R Soc Lond B Biol Sci 363:3137–3146

Robinson DL, Heien ML et al (2002) Frequency of dopamine concentration transients increases in dorsal and ventral striatum of male rats during introduction of conspecifics. J Neurosci 22:10477–10486

Roesch MR, Calu DJ et al (2007) Dopamine neurons encode the better option in rats deciding between differently delayed or sized rewards. Nature Neurosci 10:1615–1624

Roitman MF, Stuber GD et al (2004) Dopamine operates as a subsecond modulator of food seeking. J Neurosci 24:1265–1271

Rolls ET (2000) Precis of the brain and emotion. Behav Brain Sci 23:177–191 discussion 192–233

Romo R, Schultz W (1990) Dopamine neurons of the monkey midbrain: contingencies of responses to active touch during self-initiated arm movements. J Neurophysiol 63:592–606

Samejima K, Doya K et al (2003) Inter-module credit assignment in modular reinforcement learning. Neural Netw 16:985–994

Samson RD, Pare D (2005) Activity-dependent synaptic plasticity in the central nucleus of the amygdala. J Neurosci 25:1847–1855

Satoh T, Nakai S et al (2003) Correlated coding of motivation and outcome of decision by dopamine neurons. J Neurosci 23:9913–9923

Schimchowitsch S, Vuillez P et al (1991) Systematic presence of GABA-immunoreactivity in the tubero-infundibular and tubero-hypophyseal dopaminergic axonal systems: an ultrastructural immunogold study on several mammals. Exp Brain Res 83:575–586

Schoenbaum G, Chiba AA et al (1999) Neural encoding in orbitofrontal cortex and basolateral amygdala during olfactory discrimination learning. J Neurosci 19:1876–1884

Schultz W (1992) Activity of dopamine neurons in the behaving primate. Semi Neurosci 4(2):129–138

Schultz W (1998) Predictive reward signal of dopamine neurons. J Neurophysiol 80:1–27

Schultz W (2002) Getting formal with dopamine and reward. Neuron 36:241–263

Schultz W, Dayan P et al (1997) A neural substrate of prediction and reward. Science 275:1593–1599

Servan-Schreiber D, Printz H et al (1990) A network model of catecholamine effects: gain, signal-to-noise ratio, and behavior. Science 249:892–895

Seymour B, O’Doherty JP et al (2004) Temporal difference models describe higher-order learning in humans. Nature 429:664–667

Shen W, Flajolet M et al (2008) Dichotomous dopaminergic control of striatal synaptic plasticity. Science 321:848

Suri RE, Bargas J et al (2001) Modeling functions of striatal dopamine modulation in learning and planning. Neuroscience 103:65–85

Surmeier DJ, Ding J et al (2007) D1 and D2 dopamine-receptor modulation of striatal glutamatergic signaling in striatal medium spiny neurons. Trends Neurosci 30:228–235

Sutton RS (1984) Temporal credit assignment in reinforcement learning Ph.D. dissertation, Department of Computer Science, University of Massachusetts, Amherst, MA. Published as COINS Technical Report 84-2

Sutton RS (1988) Learning to predict by the methods of temporal differences. Mach Learn 3:9–44

Sutton RS, Barto AG (1990) Time-derivative models of Pavlovian reinforcement. MIT Press, Cambridge

Sutton RS, Barto AG (1998) Reinforcement learning, an introduction. MIT Press, Cambridge

Takahashi Y, Schoenbaum G et al (2008) Silencing the critics: understanding the effects of cocaine sensitization on dorsolateral and ventral striatum in the context of an actor/critic model. Front Neurosci 2:86–99

Taverna S, Pennartz CM (2003) Postsynaptic modulation of AMPA- and NMDA-receptor currents by group III metabotropic glutamate receptors in rat nucleus accumbens. Brain Res 976:60–68

Thivierge JP, Rivest F et al (2007) Spiking neurons, dopamine, and plasticity: timing is everything, but concentration also matters. Synapse 61:375–390

Thomas MJ, Beurrier C et al (2001) Long-term depression in the nucleus accumbens: a neural correlate of behavioral sensitization to cocaine. Nat Neurosci 4:1217–1223

Tremblay L, Schultz W (2000a) Modifications of reward expectation-related neuronal activity during learning in primate orbitofrontal cortex. J Neurophysiol 83:1877–1885

Tremblay L, Schultz W (2000b) Reward-related neuronal activity during go-nogo task performance in primate orbitofrontal cortex. J Neurophysiol 83:1864–1876

Vijayraghavan S, Wang M et al (2007) Inverted-U dopamine D1 receptor actions on prefrontal neurons engaged in working memory. Nat Neurosci 10:376–384

Voon V, Reynolds B, Brezing C et al (2010) Impulsive choice and response in dopamine agonist-related impulse control behaviors. Psychopharmacology (Berl) 207:645–659

Waelti P, Dickinson A et al (2001) Dopamine responses comply with basic assumptions of formal learning theory. Nature 412:43–48

Watkins C, Dayan P (1992) Q-learning. Mach Learning 8:279–292

White DJ (1993) Markov decision processes. Willey, New York

Wickens JR (2009) Synaptic plasticity in the basal ganglia. Behav Brain Res 199:119–128

Wickens JR, Horvitz JC et al (2007) Dopaminergic mechanisms in actions and habits. J Neuroscience 27:8181

Wiecki TV, Riedinger K et al (2009) A neurocomputational account of catalepsy sensitization induced by D2 receptor blockade in rats: Context dependency, extinction, and renewal. Psychopharmacology 204:265–277

Wise RA (1996a) Addictive drugs and brain stimulation reward. Annu Rev Neurosci 19:319–340

Wise RA (1996b) Neurobiology of addiction. Curr Opin Neurobiol 6:243–251

Wise RA (2004) Dopamine, learning and motivation. Nat Rev Neurosci 5:483–494

Wise RA (2005) Forebrain substrates of reward and motivation. J Comp Neurol 493:115–121

Wise RA, Hoffman DC (1992) Localization of drug reward mechanisms by intracranial injections. Synapse 10:247–263

Wise RA, Rompre PP (1989) Brain dopamine and reward. Annu Rev Psychol 40:191–225

Worgotter F, Porr B (2005) Temporal sequence learning, prediction, and control: a review of different models and their relation to biological mechanisms. Neural Comput 17:245–319

Xie X, Seung HS (2004) Learning in neural networks by reinforcement of irregular spiking. Phys Rev E Stat Nonlin Soft Matter Phys 69:041909

Yao WD, Spealman RD et al (2008) Dopaminergic signaling in dendritic spines. Biochem Pharmacol 75:2055–2069

Yavich L, MacDonald E (2000) Dopamine release from pharmacologically distinct storage pools in rat striatum following stimulation at frequency of neuronal bursting. Brain Res 870:73–79

Yin HH, Ostlund SB et al (2008) Reward-guided learning beyond dopamine in the nucleus accumbens: The integrative functions of cortico-basal ganglia networks. Eur J Neurosci 28:1437–1448

Young AM, Joseph MH et al (1992) Increased dopamine release in vivo in nucleus accumbens and caudate nucleus of the rat during drinking: a microdialysis study. Neuroscience 48:871–876