Neural mechanisms of acquired phasic dopamine responses in learning

Ahn, 2003, Independent modulation of basal and feeding-evoked dopamine efflux in the nucleus accumbens and medial prefrontal cortex by the central and basolateral amygdalar nuclei in the rat, Neuroscience, 116, 295, 10.1016/S0306-4522(02)00551-1 Amaral, 1992, Anatomical organization of the primate amygdaloid complex, 1 Andrzejewski, 2005, Instrumental learning, but not performance, requires dopamine d1-receptor activation in the amygdala, Neuroscience, 135, 335, 10.1016/j.neuroscience.2005.06.038 Apicella, 1992, Neuronal activity in monkey striatum related to the expectation of predictable environmental events, Journal of Neurophysiology, 68, 945, 10.1152/jn.1992.68.3.945 Aston-Jones, 2009, Role of lateral hypothalamic orexin neurons in reward processing and addiction, Neuropharmacology, 56, 112, 10.1016/j.neuropharm.2008.06.060 Bao, 2003, Suppression of cortical representation through backward conditioning, Proceedings of the National Academy of Sciences of the United States of America, 100, 1405, 10.1073/pnas.0337527100 Baxter, 2002, The amygdala and reward, Nature Reviews Neuroscience, 3, 563, 10.1038/nrn875 Bayer, 2005, Midbrain dopamine neurons encode a quantitative reward prediction error signal, Neuron, 47, 129, 10.1016/j.neuron.2005.05.020 Bayer, 2007, Statistics of midbrain dopamine neuron spike trains in the awake primate, Journal of Neurophysiology, 98, 1428, 10.1152/jn.01140.2006 Belova, 2007, Expectation modulates neural responses to pleasant and aversive stimuli in primate amygdala, Neuron, 55, 970, 10.1016/j.neuron.2007.08.004 Bentivoglio, 1988, The intrinsic and extrinsic organization of the thalamic intralaminar nuclei, 221 Berendse, 1992, Compartmental distribution of ventral striatal neurons projecting to the mesencephalon in the rat, Journal of Neuroscience, 12, 2079, 10.1523/JNEUROSCI.12-06-02079.1992 Keri, 2009, Reward-learning and the novelty-seeking personality: a between- and within-subjects study of the effects of dopamine agonists on young parkinson’s patients, Brain, 132, 2385, 10.1093/brain/awp094 Braver, 2000, On the control of control: The role of dopamine in regulating prefrontal function and working memory, 713 Brown, 1999, How the basal ganglia use parallel excitatory and inhibitory learning pathways to selectively respond to unexpected rewarding cues, Journal of Neuroscience, 19, 10502, 10.1523/JNEUROSCI.19-23-10502.1999 Brown, 2004, How laminar frontal cortex and basal ganglia circuits interact to control planned and reactive saccades, Neural Networks, 17, 471, 10.1016/j.neunet.2003.08.006 Brown, 1968, Auto-shaping of the pigeon’s key-peck, Journal of the Experimental Analysis of Behavior, 11, 1, 10.1901/jeab.1968.11-1 Cardinal, 2002, Emotion and motivation: the role of the amygdala, ventral striatum, and prefrontal cortex, Neuroscience and Biobehavioral Reviews, 26, 321, 10.1016/S0149-7634(02)00007-6 Cardinal, 2002, Effects of selective excitotoxic lesions of the nucleus accumbens core, anterior cingulate cortex, and central nucleus of the amygdala on autoshaping performance in rats, Behavioral Neuroscience, 116, 553, 10.1037/0735-7044.116.4.553 Chong, 2006, Dopamine-d1 and -d2 receptors differentially regulate synapsin ii expression in the rat brain, Neuroscience, 138, 587, 10.1016/j.neuroscience.2005.11.037 Comoli, 2003, A direct projection from superior colliculus to substantia nigra for detecting salient visual events, Nature Neuroscience, 6, 974, 10.1038/nn1113 Cools, 2006, Reversal learning in parkinson’s disease depends on medication status and outcome valence, Neuropsychologia, 44, 1663, 10.1016/j.neuropsychologia.2006.03.030 Cools, 2009, Striatal dopamine predicts outcome-specific reversal learning and its sensitivity to dopaminergic drug administration, The Journal of Neuroscience: The Official Journal of the Society for Neuroscience, 29, 1538, 10.1523/JNEUROSCI.4467-08.2009 Corbit, 2005, Double dissociation of basolateral and central amygdala lesions on the general and outcome-specific forms of pavlovian-instrumental transfer, Journal of Neuroscience, 25, 962, 10.1523/JNEUROSCI.4507-04.2005 Cromwell, 2003, Effects of expectations for different reward magnitudes on neuronal activity in primate striatum, Journal of Neurophysiology, 89, 2823, 10.1152/jn.01014.2002 Davis, 1994, Neurotransmission in the rat amygdala related to fear and anxiety, Trends in neurosciences, 17, 208, 10.1016/0166-2236(94)90106-6 Daw, 2003, Timing and partial observability in the dopamine system Daw, 2005, Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control, Nature Neuroscience, 8, 1704, 10.1038/nn1560 Day, 2007, The nucleus accumbens and pavlovian reward learning, The Neuroscientist, 13, 148, 10.1177/1073858406295854 Dayan, 2008, Reinforcement learning: the good, the bad and the ugly, Current Opinion in Neurobiology, 18, 185, 10.1016/j.conb.2008.08.003 Dayan, 1996, Exploration bonuses and dual control, Machine Learning, 25, 5, 10.1007/BF00115298 Deadwyler, 2004, Reward, memory and substance abuse: functional neuronal circuits in the nucleus accumbens, Neuroscience and Biobehavioral Reviews, 27, 703, 10.1016/j.neubiorev.2003.11.011 Denny, 1970 Di Ciano, 2001, Differential involvement of nmda, ampa/kainate, and dopamine receptors in the nucleus accumbens core in the acquisition and performance of pavlovian approach behavior, Journal of Neuroscience, 21, 9471, 10.1523/JNEUROSCI.21-23-09471.2001 Dinsmoor, 2001, Stimuli inevitably generated by behavior that avoids electric shock are inherently reinforcing, Journal of the Experimental Analysis of Behavior, 75, 311, 10.1901/jeab.2001.75-311 Dommett, 2005, How visual stimuli activate dopaminergic neurons at short latency, Science, 307, 1476, 10.1126/science.1107026 Doya, 2002, Metalearning and neuromodulation, Neural Networks, 15, 495, 10.1016/S0893-6080(02)00044-8 Dykman, R.A., 1965. Toward a theory of classical conditioning: cognitive, emotional, and motor components of the conditional reflex. In: Maher, B.A. (Ed.), Progress in Experimental Personality Research. Vol. 2. Elsevier Science & Technology Books, pp. 229–317. Fallon, 1992, Distribution of monoamines within the amygdala, 97 Fiorillo, 2005, Evidence that the delay-period activity of dopamine neurons corresponds to reward uncertainty rather than backpropagating td errors, Behavioral and Brain Functions, 1, 1 Fiorillo, 2000, Cholinergic inhibition of ventral midbrain dopamine neurons, Journal of Neuroscience, 20, 7855, 10.1523/JNEUROSCI.20-20-07855.2000 Floresco, 2003, Afferent modulation of dopamine neuron firing differentially regulates tonic and phasic dopamine transmission, Nature Neuroscience, 6, 968, 10.1038/nn1103 Frank, 2005, Dynamic dopamine modulation in the basal ganglia: a neurocomputational account of cognitive deficits in medicated and non-medicated parkinsonism, Journal of Cognitive Neuroscience, 17, 51, 10.1162/0898929052880093 Frank, M.J., Doll, B., Oas-Terpstra, J., Moreno, F., 2009. Prefrontal and striatal dopaminergic genes predict individual differences in exploration and exploitation. Nature Neurosci. 12, 1062–1068. Frank, 2001, Interactions between the frontal cortex and basal ganglia in working memory: a computational model, Cognitive, Affective, and Behavioral Neuroscience, 1, 137, 10.3758/CABN.1.2.137 Frank, 2006, A mechanistic account of striatal dopamine function in human cognition: psychopharmacological studies with cabergoline and haloperidol, Behavioral Neuroscience, 120, 497, 10.1037/0735-7044.120.3.497 Frank, 2004, By carrot or by stick: Cognitive reinforcement learning in parkinsonism, Science, 306, 1940, 10.1126/science.1102941 Frey, 1991, The effect of d1 dopaminergic receptor blockade during tetanization on the expression of long-term potentiation in the rat ca1 region in vitro, Neuroscience Letters, 129, 111, 10.1016/0304-3940(91)90732-9 Frey, 1990, Dopaminergic antagonists prevent long-term maintenance of posttetanic ltp in the ca1 region of rat hippocampal slices, Brain Research, 552, 69, 10.1016/0006-8993(90)91578-5 Fudge, 2000, The central nucleus of the amygdala projection to dopamine subpopulations in primates, Neuroscience, 97, 479, 10.1016/S0306-4522(00)00092-0 Fukuda, 1993, Amygdala-hypothalamic control of feeding behavior in monkey: single cell responses before and after reversible blockade of temporal cortex or amygdala projections, Behavioural Brain Research, 55, 141, 10.1016/0166-4328(93)90119-B Fuster, 1971, Reactivity of limbic neurons of the monkey to appetitive and aversive signals, Electroencephalography and Clinical Neurophysiology, 30, 281, 10.1016/0013-4694(71)90111-8 Gallagher, 2000, The amygdala and associative learning, 311 Gallagher, 1990, The amygdala central nucleus and appetitive pavlovian conditioning: lesions impair one class of conditioned behavior, Journal of Neuroscience, 14, 497 Gallagher, 1993, Preserved configural learning and spatial learning impairment in rats with hippocampal damage, Hippocampus, 2, 81, 10.1002/hipo.450020111 Gallagher, 1994, The amygdala complex: multiple roles in associative learning and attention, Proceedings of the National Academy of Sciences of the United States of America, 91, 11771, 10.1073/pnas.91.25.11771 Gerfen, 1985, The neostriatal mosaic. I. compartmental organization of projections of the striatonigral system in the rat, Journal of Comparative Neurology, 236, 454, 10.1002/cne.902360404 Gerfen, 1987, The neostriatal mosaic. Ii. Patch- and matrix-directed mesostriatal dopaminergic and non-dopaminergic systems, Journal of Neuroscience, 7, 3915, 10.1523/JNEUROSCI.07-12-03915.1987 Gibbon, 1975, Signal-food contingency and signal frequency in a continuous trials auto-shaping paradigm, Animal Learning and Behavior, 3, 317, 10.3758/BF03213453 Glimcher, 2008, Perspectives from economics and neuroeconomics understanding risk: a guide for the perplexed, Cognitive Affective & Behavioral Neuroscience, 8, 348, 10.3758/CABN.8.4.348 Grace, 1979, Paradoxical gaba excitation of nigral dopaminergic cells: indirect mediation through reticulata inhibitory neurons, European Journal of Pharmacology, 59, 211, 10.1016/0014-2999(79)90283-8 Grace, 1985, Opposing effects of striatonigral feedback pathways on midbrain dopamine cell activity, Brain Research, 333, 271, 10.1016/0006-8993(85)91581-1 Grace, 2000, Electrophysiological properties of midbrain dopamine neurons Graybiel, 1998, The basal ganglia and chunking of action repertoires, Neurobiology of Learning and Memory, 70, 119, 10.1006/nlme.1998.3843 Groenewegen, 1990, The anatomical relationship of the prefrontal cortex with the striatopallidal system the thalamus and the amygdala: evidence for a parallel organization, Progress in Brain Research, 85, 95, 10.1016/S0079-6123(08)62677-1 Groshek, 2005, Amygdala central nucleus function is necessary for learning, but not expression, of conditioned auditory orienting, Behavioral neuroscience, 119, 202, 10.1037/0735-7044.119.1.202 Han, 1997, The role of an amygdalo-nigrostriatal pathway in associative learning, Journal of Neuroscience, 17, 3913, 10.1523/JNEUROSCI.17-10-03913.1997 Hatfield, 1996, Neurotoxic lesions of basolateral, but not central, amygdala interfere with pavlovian second-order conditioning and reinforcer devaluation effects, Journal of Neuroscience, 16, 5256, 10.1523/JNEUROSCI.16-16-05256.1996 Hazy, 2006, Banishing the homunculus: making working memory work, Neuroscience, 139, 105, 10.1016/j.neuroscience.2005.04.067 Hazy, 2007, Towards an executive without a homunculus: computational models of the prefrontal cortex/basal ganglia system, Philosophical Transactions of the Royal Society of London, Series B: Biological Sciences, 362, 105, 10.1098/rstb.2007.2055 Hebb, 1949 Heldt, 2007, Lesions of the habenula produce stress- and dopamine-dependent alterations in prepulse inhibition and locomotion, Brain Research, 1073, 229 Herkenham, 1977, Efferent connections of the habenular nuclei in the rat, Journal of Comparative Neurology, 187, 19, 10.1002/cne.901870103 Holland, 1977, Conditioned stimulus as a determinant of the form of the pavlovian conditioned response, Journal of Experimental Psychology, 3, 77 Holland, 1984, Origins of behavior in pavlovian conditioning Holland, 2002, Operant and pavlovian control of visual stimulus orienting and food-related behaviors in rats with lesions of the amygdala central nucleus, Behavioral Neuroscience, 116, 577, 10.1037/0735-7044.116.4.577 Hollerman, 1998, Dopamine neurons report an error in the temporal prediction of reward during learning, Nature Neuroscience, 1, 304, 10.1038/1124 Horvitz, 2000, Mesolimbocortical and nigrostriatal dopamine responses to salient non-reward events, Neuroscience, 96, 651, 10.1016/S0306-4522(00)00019-1 Horvitz, 1997, Burst activity of ventral tegmental dopamine neurons is elicited by sensory stimuli in the awake cats, Brain Research, 759, 251, 10.1016/S0006-8993(97)00265-5 Houk, 1995, A model of how the basal ganglia generate and use neural signals that predict reinforcement, 233 Huang, 1995, D1/D5 receptor agonists induce a protein synthesis-dependent late potentiation in the ca1 region of the hippocampus, Proceedings of the National Academy of Sciences of the United States of America, 92, 2446, 10.1073/pnas.92.7.2446 Humphries, 2006, A physiologically plausible model of action selection and oscillatory activity in the basal ganglia, Journal of Neuroscience, 26, 12921, 10.1523/JNEUROSCI.3486-06.2006 Izhikevich, 2007, Solving the distal reward problem through linkage of stdp and dopamine signaling, Cerebral Cortex, 17, 2443, 10.1093/cercor/bhl152 Jenkins, 1973, Effects of the stimulus reinforcer relation on selected and unselected responses Ji, 2007, Lateral habenula stimulation inhibits rat midbrain dopamine neurons through a gaba-a receptor-mediated mechanism, Journal of Neuroscience, 27, 6923, 10.1523/JNEUROSCI.0958-07.2007 Joel, 2000, The connections of the dopaminergic system with the striatum in rats and primates: an analysis with respect to the functional and compartmental organization of the striatum, Neuroscience, 96, 451, 10.1016/S0306-4522(99)00575-8 Kakade, 2002, Dopamine: generalization and bonuses, Neural Networks, 15, 549, 10.1016/S0893-6080(02)00048-5 Kamin, 1969, Predictability, surprise, attention and conditioning Kamin, 1969, Selective association and conditioning, 42 Killcross, 1997, Different types of fear-conditioned behaviour mediated by separate nuclei within amygdala, Nature, 388, 377, 10.1038/41097 Kronforst-Collins, 1998, Lesions of the caudal area of rabbit medial prefrontal cortex impair trace eyeblink conditioning, Neurobiology of Learning and Memory, 69, 147, 10.1006/nlme.1997.3818 Lecourtier, 2008, Differential tonic influence of lateral habenula on prefrontal cortex and nucleus accumbens dopamine release, European Journal of Neuroscience, 27, 1755, 10.1111/j.1460-9568.2008.06130.x LeDoux, 2003, The emotional brain, fear, and the amygdala, Cellular and Molecular Neurobiology, 23, 727, 10.1023/A:1025048802629 Lee, 2005, Role of amygdalo-nigral circuitry in conditioning of a visual stimulus paired with food, Journal of Neuroscience, 25, 3881, 10.1523/JNEUROSCI.0416-05.2005 Legault, 2001, Novelty-evoked elevations of nucleus accumbens dopamine: dependence on impulse flow from the ventral subiculum and glutaminergic neurotransmission in the ventral tegmental area, European Journal of Neuroscience, 13, 819, 10.1046/j.0953-816x.2000.01448.x Lisman, 2005, The hippocampal-VTA loop: controlling the entry of information into long-term memory, Neuron, 46, 703, 10.1016/j.neuron.2005.05.002 Ljungberg, 1992, Responses of monkey dopamine neurons during learning of behavioral reactions, Journal of Neurophysiology, 67, 145, 10.1152/jn.1992.67.1.145 Locurto, 1981, Contributions of autoshaping to the partitioning of conditioned behavior, 101 Lustig, 2005, Not just a coincidence: frontal-striatal interactions in working memory and interval timing, Memory (Hove, England), 13, 441, 10.1080/09658210344000404 Matsumoto, 2007, Lateral habenula as a source of negative reward signals in dopamine neurons, Nature, 447, 1111, 10.1038/nature05860 Matsumoto, 2009, Representation of negative motivational value in the primate lateral habenula, Nature Neuroscience, 12, 77, 10.1038/nn.2233 Mauk, 2004, The neural basis of temporal processing, Annual Review of Neuroscience, 27, 307, 10.1146/annurev.neuro.27.070203.144247 McClure, 2005, An exploration–exploitation model based on norepinepherine and dopamine activity, Advances in Neural Information Processing Systems, 18 Miller, 2000, The amygdaloid complex: input processor for the midbrain dopaminergic nuclei and striatum, 77 Mink, 1996, The basal ganglia: Focused selection and inhibition of competing motor programs, Progress in Neurobiology, 50, 381, 10.1016/S0301-0082(96)00042-1 Mirenowicz, 1996, Preferential activation of midbrain dopamine neurons by appetitive rather than aversive stimuli, Nature, 379, 449, 10.1038/379449a0 Montague, 1997, A framework for mesencephalic dopamine systems based on predictive hebbian learning, The Journal of Neuroscience, 16, 1936, 10.1523/JNEUROSCI.16-05-01936.1996 Montague, 2004, Computational roles for dopamine in behavioural control, Nature, 431, 760, 10.1038/nature03015 Mora, 1976, Modulation during learning of the responses of neurons in the lateral hypothalamus to the sight of food, Experimental Neurology, 53, 508, 10.1016/0014-4886(76)90089-3 Moustafa, 2008, A role for dopamine in temporal decision making and reward maximization in parkinsonism, Journal of Neuroscience, 28, 12294, 10.1523/JNEUROSCI.3116-08.2008 Nakamura, 1986, Lateral hypothalamus neuron involvement in integration of natural and artificial rewards and cue signals, Journal of Neurophysiology, 55, 163, 10.1152/jn.1986.55.1.163 Niv, 2005, Dopamine, uncertainty and td learning, Behavioral and Brain Functions, 1, 1 Norgren, 1976, Taste pathways to hypothalamus and amygdala, Journal of Comparative Neurology, 166, 17, 10.1002/cne.901660103 Ono, 1986, Hypothalamic neuron involvement in integration of reward, aversion, and cue signals, Journal of Neurophysiology, 56, 63, 10.1152/jn.1986.56.1.63 Ono, 1995, Amygdala role in conditioned associative learning, Progress in Neurobiology, 46, 401, 10.1016/0301-0082(95)00008-J Ono, 1981, Monkey lateral hypothalamic neuron response to sight of food, and during bar press and ingestion, Neuroscience Letters, 21, 99, 10.1016/0304-3940(81)90065-3 O’Reilly, 2006, Making working memory work: a computational model of learning in the prefrontal cortex and basal ganglia, Neural Computation, 18, 283, 10.1162/089976606775093909 O’Reilly, 2007, Pvlv: The primary value and learned value pavlovian learning algorithm, Behavioral Neuroscience, 121, 31, 10.1037/0735-7044.121.1.31 O’Keefe, 1969, Complex sensory properties of certain amygdala units in the freely moving cat, Experimental Neurology, 23, 384, 10.1016/0014-4886(69)90086-7 Palminteri, 2009, Pharmacological modulation of subliminal learning in parkinson’s and tourette’s syndromes, Proceedings of the National Academy of Sciences of the United States of America, 106, 19179, 10.1073/pnas.0904035106 Pan, 2005, Dopamine cells respond to predicted events during classical conditioning: evidence for eligibility traces in the reward-learning network, Journal of Neuroscience, 25, 6235, 10.1523/JNEUROSCI.1478-05.2005 Parkinson, 1998, Effects of 6-ohda lesions of the rat nucleus accumbens on appetitive pavlovian conditioning, Journal of Psychopharmacology, 12, A8 Parkinson, 2002, Nucleus accumbens dopamine depletion impairs both acquisition and performance of appetitive pavlovian approach behaviour: Implications for mesoaccumbens dopamine function, Behavioural Brain Research, 137, 149, 10.1016/S0166-4328(02)00291-7 Paton, 2006, The primate amygdala represents the positive and negative value of visual stimuli during learning, Nature, 439, 865, 10.1038/nature04490 Pavlov, 1927 Petrovich, 2001, Combinatorial amygdalar inputs to hippocampal domains and hypothalamic behavior systems, Brain Research Reviews, 38, 247, 10.1016/S0165-0173(01)00080-7 Petrovich, 2002, Amygdalo-hypothalamic circuit allows learned cues to override satiety and promote eating, Journal of Neuroscience, 22, 8748, 10.1523/JNEUROSCI.22-19-08748.2002 Phillipson, 1978, Afferent projections to a10 dopaminergic neurones in the rat as shown by the retrograde transport of horseradish peroxidase, Neuroscience Letters, 9, 353, 10.1016/0304-3940(78)90208-2 Pitkanen, 2000, Connectivity of the rat amygdaloid complex, 31 Rachlin, 1969, Autoshaping of key pecking in pigeons wiht negative reinforcement, Journal of the Experimental Analysis of Behavior, 12, 521, 10.1901/jeab.1969.12-521 Redgrave, 1999, Is the short-latency dopamine response too short to signal reward error?, Trends in neurosciences, 22, 146, 10.1016/S0166-2236(98)01373-3 Rescorla, 1972, A theory of pavlovian conditioning: variation in the effectiveness of reinforcement and non-reinforcement, 64 Roesch, 2007, Dopamine neurons encode the better option in rats deciding between differently delayed or sized rewards, Nature Neuroscience, 10, 1615, 10.1038/nn2013 Roitman, 2005, Nucleus accumbens neurons are innately tuned for rewarding and aversive taste stimuli, encode their predictors, and are linked to motor output, Neuron, 45, 587, 10.1016/j.neuron.2004.12.055 Rolls, 1976, Hypothalamic neuronal responses associated with the sight of food, Brain Research, 111, 53, 10.1016/0006-8993(76)91048-9 Roozendaal, 1990, The central amygdala is involved in the conditioned but not in the meal-induced cephalic insulin response in the rat, Neuroscience Letters, 116, 210, 10.1016/0304-3940(90)90412-3 Rouillard, 1995, Effects of electrical stimulation of the central nucleus of the amygdala on the in vivo electrophysiological activity of rat nigral dopaminergic neurons, Synapse (New York, NY), 21, 348, 10.1002/syn.890210410 Sanghera, 1979, Visual responses of neurons in the dorsolateral amygdala of the alert monkey, Experimental Neurology, 63, 610, 10.1016/0014-4886(79)90175-4 Santesso, 2009, Single dose of a dopamine agonist impairs reinforcement learning in humans: evidence from event-related potentials and computational modeling of striatal-cortical function, Human Brain Mapping, 30, 1963, 10.1002/hbm.20642 Satoh, 2003, Correlated coding of motivation and outcome of decision by dopamine neurons, Journal of Neuroscience, 23, 9913, 10.1523/JNEUROSCI.23-30-09913.2003 Savastano, 1998, Time as content in pavlovian conditioning, Behavioural Processes, 44, 147, 10.1016/S0376-6357(98)00046-1 Schmajuk, 1997, Animal learning and cognition: a neural network approch Schneiderman, 1966, Interstimulus interval function of the nictitating membrane response of the rabbit under delay versus trace conditioning, Journal of Comparative and Physiological Psychology, 62, 397, 10.1037/h0023946 Schoenbaum, 1999, Neural encoding in orbitofrontal cortex and basolateral amygdala during olfactory discrimination learning, Journal of Neuroscience, 19, 1876, 10.1523/JNEUROSCI.19-05-01876.1999 Schoenbaum, 2003, Encoding predicted outcome and acquired value in orbitofrontal cortex during cue sampling depends upon input from basolateral amygdala, Neuron, 39, 855, 10.1016/S0896-6273(03)00474-4 Schultz, 1998, Predictive reward signal of dopamine neurons, Journal of Neurophysiology, 80, 1, 10.1152/jn.1998.80.1.1 Schultz, 1993, Responses of monkey dopamine neurons to reward and conditioned stimuli during successive steps of learning a delayed response task, Journal of Neuroscience, 13, 900, 10.1523/JNEUROSCI.13-03-00900.1993 Schultz, 1993, Neuronal activity in monkey ventral striatum related to the expectation of reward, Journal of Neuroscience, 12, 4595, 10.1523/JNEUROSCI.12-12-04595.1992 Schultz, 1997, A neural substrate of prediction and reward, Science, 275, 1593, 10.1126/science.275.5306.1593 Schultz, 2000, Neuronal coding of prediction errors, Annual Review of Neuroscience, 23, 473, 10.1146/annurev.neuro.23.1.473 Schultz, 1990, Dopamine neurons of the monkey midbrain: contingencies of responses to stimuli eliciting immediate behavioral reactions, Journal of Neurophysiology, 63, 607, 10.1152/jn.1990.63.3.607 Semba, 1992, Afferent connections of the laterodorsal and the pedunculopontine tegmental nuclei in the rat: a retro- and antero-grade transport and immunohistochemical study, Journal of Comparative Neurology, 323, 387, 10.1002/cne.903230307 Shepard, 2006, The presence of absence: habenular regulation of dopamine neurons and the encoding of negative outcomes, Schizophrenia Bulletin, 32, 417, 10.1093/schbul/sbj083 Smith, 1990, The neural network of the basal ganglia as revealed by the study of synaptic connections of identified neurones, Trends in Neuroscience, 13, 259, 10.1016/0166-2236(90)90106-K Smith, 1968, CS–US interval and us intensity in classical conditioning of the rabbit’s nictitating membrane response, Journal of Comparative and Physiological Psychology, 66, 679, 10.1037/h0026550 Sokolov, 1963, Higher nervous functions: the orienting reflex, Annual Review of Physiology, 25, 545, 10.1146/annurev.ph.25.030163.002553 Steinfels, 1983, Response of dopamine neurons in cat to auditory stimuli presented across the sleep-waking cycle, Brain Research, 277, 150, 10.1016/0006-8993(83)90917-4 Stuber, 2008, Reward-predictive cues enhance excitatory synaptic strength onto midbrain dopamine neurons, Science, 321, 1690, 10.1126/science.1160873 Suri, 1999, A neural network model with dopamine-like reinforcement signal that learns a spatial delayed response task, Neuroscience, 91, 871, 10.1016/S0306-4522(98)00697-6 Suri, 2001, Temporal difference model reproduces anticipatory neural activity, Neural Computation, 13, 841, 10.1162/089976601300014376 Surmeier, 2007, D1 and d2 dopamine-receptor modulation of striatal glutamatergic signaling in striatal medium spiny neurons, Trends in Neurosciences, 30, 228, 10.1016/j.tins.2007.03.008 Sutton, 1988, Learning to predict by the method of temporal differences, Machine Learning, 3, 9, 10.1007/BF00115009 Sutton, 1990, Time-derivative models of pavlovian reinforcement, 497 Sutton, 1998 Takayama, 1991, Glutamate-immunoreactive neurons of the central amygdaloid nucleus projecting to the subretrofacial nucleus of shr and wky rats: a double-labeling study, Neuroscience Letters, 134, 62, 10.1016/0304-3940(91)90509-R Tan, 2008, A local circuit model of learned striatal and dopamine cell responses under probabilistic schedules of reward, Journal of Neuroscience, 28, 10062, 10.1523/JNEUROSCI.0259-08.2008 Terrace, 1981, Introduction: autoshaping and two-factor learning theory, 1 Thorndike, 1911 Tobler, 2003, Coding of predicted reward omission by dopamine neurons in a conditioned inhibition paradigm, Journal of Neuroscience, 23, 10402, 10.1523/JNEUROSCI.23-32-10402.2003 Tomie, 1981, Effect of unpredictable food on the subsequent acquisition of autoshaping: analysis of the context-blocking hypothesis, 181 Ungless, 2004, Uniform inhibition of dopamine neurons in the ventral tegmental area by aversive stimuli, Science (New York, NY), 303, 2040, 10.1126/science.1093360 Waelti, 2001, Dopamine responses comply with basic assumptions of formal learning theory, Nature, 412, 43, 10.1038/35083500 Wallace, 1992, Organization of amygdaloid projections to brainstem dopaminergic, noradrenergic, and adrenergic cell groups in the rat, Brain Research Bulletin, 28, 447, 10.1016/0361-9230(92)90046-Z Wasserman, 1981, Response evocation in autoshaping: contributions of cognitive and comparative-evolutionary analyses to an understanding of directed action, 21 Weible, 2000, Cortical involvement in acquisition and extinction of trace eyeblink conditioning, Behavioral Neuroscience, 114, 1058, 10.1037/0735-7044.114.6.1058 Wessells, 1974, The effects of reinforcement upon the prepecking behaviors in pigeons in the autoshaping experiment, Journal of the Experimental Analysis of Behavior, 21, 125, 10.1901/jeab.1974.21-125 Wickens, 1995, Cellular models of reinforcement, 187 Wickens, 1996, Dopamine reverses the depression of rat corticostriatal synapses which normally follows high-frequency stimulation of cortex in vitro, Neuroscience, 70, 1, 10.1016/0306-4522(95)00436-M Wickens, 2007, Dopaminergic mechanisms in actions and habits, Journal of Neuroscience, 27, 8181, 10.1523/JNEUROSCI.1671-07.2007 Wittmann, 2008, Striatal activity underlies novelty-based choice in humans, Neuron, 58, 967, 10.1016/j.neuron.2008.04.027