Multiple reward signals in the brain

Fibiger, H. C. & Phillips, A. G. in Handbook of Physiology—The Nervous System Vol. IV (ed. Bloom, F. E.) 647– 675 (Williams and Wilkins, Baltimore, Maryland, 1986 ).

Wise, R. A. & Hoffman, D. C. Localization of drug reward mechanisms by intracranial injections. Synapse 10, 247–263 (1992).

Robinson, T. E. & Berridge, K. C. The neural basis for drug craving: an incentive-sensitization theory of addiction. Brain Res. Rev. 18, 247–291 (1993).

Robbins, T. W. & Everitt, B. J. Neurobehavioural mechanisms of reward and motivation. Curr. Opin. Neurobiol. 6, 228–236 (1996)

Louilot, A., LeMoal, M. & Simon, H. Differential reactivity of dopaminergic neurons in the nucleus accumbens in response to different behavioural situations. An in vivo voltammetric study in free moving rats. Brain Res. 397, 395–400 ( 1986).

Church, W. H. & Justice, J. B. Jr, Neill, D. B. Detecting behaviourally relevant changes in extracellular dopamine with microdialysis. Brain Res. 41, 397–399 (1987).

Young, A. M. J., Joseph, M. H. & Gray, J. A. Increased dopamine release in vivo in nucleus accumbens and caudate nucleus of the rat during drinking: a microdialysis study. Neuroscience 48, 871– 876 (1992).

Wilson, C., Nomikos, G. G., Collu, M. & Fibiger, H. C. Dopaminergic correlates of motivated behaviour: importance of drive. J. Neurosci. 15, 5169–5178 (1995).

Fiorino, D. F., Coury, A. & Phillips, A. G. Dynamic changes in nucleus accumbens dopamine efflux during the Coolidge effect in male rats. J. Neurosci. 17, 4849–4855 (1997).

Thut, G. et al. Activation of the human brain by monetary reward. NeuroReport 8, 1225–1228 (1997).

Rogers, R. D. et al. Choosing between small, likely rewards and large, unlikely rewards activates inferior and orbital prefrontal cortex. J. Neurosci. 19, 9029–9038 ( 1999)

Elliott, R., Friston, K. J. & Dolan, R. J. Dissociable neural responses in human reward systems . J. Neurosci. 20, 6159– 6165 (2000)

Pavlov, P. I. Conditioned Reflexes (Oxford Univ. Press, London, 1927 ).

Rescorla, R. A. & Wagner, A. R. in Classical Conditioning II: Current Research and Theory (eds Black, A. H. and Prokasy, W. F.) 64–99 (Appleton Century Crofts, New York, 1972).

Mackintosh, N. J. A theory of attention: variations in the associability of stimulus with reinforcement . Psychol. Rev. 82, 276– 298 (1975).

Pearce, J. M. & Hall, G. A model for Pavlovian conditioning: variations in the effectiveness of conditioned but not of unconditioned stimuli . Psychol. Rev. 87, 532– 552 (1980).

Schultz, W. Responses of midbrain dopamine neurons to behavioural trigger stimuli in the monkey. J. Neurophysiol. 56, 1439– 1462 (1986).

Romo, R. & Schultz, W. Dopamine neurons of the monkey midbrain: contingencies of responses to active touch during self-initiated arm movements . J. Neurophysiol. 63, 592– 606 (1990).

Schultz, W. & Romo, R. Dopamine neurons of the monkey midbrain: contingencies of responses to stimuli eliciting immediate behavioural reactions . J. Neurophysiol. 63, 607– 624 (1990).

Ljungberg, T., Apicella, P. & Schultz, W. Responses of monkey dopamine neurons during learning of behavioural reactions. J. Neurophysiol. 67, 145–163 (1992).

Strecker, R. E. & Jacobs, B. L. Substantia nigra dopaminergic unit activity in behaving cats: effect of arousal on spontaneous discharge and sensory evoked activity. Brain Res. 361 , 339–350 (1985).

Horvitz, J. C. Mesolimbocortical and nigrostriatal dopamine responses to salient non-reward events. Neuroscience 96, 651– 656 (2000).

Mirenowicz, J. & Schultz, W. Preferential activation of midbrain dopamine neurons by appetitive rather than aversive stimuli. Nature 379, 449–451 ( 1996).

Guarraci, F. A. & Kapp, B. S. An electrophysiological characterization of ventral tegmental area dopaminergic neurons during differential Pavlovian fear conditioning in the awake rabbit. Behav. Brain Res. 99, 169–179 ( 1999).

Schultz, W. Activity of dopamine neurons in the behaving primate. Semin. Neurosci. 4, 129–138 ( 1992).

Redgrave, P., Prescott, T. J. & Gurney, K. Is the short-latency dopamine response too short to signal reward? Trends Neurosci. 22, 146– 151 (1999).

Hollerman, J. R. & Schultz, W. Dopamine neurons report an error in the temporal prediction of reward during learning. Nature Neurosci. 1, 304–309 (1998).

Schultz, W. Predictive reward signal of dopamine neurons. J. Neurophysiol. 80, 1–27 (1998 ).

Schultz, W. & Dickinson, A. Neuronal coding of prediction errors. Annu. Rev. Neurosci. 23, 473– 500 (2000).

Mirenowicz, J. & Schultz, W. Importance of unpredictability for reward responses in primate dopamine neurons. J. Neurophysiol. 72, 1024–1027 (1994).

Schultz, W., Dayan, P. & Montague, R. R. A neural substrate of prediction and reward. Science 275, 1593–1599 ( 1997).

Montague, P. R., Dayan, P. & Sejnowski, T. J. A framework for mesencephalic dopamine systems based on predictive Hebbian learning. J. Neurosci. 16, 1936–1947 (1996).

Sutton, R. S. & Barto, A. G. Toward a modern theory of adaptive networks: expectation and prediction. Psychol. Rev. 88, 135–170 (1981). This paper proposed a very effective 'temporal difference' reinforcement learning model that computes a prediction error over time. The teaching signal incorporates primary reinforcers and conditioned stimuli, and resembles in all aspects the response of dopamine neurons to rewards and conditioned, reward-predicting stimuli, although dopamine neurons also report novel stimuli.

Tesauro, G. TD-Gammon, a self-teaching backgammon program, achieves master-level play . Neural Comp. 6, 215–219 (1994).

Suri, R. E. & Schultz, W. Learning of sequential movements by neural network model with dopamine-like reinforcement signal. Exp. Brain Res. 121, 350–354 (1998).

Suri, R. & Schultz, W. A neural network with dopamine-like reinforcement signal that learns a spatial delayed response task. Neuroscience 91, 871–890 (1999).

Schultz, W. & Romo, R. Responses of nigrostriatal dopamine neurons to high intensity somatosensory stimulation in the anesthetized monkey . J. Neurophysiol. 57, 201– 217 (1987).

Abercrombie, E. D., Keefe, K. A., DiFrischia, D. S. & Zigmond, M. J. Differential effect of stress on in vivo dopamine release in striatum, nucleus accumbens, and medial frontal cortex. J. Neurochem. 52, 1655–1658 (1989).

Hikosaka, O., Sakamoto, M. & Usui, S. Functional properties of monkey caudate neurons. III. Activities related to expectation of target and reward. J. Neurophysiol. 61, 814–832 ( 1989).

Apicella, P., Ljungberg, T., Scarnati, E. & Schultz, W. Responses to reward in monkey dorsal and ventral striatum. Exp. Brain Res. 85, 491–500 ( 1991).

Apicella, P., Scarnati, E. & Schultz, W. Tonically discharging neurons of monkey striatum respond to preparatory and rewarding stimuli. Exp. Brain Res. 84, 672–675 (1991).

Lavoie, A. M. & Mizumori, S. J. Y. Spatial-, movement- and reward-sensitive discharge by medial ventral striatum neurons of rats. Brain Res. 638, 157–168 ( 1994).

Bowman, E. M., Aigner, T. G. & Richmond, B. J. Neural signals in the monkey ventral striatum related to motivation for juice and cocaine rewards. J. Neurophysiol. 75, 1061–1073 (1996).

Shidara, M., Aigner, T. G. & Richmond, B. J. Neuronal signals in the monkey ventral striatum related to progress through a predictable series of trials. J. Neurosci. 18, 2613–2625 ( 1998).

Matsumura, M., Kojima, J., Gardiner, T. W. & Hikosaka, O. Visual and oculomotor functions of monkey subthalamic nucleus. J. Neurophysiol. 67, 1615–1632 (1992).

Schultz, W. Activity of pars reticulata neurons of monkey substantia nigra in relation to motor, sensory and complex events. J. Neurophysiol. 55, 660–677 (1986).

Niki, H., Sakai, M. & Kubota, K. Delayed alternation performance and unit activity of the caudate head and medial orbitofrontal gyrus in the monkey. Brain Res. 38, 343–353 ( 1972).

Niki, H. & Watanabe, M. Prefrontal and cingulate unit activity during timing behaviour in the monkey. Brain Res. 171 , 213–224 (1979).

Rosenkilde, C. E., Bauer, R. H. & Fuster, J. M. Single cell activity in ventral prefrontal cortex of behaving monkeys. Brain Res. 209, 375 –394 (1981).

Watanabe, M. The appropriateness of behavioural responses coded in post-trial activity of primate prefrontal units. Neurosci. Lett. 101, 113–117 (1989).

Tremblay, L. & Schultz, W. Reward-related neuronal activity during go–no go task performance in primate orbitofrontal cortex. J. Neurophysiol. 83, 1864–1876 (2000).

Niki, H. & Watanabe, M. Cingulate unit activity and delayed response. Brain Res. 110, 381– 386 (1976).

Nishijo, H., Ono, T. & Nishino, H. Single neuron responses in amygdala of alert monkey during complex sensory stimulation with affective significance. J. Neurosci. 8, 3570–3583 ( 1988).A paper written by one of the few groups investigating neurons in the primate amygdala in relation to reward-related stimuli. They reported a number of different responses to the presentation of natural rewards.

Nakamura, K., Mikami, A. & Kubota, K. Activity of single neurons in the monkey amygdala during performance of a visual discrimination task. J. Neurophysiol. 67, 1447–1463 (1992).

Burton, M. J., Rolls, E. T. & Mora, F. Effects of hunger on the responses of neurons in the lateral hypothalamus to the sight and taste of food. Exp. Neurol. 51, 668–677 ( 1976).

Tremblay, L. & Schultz, W. Relative reward preference in primate orbitofrontal cortex. Nature 398, 704– 708 (1999).

Pratt, W. E. & Mizumori, S. J. Y. Characteristics of basolateral amygdala neuronal firing on a spatial memory task involving differential reward . Behav. Neurosci. 112, 554– 570 (1998).

Thorpe, S. J., Rolls, E. T. & Maddison, S. The orbitofrontal cortex: neuronal activity in the behaving monkey. Exp. Brain Res. 49, 93– 115 (1983).

Rolls, E. T., Murzi, E., Yaxley, S., Thorpe, S. J. & Simpson, S. J. Sensory-specific satiety: food-specific reduction in responsiveness of ventral forebrain neurons after feeding in the monkey . Brain Res. 368, 79–86 (1986).

Rolls, E. T., Sienkiewicz, Z. J. & Yaxley, S. Hunger modulates the responses to gustatory stimuli of single neurons in the caudolateral orbitofrontal cortex of the macaque monkey. Eur. J. Neurosci. 1, 53– 60 (1989).

Rolls, E. T., Scott, T. R., Sienkiewicz, Z. J. & Yaxley, S. The responsiveness of neurons in the frontal opercular gustatory cortex of the macaque monkey is independent of hunger. J. Physiol. 397, 1–12 (1988).

Apicella, P., Legallet, E. & Trouche, E. Responses of tonically discharging neurons in the monkey striatum to primary rewards delivered during different behavioural states . Exp. Brain Res. 116, 456– 466 (1997).

Apicella, P., Ravel, S., Sardo, P. & Legallet, E. Influence of predictive information on responses of tonically active neurons in the monkey striatum. J. Neurophysiol. 80, 3341– 3344 (1998).

Apicella, P., Legallet, E. & Trouche, E. Responses of tonically discharging neurons in monkey striatum to visual stimuli presented under passive conditions and during task performance. Neurosci. Lett. 203, 147– 150 (1996).

Williams, G. V., Rolls, E. T., Leonard, C. M. & Stern, C. Neuronal responses in the ventral striatum of the behaving monkey. Behav. Brain Res. 55, 243–252 (1993).

Aosaki, T. et al. Responses of tonically active neurons in the primate's striatum undergo systematic changes during behavioural sensorimotor conditioning. J. Neurosci. 14, 3969–3984 (1994).

Apicella, P., Scarnati, E., Ljungberg, T. & Schultz, W. Neuronal activity in monkey striatum related to the expectation of predictable environmental events. J. Neurophysiol. 68, 945–960 (1992).

Schultz, W., Apicella, P., Scarnati, E. & Ljungberg, T. Neuronal activity in monkey ventral striatum related to the expectation of reward. J. Neurosci. 12, 4595– 4610 (1992).

Schoenbaum, G., Chiba, A. A. & Gallagher, M. Orbitofrontal cortex and basolateral amygdala encode expected outcomes during learning. Nature Neurosci. 1, 155–159 (1998).

Hikosaka, K. & Watanabe, M. Delay activity of orbital and lateral prefrontal neurons of the monkey varying with different rewards. Cerebral Cortex 10, 263–271 (2000).

Hollerman, J. R., Tremblay, L. & Schultz, W. Influence of reward expectation on behavior-related neuronal activity in primate striatum. J. Neurophysiol. 80, 947–963 (1998).

Tremblay, L., Hollerman, J. R. & Schultz, W. Modifications of reward expectation-related neuronal activity during learning in primate striatum. J. Neurophysiol. 80, 964–977 ( 1998).

Tremblay, L. & Schultz, W. Modifications of reward expectation-related neuronal activity during learning in primate orbitofrontal cortex. J. Neurophysiol. 83, 1877–1885 (2000).

Dickinson, A. & Balleine, B. Motivational control of goal-directed action. Animal Learn. Behav. 22, 1– 18 (1994).

Okano, K. & Tanji, J. Neuronal activities in the primate motor fields of the agranular frontal cortex preceding visually triggered and self-paced movement. Exp. Brain Res. 66, 155–166 (1987).

Romo, R. & Schultz, W. Neuronal activity preceding self-initiated or externally timed arm movements in area 6 of monkey cortex. Exp. Brain Res. 67, 656–662 (1987).

Kurata, K. & Wise, S. P. Premotor and supplementary motor cortex in rhesus monkeys: neuronal activity during externally- and internally-instructed motor tasks. Exp. Brain Res. 72, 237– 248 (1988).

Schultz, W. & Romo, R. Neuronal activity in the monkey striatum during the initiation of movements. Exp. Brain Res. 71, 431–436 (1988).

Watanabe, M. Reward expectancy in primate prefrontal neurons. Nature 382, 629–632 (1996). The first demonstration that expected rewards influence the behaviour-related activity of neurons in a manner that is compatible with a goal-directed account. Neurons in primate dorsolateral prefrontal cortex show different activities depending on the expected reward during a typical prefrontal spatial delayed response task.

Leon, M. I. & Shadlen, M. N. Effect of expected reward magnitude on the responses of neurons in the dorsolateral prefrontal cortex of the macaque . Neuron 24, 415–425 (1999).

Kawagoe, R., Takikawa, Y. & Hikosaka, O. Expectation of reward modulates cognitive signals in the basal ganglia. Nature Neurosci. 1, 411 –416 (1998).

Liu, Z. & Richmond, B. J. Response differences in monkey TE and perirhinal cortex: stimulus association related to reward schedules . J. Neurophysiol. 83, 1677– 1692 (2000).The first demonstration of the prominent reward relationships in neurons of temporal cortex. Neuronal responses to task stimuli in the primate perirhinal cortex were profoundly affected by the distance to reward, whereas neuronal responses in the neighbouring TE area predominantly reflected the visual features of the stimuli.

Platt, M. L. & Glimcher, P. W. Neural correlates of decision variables in parietal cortex. Nature 400, 233–238 (1999).Neurons in primate parietal association cortex were sensitive to two key variables of game theory and decision making: the quantity and the probability of the outcomes. Based on studies of choice behaviour and decision making in human economics and animal psychology, this is the first application of principles of decision theory in primate neurophysiology.

Shima, K. & Tanji, J. Role for cingulate motor area cells in voluntary movement selection based on reward. Science 282, 1335–1338 (1998). Neurons in the primate cingulate motor area were active when animals switched to a different movement when continuing to perform the current movement would have produced less reward. This study is interesting from the point of view of movement selection and also for the influence of rewards on behavioural choices.

Kelley, A. E. Functional specificity of ventral striatal compartments in appetitive behaviours . Ann. NY Acad. Sci. 877, 71– 90 (1999).

Carelli, R. M., King, V. C., Hampson, R. E. & Deadwyler, S. A. Firing patterns of nucleus accumbens neurons during cocaine self-administration . Brain Res. 626, 14–22 (1993).The first demonstration of drug-related changes in neuronal activity in one of the key structures of drug addiction, the nucleus accumbens. Two important neuronal patterns were found in the rat: activity preceding lever presses that led to acquisition of the drug and activity following drug delivery. These findings have been consistently reproduced by other goups.

Chang, J. Y., Sawyer, S. F., Lee, R. S. & Woodward, D. J. Electrophysiological and pharmacological evidence for the role of the nucleus accumbens in cocaine self-administration in freely moving rats. J. Neurosci. 14, 1224–1244 (1994).

Carelli, R. M. & Deadwyler, S. A. A comparison of nucleus accumbens neuronal firing patterns during cocaine self-administration and water reinforcement in rats. J. Neurosci. 14, 7735–7746 (1994).

Chang, J. Y., Janak, P. H. & Woodward, D. J. Comparison of mesocorticolimbic neuronal responses during cocaine and heroin self-administration in freely moving rats. J. Neurosci. 18, 3098–3115 (1998).

Carelli, R. M., Ijames, S. G. & Crumling, A. J. Evidence that separate neural circuits in the nucleus accumbens encode cocaine versus 'natural' (water and food) reward. J. Neurosci. 20, 4255–4266 (2000).

Chang, J. Y., Paris, J. M., Sawyer, S. F., Kirillov, A. B. & Woodward, D. J. Neuronal spike activity in rat nucleus accumbens during cocaine self-administration under different fixed-ratio schedules. J. Neurosci. 74, 483– 497 (1996).

Peoples, L. L., Uzwiak, A. J., Gee, F. & West, M. O. Operant behaviour during sessions of intravenous cocaine infusion is necessary and sufficient for phasic firing of single nucleus accumbens neurons. Brain Res. 757, 280–284 ( 1997).

West, M. O., Peoples, L. L., Michael, A. J., Chapin, J. K. & Woodward, D. J. Low-dose amphetamine elevates movement-related firing of rat striatal neurons. Brain Res. 745, 331–335 (1997).

Peoples, L. L. & West, M. O. Phasic firing of single neurons in the rat nucleus accumbens correlated with the timing of intravenous cocaine self-administration. J. Neurosci. 16, 3459–3473 (1996).

Peoples, L. L., Gee, F., Bibi, R. & West, M. O. Phasic firing time locked to cocaine self-infusion and locomotion: dissociable firing patterns of single nucleus accumbens neurons in the rat. J. Neurosci. 18, 7588–7598 (1998).

Calabresi, P., Maj, R., Pisani, A., Mercuri, N. B. & Bernardi, G. Long-term synaptic depression in the striatum: physiological and pharmacological characterization. J. Neurosci. 12, 4224–4233 (1992).

Otmakhova, N. A. & Lisman, J. E. D1/D5 dopamine receptor activation increases the magnitude of early long-term potentiation at CA1 hippocampal synapses. J. Neurosci. 16, 7478–7486 (1996).

Wickens, J. R., Begg, A. J. & Arbuthnott, G. W. Dopamine reverses the depression of rat corticostriatal synapses which normally follows high-frequency stimulation of cortex in vitro. Neuroscience 70, 1– 5 (1996).

Calabresi, P. et al. Abnormal synaptic plasticity in the striatum of mice lacking dopamine D2 receptors. J. Neurosci. 17, 4536–4544 (1997).