Externally Provided Rewards Increase Internal Preference, but Not as Much as Preferred Ones Without Extrinsic Rewards
Tóm tắt
It is well known that preferences are formed through choices, known as choice-induced preference change (CIPC). However, whether value learned through externally provided rewards influences the preferences formed through CIPC remains unclear. To address this issue, we used tasks for decision-making guided by reward provided by the external environment (externally guided decision-making; EDM) and for decision-making guided by one’s internal preference (internally guided decision-making; IDM). In the IDM task, we presented stimuli with learned value in the EDM and novel stimuli to examine whether the value in the EDM affects preferences. Stimuli reinforced by rewards given in the EDM were reflected in the IDM’s initial preference and further increased through CIPC in the IDM. However, such stimuli were not as strongly preferred as the most preferred novel stimulus in the IDM (superiority of intrinsically learned values; SIV), suggesting that the values learned by the EDM and IDM differ. The underlying process of this phenomenon is discussed in terms of the fundamental self-hypothesis.
Từ khóa
Tài liệu tham khảo
Akaishi, R., Umeda, K., Nagase, A., & Sakai, K. (2014). Autonomous mechanism of internal choice estimate underlies decision inertia. Neuron, 81, 195–206. https://doi.org/10.1016/j.neuron.2013.10.018
Aridan, N., Pelletier, G., Fellows, L. K., & Schonberg, T. (2019). Is ventromedial prefrontal cortex critical for behavior change without external reinforcement? Neuropsychologia, 124, 208–215. https://doi.org/10.1016/j.neuropsychologia.2018.12.008
Bai, Y., Nakao, T., Xu, J., Qin, P., Chaves, P., Heinzel, A., Duncan, N., Lane, T., Yen, N. S., Tsai, S. Y., & Northoff, G. (2016). Resting state glutamate predicts elevated pre-stimulus alpha during self-relatedness: A combined EEG-MRS study on “rest-self overlap.” Social Neuroscience, 11(3). https://doi.org/10.1080/17470919.2015.1072582
Bechara, A., Damasio, H., Tranel, D., & Damasio, A. R. (1997). Deciding advantageously before knowing the advantageous strategy. Science, 275, 1293–1295. https://doi.org/10.1126/science.275.5304.1293
Bechara, A., Damasio, H., Tranel, D., & Damasio, A. R. (2005). The Iowa Gambling Task and the somatic marker hypothesis: Some questions and answers. Trends in Cognitive Sciences, 9, 159–162. https://doi.org/10.1016/j.tics.2005.02.002
Behrens, T. E. J., Woolrich, M. W., Walton, M. E., & Rushworth, M. F. S. (2007). Learning the value of information in an uncertain world. Nature Neuroscience, 10(9), 1214–1221. https://doi.org/10.1038/nn1954
Biele, G., Rieskamp, J., Krugel, L. K., & Heekeren, H. R. (2011). The neural basis of following advice. PLoS Biology, 9(6), e1001089. https://doi.org/10.1371/journal.pbio.1001089
Bouton, M. E., & Moody, E. W. (2004). Memory processes in classical conditioning. Neuroscience and Biobehavioral Reviews, 28(7), 663–674. https://doi.org/10.1016/j.neubiorev.2004.09.001
Brehm, J. W. (1956). Postdecision changes in the desirability of alternatives. Journal of Abnormal and Social Psychology, 52(3), 384–389. https://doi.org/10.1037/h0041006
Camille, N., Griffiths, C. A., Vo, K., Fellows, L. K., & Kable, J. W. (2011). Ventromedial frontal lobe damage disrupts value maximization in humans. Journal of Neuroscience, 31(20), 7527–7532. https://doi.org/10.1523/JNEUROSCI.6527-10.2011
Chen, M. K., & Risen, J. L. (2010). How choice affects and reflects preferences: Revisiting the free-choice paradigm. Journal of Personality and Social Psychology, 99(4), 573–594. https://doi.org/10.1037/a0020217
Colosio, M., Shestakova, A., Nikulin, V. V., Blagovechtchenski, E., & Klucharev, V. (2017). Neural mechanisms of cognitive dissonance (Revised): An EEG study. Journal of Neuroscience, 37(20), 5074–5083. https://doi.org/10.1523/JNEUROSCI.3209-16.2017
Daw, N. D., & Doya, K. (2006). The computational neurobiology of learning and reward. Current Opinion in Neurobiology, 16(2), 199–204. https://doi.org/10.1016/j.conb.2006.03.006
Dayan, P., & Abbott, L. F. (2001). Theoretical neuroscience: Computational and mathematical modeling of neural systems. MIT Press.
Dayan, P., & Balleine, B. W. (2002). Reward, motivation, and reinforcement learning. Neuron, 36(2), 285–298. https://doi.org/10.1016/S0896-6273(02)00963-7
de Greck, M., Rotte, M., Paus, R., Moritz, D., Thiemann, R., Proesch, U., Bruer, U., Moerth, S., Tempelmann, C., Bogerts, B., & Northoff, G. (2008). Is our self based on reward? Self-relatedness recruits neural activity in the reward system. NeuroImage, 39(4), 2066–2075. https://doi.org/10.1016/j.neuroimage.2007.11.006
Dickinson, A., & Balleine, B. (1995). Motivational control of instrumental action. Current Directions in Psychological Science, 4(5), 162–167. https://doi.org/10.1111/1467-8721.ep11512272
Endo, N., Saiki, J., Nakao, Y., & Saito, H. (2003). Perceptual judgments of novel contour shapes and hierarchical descriptions of geometrical properties. Jpn J Psychol, 74, 346–353. https://doi.org/10.4992/jjpsy.74.346
Enzi, B., de Greck, M., Prösch, U., Tempelmann, C., & Northoff, G. (2009). Is our self nothing but reward? Neuronal overlap and distinction between reward and personal relevance and its relation to human personality. PLoS ONE, 4(12), e8429. https://doi.org/10.1371/journal.pone.0008429
Fellinger, R., Klimesch, W., Gruber, W., Freunberger, R., & Doppelmayr, M. (2011). Pre-stimulus alpha phase-alignment predicts P1-amplitude. Brain Research Bulletin, 85(6). https://doi.org/10.1016/j.brainresbull.2011.03.025
Fellows, L. K., & Farah, M. J. (2007). The role of ventromedial prefrontal cortex in decision making: Judgment under uncertainty or judgment per se? Cerebral Cortex, 17(11), 2669–2674. https://doi.org/10.1093/cercor/bhl176
Festinger, L. (1957). A theory of social cognitive dissonance. Stanford University Press.
Gelman, A., Meng, X. L., & Stern, H. (1996). Posterior predictive assessment of model fitness via realized discrepancies. Statistica Sinica, 6(4), 733–760. http://www.jstor.org/stable/24306036
Gläscher, J., Daw, N., Dayan, P., & O’Doherty, J. P. (2010). States versus rewards: Dissociable neural prediction error signals underlying model-based and model-free reinforcement learning. Neuron, 66(4), 585–594. https://doi.org/10.1016/j.neuron.2010.04.016
Gluth, S., Rieskamp, J., & Büchel, C. (2014). Neural evidence for adaptive strategy selection in value-based decision-making. Cerebral Cortex, 24(8), 2009–2021. https://doi.org/10.1093/cercor/bht049
Guitart-Masip, M., Duzel, E., Dolan, R., & Dayan, P. (2014). Action versus valence in decision making. Trends in Cognitive Sciences, 18(4). https://doi.org/10.1016/j.tics.2014.01.003
Hampton, A. N., Bossaerts, P., & O’Doherty, J. P. (2008). Neural correlates of mentalizing-related computations during strategic interactions in humans. Proceedings of the National Academy of Sciences of the United States of America, 105(18), 6741–6746. https://doi.org/10.1073/pnas.0711099105
Hauser, T. U., Iannaccone, R., Walitza, S., Brandeis, D., & Brem, S. (2015). Cognitive flexibility in adolescence: Neural and behavioral mechanisms of reward prediction error processing in adaptive decision making during development. NeuroImage, 104, 347–354. https://doi.org/10.1016/j.neuroimage.2014.09.018
Humphreys, G. W., & Sui, J. (2016). Attentional control and the self: The self-attention network (SAN). Cognitive Neuroscience, 7(1–4), 5–17. https://doi.org/10.1080/17588928.2015.1044427
Ito, M., & Doya, K. (2009). Validation of decision-making models and analysis of decision variables in the rat basal ganglia. Journal of Neuroscience, 29(31), 9861–9874. https://doi.org/10.1523/JNEUROSCI.6157-08.2009
Izuma, K., & Murayama, K. (2013). Choice-induced preference change in the free-choice paradigm: A critical methodological review. Frontiers in Psychology, 4, 41. https://doi.org/10.3389/fpsyg.2013.00041
Izuma, K., Matsumoto, M., Murayama, K., Samejima, K., Sadato, N., & Matsumoto, K. (2010). Neural correlates of cognitive dissonance and choice-induced preference change. Proceedings of the National Academy of Sciences of the United States of America, 107(51), 22014–22019. https://doi.org/10.1073/pnas.1011879108
Johansson, P., Hall, L., Tärning, B., Sikström, S., & Chater, N. (2014). Choice blindness and preference change: You will like this paper better if you (believe you) chose to read it! Journal of Behavioral Decision Making, 27(3). https://doi.org/10.1002/bdm.1807
Kass, R. E., & Raftery, A. E. (1995). Bayes factors. Journal of the American Statistical Association, 90(430), 773–795. https://doi.org/10.1080/01621459.1995.10476572
Katahira, K., Fujimura, T., Okanoya, K., & Okada, M. (2011). Decision-making based on emotional images. Frontiers in Psychology, 2, 311. https://doi.org/10.3389/fpsyg.2011.00311
Katahira, K., Yuki, S., & Okanoya, K. (2017). Model-based estimation of subjective values using choice tasks with probabilistic feedback. Journal of Mathematical Psychology, 79. https://doi.org/10.1016/j.jmp.2017.05.005
Koster, R., Duzel, E., & Dolan, R. J. (2015). Action and valence modulate choice and choice-induced preference change. PLoS ONE, 10(3), e0119682. https://doi.org/10.1371/journal.pone.0119682
Kunisato, Y., Okamoto, Y., Ueda, K., Onoda, K., Okada, G., Yoshimura, S., Suzuki, S. I., Samejima, K., & Yamawaki, S. (2012). Effects of depression on reward-based decision making and variability of action in probabilistic learning. Journal of Behavior Therapy and Experimental Psychiatry, 43(4). https://doi.org/10.1016/j.jbtep.2012.05.007
Lee, D., & Daunizeau, J. (2020). Choosing what we like vs liking what we choose: How choice-induced preference change might actually be instrumental to decision-making. PLoS ONE, 15(5), e0231081. https://doi.org/10.1371/journal.pone.0231081
Lindström, B., Selbing, I., Molapour, T., & Olsson, A. (2014). Racial bias shapes social reinforcement learning. Psychological Science, 25(3), 711–719. https://doi.org/10.1177/0956797613514093
Marco-Pallarés, J., Cucurell, D., Cunillera, T., García, R., Andrés-Pueyo, A., Münte, T. F., & Rodríguez-Fornells, A. (2008). Human oscillatory activity associated to reward processing in a gambling task. Neuropsychologia, 46(1), 241–248. https://doi.org/10.1016/j.neuropsychologia.2007.07.016
Marco-Pallarés, J., Münte, T. F., & Rodríguez-Fornells, A. (2015). The role of high-frequency oscillatory activity in reward processing and learning. Neuroscience and Biobehavioral Reviews, 49, 1–7. https://doi.org/10.1016/j.neubiorev.2014.11.014
Miyagi, M., Miyatani, M., & Nakao, T. (2017). Relation between choice-induced preference change and depression. PLoS ONE, 12(6), e0180041. https://doi.org/10.1371/journal.pone.0180041
Nakamura, K., & Kawabata, H. (2013). I choose, therefore i like: Preference for faces induced by arbitrary choice. PLoS ONE, 8(8), e72071. https://doi.org/10.1371/journal.pone.0072071
Nakao, T., Ohira, H., & Northoff, G. (2012). Distinction between externally vs. Internally guided decision-making: Operational differences, meta-analytical comparisons and their theoretical implications. Frontiers in Neuroscience, 6, 1–26. https://doi.org/10.3389/fnins.2012.00031
Nakao, T., Bai, Y., Nashiwa, H., & Northoff, G. (2013). Resting-state EEG power predicts conflict-related brain activity in internally guided but not in externally guided decision-making. NeuroImage, 66, 9–21. https://doi.org/10.1016/j.neuroimage.2012.10.034
Nakao, T., Kanayama, N., Katahira, K., Odani, M., Ito, Y., Hirata, Y., Nasuno, R., Ozaki, H., Hiramoto, R., Miyatani, M., & Northoff, G. (2016). Post-response βγ power predicts the degree of choice-based learning in internally guided decision-making. Scientific Reports, 6, 32477. https://doi.org/10.1038/srep32477
Nakao, T., Miyagi, M., Hiramoto, R., Wolff, A., Gomez-Pilar, J., Miyatani, M., & Northoff, G. (2019). From neuronal to psychological noise – Long-range temporal correlations in EEG intrinsic activity reduce noise in internally-guided decision making. NeuroImage, 201, 116015. https://doi.org/10.1016/j.neuroimage.2019.116015
Niv, Y., Edlund, J. A., Dayan, P., & O’Doherty, J. P. (2012). Neural prediction errors reveal a risk-sensitive reinforcement-learning process in the human brain. Journal of Neuroscience, 32(2), 551–562. https://doi.org/10.1523/JNEUROSCI.5498-10.2012
Northoff, G. (2016). Is the self a higher-order or fundamental function of the brain? The “basis model of self-specificity” and its encoding by the brain’s spontaneous activity. Cognitive Neuroscience, 7(1–4), 203–222. https://doi.org/10.1080/17588928.2015.1111868
Northoff, G., Vatansever, D., Scalabrini, A., & Stamatakis, E. A. (2022). Ongoing brain activity and its role in cognition: Dual versus baseline models. Neuroscientist, 29(4). https://doi.org/10.1177/10738584221081752
O’Doherty, J. P., Hampton, A., & Kim, H. (2007). Model-based fMRI and its application to reward learning and decision making. Annals of the New York Academy of Sciences, 1104, 35–53. https://doi.org/10.1196/annals.1390.022
Ohira, H., Fukuyama, S., Kimura, K., Nomura, M., Isowa, T., Ichikawa, N., Matsunaga, M., Shinoda, J., & Yamada, J. (2009). Regulation of natural killer cell redistribution by prefrontal cortex during stochastic learning. NeuroImage, 47(3). https://doi.org/10.1016/j.neuroimage.2009.04.088
Ohira, H., Ichikawa, N., Nomura, M., Isowa, T., Kimura, K., Kanayama, N., Fukuyama, S., Shinoda, J., & Yamada, J. (2010). Brain and autonomic association accompanying stochastic decision-making. NeuroImage, 49(1). https://doi.org/10.1016/j.neuroimage.2009.07.060
Palminteri, S., Lefebvre, G., Kilford, E. J., & Blakemore, S. J. (2017). Confirmation bias in human reinforcement learning: Evidence from counterfactual feedback processing. PLoS Computational Biology, 13(8), e1005684. https://doi.org/10.1371/journal.pcbi.1005684
Peirce, J., Gray, J. R., Simpson, S., MacAskill, M., Höchenberger, R., Sogo, H., Kastman, E., & Lindeløv, J. K. (2019). PsychoPy2: Experiments in behavior made easy. Behavior Research Methods, 51(1), 195–203. https://doi.org/10.3758/s13428-018-01193-y
Qin, P., Wang, M., & Northoff, G. (2020). Linking bodily, environmental and mental states in the self—A three-level model based on a meta-analysis. Neuroscience and Biobehavioral Reviews, 115, 77–95. https://doi.org/10.1016/j.neubiorev.2020.05.004
R Core Team. (2020). R: A language and environment for statistical computing. R Foundation for Statistical Computing. http://www.R-project.org
Rescorla, R. A., & Wagner, A. R. (1972). A theory of Pavlovian conditioning: Variations in the effectiveness of reinforcement and nonreinforcement. In A. H. Black & W. F. Prokasy (Eds.), Classical conditioning II: Current research and theory (pp. 64–99). Appleton-Century-Crofts.
Schönberg, T., Daw, N. D., Joel, D., & O’Doherty, J. P. (2007). Reinforcement learning signals in the human striatum distinguish learners from nonlearners during reward-based decision making. Journal of Neuroscience, 27(47), 12860–12867. https://doi.org/10.1523/JNEUROSCI.2496-07.2007
Stan Development Team. (2020). RStan: The R interface to Stan. R package version 2.21.2. http://mc-stan.org/
Stevenson, J. G., & Clayton, F. L. (1970). A response duration schedule: Effects of training, extinction, and deprivation. Journal of the Experimental Analysis of Behavior, 13(3), 359–367. https://doi.org/10.1901/jeab.1970.13-359
Sugawara, M., & Katahira, K. (2021). Dissociation between asymmetric value updating and perseverance in human reinforcement learning. Scientific Reports, 11(1), 3574. https://doi.org/10.1038/s41598-020-80593-7
Sui, J., & Gu, X. (2017). Self as object: Emerging trends in self research. Trends in Neurosciences, 40(11), 643–653. https://doi.org/10.1016/j.tins.2017.09.002
Sui, J., & Humphreys, G. W. (2015). The integrative self: How self-reference integrates perception and memory. Trends in Cognitive Sciences, 19(12), 719–728. https://doi.org/10.1016/j.tics.2015.08.015
Sutton, R. S., & Barto, A. G. (1998). Introduction to reinforcement learning. MIT Press.
Thorndike, E. L. (1898). Animal intelligence: An experimental study of the associative processes in animals. The Psychological Review: Monograph Supplements, 2(4), i–109. https://doi.org/10.1037/h0092987
Ugazio, G., Grueschow, M., Polania, R., Lamm, C., Tobler, P., & Ruff, C. (2021). Neuro-computational foundations of moral preferences. Social Cognitive and Affective Neuroscience, 17, 253–265. https://doi.org/10.1093/scan/nsab100
Vinckier, F., Rigoux, L., Kurniawan, I. T., Hu, C., Bourgeois-Gironde, S., Daunizeau, J., & Pessiglione, M. (2019). Sour grapes and sweet victories: How actions shape preferences. PLoS Computational Biology, 15(1), e1006499. https://doi.org/10.1371/journal.pcbi.1006499
Watanabe, S. (2013). A widely applicable Bayesian information criterion. Journal of Machine Learning Research, 14(1), 867–897.
Wilson, R. C., & Collins, A. G. E. (2019). Ten simple rules for the computational modeling of behavioral data. eLife, 8, e49547. https://doi.org/10.7554/eLife.49547
Wolff, A., Gomez-Pilar, J., Nakao, T., & Northoff, G. (2019). Interindividual neural differences in moral decision-making are mediated by alpha power and delta/theta phase coherence. Scientific Reports, 9(1), 4432. https://doi.org/10.1038/s41598-019-40743-y
Yacubian, J., Gläscher, J., Schroeder, K., Sommer, T., Braus, D. F., & Büchel, C. (2006). Dissociable systems for gain- and loss-related value predictions and errors of prediction in the human brain. Journal of Neuroscience, 26(37), 9530–9537. https://doi.org/10.1523/JNEUROSCI.2915-06.2006
Zhang, Y., Wang, F., & Sui, J. (2022). Decoding individual differences in self-prioritization from the resting-state functional connectome. Research Square. https://doi.org/10.21203/rs.3.rs-2204324/v1
Zhu, J., Hashimoto, J., Katahira, K., Hirakawa, M., & Nakao, T. (2021). Computational modeling of choice-induced preference change: A reinforcement-learning-based approach. PLoS ONE, 16(1), e0244434. https://doi.org/10.1371/journal.pone.0244434