Available online at www.sciencedirect.com
Learning and motivation in the human striatum Daphna Shohamy The past decade has seen a dramatic change in our understanding of the role of the striatum in behavior. Early perspectives emphasized a role for the striatum in habitual learning of stimulus–response associations and sequences of actions. Recent advances from human neuroimaging research suggest a broader role for the striatum in motivated learning. New findings demonstrate that the striatum represents multiple learning signals and highlight the contribution of the striatum across many cognitive domains and contexts. Recent findings also emphasize interactions between the striatum and other specialized brain systems for learning. Together, these findings suggest that the striatum contributes to a distributed network that learns to select actions based on their predicted value in order to optimize behavior. Address Department of Psychology, Columbia University, New York, NY 10025, United States Corresponding author: Shohamy, Daphna (
[email protected])
Current Opinion in Neurobiology 2011, 21:408–414 This review comes from a themed issue on Behavioral and Cognitive Neuroscience Edited by Ann Graybiel and Richard Morris Available online 12th June 2011 0959-4388/$ – see front matter # 2011 Elsevier Ltd. All rights reserved. DOI 10.1016/j.conb.2011.05.009
Introduction It is hard to believe that only a decade ago it was still hotly debated whether the striatum or other basal ganglia nuclei contribute to cognitive function, in addition to their obvious role in motor control. In recent years the question has reversed itself: Given the wealth of reports indicating a broad role for the striatum in many aspects of cognitive function – ranging from sequence learning to social cooperation – it seems that the crucial question now is: when does the striatum not contribute to cognition? That is, how can we understand the specific and selective role of the striatum in cognitive function? Recent reports indicating that the striatum contributes to learning and motivation have been largely inspired by findings from systems and computational neuroscience demonstrating an important role for the striatum and its dopaminergic inputs in learning to predict reward. Current Opinion in Neurobiology 2011, 21:408–414
Together with extensive converging evidence indicating a role for the basal ganglia in habitual learning of stimulus–response associations [1–4], addiction [3], and movement [5,6], these findings emphasized a crucial role for this circuit in learning to predict rewards and acting to obtain them. Here, we focus on recent human functional magnetic resonance imaging (fMRI) studies of the striatum. Together, these studies suggest an even broader role for the striatum in learning than previously recognized. This conclusion emerges from three complementary directions. First, recent fMRI studies have aimed to characterize the specific nature of learning signals in the striatum, raising questions about its selective role in reward-driven habits. Second, emerging data indicate that the striatum contributes to a wide range of sophisticated behaviors and to the representation of many kinds of outcomes. Third, new studies are taking advantage of the ability of fMRI to characterize broad neural networks and are providing a more detailed understanding of how interactions between the striatum and other specialized learning systems in the brain shape behavior. The following review focuses on these novel insights, considers the challenges they pose for developing an integrated understanding of the role of the striatum in learning, and discusses promising future directions that may help address these challenges.
The striatum and feedback-based learning Understanding the selective role of the striatum in human learning
In the past decade there have been substantial advances in understanding the role of midbrain dopamine neurons and their striatal targets in learning from reward (for review [7]). The emergence of these findings has had an enormous influence on our understanding of the neural bases of learning. Although it had been previously recognized that the striatum plays an important role in learning, the common view was that it is specialized for implicit (i.e. unconscious) learning of habits (e.g. [1,2]). The novel emphasis on reinforcement, however, suggested a special role for the striatum in learning that is based on trial-by-trial feedback, regardless of whether it is implicit or not. This idea and the data that support it led to a novel conceptualization of the role of the striatum and helped resolved previous inconsistencies in the literature (e.g. [8,9]). These ideas also led a wealth of fMRI studies in humans, which aimed to identify the selective role of the striatum in learning. Converging evidence indicates that the striatum supports the ability to gradually learn, based www.sciencedirect.com
The striatum and Learning Shohamy 409
on trial-by-trial feedback, to update responses (e.g. [10– 13]). Within this context, distinct striatal subregions are thought to contribute to learning in different ways. In humans, the most common distinction has been between a role for the ventral striatum in responding to reward prediction errors, and for the dorsal striatum in learning from these prediction errors to select appropriate actions (e.g. [14]) (for a more detailed discussion of how subregions of striatum support learning, see Bornstein and Daw, this issue).
that a reward prediction error – computationally equivalent to that signaled by midbrain dopamine neurons – correlates with BOLD activation in the ventral striatum, presumably reflecting dopaminergic inputs there (e.g. [22–26]; for review [27]). This signal is thought to be central to the learning of habitual stimulus-reward associations and is referred to computationally as ‘model-free’ learning (because it involves learning directly from experience rather than using an existing model of the task structure).
Feedback-based learning of habits, goals, or both?
Expanding on these findings, recent studies have been using this approach to compare different kinds of learning signals and their neural substrates [8,20,21]. For example, one recent study revealed that activation in the striatum during learning was related to multiple forms of learning [20]. Specifically, activation in the ventral striatum correlated with ‘model-free’ learning of habitual stimulus-reward associations, as shown previously. But, an overlapping region in the ventral striatum also correlated with so-called ‘model-based’ learning, which involves more abstract learning about task structure and states, rather than stimulus–response associations (for a related study see also [21]).
An important question which these studies leave open is to what extent is feedback-based learning in the human striatum truly habitual in nature. To get at this question, one approach has been to test, after learning, how people use what they learned. Habits are typically defined as behavior that is not sensitive to the value of its outcomes and in that respect is rigid or inflexible. In rodent conditioning studies, a series of studies demonstrate a key role for dorsolateral striatum in driving habitual responses that are expressed even when the animal no longer desires their outcomes (assessed using a devaluation procedure; for review [15]). Adapting this approach to humans learning to respond for food rewards, fMRI has revealed that the emergence of habits is associated with changes in activation in the dorsal striatum during learning [16] (see Figure 1). Using a somewhat different approach, another recent study has shown that the dorsal striatum supports learning of inflexible stimulus–response associations particularly when people are distracted while they are learning [17]. Here, learning involved cognitive feedback (rather than primary rewards); habits were defined as the inability to flexibly express the learned associations on a later probe test. Notably, in both of these studies, as in many others, the selective role for the striatum in habit learning is thought to be complemented by other distinct brain regions that are specialized for other forms of learning (e.g. goal-directed behaviors in the orbitofrontal cortex (OFC) [18] or mnemonically flexible behaviors in the hippocampus [2,17,19]). Recent findings, however, raise questions about this parcellation and especially about the unique role of the striatum in feedback-based learning of habits, as detailed below. Rather than focusing on post-learning behaviors, these studies have combined computational models with fMRI to identify brain regions where trialby-trial fluctuations in the BOLD signal vary with specific computational variables, as learning takes place [11,20,21–24]. Early fMRI studies in humans focused on classical conditioning paradigms and primary rewards such as juice, literally replicating paradigms from animal electrophysiological studies [23,24]. The ubiquitous finding is www.sciencedirect.com
This surprising result stands in contrast to other evidence for clear anatomical segregation between different learning signals, both within the striatum (e.g. [14]; for review see [15]) or between the striatum and other brain regions (e.g. [13,21]; for review see [18]). A challenge for future research will be to reconcile these disparate findings towards a unified account of the role of the striatum in multiple forms of learning.
The striatum supports learning and motivation across domains In parallel to the attempts to characterize distinct types of learning and their neural substrates, there has also been much effort to determine whether these neurobiological learning mechanisms encompass the wealth of stimuli, outcomes and contexts which are characteristic of, and in some cases unique to, human learning. Learning signals across domains
There is now ample evidence that reward prediction error signals in the ventral striatum are generated by expectations and receipt of a wide variety of rewards, including money, verbal and cognitive feedback, food, and music [12,25,26,28–36]. Of particular importance for neurobiological theories of learning, recent findings in both animals and humans indicate that the striatum supports learning to predict both positive and negative outcomes [37–42]. Beyond simple reinforcement, however, these signals also account for a wide variety of complex social behaviors, including reciprocity in rewards [43], social fairness [44], social learning [45–47], and overbidding in auctions [41]. Current Opinion in Neurobiology 2011, 21:408–414
410 Behavioral and Cognitive Neuroscience
Figure 1
(a)
Multiple learning signals
(b)
Motivation
Reward learing (model-free)
Habits (devaluation)
Motivated memory formation
Reward learing (model-based)
Habits (inflexible)
“Over”motivation Goal-directed memory retrieval
Current Opinion in Neurobiology
Location of learning and motivation signals in the striatum. (a) Multiple learning signals that relate to different computations elicit nearby and partly overlapping activation in the ventral striatum [20], while responses specifically linked to habitual behavior elicit activation in more posterior regions [16,17]. Peak coordinates from published papers were plotted on coronal slices of an average structural MRI. (b) Effects of motivation on activation in the striatum across multiple paradigms. The plotted signals were related to motivation and goal achievement and, notably, were not directly related to the presentation of feedback or rewards [49,50,54].
Motivation to learn
Collectively, these findings suggest a broad role for the striatum in learning to modify actions based on predicted outcomes. This provides an obvious link to motivated, goal-directed behavior. Notably, truly motivated behavior has traditionally been dissociated from habitual responses to obtain rewards, again demonstrating quite a broad role for the striatum in many different processes. One important recent advance has been the demonstration that striatal signals encode not just extrinsic rewards, but also intrinsic motivation [48]. This study leverages a well-known psychological phenomenon whereby extrinsically controlled rewards, such as money, can hamper intrinsic motivation. Here, performance and brain activation were compared between two groups that both played a simple ‘stopwatch’ computer game that required rapid responding to cues on a computer screen. One group received performance-based rewards (money), while another group played the same game but did not receive performance-based rewards. When given the option to keep playing the game during their ‘free’ time, the group that received performance-based rewards was much less likely to choose to play compared with the group that did not receive rewards. In parallel to this behavioral difference between the groups, fMRI revealed that the decrease in motivation due to extrinsic reward was related to decreased responses in the anterior striatum and in the prefrontal cortex (PFC), suggesting a key role for the striatum in the integration of extrinsic reward value and intrinsic motivation. Other negative effects due to extrinsic reward (in this case ‘overmotivation’) also appear to operate via motivational signals in the striatum [49] (see Figure 1). Current Opinion in Neurobiology 2011, 21:408–414
The idea that the striatum may contribute to multiple mechanisms that underlie motivated behavior also has implications for more abstract forms of knowledge, such as memory for facts, events or objects [50,51–53]. Intriguingly, in such cases, activation in the ventral striatum predicted the beneficial effects of motivation on long-term memory, along with cooperative interactions between midbrain dopamine regions and the hippocampus [50]. Finally, responses in the striatum also relate to goal satisfaction in tests of episodic memory retrieval [54,55]. Notably, these learning situations do not involve repeated responses, nor extrinsic rewards, nor any outcome or feedback, and thus differ substantially from the repeatedly reinforced stimulus–response associations typically associated with the striatum (see also [56]). Feedback, motivation and learning
The demonstration of such a wide variety of motivational signals in the striatum raises questions about whether this region plays a role that is much broader than learning from feedback, or, alternatively, whether our traditional framing of feedback as explicit reinforcement is too narrow. Interestingly, in the birdsong literature it has been shown that performance itself – without additional explicit reinforcement – serves as a form of feedback that supports learned behaviors in the avian basal ganglia [57]. As in the studies reviewed above, these behaviors do not fit the canonical definition of a feedback-based learning task that typically involves the selection of a specific action in order to obtain a specific reward. Together, these varied findings emphasize the broad importance of feedback in driving behavior even when this feedback is not explicit (see also [58]).
Interactions between the striatum and other learning systems The striatum is known to interact with a wide network of cortical regions [59]. Recent years have seen exciting progress in understanding the specific role of the OFC in learning value and interacting with the striatum to guide actions (for review [18,60]). As nicely demonstrated in a recent rat lesion study, the OFC together with the ventral striatum represent not only reward value, but also reward identity, suggesting, yet again, that both these regions may encode rather richer representations of outcomes than previously thought [61]. New advances in fMRI data analysis make it increasingly possible to ask questions not only about parallel contributions of interconnected regions, but also their dynamic interactions over the course of learning. For example, one recent study used dynamic causal modeling of BOLD activity to demonstrate that striatal prediction error responses can tune plasticity in visual and motor cortex [62]. www.sciencedirect.com
The striatum and Learning Shohamy 411
Finally, there has been a surge of interest in understanding interactions between the striatum and other specialized learning systems, most notably the hippocampus [13,63–66,67,94]. Although the striatum and the hippocampus were traditionally thought to support two distinct and independent memory systems [2], an emerging literature has focused on characterizing their joint and interactive contributions to learning [13,17,65,66,68]. Anatomically, both the striatum and the hippocampus connect with PFC, a likely pathway for mediating interactions between them [59,69–72]. Functionally, both competitive and cooperative interactions between the hippocampus and the striatum, as well as parallel contributions, have been reported [17,66,68,73]. These have emerged in a number of recent studies that involve multitrial reinforcement learning paradigms (e.g. [13,28,67]). Notably, cooperative interactions between the hippocampus and the striatum also emerge in very different kinds of learning situations, even those which involve no choice or feedback, and indeed no real ‘performance’ beyond passive viewing [50,74]. Providing a possible explanation for these varied effects, a recent study reported that responses in the striatum encode value even during passive viewing, and that such responses predict later choice behavior [75]. Yet other findings indicate that in contrast to its traditional depiction as a neutral ‘cognitive’ memory store, long-term memory in the hippocampus is also modulated by dopamine, expectations and value [53,63,76,77]. Together, these findings suggest that information about affect and value may be encoded by multiple interacting brain systems for learning (for review [63]). These findings also emphasize the link between mnemonic processes, learning and decision making (for review [78]), an idea that has received recent empirical support [79].
Challenges and open questions Determining the necessary role for the striatum in learning and motivation
An important limitation of fMRI is that it cannot indicate the causal, essential role of a brain region for a particular function. Thus, moving forward, it will be crucial to test predictions that emerge from fMRI research in both animal and human lesion studies that can test the necessary role of the striatum across varied learning situations. Some important progress on this front has emerged from studies with Parkinson’s disease patients, which in its early stages involves relatively selective loss of nigro-striatal dopamine projections. Indeed, Parkinson’s patients have provided important insight into the role of the striatum in learning (for review see [80–82]), including evidence directly relevant to the role of the striatum in feedback and reward based learning [9,81,83–85]. Notably, data from Parkinson’s patients so far indicate a rather selective pattern of deficits in www.sciencedirect.com
learning from reinforcement. Future studies are necessary to test the selectivity of these effects in light of the recent broad findings from fMRI to allow a more precise understanding of when the striatum is involved, and necessary, for learning. The role for dopamine in learning in the striatum
A related challenge is to determine the crucial role of dopamine in learning and motivation. One approach to probe this question has been to use Parkinson’s disease as a model with which to directly test the role of dopamine in learning, by testing patients either on or off dopaminergic drugs [83,84,86]. Together with genetic studies and PET imaging [8,87–89,90], such studies afford insight into dopaminergic mechanisms underlying learning, providing an important complement to BOLD fMRI, which does not provide insight about dopamine, per se. Implications for disorders of learning and motivation
The many recent advances in our basic understanding of how and when the striatum contributes to learning and motivation have important implications for psychiatric and neurological disease and for changes in learning and motivation across the life span (e.g. [91–93]). Initial progress in these domains demonstrates the promising potential of leveraging basic findings from computational, neurophysiological and cognitive neuroscience to gain a better understanding of the neural mechanisms that support learning and motivation in both health and disease.
Acknowledgements Thanks to R.A. Adcock, L. Atlas, E.K. Braun, N. Daw, K. Foerde, I. Kahn, G.E. Wimmer and S. Wood for helpful comments and to support from an NSF Career Development Award, a NARSAD Young Investigator Award, and the Michael J. Fox Foundation.
References and recommended reading Papers of particular interest, published within the annual period of review, have been highlighted as: of special interest of outstanding interest 1.
Gabrieli JD: Cognitive neuroscience of human memory. Annu Rev Psychol 1998, 49:87-115.
2.
Knowlton BJ, Mangels JA, Squire LR: A neostriatal habit learning system in humans. Science 1996, 273:1399-1402.
3.
Everitt BJ, Robbins TW: Neural systems of reinforcement for drug addiction: from actions to habits to compulsion. Nat Neurosci 2005, 8:1481-1489.
4.
Jog MS, Kubota Y, Connolly CI, Hillegaart V, Graybiel AM: Building neural representations of habits. Science 1999, 286:1745-1749.
5.
Albin RL, Young AB, Penney JB: The functional anatomy of basal ganglia disorders. Trends Neurosci 1989, 12:366-375.
6.
DeLong MR: Primate models of movement disorders of basal ganglia origin. Trends Neurosci 1990, 13:281-285.
7.
Schultz W: Predictive reward signal of dopamine neurons. J Neurophysiol 1998, 80:1-27. Current Opinion in Neurobiology 2011, 21:408–414
412 Behavioral and Cognitive Neuroscience
8.
Frank MJ, Doll BB, Oas-Terpstra J, Moreno F: Prefrontal and striatal dopaminergic genes predict individual differences in exploration and exploitation. Nat Neurosci 2009, 12:1062-1068.
9.
Shohamy D, Myers CE, Grossman S, Sage J, Gluck MA, Poldrack RA: Cortico-striatal contributions to feedback-based learning: converging data from neuroimaging and neuropsychology. Brain 2004, 127:851-859.
10. Cools R, Clark L, Owen AM, Robbins TW: Defining the neural mechanisms of probabilistic reversal learning using eventrelated functional magnetic resonance imaging. J Neurosci 2002, 22:4563-4567. 11. Daw ND, O’Doherty JP, Dayan P, Seymour B, Dolan RJ: Cortical substrates for exploratory decisions in humans. Nature 2006, 441:876-879. 12. Delgado MR, Miller MM, Inati S, Phelps EA: An fMRI study of reward-related probability learning. Neuroimage 2005, 24:862-873. 13. Poldrack RA, Clark J, Pare-Blagoev EJ, Shohamy D, Creso Moyano J, Myers C, Gluck MA: Interactive memory systems in the human brain. Nature 2001, 414:546-550. 14. O’Doherty J, Dayan P, Schultz J, Deichmann R, Friston K, Dolan RJ: Dissociable roles of ventral and dorsal striatum in instrumental conditioning. Science 2004, 304:452-454. 15. Yin HH, Knowlton BJ: The role of the basal ganglia in habit formation. Nat Rev Neurosci 2006, 7:464-476. 16. Tricomi E, Balleine BW, O’Doherty JP: A specific role for posterior dorsolateral striatum in human habit learning. Eur J Neurosci 2009, 29:2225-2232. 17. Foerde K, Knowlton BJ, Poldrack RA: Modulation of competing memory systems by distraction. Proc Natl Acad Sci USA 2006, 103:11778-11783. 18. Rangel A, Hare T: Neural computations associated with goaldirected choice. Curr Opin Neurobiol 2010, 20:262-270. 19. Shohamy D, Wagner AD: Integrating memories in the human brain: hippocampal-midbrain encoding of overlapping events. Neuron 2008, 60:378-389. 20. Daw N, Gershman SJ, Dayan P, Dolan RJ: Model-based influences on humans’ choices and striatal prediction errors. Neuron 2011, 69:1204-1215. This paper used fMRI and computational models of learning and demonstrated overlapping responses to two distinct learning signals: one thought to correspond to retrospective learning of stimulus–response associations (‘model-free’), the other thought to correspond to prospective learning of the task structure (‘model-based’). 21. Glascher J, Daw N, Dayan P, O’Doherty JP: States versus rewards: dissociable neural prediction error signals underlying model-based and model-free reinforcement learning. Neuron 2010, 66:585-595. 22. Hare TA, O’Doherty J, Camerer CF, Schultz W, Rangel A: Dissociating the role of the orbitofrontal cortex and the striatum in the computation of goal values and prediction errors. J Neurosci 2008, 28:5623-5630. 23. McClure SM, Berns GS, Montague PR: Temporal prediction errors in a passive learning task activate human striatum. Neuron 2003, 38:339-346. 24. O’Doherty JP, Dayan P, Friston K, Critchley H, Dolan RJ: Temporal difference models and reward-related learning in the human brain. Neuron 2003, 38:329-337. 25. Daw ND, O’Doherty JP, Dayan P, Seymour B, Dolan RJ: Cortical substrates for exploratory decisions in humans. Nature 2006. 26. Delgado MR, Nystrom LE, Fissell C, Noll DC, Fiez JA: Tracking the hemodynamic responses to reward and punishment in the striatum. J Neurophysiol 2000, 84:3072-3077. 27. Daw ND, Doya K: The computational neurobiology of learning and reward. Curr Opin Neurobiol 2006, 16:199-204. Current Opinion in Neurobiology 2011, 21:408–414
28. Dickerson KC, Li J, Delgado MR: Parallel contributions of distinct human memory systems during probabilistic learning. Neuroimage 2011, 55:266-276. 29. Lin A, Adolphs R, Rangel A: Social and monetary reward learning engage overlapping neural substrates. Soc Cogn Affect Neurosci 2011. 30. Stice E, Yokum S, Blum K, Bohon C: Weight gain is associated with reduced striatal response to palatable food. J Neurosci 2010, 30:13105-13109. 31. Aron AR, Shohamy D, Clark J, Myers C, Gluck MA, Poldrack RA: Human midbrain sensitivity to cognitive feedback and uncertainty during classification learning. J Neurophysiol 2004, 92:1144-1152. 32. Salimpoor VN, Benovoy M, Larcher K, Dagher A, Zatorre RJ: Anatomically distinct dopamine release during anticipation and experience of peak emotion to music. Nat Neurosci 2011, 14:257-262. 33. Haruno M, Kawato M: Different neural correlates of reward expectation and reward expectation error in the putamen and caudate nucleus during stimulus-action-reward association learning. J Neurophysiol 2006, 95:948-959. 34. Haruno M, Kuroda T, Doya K, Toyama K, Kimura M, Samejima K, Imamizu H, Kawato M: A neural correlate of reward-based behavioral learning in caudate nucleus: a functional magnetic resonance imaging study of a stochastic decision task. J Neurosci 2004, 24:1660-1665. 35. King-Casas B, Tomlin D, Anen C, Camerer CF, Quartz SR, Montague PR: Getting to know you: reputation and trust in a two-person economic exchange. Science 2005, 308:78-83. 36. Tricomi EM, Delgado MR, Fiez JA: Modulation of caudate activity by action contingency. Neuron 2004, 41:281-292. 37. Bromberg-Martin ES, Matsumoto M, Hikosaka O: Dopamine in motivational control: rewarding, aversive, and alerting. Neuron 2010, 68:815-834. 38. Salas R, Baldwin P, de Biasi M, Montague PR: BOLD responses to negative reward prediction errors in human habenula. Front Hum Neurosci 2010, 4:36. 39. Robinson OJ, Frank MJ, Sahakian BJ, Cools R: Dissociable responses to punishment in distinct striatal regions during reversal learning. Neuroimage 2010, 51:1459-1467. 40. Seymour B, O’Doherty JP, Dayan P, Koltzenburg M, Jones AK, Dolan RJ, Friston KJ, Frackowiak RS: Temporal difference models describe higher-order learning in humans. Nature 2004, 429:664-667. 41. Delgado MR, Schotter A, Ozbay EY, Phelps EA: Understanding overbidding: using the neural circuitry of reward to design economic auctions. Science 2008, 321:1849-1852. 42. Balleine BW, Delgado MR, Hikosaka O: The role of the dorsal striatum in reward and decision-making. J Neurosci 2007, 27:8161-8165. 43. Phan KL, Sripada CS, Angstadt M, McCabe K: Reputation for reciprocity engages the brain reward center. Proc Natl Acad Sci USA 2010, 107:13099-13104. 44. Tricomi E, Rangel A, Camerer CF, O’Doherty JP: Neural evidence for inequality-averse social preferences. Nature 2010, 463:1089-1091. The authors examine the neural basis of inequality-averse preferences. Activity in a subject’s ventral striatum and ventromedial prefrontal cortex was more responsive to the transfer of money to others when the subject was first given a large endowment of money, while the opposite pattern was seen when the subject was not given the endowment. 45. Campbell-Meiklejohn DK, Bach DR, Roepstorff A, Dolan RJ, Frith CD: How the opinion of others affects our valuation of objects. Curr Biol 2010, 20:1165-1170. 46. Mobbs D, Yu R, Meyer M, Passamonti L, Seymour B, Calder AJ, Schweizer S, Frith CD, Dalgleish T: A key role for similarity in vicarious reward. Science 2009, 324:900. www.sciencedirect.com
The striatum and Learning Shohamy 413
47. Burke CJ, Tobler PN, Baddeley M, Schultz W: Neural mechanisms of observational learning. Proc Natl Acad Sci USA 2010, 107:14431-14436. 48. Murayama K, Matsumoto M, Izuma K, Matsumoto K: Neural basis of the undermining effect of monetary reward on intrinsic motivation. Proc Natl Acad Sci USA 2010, 107:20911-20916. The authors demonstrate that activity in the anterior striatum and prefrontal areas decreased along with the ‘undermining effect,’ the phenomenon in which performance-based extrinsic reward undermines a person’s intrinsic motivation to engage in a task.
63. Shohamy D, Adcock RA: Dopamine and adaptive memory. Trends Cogn Sci 2010, 14:464-472. 64. van der Meer MA, Johnson A, Schmitzer-Torbert NC, Redish AD: Triple dissociation of information processing in dorsal striatum, ventral striatum, and hippocampus on a learned spatial decision task. Neuron 2010, 67:25-32. 65. Johnson A, van der Meer MA, Redish AD: Integrating hippocampus and striatum in decision-making. Curr Opin Neurobiol 2007, 17:692-697.
49. Mobbs D, Hassabis D, Seymour B, Marchant JL, Weiskopf N, Dolan RJ, Frith CD: Choking on the money: reward-based performance decrements are associated with midbrain activity. Psychol Sci 2009, 20:955-962.
66. Voermans NC, Petersson KM, Daudey L, Weber B, Van Spaendonck KP, Kremer HP, Fernandez G: Interaction between the human hippocampus and the caudate nucleus during route recognition. Neuron 2004, 43:427-435.
50. Adcock RA, Thangavel A, Whitfield-Gabrieli S, Knutson B, Gabrieli JD: Reward-motivated learning: mesolimbic activation precedes memory formation. Neuron 2006, 50:507-517. The authors discover that cues indicating high-value rewards for memorizing upcoming scenes lead to greater activation in the VTA, nucleus accumbens, and hippocampus, along with superior memory performance later. Greater correlation between the hippocampus and VTA is associated with enhanced long-term memory, suggesting reward motivation may promote memory formation via DA release in the hippocampus before learning.
67. Mattfeld AT, Stark CE: Striatal and medial temporal lobe functional interactions during visuomotor associative learning. Cereb Cortex 2011, 21:647-658. The authors demonstrate coupling in activity in the ventral striatum and MTL during learning of an arbitrary associative learning task, and the opposite pattern in the associative striatum and MTL. Distinct areas throughout the striatum and MTL are modulated by probability correct or rate of learning.
51. Kang MJ, Hsu M, Krajbich IM, Loewenstein G, McClure SM, Wang JT, Camerer CF: The wick in the candle of learning: epistemic curiosity activates reward circuitry and enhances memory. Psychol Sci 2009, 20:963-973. 52. Fenker DB, Schoenfeld MA, Waldmann MR, Schuetze H, Heinze HJ, Duezel E: ‘‘Virus and epidemic’’: causal knowledge activates prediction error circuitry. J Cogn Neurosci 2010, 22:2151-2163. 53. Wittmann BC, Schott BH, Guderian S, Frey JU, Heinze HJ, Duzel E: Reward-related FMRI activation of dopaminergic midbrain is associated with enhanced hippocampus-dependent longterm memory formation. Neuron 2005, 45:459-467. 54. Han S, Huettel SA, Raposo A, Adcock RA, Dobbins IG: Functional significance of striatal responses during episodic decisions: recovery or goal attainment? J Neurosci 2010, 30:4767-4775. 55. Tricomi E, Fiez JA: Feedback signals in the caudate reflect goal achievement on a declarative memory task. Neuroimage 2008, 41:1154-1167. 56. Turk-Browne NB, Scholl BJ, Chun MM, Johnson MK: Neural evidence of statistical learning: efficient detection of visual regularities without awareness. J Cogn Neurosci 2009, 21:1934-1945. 57. Brainard MS, Doupe AJ: Interruption of a basal gangliaforebrain circuit prevents plasticity of learned vocalizations. Nature 2000, 404:762-766. 58. Pessiglione M, Petrovic P, Daunizeau J, Palminteri S, Dolan RJ, Frith CD: Subliminal instrumental conditioning demonstrated in the human brain. Neuron 2008, 59:561-567. 59. Alexander GE, DeLong MR, Strick PL: Parallel organization of functionally segregated circuits linking basal ganglia and cortex. Annu Rev Neurosci 1986, 9:357-381. 60. Schoenbaum G, Esber GR: How do you (estimate you will) like them apples? Integration as a defining trait of orbitofrontal function. Curr Opin Neurobiol 2010, 20:205-211. 61. McDannald MA, Lucantonio F, Burke KA, Niv Y, Schoenbaum G: Ventral striatum and orbitofrontal cortex are both required for model-based, but not model-free, reinforcement learning. J Neurosci 2011, 31:2700-2705. 62. den Ouden HE, Daunizeau J, Roiser J, Friston KJ, Stephan KE: Striatal prediction error modulates cortical coupling. J Neurosci 2010, 30:3210-3219. The authors use a probabilistic, cue-stimulus association task to characterize the role of striatal trial-by-trial prediction error activity in controlling the efficacy of visuomotor connections. These data show the influence of surprising stimuli on premotor activity, and provide empirical evidence for a central role for prediction error-dependent plasticity in learning. www.sciencedirect.com
68. Hartley T, Burgess N: Complementary memory systems: competition, cooperation and compensation. Trends Neurosci 2005, 28:169-170. 69. Goldman-Rakic PS, Selemon LD, Schwartz ML: Dual pathways connecting the dorsolateral prefrontal cortex with the hippocampal formation and parahippocampal cortex in the rhesus monkey. Neuroscience 1984, 12:719-743. 70. Fuster JM: The Prefrontal Cortex: Anatomy, Physiology, and Neuropsychology of the Frontal Lobe. Philadelphia: LippincottRaven; 1997. 71. Haber SN: The primate basal ganglia: parallel and integrative networks. J Chem Neuroanat 2003, 26:317-330. 72. Suzuki WA, Amaral DG: Functional neuroanatomy of the medial temporal lobe memory system. Cortex 2004, 40:220-222. 73. Doeller CF, King JA, Burgess N: Parallel striatal and hippocampal systems for landmarks and boundaries in spatial memory. Proc Natl Acad Sci USA 2008, 105:5915-5920. 74. Sadeh T, Shohamy D, Levy DR, Reggev N, Maril A: Cooperation between the hippocampus and the striatum during episodic encoding. J Cogn Neurosci 2011. 75. Levy I, Lazzaro SC, Rutledge RB, Glimcher PW: Choice from nonchoice: predicting consumer preferences from blood oxygenation level-dependent signals obtained during passive viewing. J Neurosci 2011, 31:118-125. 76. Axmacher N, Cohen MX, Fell J, Haupt S, Dumpelmann M, Elger CE, Schlaepfer TE, Lenartz D, Sturm V, Ranganath C: Intracranial EEG correlates of expectancy and memory formation in the human hippocampus and nucleus accumbens. Neuron 2010, 65:541-549. 77. Lisman JE, Grace AA: The hippocampal-VTA loop: controlling the entry of information into long-term memory. Neuron 2005, 46:703-713. 78. Daw ND, Shohamy D: The cognitive neuroscience of motivation and learning. Soc Cogn 2008, 26:593-620. 79. Peters J, Buchel C: Episodic future thinking reduces reward delay discounting through an enhancement of prefrontalmediotemporal interactions. Neuron 2010, 66:138-148. 80. Shohamy D, Myers CE, Kalanithi J, Gluck MA: Basal ganglia and dopamine contributions to probabilistic category learning. Neurosci Biobehav Rev 2008, 32:219-236. 81. Grahn JA, Parkinson JA, Owen AM: The role of the basal ganglia in learning and memory: neuropsychological studies. Behav Brain Res 2009, 199:53-60. 82. Rowe JB, Hughes L, Ghosh BC, Eckstein D, Williams-Gray CH, Fallon S, Barker RA, Owen AM: Parkinson’s disease and dopaminergic therapy – differential effects on movement, reward and cognition. Brain 2008, 131:2094-2105. Current Opinion in Neurobiology 2011, 21:408–414
414 Behavioral and Cognitive Neuroscience
83. Frank MJ, Seeberger LC, O’Reilly RC: By carrot or by stick: cognitive reinforcement learning in parkinsonism. Science 2004, 306:1940-1943.
89. Keri S, Moustafa AA, Myers CE, Benedek G, Gluck MA: {alpha}Synuclein gene duplication impairs reward learning. Proc Natl Acad Sci USA 2010, 107:15992-15994.
84. Cools R, Lewis SJ, Clark L, Barker RA, Robbins TW: L-DOPA disrupts activity in the nucleus accumbens during reversal learning in Parkinson’s disease. Neuropsychopharmacology 2007, 32:180-189.
90. Krugel LK, Biele G, Mohr PN, Li SC, Heekeren HR: Genetic variation in dopaminergic neuromodulation influences the ability to rapidly and flexibly adapt decisions. Proc Natl Acad Sci USA 2009, 106:17951-17956. The authors demonstrate an advantage in reward-based learning in those with the Val/Val genotype of the Val158Met COMT polymorphism, characterized by faster, more flexible learning, as well as greater and more differentiated striatal fMRI responses to predictions errors.
85. Schonberg T, O’Doherty JP, Joel D, Inzelberg R, Segev Y, Daw ND: Selective impairment of prediction error signaling in human dorsolateral but not ventral striatum in Parkinson’s disease patients: evidence from a model-based fMRI study. Neuroimage 2010, 49:772-781. 86. Shohamy D, Myers CE, Geghman KD, Sage J, Gluck MA: L-dopa impairs learning, but spares generalization, in Parkinson’s disease. Neuropsychologia 2006, 44:774-784. 87. Lotze M, Reimold M, Heymans U, Laihinen A, Patt M, Halsband U: Reduced ventrolateral fMRI response during observation of emotional gestures related to the degree of dopaminergic impairment in Parkinson disease. J Cogn Neurosci 2009, 21:1321-1331. 88. Schott BH, Minuzzi L, Krebs RM, Elmenhorst D, Lang M, Winz OH, Seidenbecher CI, Coenen HH, Heinze HJ, Zilles K et al.: Mesolimbic functional magnetic resonance imaging activations during reward anticipation correlate with rewardrelated ventral striatal dopamine release. J Neurosci 2008, 28:14311-14319.
Current Opinion in Neurobiology 2011, 21:408–414
91. Maia TV, Frank MJ: From reinforcement learning models to psychiatric and neurological disorders. Nat Neurosci 2011, 14:154-162. 92. Van Leijenhorst L, Moor BG, Op de Macks ZA, Rombouts SA, Westenberg PM, Crone EA: Adolescent risky decision-making: neurocognitive development of reward and control regions. Neuroimage 2010, 51:345-355. 93. Cohen JR, Asarnow RF, Sabb FW, Bilder RM, Bookheimer SY, Knowlton BJ, Poldrack RA: A unique adolescent response to reward prediction errors. Nat Neurosci 2010, 13:669-671. 94. DeCouteau WE, Thorn C, Gibson DJ, Courtemanche R, Mitra P, Kubota Y, Graybiel AM: Learning-related coordination of striatal and hippocampal theta rhythms during acquisition of a procedural maze task. Proc Natl Acad Sci USA 2007, 104:5644-5649.
www.sciencedirect.com