2. Scope of the work

Addiction is a complex phenomenon, with causes that can be identified from many perspectives (Volkow & Li Reference Volkow and Li2005a; West Reference West2001), including social (Davis & Tunks Reference Davis and Tunks1991), environmental (DeFeudis Reference DeFeudis1978; Dickerson & O'Connor Reference Dickerson and O'Connor2006; Maddahian et al. Reference Maddahian, Newcomb and Bentler1986; Morgan et al. Reference Morgan, Grant, Gage, Mach, Kaplan, Prioleau, Nader, Buchheimer, Ehrenkaufer and Nader2002), legal (Dickerson & O'Connor Reference Dickerson and O'Connor2006; Kleber et al., Reference Kleber, Califano, Demers, Lowinson, Ruiz, Millman and Langrod1997; MacCoun Reference MacCoun1993), as well as psychological and neurobiological (Goldman et al. Reference Goldman, Brown, Christiansen, lsn and lsn1987; Reference Goldman, Boca, Darkes, Leonard and lsn1999; Heyman Reference Heyman1996; Reference Heyman2000; Koob & Le Moal Reference Koob and Le Moal2006; Redish Reference Redish2004; Robinson Reference Robinson2004; Robinson & Berridge Reference Robinson and Berridge2003; Tiffany Reference Tiffany1990), economic (Ainslie Reference Ainslie1992; Reference Ainslie2001; Becker & Murphy Reference Becker and Murphy1988; Bernheim & Rangel Reference Bernheim and Rangel2004; Hursh Reference Hursh1991; Hursh et al. Reference Hursh, Galuska, Winger and Woods2005), and genetic (Crabbe Reference Crabbe2002; Goldman et al. Reference Goldman, Oroszi and Ducci2005; Hiroi & Agatsuma Reference Hiroi and Agatsuma2005) perspectives. All of these perspectives have explanatory power as to the causes of addiction, and all of them provide suggested methods of treatment of addiction. However, a thorough treatment of addiction from all of these perspectives is beyond the scope of a paper such as this one. In this target article, we address an explanation for addictive decisions based on animal learning theory, the neuroscience of learning and memory, human decision-making, and neuroeconomics, which we argue have converged on a unified theory of decision-making as arising from an interaction between two learning systems (a quickly learned, flexible, but computationally expensive-to-execute planning system and a slowly learned, inflexible, but computationally inexpensive-to-execute habit system).

3. Making decisions

Theories of how animals make decisions have been developed over the last 50 years in the fields of economics (Ainslie Reference Ainslie1992, Reference Ainslie2001; Becker & Murphy Reference Becker and Murphy1988; Bernheim & Rangel Reference Bernheim and Rangel2004; Bickel & Marsch Reference Bickel and Marsch2001; Glimcher & Rustichini Reference Glimcher and Rustichini2004; Petry & Bickel Reference Petry and Bickel1998), psychology and neuroscience (Daw Reference Daw2003; Glimcher Reference Glimcher2003; Hastie Reference Hastie2001; Herrnstein Reference Herrnstein1997; Heyman Reference Heyman1996; Kahneman et al. Reference Kahneman, Slovic and Tversky1982; Kahneman & Tversky Reference Kahneman and Tversky2000; Sanfey et al. Reference Sanfey, Loewenstein, McClure and Cohen2006; Slovic et al. Reference Slovic, Fischhoff and Lichtenstein1977), and machine learning (Sutton & Barto Reference Sutton and Barto1998). These literatures have converged on the concept that decisions are based on the prediction of value or expected utility of the decision.Footnote ¹ These terms can be defined as the total, expected, future reward, taking into account the probability of receiving the reward and any delay before the reward is received. In these analyses, costs are typically included as negative rewards, but they can also be included separately in some formulations. If the agent can correctly predict the value (total discountedFootnote ² reward minus total expected cost) of its actions, then it can make appropriate decisions about which actions to take. The theories of addiction that have been proposed (Table 1) all have the effect of changing the prediction of value or cost in ways that make the agent continue to repeatedly return to seeking of the addictive drug or maladaptive behavior.

There are two potential methods from which one can derive the value of taking some action (Bernheim & Rangel Reference Bernheim and Rangel2004; Daw et al. Reference Daw, Niv and Dayan2005; Redish & Johnson Reference Redish and Johnson2007; Sutton & Barto Reference Sutton and Barto1998): forward-search and caching. In the first case (forward-search), one considers the possible consequences of one's actions – the agent realizes that if it takes this action in this situation, this will occur, and it will get this reward, but if it does something else, there will be different consequences, and it will get a different reward. In the other case (caching), the agent has learned to associate a specific action with a given situation – over time, the agent has learned that the best thing to do in this situation is to take this action. The forward-search system takes time to execute (because one has to mentally trace down possible paths), but is very flexible. That flexibility means that it is safe to learn quickly. Learning potential consequences of one's actions does not commit one to an action; rather it opens the possibility of considering the consequences of an action before selecting that action. In contrast, the caching system is very fast to execute (because one simply has to retrieve the best action for a given situation), but is very rigid. That inflexibility means that it would be dangerous to learn the stimulus-action relationships stored in the habit system too quickly.

This dichotomy can be related to the question of when to stop a search process (Nilsson et al. Reference Nilsson, Shapiro, Gage, Olton and Bjorklund1987; Simon Reference Simon1955). Incomplete search processes may be available in which temporarily cached values are accessed to cut off parts of the search tree, similar to heuristic search processes studied in the classic artificial intelligence literature (Nilsson et al. Reference Nilsson, Shapiro, Gage, Olton and Bjorklund1987; Rich & Knight Reference Rich and Knight1991; Russell & Norvig Reference Russell and Norvig2002). Similarly, one can imagine that only some of the potential paths are searched in any decision. Finding an optimal solution takes time, and there is a tradeoff between search time and the optimality of the solution found (Simon Reference Simon1955). From an evolutionary perspective, a quickly found, acceptable solution may be more efficient than a slowly found optimal solution (Gigerenzer Reference Gigerenzer2001; Gigerenzer & Goldstein Reference Gigerenzer and Goldstein1996; Simon Reference Simon1955). A true caching system, however, does not entail a search process and should not be considered to be equivalent to a single step of the search process (Daw et al. Reference Daw, Niv and Dayan2005; Gigerenzer Reference Gigerenzer2001). A single step of the search process would identify the consequence of that step, allowing changes in that consequence to change performance without relearning. In contrast, the caching system compares a stored value with an action taken in a given situation and does not identify the consequence during performance, which means that it cannot change its reactions to changes in the value of that consequence. This distinction can be seen in the devaluation literature, discussed further on.

A number of literatures have converged on a division between learning systems that match these two systems. In the animal navigation literature, these two systems are referred to as the cognitive map and route systems,Footnote ³ respectively (O'Keefe & Nadel Reference O'Keefe and Nadel1978; Redish Reference Redish1999). In the animal learning-theory literature, these systems can be identified as three separate systems, a Pavlovian learning system (situation-outcome, ), an instrumental learning system (action-outcome, ), and a habit learning system ().Footnote ⁴

They have also been referred to as cognitive and habit learning systems (Mishkin & Appenzeller Reference Mishkin and Appenzeller1987; Poldrack & Packard Reference Poldrack and Packard2003; Saint-Cyr et al. Reference Saint-Cyr, Taylor and Lang1988; Yin & Knowlton Reference Yin and Knowlton2006), and match closely the distinction made between declarative and procedural learning (Cohen & Eichenbaum Reference Cohen and Eichenbaum1993; Cohen & Squire Reference Cohen and Squire1980; Redish Reference Redish1999; Squire Reference Squire1987; Squire et al. Reference Squire, Cohen, Nadel, Weingartner and Parker1984) and between explicit and implicit learning systems (Clark & Squire Reference Clark and Squire1998; Curran Reference Curran1995; Doyon et al. Reference Doyon, Laforce, Bouchard, Gaudreau, Roy, Poirer, Bedard, Bedard and Bouchard1998; Ferraro et al. Reference Ferraro, Balota and Connor1993; Forkstam & Petersson Reference Forkstam and Petersson2005; Knopman & Nissen Reference Knopman and Nissen1987; Reference Knopman and Nissen1991; Nissen et al. Reference Nissen, Knopman and Schacter1987; Willingham et al. Reference Willingham, Nissen and Bullemer1989), as well as between controlled and automatic processing theories (Kahneman & Frederick Reference Kahneman, Frederick, Gilovich, Griffin and Kahneman2002; Schneider & Chein Reference Schneider and Chein2003; Schneider & Shiffrin Reference Schneider and Shiffrin1977). We argue that these diverse literatures have converged on a pair of decision-making systems, which can be understood as (1) a flexible, cognitive, planning system and (2) a rigid, automatic, habit-based system.

This dichotomy is related to the historical debate on “expectancies” in the classic animal learning theory literature (Bolles Reference Bolles1972; Hull Reference Hull1943; Reference Hull1952; Munn Reference Munn1950; Tolman Reference Tolman1938; Reference Tolman1939; Reference Tolman1948). Tolman (Reference Tolman1938; Reference Tolman1939; Reference Tolman1948) argued that animals maintain an expectancy of their potential future consequences (including an expectancy of any rewarding component), and that this provided for latent learning effects as well as fast changes in choices in response to changes in provided needs, whereas Hull (Reference Hull1943; Reference Hull1952) argued that animals learn simple associations of stimuli and responses, allowing for the slow development of automation (Carr & Watson Reference Carr and Watson1908; Dennis Reference Dennis1932). As noted by Guthrie (Reference Guthrie1935; see Balleine & Ostlund Reference Balleine and Ostlund2007; Bolles Reference Bolles1972), one implication of Tolman's cognitive expectancies theories would be a delay in choosing. Just such a delay is seen in early learning, particularly in tasks that require the planning system. At choice points, rats faced with difficult decisions pause and vicariously sample the different choices before committing to a decision (Brown Reference Brown1992; Meunzinger Reference Meunzinger1938; Tolman Reference Tolman1938; Reference Tolman1939). This “vicarious trial and error” (VTE) behavior is abolished with hippocampal lesions (Hu & Amsel Reference Hu and Amsel1995), and is related to hippocampal activity on hippocampal-dependent tasks (Hu et al. Reference Hu, Xu and Gonzalez-Lima2006). Recent neural ensemble recording data have found that hippocampal firing patterns transiently represent locations ahead of the animal at choice points during VTE-like behaviors (Johnson & Redish Reference Johnson and Redish2007). Once tasks have been overtrained, these VTE movements disappear (Hu et al. Reference Hu, Xu and Gonzalez-Lima2006; Munn Reference Munn1950; Tolman Reference Tolman1938), as do the forward representations (Johnson & Redish Reference Johnson and Redish2007), suggesting that VTE may be a signal of the active processing in the planning system (Buckner & Carroll, Reference Buckner and Carroll2007; Johnson & Redish Reference Johnson and Redish2007; Tolman Reference Tolman1938; Reference Tolman1939).

These two systems mirror the classical two-process theory in psychology (Domjan Reference Domjan1998; Gray Reference Gray1975) and the more recent distinction between stimulus-stimulus (SS, ), stimulus-outcome (SO, SAO, ), action-outcome (AO, ), and stimulus-response or stimulus-action (SA, ) (Balleine & Ostlund Reference Balleine and Ostlund2007; Dickinson Reference Dickinson1985) (see Table 2). The first () entails the recognition of a causal sequence but does not entail an actual decision. The second () is classical Pavlovian conditioning and entails an action taken in response to a situation in anticipation of a given outcome (Domjan Reference Domjan1998; Pavlov Reference Pavlov1927; Rescorla Reference Rescorla1988). The third () is classical instrumental conditioning (Balleine & Ostlund Reference Balleine and Ostlund2007; Domjan Reference Domjan1998; Ferster & Skinner Reference Ferster and Skinner1957) and entails an action taken to achieve an outcome, even in the absence of an immediate stimulus. It is important to note, however, that action-outcome associations do still include stimuli in the form of the context (actions are not taken at all times but rather only within certain facilitating contexts).Footnote ⁵ The fourth () entails an association between a situation and an action and denotes habit learning (Domjan Reference Domjan1998; Hull Reference Hull1943; Reference Hull1952).

Table 2.

Learning theory and decision-making

These four associations can be differentiated in terms of their expectancies (Table 2). associations entail an expectancy of an outcome, but with no decision, there is no necessary further processing of that outcome, although there is likely to be an emotional preparation of some sort. If an animal can do something to prepare for, produce, or change that outcome, then the association becomes one of situation-action-outcome (). If there is no immediate stimulus triggering the action, then the association becomes an association. Because associations continue to include a contextual gating component, the association is truly an association. Although there are anatomical reasons to separate from associations (Balleine & Ostlund Reference Balleine and Ostlund2007; Ostlund & Balleine Reference Ostlund and Balleine2007; Yin et al. Reference Yin, Knowlton and Balleine2005), for our purposes, they can be treated similarly: they both entail an expectancy of an outcome that must be evaluated to produce an expectancy of a value. This means they both require a planning component and can be differentiated from habit learning in which situations are directly associated with actions (). In the association, situation-action pairs entail a direct expectancy of a value, which can then drive the action, even in the absence of a recognition of the outcome.

Following this distinction, we categorize these four association systems into three decision systems: an observation system, which does not make decisions and will not be discussed further; a planning system, which takes a given situation (derived from stimuli, context, or a combination thereof), predicts an outcome, and evaluates that outcome; and a habit system, which takes a given situation (derived from stimuli, context, or a combination thereof) and identifies the best remembered action to take.

These systems, of course, exist within overlapping and interacting structures (Balleine & Ostlund Reference Balleine and Ostlund2007; Corbit et al. Reference Corbit, Muir and Balleine2001; Dayan & Balleine Reference Dayan and Balleine2002; Devan & White Reference Devan and White1999; Kelley Reference Kelley1999a; Reference Kelley1999b; Voorn et al. Reference Voorn, Vanderschuren, Groenewegen, Robbins and Pennartz2004; Yin et al. Reference Yin, Knowlton and Balleine2006; Yin & Knowlton Reference Yin and Knowlton2006). The flexible planning system involves the entorhinal cortex (Corbit & Balleine Reference Corbit and Balleine2000), hippocampus (O'Keefe & Nadel Reference O'Keefe and Nadel1978; Packard & McGaugh Reference Packard and McGaugh1996; Redish Reference Redish1999), the ventral and dorsomedial striatum (Devan & White Reference Devan and White1999; Martin Reference Martin2001; Mogenson Reference Mogenson and Bandler1984; Mogenson et al. Reference Mogenson, Jones and Yim1980; Pennartz et al. Reference Pennartz, Lee, Verheul, Lipa, Barnes and McNaughton2004; Schoenbaum et al. Reference Schoenbaum, Setlow and Ramus2003; Yin et al. Reference Yin, Knowlton and Balleine2005), prelimbic medial prefrontal cortex (Jung et al. Reference Jung, Qin, McNaughton and Barnes1998; Killcross & Coutureau Reference Killcross and Coutureau2003; Ragozzino et al. Reference Ragozzino, Detrick and Kesner1999), and orbitofrontal cortex (Davis et al. Reference Davis, Donahue, Discenza, Waite and Ramus2006; Padoa-Schioppa & Assad Reference Padoa-Schioppa and Assad2006; Schoenbaum et al. Reference Schoenbaum, Setlow and Ramus2003; Reference Schoenbaum, Roesch and Stalnaker2006a; Reference Schoenbaum, Setlow, Saddoris and Gallagher2006b; Schoenbaum & Roesch Reference Schoenbaum and Roesch2005). The habit system involves the dorsolateral striatum (Barnes et al. Reference Barnes, Kubota, Hu, Jin and Graybiel2005; Packard & McGaugh Reference Packard and McGaugh1996; Schmitzer-Torbert & Redish Reference Schmitzer-Torbert and Redish2004; Yin & Knowlton Reference Yin and Knowlton2004; Reference Yin and Knowlton2006), the infralimbic medial prefrontal cortex (Coutureau & Killcross Reference Coutureau and Killcross2003; Killcross & Coutureau Reference Killcross and Coutureau2003) as well as the parietal cortex (DiMattia & Kesner Reference DiMattia and Kesner1988; Kesner et al. Reference Kesner, Farnsworth and DiMattia1989) (see Table 3).

Table 3.

Two systems

3.1. Transitions between decision systems

Behavior generally begins with flexible planning systems but, for repeated behaviors, can become driven by the less-flexible (but also less computationally expensive) habit systems. Examples of this development are well known from our experiences. For example, the first time we drive to a new job, we need a travel plan; we pay attention to street-signs and other landmarks. But after driving that same trip every day for years, the trip requires less and less attention, freeing up resources for other cognitive processes such as planning classes, papers, or dinner. The flexible system, however, generally remains available, as when road construction closes one's primary route to work and one now needs to identify a new route. Errors can exist within both systems, as for example, a misjudged plan or a trip so automatic, that if one is not paying attention, one might accidentally find oneself having driven to work even though one planned to go somewhere else. This interaction is well-studied in the animal literature, including the overlaying of planning by habit systems (Dickinson Reference Dickinson1980; Hikosaka et al. Reference Hikosaka, Nakahara, Rand, Sakai, Lu, Nakamura, Miyachi and Doya1999; Packard & McGaugh Reference Packard and McGaugh1996; Schmitzer-Torbert & Redish Reference Schmitzer-Torbert and Redish2002), restoration of planning in the face of changes (Gray & McNaughton Reference Gray and McNaughton2000; Isoda & Hikosaka Reference Isoda and Hikosaka2007; Sakagami et al. Reference Sakagami, Pan and Uttl2006), and conflict between the two systems (Gold Reference Gold2004; McDonald & White Reference McDonald and White1994; Packard Reference Packard1999; Poldrack & Packard Reference Poldrack and Packard2003; Redish et al. Reference Redish, Rosenzweig, Bohanick, McNaughton and Barnes2000).

Four well-studied examples in the animal literature are the transfer of place strategies to response strategies in the plus-maze (Chang & Gold Reference Chang and Gold2004; Packard & McGaugh Reference Packard and McGaugh1996; Yin & Knowlton Reference Yin and Knowlton2004), the development of the regularity of behavioral paths (Barnes et al. Reference Barnes, Kubota, Hu, Jin and Graybiel2005; Jog et al. Reference Jog, Kubota, Connolly, Hillegaart and Graybiel1999; Schmitzer-Torbert & Redish Reference Schmitzer-Torbert and Redish2002), the disappearance of devaluation in animal learning studies (Adams & Dickinson Reference Adams and Dickinson1981; Balleine & Dickinson Reference Balleine and Dickinson1998; Colwill & Rescorla Reference Colwill and Rescorla1985; Tang et al. Reference Tang, Pawlak, Volodymyr and West2007), and the inhibition of habitual responses in go/no-go tasks (Gray & McNaughton Reference Gray and McNaughton2000; Husain et al. Reference Husain, Parton, Hodgson, Mort and Rees2003; Isoda & Hikosaka Reference Isoda and Hikosaka2007).

In the plus-maze, animals are trained to take an action that can be solved either by going to a specific place (Tolman et al. Reference Tolman, Ritchie and Kalish1946) or by taking an action in response to being placed on the maze (Hull Reference Hull1952). These algorithms can be differentiated by an appropriately designed probe trial (Barnes et al. Reference Barnes, Nadel and Honig1980; Packard & McGaugh Reference Packard and McGaugh1996; Restle Reference Restle1957). Rats on this task (and on other similar tasks) first use a place strategy, which then evolves into a response strategy (McDonald & White Reference McDonald and White1994; Packard & McGaugh Reference Packard and McGaugh1996; Yin & Knowlton Reference Yin and Knowlton2004). Place strategies depend on hippocampal, as well as ventral and dorsomedial striatal integrity, while response strategies depend on dorsolateral striatal integrity (Packard & McGaugh Reference Packard and McGaugh1996; Yin & Knowlton Reference Yin and Knowlton2004; Reference Yin and Knowlton2006; Yin et al. Reference Yin, Knowlton and Balleine2005).

In tasks in which animals are provided a general task with specific cases that change from day to day or session to session, animals can learn the specific instantiations very quickly. In these tasks, behavioral accuracy improves quickly, followed by a slower development of a regularity in the actions taken by the animal (rats, Jog et al. Reference Jog, Kubota, Connolly, Hillegaart and Graybiel1999; Schmitzer-Torbert & Redish Reference Schmitzer-Torbert and Redish2002; monkeys, Hikosaka et al. Reference Hikosaka, Nakahara, Rand, Sakai, Lu, Nakamura, Miyachi and Doya1999; Rand et al. Reference Rand, Hikosaka, Miyachi, Lu and Miyashita1998; Reference Rand, Hikosaka, Miyachi, Lu, Nakamura, Kitaguchi and Shimo2000; humans, Nissen & Bullemer Reference Nissen and Bullemer1987; Willingham et al. Reference Willingham, Nissen and Bullemer1989). In these tasks, the early (accurate, flexible, and slower) behavior is dependent on hippocampal integrity and correlated to hippocampal activity (Ferraro et al. Reference Ferraro, Balota and Connor1993; Johnson & Redish Reference Johnson and Redish2007; Knopman & Nissen Reference Knopman and Nissen1987), whereas later (also accurate, but inflexible and faster) behavior is dependent on dorsolateral striatal integrity and correlated to dorsolateral striatal activity (Barnes et al. Reference Barnes, Kubota, Hu, Jin and Graybiel2005; Doyon et al. Reference Doyon, Laforce, Bouchard, Gaudreau, Roy, Poirer, Bedard, Bedard and Bouchard1998; Hikosaka et al. Reference Hikosaka, Miyashita, Miyachi, Sakai and Lu1998; Jackson et al. Reference Jackson, Jackson, Harrison, Henderson and Kennard1995; Jog et al. Reference Jog, Kubota, Connolly, Hillegaart and Graybiel1999; Knopman & Nissen Reference Knopman and Nissen1991).

The implication of multiple decision-making systems on the calculation of value can also be seen in the effect of these two decision systems on changes in the valuation of a reward (Adams & Dickinson Reference Adams and Dickinson1981; Balleine & Dickinson Reference Balleine and Dickinson1998; Colwill & Rescorla Reference Colwill and Rescorla1985; Dickinson Reference Dickinson1980; Reference Dickinson1985). Classically, these differences are measured by (1) training an agent to take an action (or a sequence of actions) to receive a reward R, and then, (2) changing the value of reward R to the agent, usually in a different context. The value of a reward can be changed by providing excess amounts of the reward (satiation, Balleine & Dickinson Reference Balleine and Dickinson1998) or by pairing the reward with an aversive stimulus, such as lithium chloride (devaluation, Adams & Dickinson Reference Adams and Dickinson1981; Colwill & Rescorla Reference Colwill and Rescorla1985; Holland & Rescorla Reference Holland and Rescorla1975; Holland & Straub Reference Holland and Straub1979; Nelson & Killcross Reference Nelson and Killcross2006; Schoenbaum et al. Reference Schoenbaum, Roesch and Stalnaker2006a). Finally, (3) the agent is provided the chance to take the action. If the action-selection process takes into account the current value of the reward, then the agent will modify its actions in response to the change, but if the action-selection process is an association between the situation and the action (hence does not take into account the value of the reward), the agent will not modify its response. Lesions to ventral striatum (Corbit et al. Reference Corbit, Muir and Balleine2001; Schoenbaum et al. Reference Schoenbaum, Stalnaker and Roesch2006c) and prelimbic medial prefrontal cortex (Killcross & Coutureau Reference Killcross and Coutureau2003) or orbitofrontal cortex (Ostlund & Balleine Reference Ostlund and Balleine2007; Schoenbaum et al. Reference Schoenbaum, Roesch and Stalnaker2006a; Reference Schoenbaum, Setlow, Saddoris and Gallagher2006b) discourage devaluation, whereas lesions to dorsolateral striatum (Yin et al. Reference Yin, Knowlton and Balleine2004; Yin & Knowlton Reference Yin and Knowlton2004; Yin et al. Reference Yin, Knowlton and Balleine2006) and infralimbic cortex (Coutureau & Killcross Reference Coutureau and Killcross2003; Killcross & Coutureau Reference Killcross and Coutureau2003) encourage devaluation processes. Lesions to entorhinal cortex (Corbit & Balleine Reference Corbit and Balleine2000) and dorsomedial striatum (Adams et al. Reference Adams, Kesner and Ragozzino2001; Ragozzino et al. Reference Ragozzino, Jih and Tzavos2002a; Reference Ragozzino, Ragozzino, Mizumori and Kesner2002b; Yin et al. Reference Yin, Knowlton and Balleine2005) disrupt flexibility in the face of predictability changes (contingency degradation), whereas lesions to dorsolateral striatum do not (Yin & Knowlton Reference Yin and Knowlton2006).

It is important to note that not all transitions need be from planning strategies to habit strategies. Because planning strategies are flexible and learned quickly, while habit-based strategies are more rigid and learned more slowly, many tasks are solved in their early stages through the planning system and in their late stages through the habit system (Dickinson Reference Dickinson1980; Hikosaka et al. Reference Hikosaka, Nakahara, Rand, Sakai, Lu, Nakamura, Miyachi and Doya1999; Packard & McGaugh Reference Packard and McGaugh1996; Restle Reference Restle1957). But the habit system can also learn in the absence of an available planning system (Cohen & Squire Reference Cohen and Squire1980; Day et al. Reference Day, Weisend, Sutherland and Schallert1999; Knowlton et al. Reference Knowlton, Squire and Gluck1994; Mishkin et al. Reference Mishkin, Malamut, Bachevalier, Lynch, McGaugh and Weinberger1984). Under appropriate conditions, well-developed automated responses can be overridden by controlled (planning-like) systems as in go/no-go tasks (Goldman et al. Reference Goldman, Rosvold and Mishkin1970; Gray & McNaughton Reference Gray and McNaughton2000; Isoda & Hikosaka Reference Isoda and Hikosaka2007; Iversen & Mishkin Reference Iversen and Mishkin1970) or reversal learning (Hirsh Reference Hirsh1974; Mackintosh Reference Mackintosh1974; Ragozzino et al. Reference Ragozzino, Jih and Tzavos2002a). Which system drives behavior at which time depends on parameters of the specific task (Curran Reference Curran1995; McDonald & White Reference McDonald and White1994; O'Keefe & Nadel Reference O'Keefe and Nadel1978; Redish Reference Redish1999) and may even differ between individuals under identical experimental conditions. In many cases, identical behaviors can be driven by the two systems, and only specialized probe trials can differentiate them (Barnes Reference Barnes1979; Curran Reference Curran2001; Hikosaka et al. Reference Hikosaka, Nakahara, Rand, Sakai, Lu, Nakamura, Miyachi and Doya1999).

3.2. The planning system

The planning system requires recognition of a situation and/or context S, identification of the consequences of taking action a in situation S (recognition of a means of achieving outcome O), and the evaluation of the value of achieving outcome O. This system selects the most appropriate action by considering the potential consequences of that action. The key behavioral parameters involved in this system are fast storage and slow retrieval. As noted earlier, retrieval within this system can be slow because the calculation of value at each step requires processing through the consideration of possibilities. Because the consideration of possibilities does not commit one to a single choice, this system is flexible in its behavioral choices. Because the value of taking action a in situation S is calculated from the value of achieving expected outcome O, which is calculated online from the current needs of the agent, if the desire (need) for the outcome is changed (even in another context), the value calculation can reflect that change.

Computationally, the planning system is likely to require three interacting components: a situation-recognition component, which classifies the complex interaction of contexts and stimuli to identify the situation in which the animal finds itself; a prediction component, which calculates the consequences of potential actions; and an evaluative component, which calculates the value of those consequences (taking into account the time, effort, and probability of receiving reward).

The situation-recognition component entails a categorization process, in which the set of available cues and contexts must be integrated with the agent's memory so as to produce a classification of the situation. This system is likely to arise in cortical sensory and association systems through competitive learning (Arbib Reference Arbib1995; Grossberg Reference Grossberg1976; Redish et al. Reference Redish, Jensen, Johnson and Kurth-Nelson2007; Rumelhart & McClelland Reference Rumelhart and McClelland1986). Mathematically, the cortical recognition system can be modeled with attractor network dynamics (Durstewitz et al. Reference Durstewitz, Kelc and Gunturkun1999; Reference Durstewitz, Seamans and Sejnowski2000; Kohonen Reference Kohonen1984; Laing & Chow Reference Laing and Chow2001; Redish Reference Redish1999; Seamans & Yang Reference Seamans and Yang2004; Wilson & Cowan Reference Wilson and Cowan1972; Reference Wilson and Cowan1973), in which a partial pattern can be completed to form a remembered pattern through recurrent connections within the structure (Hebb Reference Hebb1949/2002; Hertz et al. Reference Hertz, Krogh and Palmer1991; Hopfield Reference Hopfield1982). This content addressable memory provides a categorization process transforming the observed set of cues to a defined (remembered) situation that can be reasoned from (Redish et al. Reference Redish, Jensen, Johnson and Kurth-Nelson2007).

The prediction component entails a prediction of the probability that the agent will reach situation s _t+1, given that it takes action a in situation s _t: P(s _t+1|s _t, a). This functionality has been suggested to lie in the hippocampus (Jensen & Lisman Reference Jensen and Lisman1998; Reference Jensen and Lisman2005; Johnson & Redish Reference Johnson and Redish2007; Koene et al. Reference Koene, Gorchetchnikov, Cannon and Hasselmo2003) or frontal cortex (Daw et al. Reference Daw, Niv and Dayan2005). The hippocampus has been identified with stimulus-stimulus associations (Devenport Reference Devenport1979; Reference Devenport1980; Devenport & Holloway Reference Devenport and Holloway1980; Hirsh Reference Hirsh1974; Mackintosh Reference Mackintosh1974; White & McDonald Reference White and McDonald2002), episodic memory (Cohen & Eichenbaum Reference Cohen and Eichenbaum1993; Ferbinteanu & Shapiro Reference Ferbinteanu and Shapiro2003; Ferbinteanu et al. Reference Ferbinteanu, Kennedy and Shapiro2006; Squire Reference Squire1987), flexible behavior (Devenport et al. Reference Devenport, Devenport and Holloway1981b; Gray & McNaughton Reference Gray and McNaughton2000), including flexible navigation behavior (the cognitive map, i.e., spatial associations between stimuli; O'Keefe & Nadel Reference O'Keefe and Nadel1978), as well as in sequence learning (Agster et al. Reference Agster, Fortin and Eichenbaum2002; Cohen & Eichenbaum Reference Cohen and Eichenbaum1993; Fortin et al. Reference Fortin, Agster and Eichenbaum2002; Levy Reference Levy1996; Levy et al. Reference Levy, Sanyal, Rodriguez, Sullivan and Wu2005) (see Redish [1999] for review). Similar functionality has been proposed to lie in the frontal cortex (Daw et al. Reference Daw, Niv and Dayan2005), which has long been associated with the ability to recategorize situations (Baddeley Reference Baddeley1986; Clark & Robbins Reference Clark and Robbins2002; Dalley et al. Reference Dalley, Cardinal and Robbins2004; Isoda & Hikosaka Reference Isoda and Hikosaka2007; Jentsch & Taylor Reference Jentsch and Taylor1999; Quirk et al. Reference Quirk, Garcia and González-Lima2006; Rushworth et al. Reference Rushworth, Buckley, Behrens, Walton and Bannerman2007) with the storage of delayed events (Baddeley Reference Baddeley1986; Fuster Reference Fuster1997; Goldman-Rakic et al. Reference Goldman-Rakic, Funahashi and Bruce1990) and the anticipation of reward (Davis et al. Reference Davis, Donahue, Discenza, Waite and Ramus2006; Fuster Reference Fuster1997; Watanabe Reference Watanabe2007), as well as with sequence planning (Averbeck & Lee Reference Averbeck and Lee2007; Kolb Reference Kolb, Kolb and Tees1990; Mushiake et al. Reference Mushiake, Saito, Sakamoto, Itoyama and Tanji2006; Owen Reference Owen1997).

The evaluative component allows the calculation of value with each predicted outcome. Anatomically, the evaluative component is likely to include the amygdala (Aggleton Reference Aggleton1993; Dayan & Balleine Reference Dayan and Balleine2002; Phelps & LeDoux Reference Phelps and LeDoux2005; Rodrigues et al. Reference Rodrigues, Schafe and LeDoux2004; Schoenbaum et al. Reference Schoenbaum, Setlow and Ramus2003), the ventral striatum (nucleus accumbens) (Daw Reference Daw2003; Kelley Reference Kelley1999a; Reference Kelley1999b; Kelley & Berridge Reference Kelley and Berridge2002; Mogenson Reference Mogenson and Bandler1984; Pennartz et al. Reference Pennartz, Groenewegen and Lopes da Silva1994; Stefani & Moghaddam Reference Stefani and Moghaddam2006; Wilson & Bowman Reference Wilson and Bowman2005) and associated structures (Tindell et al. Reference Tindell, Berridge and Aldridge2004; Reference Tindell, Smith, Pecina, Berridge and Aldridge2006), and/or the orbitofrontal cortex (Feierstein et al. Reference Feierstein, Quirk, Uchida, Sosulski and Mainen2006; Padoa-Schioppa & Assad Reference Padoa-Schioppa and Assad2006; Plassmann et al. Reference Plassmann, O'Doherty and Rangel2007; Sakagami & Pan Reference Sakagami and Pan2007; Schoenbaum et al. Reference Schoenbaum, Setlow and Ramus2003; Reference Schoenbaum, Roesch and Stalnaker2006a; Volkow et al. Reference Volkow, Fowler and Wang2003). Neurons in the ventral striatum show reward correlates (Carelli Reference Carelli2002; Carelli et al. Reference Carelli, Ijames and Crumling2000; Carelli & Wondolowski Reference Carelli and Wondolowski2003; Lavoie & Mizumori Reference Lavoie and Mizumori1994; Martin & Ono Reference Martin and Ono2000; Miyazaki et al. Reference Miyazaki, Mogi, Araki and Matsumoto1998) and anticipate predicted reward (Martin & Ono Reference Martin and Ono2000; Miyazaki et al. Reference Miyazaki, Mogi, Araki and Matsumoto1998; Schultz et al. Reference Schultz, Apicella, Scarnati and Ljungberg1992; Yun et al. Reference Yun, Wakabayashi, Fields and Nicola2004). Neurons in the ventral pallidum are associated with the identification of hedonic signals (Tindell et al. Reference Tindell, Berridge and Aldridge2004; Reference Tindell, Smith, Pecina, Berridge and Aldridge2006). Both the hippocampus and prefrontal cortex project to ventral striatum (Finch Reference Finch1996; McGeorge & Faull Reference McGeorge and Faull1989; Swanson Reference Swanson2000), and ventral striatal firing patterns reflect hippocampal and prefrontal neural activity (Goto & Grace Reference Goto and Grace2005a; Reference Goto and Grace2005b; Kalivas et al. Reference Kalivas, Volkow and Seamans2005; Martin Reference Martin2001; Pennartz et al. Reference Pennartz, Lee, Verheul, Lipa, Barnes and McNaughton2004). Neurons in the orbitofrontal cortex encode parameters relating the value of potential choices (Padoa-Schioppa & Assad Reference Padoa-Schioppa and Assad2006; Schoenbaum & Roesch Reference Schoenbaum and Roesch2005).

These structures all receive strong dopaminergic input from the ventral tegmental area. Neurophysiologically, dopamine signals in the ventral striatum – measured by neural recordings from dopaminergic projection neurons (Schultz Reference Schultz1998; Reference Schultz2002) and from voltammetry signals in the ventral striatum itself (Roitman et al. Reference Roitman, Stuber, Phillips, Wightman and Carelli2004; Stuber et al. Reference Stuber, Wightman and Carelli2005) – show increased firing to unexpected rewards and to unexpected cues predicting rewards. In computational models of the habit system, these signals have been hypothesized to carry value-prediction error information (see further on). Much of the data seems to support a similar role for dopamine from ventral tegmental sources (de la Fuente-Fernandez et al. Reference de la Fuente-Fernandez, Phillips, Zamburlini, Sossi, Calne, Ruth and Stoessl2002; Ljungberg et al. Reference Ljungberg, Apicella and Schultz1992; Roitman et al. Reference Roitman, Stuber, Phillips, Wightman and Carelli2004; Stuber et al. Reference Stuber, Wightman and Carelli2005; Ungless et al. Reference Ungless, Magill and Bolam2004). However, detailed, anatomically instantiated computational models are not yet available for the planning system. Theories addressing dopamine's role in the planning system have included motivation and effort (Berridge 2006; Berridge & Robinson Reference Berridge and Robinson1998; Reference Berridge and Robinson2003; Niv et al. Reference Niv, Daw, Joel and Dayan2007; Robbins & Everitt Reference Robbins and Everitt2006; Salamone & Correa Reference Salamone and Correa2002; Salamone et al. Reference Salamone, Correa, Mingote and Weber2005; Reference Salamone, Correa, Farrar and Mingote2007) and learning (Ikemoto & Panksepp Reference Ikemoto and Panksepp1999; Reynolds et al. Reference Reynolds, Hyland and Wickens2001). An important open question, however, is to what extent dopamine is carrying the actual signal of motivation (Berridge Reference Berridge2007) and to what extent dopamine's effects are dependent on corticostriatal synapses (Anagnostaras et al. Reference Anagnostaras, Schallert and Robinson2002; Li et al. Reference Li, Acerbo and Robinson2004; McFarland & Kalivas Reference McFarland and Kalivas2001; McFarland et al. Reference McFarland, Lapish and Kalivas2003; Nicola & Malenka Reference Nicola and Malenka1998; Reynolds & Wickens Reference Reynolds and Wickens2002). Finally, dopamine in the prefrontal cortex has also been hypothesized to have a role in controlling the depth of the categorization process (Durstewitz et al. Reference Durstewitz, Kelc and Gunturkun1999; Reference Durstewitz, Seamans and Sejnowski2000; Redish et al. Reference Redish, Jensen, Johnson and Kurth-Nelson2007; Seamans et al. Reference Seamans, Gorelova, Durstewitz and Yang2001; Seamans & Yang Reference Seamans and Yang2004; Tanaka Reference Tanaka2002; Reference Tanaka2006).

Neuropharmacologically, these systems, particularly the ventral striatum, are also highly dependent on mechanisms involving opioid signaling. Opioid signaling has been hypothesized to be involved in hedonic processes (Berridge & Robinson Reference Berridge and Robinson1998; Reference Berridge and Robinson2003; Kelley et al. Reference Kelley, Bakshi, Haber, Steininger, Will and Zhang2002). Consistent with these ideas, Levine and colleagues (Arbisi et al. Reference Arbisi, Billington and Levine1999; Levine & Billington Reference Levine and Billington2004) report that opioid antagonists directly interfere with the reported qualia of hedonic pleasure in humans eating sweet liquids, without interfering in taste discrimination. We have suggested that the multiple opioid receptors in the mammalian brain (μ, κ, δ; De Vries & Shippenberg Reference De Vries and Shippenberg2002; Herz Reference Herz1997; Reference Herz1998) are associated with an evaluation process identifying positive (euphorigenic, signaled by μ-opioid activation) and negative (dysphorigenic, signaled by κ-opioid signaling) evaluations (Redish & Johnson Reference Redish and Johnson2007). Whereas μ-receptor agonists are rewarding, euphorigenic, and support self-administration, κ-receptor agonists are aversive, dysphoric, and interfere with self-administration (Bals-Kubik et al. Reference Bals-Kubik, Herz and Shippenberg1989; Chavkin et al. Reference Chavkin, James and Goldstein1982; De Vries & Shippenberg Reference De Vries and Shippenberg2002; Herz Reference Herz1997, Reference Herz1998; Kieffer Reference Kieffer1999; Matthes et al. Reference Matthes, Maldonado, Simonin, Valverde, Slowe, Kitchen, Befort, Dierich, Meur, Dolĺe, Tzavara, Hanoune, Roques and Kieffer1996; Meyer & Mirin Reference Meyer and Mirin1979; Mucha & Herz Reference Mucha and Herz1985).Footnote ⁶

We have also proposed that part of the evaluation mechanism occurring during the search process (calculating the expected value from the agent's needs and the expected outcomes given a relation) may also involve the opioid system (Redish & Johnson Reference Redish and Johnson2007). This would predict a release of μ-opioid agonists (e.g., enkephalins) in anticipation of extreme rewards. Rats placed in a drug-associated location show a dramatic increase in released enkephalin in the nucleus accumbens relative to being placed in a saline-associated compartment, presumably in anticipation arising from the drug-associated compartment (Mas-Nieto et al. Reference Mas-Nieto, Wilson, Cupo, Roques and Noble2002).Footnote ⁷

3.3. The habit system

In contrast to the complexity of the planning system, the habit system entails a simple association between situation and action. Thus, the habit system requires recognition of a situation S, and a single, identified action to take within that situation. This simplicity allows the habit system to react quickly. However, this simplicity also makes the habit system rigid. A learned association essentially commits the agent to take action a in situation S. This means that it would be dangerous to store an association that was not reliable. Therefore, habit associations should only be stored after extensive experience with a consistent relation.

In contrast to the planning system, the habit system does not include any consideration of the potential outcome (i.e., there is no O term in the relation; Table 2). Therefore, in contrast to the planning system, the habit system does not include a prediction of available outcomes and cannot evaluate those potential outcomes online. Hence, it cannot take into account the current perceived needs (desires) of the agent. The habit system is still sensitive to the overall arousal levels of the agent. Thus, a hungry rat will run faster and work harder than a satiated rat (Bolles Reference Bolles1967; Munn Reference Munn1950; Niv et al. Reference Niv, Daw, Joel and Dayan2007). However, because the habit system does not reflect the current desires of the agent, the habit system will not modify responses when a reward is devalued. Similarly, the habit system cannot select multiple actions in response to a single situation.Footnote ⁹ This means that in navigation, the habit system can only take a single action in response to a given situation. For example, a rat with a damaged planning system cannot decide to turn left for water when it is thirsty on one day and turn right for food when it is hungry on the next day.Footnote ¹⁰

Computationally, there are very good models of how the habit system might work. These models have generally been based on the temporal difference instantiation of reinforcement learning (Daw Reference Daw2003; Daw et al. Reference Daw, Niv and Dayan2005; Reference Daw, Courville and Touretzky2006; Dayan et al. Reference Dayan, Kakade and Montague2000; Doya Reference Doya2000b; Montague et al. Reference Montague, Dayan and Sejnowski1996; Redish Reference Redish2004; Schultz et al. Reference Schultz, Dayan and Montague1997; Suri & Schultz Reference Suri and Schultz1999; Sutton & Barto Reference Sutton and Barto1998). In the simplest version of this model, each situation-action pair is associated with a value (termed Q(S, a); Sutton & Barto Reference Sutton and Barto1998).Footnote ¹¹ When an agent takes an action from a situation, the agent can compare the expected value of taking action a in situation S (i.e., Q(S, a)) with the observed value (the reward received minus the cost spent plus the value of being in the state the agent ended up in):

(1)

where R(t) is the reward observed, C(t) the cost spent, the most value one can get from the situation the agent finds itself in (S _new), and Q(S _old, a) is the estimated value of taking action a in the situation the agent was in (S _old). By updating Q(S _old, a) by δ, the agent moves its estimate Q(S _old, a) closer to its true value; γ is a discounting parameter (γ<1) which ensures that the time required to reach future rewards is taken into account (Daw Reference Daw2003; Sutton & Barto Reference Sutton and Barto1998). These models can be trained to learn complex situation-action sequences.

The slow development of a habit association has been most studied in contrast to the fast planning system. Lesions or inactivation of the dorsolateral striatum (Yin & Knowlton Reference Yin and Knowlton2004; Yin et al. Reference Yin, Knowlton and Balleine2004; Yin et al. Reference Yin, Knowlton and Balleine2006) and the infralimbic cortex (Coutureau & Killcross Reference Coutureau and Killcross2003; Killcross & Coutureau Reference Killcross and Coutureau2003) prevent the loss of devaluation with experience. Lesions or inactivation of the dorsolateral striatum (McDonald & White Reference McDonald and White1994; Packard & McGaugh Reference Packard and McGaugh1992; Reference Packard and McGaugh1996; Potegal Reference Potegal1972; White & McDonald Reference White and McDonald2002; Yin & Knowlton Reference Yin and Knowlton2004) or the parietal cortices (DiMattia & Kesner Reference DiMattia and Kesner1988; Kesner et al. Reference Kesner, Farnsworth and DiMattia1989) shift rats from response strategy to map strategies in navigation tasks.

As in the planning system, the habit system requires a situation-recognition component, in which the set of cues and contexts is integrated with the agent's memory to classify the situation, to allow retrieval of the correct association. As with our discussion in the planning system, we suggest that cortical sensory and association systems classify situations through competitive learning processes (Arbib Reference Arbib1995; Grossberg Reference Grossberg1976; Redish et al. Reference Redish, Jensen, Johnson and Kurth-Nelson2007; Rumelhart & McClelland Reference Rumelhart and McClelland1986). Although there are no neurophysiological data suggesting that this situation categorization system is anatomically separate from that used for the planning system, the identification of the habit system with networks involving dorsal and lateral aspects of striatum, and the planning system with networks involving more ventral and medial aspects of striatum, suggest that the specific cortical systems involved may differ. The association itself, including the mechanisms by which the situation signals are finally categorized to achieve a single action decision, have been hypothesized to include the afferent connections from cortex to dorsolateral striatum (Beiser et al. Reference Beiser, Hua and Houk1997; Houk et al. Reference Houk, Adams, Barto, Houk, Davis and Beiser1995; Samejima et al. Reference Samejima, Ueda, Doya and Kimura2005; Wickens Reference Wickens1993; Wickens et al. Reference Wickens, Reynolds and Hyland2003).

Neuropharmacologically, the habit system receives strong dopamine inputs from the substantia nigra pars compacta (SNpc). The dopamine signal has been well studied in the primate (Bayer & Glimcher Reference Bayer and Glimcher2005; Ljungberg et al. Reference Ljungberg, Apicella and Schultz1992; Mirenowicz & Schultz Reference Mirenowicz and Schultz1994; Schultz Reference Schultz1998; Reference Schultz2002; Waelti et al. Reference Waelti, Dickinson and Schultz2001). Like the dopamine neurons in the ventral tegmental area, dopamine neurons in SNpc increase firing in response to unexpected increases in expected value (via unexpected rewards or via cues that lead to an increased expectation of reward), and decrease firing in response to unexpected decreases in value (via lack of expected reward or via cues that lead to a decrease in the expectation of reward). This signal has been identified with the value-error signal δ in the temporal difference reinforcement learning algorithm (Barto Reference Barto, Houk, Davis and Beiser1995; Montague et al. Reference Montague, Dayan, Person and Sejnowski1995; Reference Montague, Dayan and Sejnowski1996; Schultz et al. Reference Schultz, Dayan and Montague1997), which can provide dopamine a role in training up associations. Dopamine has been shown to be critical to the learning of habitual associations (Faure et al. Reference Faure, Haberland, Fran and Massioui2005). However, the role of dopamine in learning and performance is still controversial (Berridge Reference Berridge2007; Cagniard et al. Reference Cagniard, Beeler, Britt, McGehee, Marinelli and Zhuang2006; Frank et al. Reference Frank, Seeberger and O'Reilly2004; Niv et al. Reference Niv, Daw, Joel and Dayan2007).

Cellularly, neurons in the dorsal striatum have been found to represent key parameters of the temporal difference reinforcement learning algorithm: for example, situation-action associations (Barnes et al. Reference Barnes, Kubota, Hu, Jin and Graybiel2005; Carelli & West Reference Carelli and West1991; Daw Reference Daw2003; Gardiner & Kitai Reference Gardiner and Kitai1992; Hikosaka et al. Reference Hikosaka, Nakahara, Rand, Sakai, Lu, Nakamura, Miyachi and Doya1999; Hikosaka et al. Reference Hikosaka, Nakamura and Nakahara2006; Jog et al. Reference Jog, Kubota, Connolly, Hillegaart and Graybiel1999; Kermadi et al. Reference Kermadi, Jurquet, Arzi and Joseph1993; Kermadi & Joseph Reference Kermadi and Joseph1995; Matsumoto et al. Reference Matsumoto, Hanakawa, Maki, Graybiel and Kimura1999; Miyachi et al. Reference Miyachi, Hikosaka, Miyashita, Kárádi and Rand1997; Schmitzer-Torbert & Redish Reference Schmitzer-Torbert and Redish2004; Tremblay et al. Reference Tremblay, Hollerman and Schultz1998); reward delivery (Daw Reference Daw2003; Schmitzer-Torbert & Redish Reference Schmitzer-Torbert and Redish2004; White & Hiroi Reference White and Hiroi1998;); and value signals (Daw Reference Daw2003; Kawagoe et al. Reference Kawagoe, Takikawa and Hikosaka2004; Nakahara et al. Reference Nakahara, Itoh, Kawagoe, Takikawa and Hikosaka2004). These signals develop in parallel with the development of automated behaviors (Barnes et al. Reference Barnes, Kubota, Hu, Jin and Graybiel2005; Itoh et al. Reference Itoh, Nakahara, Hikosaka, Kawagoe, Takikawa and Aihara2003; Jog et al. Reference Jog, Kubota, Connolly, Hillegaart and Graybiel1999; Samejima et al. Reference Samejima, Ueda, Doya and Kimura2005; Schmitzer-Torbert & Redish Reference Schmitzer-Torbert and Redish2004; Tang et al. Reference Tang, Pawlak, Volodymyr and West2007), through an interaction with dopamine signals (Arbuthnott & Wickens Reference Arbuthnott and Wickens2007; Centonze et al. Reference Centonze, Gubellini, Picconi, Calabresi, Giacomini and Bernardi1999; Picconi et al. Reference Picconi, Centonze, Håkansson, Bernardi, Greengard, Fisone, Cenci and Calabresi2003; Reynolds & Wickens Reference Reynolds and Wickens2002). Functional imaging data from humans playing sequential games show similar correlates to value, δ, and other parameters of these models (McClure et al. Reference McClure, Berns and Montague2003; Reference McClure, Laibson, Loewenstein and Cohen2004; O'Doherty Reference O'Doherty2004; O'Doherty et al. Reference O'Doherty, Dayan, Schultz, Deichmann, Friston and Dolan2004; Seymour et al. Reference Seymour, O'Doherty, Dayan, Koltzenburg, Jones, Dolan, Friston and Frackowlak2004; Tanaka et al. Reference Tanaka, Doya, Okada, Ueda, Okamoto and Yamawaki2004a).

3.4. Interactions between planning and habit systems

Generally, the planning system is engaged early; but with experience, behavioral control in repetitive tasks is transferred to the habit system. This has been observed in the navigation (O'Keefe & Nadel Reference O'Keefe and Nadel1978; Packard & McGaugh Reference Packard and McGaugh1996; Redish Reference Redish1999), animal conditioning (Balleine & Dickinson Reference Balleine and Dickinson1998; Dickinson Reference Dickinson1985; Yin et al. Reference Yin, Knowlton and Balleine2006), and human learning (Jackson et al. Reference Jackson, Jackson, Harrison, Henderson and Kennard1995; Knopman & Nissen Reference Knopman and Nissen1991; Poldrack et al. Reference Poldrack, Clark, Paré-Blagoev, Shohamy, Moyano, Myers and Gluck2001) literatures. However, in tasks which require behavioral flexibility, behavioral control can remain with the planning system, even in highly trained animals (Gray & McNaughton Reference Gray and McNaughton2000; Killcross & Coutureau Reference Killcross and Coutureau2003; McDonald & White Reference McDonald and White1994; Morris et al. Reference Morris, Garrud, Rawlins and O'Keefe1982; White & McDonald Reference White and McDonald2002).

For many tasks, both systems can drive behavior in the absence of the other (Cohen & Squire Reference Cohen and Squire1980; Nadel Reference Nadel, Schacter and Tulving1994; O'Keefe & Nadel Reference O'Keefe and Nadel1978; Squire Reference Squire1987), but some tasks can only be solved by one system or the other. For example, the water maze requires a flexible response to reach a hidden platform and requires hippocampal integrity to be learned quickly (Morris et al. Reference Morris, Garrud, Rawlins and O'Keefe1982; for a review, see Redish Reference Redish1999). If the flexibility of the required response is decreased (by, for example, starting the animal in the same location each trial), then the hippocampus no longer becomes necessary to reach the platform (Eichenbaum et al. Reference Eichenbaum, Stewart and Morris1990). Other tasks, such as mirror-writing or the serial reaction time task, require the slow recognition of regularities in situation-action associations, and they are learned at similar speeds by patients with damaged and intact planning systems (Cohen & Squire Reference Cohen and Squire1980; Ferraro et al. Reference Ferraro, Balota and Connor1993; Knopman & Nissen Reference Knopman and Nissen1987). Patients with damaged lateral striatal systems are impaired on these habit-based tasks (Doyon et al. Reference Doyon, Laforce, Bouchard, Gaudreau, Roy, Poirer, Bedard, Bedard and Bouchard1998; Ferraro et al. Reference Ferraro, Balota and Connor1993; Knopman & Nissen Reference Knopman and Nissen1991; Smith et al. Reference Smith, Brandt and Shadmehr2000; Yin & Knowlton Reference Yin and Knowlton2006). For some tasks, the planning system can “train up” the habit system, potentially through replay events occurring during subsequent sleep states (Buzsáki Reference Buzsáki1996; Hoffmann & McNaughton Reference Hoffmann and McNaughton2002; Marr Reference Marr1971; Pavlides & Winson Reference Pavlides and Winson1989; Redish Reference Redish1999; Redish & Touretzky Reference Redish and Touretzky1998; Wilson & McNaughton Reference Wilson and McNaughton1994). This transfer of information between systems can explain observations of incomplete retrograde amnesia with certain lesions (consolidation, Cohen & Squire Reference Cohen and Squire1980; Nadel & Bohbot Reference Nadel and Bohbot2001; Nadel & Moscovitch Reference Nadel and Moscovitch1997; Redish Reference Redish1999; Squire Reference Squire1987) but predicts that “consolidated memories” will be less flexible than unconsolidated memories (Redish & Touretzky Reference Redish and Touretzky1998).

The question of which system drives behavior when the two are put into conflict has only begun to be addressed computationally (Daw et al. Reference Daw, Niv and Dayan2005) and experimentally (Isoda & Hikosaka Reference Isoda and Hikosaka2007), but there is a large literature on behavioral inhibition, in which a changed, novel, or potentially dangerous or costly behavior is inhibited (Gray & McNaughton Reference Gray and McNaughton2000). This system seems to involve the prefrontal (Sakagami et al. Reference Sakagami, Pan and Uttl2006) and/or the hippocampal system (Gray & McNaughton Reference Gray and McNaughton2000), depending on the specific conditions involved. Whether the interaction entails the planning system overriding a developed habit (Gray & McNaughton Reference Gray and McNaughton2000) or an external system that mediates control between the two (Isoda & Hikosaka Reference Isoda and Hikosaka2007) is still unresolved. Whether such an external mediator can be identified with executive control (presumably, in the prefrontal cortex, Baddeley Reference Baddeley1986; Barkley Reference Barkley2001; Barkley et al. Reference Barkley, Edwards, Laneri, Fletcher and Metevia2001) is still a matter of open research.

• Vulnerability 8: Selective inhibition of the planning system

The habit system is inflexible, reacts quickly, “without thinking,” whereas the planning system is highly flexible and allows the consideration of possibilities. The habit and planning systems consist of different anatomical substrates. Pharmacological agents that preferentially impair function in structures involved in the planning system or preferentially enhance function in structures involved in the habit system would encourage the automation of behaviors. A shift back from habits to planned behaviors is known to involve the prefrontal cortex (Dalley et al. Reference Dalley, Cardinal and Robbins2004; Husain et al. Reference Husain, Parton, Hodgson, Mort and Rees2003; Isoda & Hikosaka Reference Isoda and Hikosaka2007; Iversen & Mishkin Reference Iversen and Mishkin1970), and has been hypothesized to involve executive function (Barkley Reference Barkley2001; Barkley et al. Reference Barkley, Edwards, Laneri, Fletcher and Metevia2001; Tomita et al. Reference Tomita, Ohbayashi, Nakahara, Hasegawa and Myashita1999). If pre-existing dysfunction exists within the inter-system control or if pharmacological agents disrupt this inter-system control, then the agent would develop habits quickly and would have difficulty disrupting those established habits. This vulnerability is distinguishable from the specific planning and habit vulnerabilities by its disruption of function of the planning system and/or its disruption of the inter-system conflict resolution mechanism. Thus, the other vulnerabilities affecting the planning system lead the planning system to make the incorrect choice; Vulnerability 8 makes it difficult for the planning system to correct a misguided habit system (Bechara Reference Bechara2005; Bechara et al. Reference Bechara, Dolan, Denburg, Hindes, Andersen and Nathan2001; Bickel et al. Reference Bickel, Miller, Yi, Kowal, Lindquist and Pitcock2007; Gray & McNaughton Reference Gray and McNaughton2000; Jentsch & Taylor Reference Jentsch and Taylor1999; Lubman et al. Reference Lubman, Yücel and Pantelis2004; Verdejo-Garcia et al. Reference Verdejo-Garcia, Rivas-Pérez, López-Torrecillas and Pérez-Garcia2006).

3.5. Summary: Decision-making systems

The decision-making system in the mammal is hypothesized to consist of two subsystems: a planning system, based on the evaluation of potential possibilities (e.g., relationships), and a habit system, based on the association of specific actions with specific situations (e.g., associations), both of which require a situation-recognition system, in which observed cues are categorized into situations (e.g., the S terms in the previous formulations). Correct decision-making depends on the integrity of each of these systems (see Figure 1).

Figure 1.

Structure of decision making in mammalian agents. Components of the more flexible, planning-based system are shown in light gray; components of the less flexible, habit-based system are shown in dark gray. Components involved in both are shown in gradient color.

3.6. Additional failure points

The eight vulnerabilities identified so far are certainly an incomplete list of the potential failure points of the decision-making system. The description of the decision-making system is, by necessity, incomplete. For example, we have not addressed the question of discounting and impulsivity. Nor have we addressed the question of learning rates.

• Vulnerability 9: Over-fast discounting processes

Both the planning and habit systems need to take into account the probability and the delay before an expected goal will be achieved (Ainslie Reference Ainslie1992; Reference Ainslie2001; Mazur Reference Mazur2001; Stephens & Krebs Reference Stephens and Krebs1987). In the planning system, this can be calculated online from the expected value of the expected goal given the searched sequence; in the habit system, this would have to be cached as part of the stored value function. The specific mechanism (and even the specific discounting function) are still a source of much controversy (for a review, see Redish & Kurth-Nelson [in press] in Madden et al. [in press]), but the long-term discounting of future rewards is well established. If an agent discounts too strongly, it will overemphasize near-term rewards and underemphasize far-future costs. Because addictions often involve near-term pleasures and far-future costs, faster-than-normal discounting could drive an agent to underestimate the far-future costs and to choose those near-term pleasures. A number of studies have found that addicts discount faster than non-addicts (Alessi & Petry Reference Alessi and Petry2003; Bickel & Marsch Reference Bickel and Marsch2001; Kirby et al. Reference Kirby, Petry and Bickel1999; Madden et al. Reference Madden, Petry, Badger and Bickford1997; Madden et al. Reference Madden, Bickel and Jacobs1999; Odum et al. Reference Odum, Madden and Bickel2002; Petry Reference Petry2001; Petry & Bickel Reference Petry and Bickel1998; Petry et al. Reference Petry, Bickel and Arnett1998; Vuchinich & Simpson Reference Vuchinich and Simpson1998).

• Vulnerability 10: Changes in learning processes

Other unincorporated components include changes in learning processes (such as over-attention to cues or learning rates being too high or too low). More detailed models of each of the systems will be required before it will be possible to make strong claims about the consequences of such a potential failure point.

However, the hypothesis put forward in this paper that addiction is a consequence of falling victim to vulnerabilities (failure modes) in the decision-making system lays out a research paradigm with important consequences both for what (and how) research questions should be addressed as well as for drug-treatment paradigms and drug-control policies.

These vulnerabilities in the decision-making system may arise from individual predisposition (either due to genetic or social/environmental factors) as well as from drug- or behavior-driven interactions with the decision system. In the second half of this article, we address the interactions and implications of each of the previously identified vulnerabilities with drugs and behaviors of abuse as well as the policy and treatment consequences of the theory.

4. Addiction as vulnerabilities in decision-making

The unified framework for decision-making described above has potential access points through which it can be driven to make maladaptive choices, particularly choices which entail seeking of certain drugs or behaviors. Ten key vulnerabilities can be directly identified with this unified decision-making system as outlined earlier. They are summarized in Table 4 and related to the current theories in Table 5.

Table 4.

Failure modes in the decision-making system provide a taxonomy of vulnerabilities to addiction

Note. Because Vulnerability 6 affects the situation S term in both planning and habit systems, we identify it as affecting “situation-recognition.” Vulnerability 8 affects the interaction between the planning and habit systems. Vulnerabilities 9 and 10 can affect components of the planning and habit systems. Detailed models of the effects of these last two vulnerabilities on the systems are as yet unavailable.

Table 5.

Relation between identified vulnerabilities and current theories of addiction

^aBecker and Murphy (Reference Becker and Murphy1988); Harris & Gewirtz (Reference Harris and Gewirtz2005); Koob and Le Moal (Reference Koob and Le Moal2006).

^bBecker and Murphy (Reference Becker and Murphy1988); Koob and Le Moal (Reference Koob and Le Moal1997; 2001; Reference Koob and Le Moal2005; 2006); Solomon and Corbit (Reference Solomon and Corbit1973; Reference Solomon and Corbit1974).

^cKalivas and Volkow (Reference Kalivas and Volkow2005); Volkow et al. (Reference Volkow, Fowler and Wang2003; Reference Volkow, Fowler, Wang and Swanson2004); Wise (Reference Wise2004).

^dBerridge and Robinson (Reference Berridge and Robinson1998; 2003); Robinson and Berridge (Reference Robinson and Berridge1993; Reference Robinson and Berridge2001; Reference Robinson and Berridge2003; Reference Robinson and Berridge2004).

^eDrummond (Reference Drummond2001); Goldman et al. (Reference Goldman, Brown, Christiansen, lsn and lsn1987; Reference Goldman, Boca, Darkes, Leonard and lsn1999); Halikas (Reference Halikas, Lowinson, Ruiz, Millman and Langrod1997); Hommer (Reference Hommer1999).

^fBernheim and Rangel (Reference Bernheim and Rangel2004); Di Chiara (Reference Di Chiara1999); Redish (Reference Redish2004).

^gCuster (Reference Custer1984); Griffiths (Reference Griffiths1994); Langer and Roth (Reference Langer and Roth1975); Redish et al. (Reference Redish, Jensen, Johnson and Kurth-Nelson2007); Sylvain et al. (Reference Sylvain, Ladouceur and Biosvert1997); Wagenaar (Reference Wagenaar1988).

^hAinslie (Reference Ainslie1992; 2001); Ainslie and Monterosso (Reference Ainslie and Monterosso2004); Bickel and Marsch (Reference Bickel and Marsch2001); Giordano et al. (Reference Giordano, Bickel, Loewenstein, Jacobs, Marsch and Badger2002); Odum et al. (Reference Odum, Madden and Bickel2002).

ⁱBickel et al. (Reference Bickel, Miller, Yi, Kowal, Lindquist and Pitcock2007); Everitt et al. (Reference Everitt, Dickinson and Robbins2001); Everitt and Wolf (Reference Everitt and Wolf2002); Nelson and Killcross (Reference Nelson and Killcross2006); Robbins & Everitt (Reference Robbins and Everitt1999).

^jGoldman et al. (Reference Goldman, Brown, Christiansen, lsn and lsn1987; 1999); Jones et al. (Reference Jones, Corbin and Fromme2001); Oei and Baldwin (Reference Oei and Baldwin2002).

Some of these failure modes exist as prior conditions, making an agent more vulnerable to the addictive process, whereas other failure modes are driven by direct interactions with the drugs themselves.

5. Drugs and the taxonomy of vulnerabilities

The 10 vulnerabilities listed here provide a taxonomy of potential problems with decision processes.Footnote ¹⁴ Because neuromodulators (such as acetylcholine, serotonin, norepinephrine, and dopamine) are involved throughout the decision-making system (learning relations, storing and evaluating relations, recognition of situations S, etc.), drugs of abuse are unlikely to access only one subsystem. Because there are differences in these vulnerabilities, any specific drug is also unlikely to access all 10 vulnerabilities. Because behavioral control involves the entire decision-making systems, behavioral problems such as gambling are likely to arise from an interaction of vulnerabilities. Although each vulnerability can drive an agent to return to the addictive choice, each vulnerability also produces a characteristic symptomology and can thus be separately identifiable within an agent.

Different drugs are likely to access different vulnerabilities. For example, whereas opiates are generally euphorigenic on initial use (Koob & Le Moal Reference Koob and Le Moal2006; Vulnerability 3), nicotine is often dysphoric on initial use (Heishman & Henningfield Reference Heishman and Henningfield2000; Perkins Reference Perkins2001; Perkins et al. Reference Perkins, Grobe, Weiss, Fonte and Caqquila1996; making Vulnerability 3 unlikely). However, continued use of nicotine can produce strong allostatic changes (Benowitz Reference Benowitz1996; Koob & Le Moal Reference Koob and Le Moal2006; Vulnerability 2), which produce a very strong need to return levels to normal (Fiore Reference Fiore2000). Nicotine also produces increases in the firing of dopaminergic neurons (Balfour et al. Reference Balfour, Wright, Benwell and Birrell2000; Dani & Heinemann Reference Dani and Heinemann1996; Pidoplichko et al. Reference Pidoplichko, DeBiasi, Williams and Dani1997), which suggests that it can also accesses the overvaluation vulnerability (Vulnerability 7). Cocaine use automates particularly quickly (Miles et al. Reference Miles, Everitt and Dickinson2003) and produces a very strong cued relapse (Altman et al. Reference Altman, Everitt, Robbins, Glautier, Markou, Nutt, Oretti and Phillips1996; Childress et al. Reference Childress, Ehrman, Rohsenow, Robbins, O'Brien, Lowinson, Ruiz and Millman1992; Reference Childress, Hole, Ehrman, Robbins, McLellan and O'Brien1993; O'Brien et al. Reference O'Brien, Childress, McLellan, Ehrman, O'Brien and Jaffe1992), suggesting that it also accesses the overvaluation vulnerability (Redish Reference Redish2004; Vulnerability 7), presumably through its direct effects on dopamine (Kuhar et al. Reference Kuhar, Ritz, Sharkey, Clouet, Asghar and Brown1988; Ritz et al. Reference Ritz, Lamb, Goldberg and Kuhar1987; Stuber et al. Reference Stuber, Roitman, Phillips, Carelli and Wightman2004; Reference Stuber, Wightman and Carelli2005). However, chronic cocaine use can also produce long-term changes in μ- and κ-opiate receptor levels (Shippenberg et al. Reference Shippenberg, Chefer, Zapata and Heidbreder2001), suggesting it can also access the allostatic vulnerability (Vulnerability 2). The appendix provides more details on drugs and the potential vulnerabilities involved with each.

6. Future work is still needed

The thesis of this review is that addiction arises from vulnerabilities inherent in the decision-making system within the brain. Susceptibility to these vulnerabilities arises through an interaction among the genetics of the individual, the development environment, the social milieu, and the behavioral experience of the individual. We have outlined several vulnerabilities that arise from current theories of the mammalian decision-making system. However, it is important to note that the understanding of that decision-making system is still incomplete.

Exactly what differentiates the planning and habit systems is still being debated (e.g., Daw et al. Reference Daw, Niv and Dayan2005; Dayan & Balleine Reference Dayan and Balleine2002; Redish & Johnson Reference Redish and Johnson2007). Detailed computational models of the habit system have been developed (Montague et al. Reference Montague, Dayan and Sejnowski1996; Samejima et al. Reference Samejima, Ueda, Doya and Kimura2005; Suri & Schultz Reference Suri and Schultz1999, but see Berridge Reference Berridge2007, for an alternate view), including how those systems could produce addiction-like behavior (Bernheim & Rangel Reference Bernheim and Rangel2004; Redish Reference Redish2004). But computational models of the planning system are still in their earliest stages (Daw et al. Reference Daw, Niv and Dayan2005; Johnson & Redish Reference Johnson and Redish2005; Zilli & Hasselmo, Reference Zilli and Hasselmo2008). How these systems interact to produce behavior is still unknown.

A number of open questions still remain. For example, the decision-making theories discussed in this article are primarily about reinforcement (delivery of unexpected reward) and disappointment (non-delivery of expected reward). The role of aversion (delivery of punishment) and relief (non-delivery of expected punishment) in these decision-making systems is still unresolved. Negative symptoms clearly play important roles in addiction (Gawin Reference Gawin1991; Jaffe Reference Jaffe, O'Brien and Jaffe1992; Koob & Le Moal Reference Koob and Le Moal2001; 2005; 2006; O'Brien et al. Reference O'Brien, Childress, McLellan, Ehrman, O'Brien and Jaffe1992). How to incorporate those negative symptoms beyond homeostatic (Vulnerability 1) and allostatic (Vulnerability 2) effects is still unclear. Detailed decision-making models in the face of aversion and relief may help elucidate these issues. The fear-conditioning and extinction literature (Domjan Reference Domjan1998; Myers & Davis Reference Myers and Davis2002; Reference Myers and Davis2007) and the roles of the amygdala (Paré et al. Reference Paré, Quirk and Ledoux2004; Phelps & LeDoux Reference Phelps and LeDoux2005; Rodrigues et al. Reference Rodrigues, Schafe and LeDoux2004) and prefrontal cortex (Milad & Quirk Reference Milad and Quirk2002; Quirk et al. Reference Quirk, Garcia and González-Lima2006) therein are likely to be important starting points for these models.

Similarly, the key parameters that underlie individual differences are still unknown, including whether those key parameters are genetic, environmental, or some combination thereof (Kreek et al. Reference Kreek, Nielsen, Butelman and LaForge2005; Volkow & Li Reference Volkow and Li2005b). Models of decision-making can provide candidate variables that may vary across the population, which may change susceptibilities to specific vulnerabilities and would lead to individual reactions to drugs of abuse or potentially addictive behaviors.

The key social definition of a problem addiction relates to the cost to the individual and to society of the addiction. Whereas methamphetamine addiction is a terrible burden on society and thus leads to extreme measures taken to prevent it, caffeine addiction leads to a minor inconvenience to an individual and little or no burden on society. In part, we believe that these differences arise from the different vulnerabilities impacted by these drugs. Problem gambling is often classified as an addiction due in large part to the extreme costs paid by “addicted” individuals. Whether other behaviors, such as shopping or Internet use, should be counted as addictions is an open question (Holden Reference Holden2001). In this article, we have proposed a new framework for understanding addiction. This new framework provides a new definition of addiction itself as decisions made due to failure modes in the decision-making system. How serious each failure mode is, whether it should be treated, and how it should be treated, are clinical and policy issues that need to be addressed in the future.

The list of vulnerabilities laid out in this target article are certainly incomplete. There are likely to be other processes beyond decision-making that can drive errors, including errors in probability recognition (Kahneman et al. Reference Kahneman, Slovic and Tversky1982), different responses to gains and losses (Kahneman & Tversky Reference Kahneman and Tversky2000), and errors in memory itself (Schacter Reference Schacter2001). We have not fully explored the potential interactions between the decision-making vulnerabilities, nor have we fully explored the interaction between decision-making vulnerabilities and other memory-based errors.

Clinically, we cannot yet relate these potential vulnerabilities to other action-selection and decision-making disorders such as obsessive-compulsive disorder, Tourette's syndrome, depression, mania disorders, anxiety disorders, impulsivity disorders, and so on. However, we believe that the paradigm laid out in this article (taking a basic-science understanding of action-selection and decision-making and identifying failure modes) is likely to be fruitful for understanding many other psychiatric disorders beyond addiction.

More importantly, however, we do not yet have specific clinical instruments with which to identify the presence or absence of each vulnerability within an individual, nor do we have specific clinical treatments (pharmacological, behavioral, or otherwise) to suggest. Our hope, however, is that the framework laid out in this article and the identification of these vulnerabilities can lead to research aimed at identification and treatment.

More work elucidating an understanding of the mammalian decision-making system is clearly needed, but we believe that the current understanding of this system can already illuminate addictive processes. It is our belief that an interaction between basic science research on decision-making, basic science research on the neurophysiological effects of addictive substances and behaviors, and the clinical consequences of addiction will illuminate both processes and will provide new avenues for the treatment of addiction.

The origin of addictions by means of unnatural decision Ahmed, Serge H. Movement Adaptation Cognition, Université Bordeaux 2, Université Bordeaux 1, CNRS, UMR5227, Bordeaux 33076, France. sahmed@u-bordeaux2.frhttp://www.inb.u-bordeaux2.frVulnerabilities to addiction must have their impact through the common currency of discounted rewardFootnote ¹ Ainslie, George Veterans Affairs Medical Center, Coatesville, PA 19320. George.Ainslie@va.govwww.picoeconomics.comAddiction, procrastination, and failure points in decision-making systems Andreou, Chrisoula Department of Philosophy, University of Utah, Salt Lake City, UT 84112. c.andreou@utah.eduhttp://www.hum.utah.edu/philosophy/?module=facultyDetails&personId=137&orgId=300Computing motivation: Incentive salience boosts of drug or appetite states Berridge, Kent C., Zhang, Jun and Aldridge, J. Wayne Department of Psychology, University of Michigan, Ann Arbor, MI 48109; Departments of Neurology and Psychology, University of Michigan, Ann Arbor, MI 48109. berridge@umich.eduhttp://www-personal.umich.edu/~berridge/junz@umich.edu http://www.lsa.umich.edu/psych/junz/jwaynea@umich.edu http://sitemaker.umich.edu/aldridgelab/homeAddiction science as a hedgehog and as a fox Bickel, Warren K. and Yi, Richard Department of Psychiatry, Center for Addiction Research, University of Arkansas for Medical Sciences, Little Rock, AR 72205. WBickel@uams.eduRYi@uams.eduSocial influence and vulnerability Boden, Joseph M. Christchurch Health and Development Study, Christchurch School of Medicine and Health Sciences, University of Otago, PO Box 4345, Christchurch, New Zealand. joseph.boden@otago.ac.nzImpulsivity, dual diagnosis, and the structure of motivated behavior in addiction Chambers, R. Andrew Department of Psychiatry, Indiana University School of Medicine, Institute of Psychiatric Research, Indianapolis, IN 46202. robchamb@iupui.eduGambling and decision-making: A dual process perspective Coventry, Kenny R. Cognition and Communication Research Centre, School of Psychology and Sport Sciences, University of Northumbria at Newcastle, Newcastle-Upon-Tyne, NE1 8ST, United Kingdom. kenny.coventry@northumbria.ac.ukhttp://kenny.coventry.googlepages.com/Different vulnerabilities for addiction may contribute to the same phenomena and some additional interactions Goudie, Andrew James, Field, Matt and Cole, Jon School of Psychology, University of Liverpool, Liverpool L69 7ZA, United Kingdom. ajg@liverpool.ac.ukm.field@liverpool.ac.ukj.c.cole@liverpool.ac.ukThe biopsychosocial and “complex” systems approach as a unified framework for addiction Griffiths, Mark D. Department of Gambling Studies, and Director, International Gaming Research Unit, Nottingham Trent University, Nottingham, NG1 4BU, United Kingdom. mark.griffiths@ntu.ac.ukhttp://www.ntu.ac.uk/research/school_research/social/staff/51652gp.htmlNeither necessary nor sufficient for addiction Hardcastle, Valerie Gray McMicken College of Arts and Sciences, University of Cincinnati, Cincinnati, OH 45221. Valerie.hardcastle@uc.eduhttp://asweb.artsci.uc.edu/collegemain/faculty_staff/profile_details.aspx?ePID=MjAyMjYwHuman drug addiction is more than faulty decision-making Hart, Carl L. and Krauss, Robert M. Division on Substance Abuse, New York State Psychiatric Institute and Department of Psychiatry, College of Physicians and Surgeons of Columbia University, New York, NY 10032; Department of Psychology, Columbia University, New York, NY 10027. clh42@columbia.eduhttp://substanceabuse.columbia.edu/hart.htm rmk@psych.columbia.eduhttp://www.columbia.edu/cu/psychology/commlab/Are addictions “biases and errors” in the rational decision process? Khalil, Elias L. Department of Economics, Monash University, Clayton, VIC 3800, Australia. elias.khalil@buseco.monash.edu.auwww.eliaskhalil.comRole of affective associations in the planning and habit systems of decision-making related to addiction Kiviniemi, Marc T. and Bevins, Rick A. Department of Health Behavior, State University of New York at Buffalo, Buffalo, NY 14214; mtk8@buffalo.eduhttp://sphhp.buffalo.edu/hb/faculty/kiviniemi_marc.php; Department of Psychology, University of Nebraska–Lincoln, Lincoln, NE 68588-0308. rbevins1@unl.eduhttp://www.unl.edu/psychoneuropharm/Negative affects are parts of the addiction syndrome Le Moal, Michel Neurocentre Magendie, Institut François Magendie, Université Victor Segalen – Bordeaux 2, 33077 Bordeaux cedex, France. lemoal@bordeaux.inserm.frExpanding the range of vulnerabilities to pathological gambling: A consideration of over-fast discounting processes Lejuez, Carl W. and Potenza, Marc N. Department of Psychology, Center for Addictions, Personality, and Emotion Research, University of Maryland, College Park, MD 20742; Department of Psychiatry and Child Study Center, Yale University School of Medicine, Connecticut Mental Health Center, New Haven, CT 06519. clejuez@psyc.umd.eduwww.addiction.umd.edu marc.potenza@yale.eduwww.med.yale.edu/psych/faculty/potenza.htmlAddiction: More than innate rationality Lende, Daniel H. Department of Anthropology, University of Notre Dame, Notre Dame, IN 46556. dlende@nd.eduhttp://neuroanthropology.wordpress.com/Bridging the gap between science and drug policy: From “what” and “how” to “whom” and “when” MacCoun, Robert J. Goldman School of Public Policy and Boalt School of Law, University of California at Berkeley, Berkeley, CA 94720-7320. maccoun@berkeley.eduhttp://ist-socrates.berkeley.edu/~maccoun/Linking addictions to everyday habits and plans Neal, David T. and Wood, Wendy Psychology and Neuroscience, Duke University, Durham, NC 27708. dneal@duke.eduwww.duke.edu/~dneal/ wendy.wood@duke.eduwww.duke.edu/~wwood/The disunity of Pavlovian and instrumental values Ostlund, Sean B. and Balleine, Bernard W. Department of Psychiatry and Biobehavioral Sciences, University of California, Los Angeles, Los Angeles, CA 90095; Department of Psychology, University of California, Los Angeles, Los Angeles, CA 90095. sostlund@ucla.eduballeine@psych.ucla.eduTiming models of reward learning and core addictive processes in the brain Ross, Don Finance, Economics and Quantitative Methods, Department of Philosophy, University of Alabama at Birmingham, Birmingham, AL 35294-1260; Department of Economics, University of Cape Town, Rondebosch 7701, South Africa. don.ross@uct.ac.zahttp://www.uab.edu/philosophy/ross.html http://www.commerce.uct.ac.za/Economics/staff/dross/default.aspCue fascination: A new vulnerability in drug addiction Sarnecki, John, Traynor, Rebecca and Clune, Michael Department of Philosophy, University of Toledo, Toledo, OH, 43606-3390; Department of English, University of South Florida, Tampa, FL 33620-5550. john.sarnecki@utoledo.eduhttp://homepages.utoledo.edu/jsarnec/index.htm rebecca.traynor@utoledo.edumclune@cas.usf.eduhttp://english.usf.edu/faculty/mclune/E pluribus unum? A new take on addiction by Redish et al. Stalnaker, Thomas and Schoenbaum, Geoffrey Department of Anatomy and Neurobiology, University of Maryland School of Medicine, Baltimore, MD 21201; Departments of Anatomy and Neurobiology and Psychiatry, University of Maryland School of Medicine, and Adjunct, UMBC Department of Psychology, Baltimore, MD 21201. tstal002@umaryland.eduschoenbg@schoenbaumlab.orghttp://www.schoenbaumlab.orgA mismatch with dual process models of addiction rooted in psychology Wiers, Reinout W., Havermans, Remco, Deutsch, Roland and Stacy, Alan W. Department of Clinical Psychological Science, Maastricht University, 6200 MD Maastricht, The Netherlands; Behavioural Science Institute, Radboud University Nijmegen, 6500 HC Nijmegen, The Netherlands; Department of Social Psychology, Würzburg University, 97070 Würzburg, Germany; Institute for Health Promotion and Disease Prevention Research, University of Southern California, Los Angeles, CA 91803. r.wiers@psychology.unimaas.nlr.havermans@psychology.unimaas.nldeutsch@psychologie.uni-wuerzburg.deastacy@usc.eduThe elephantine shape of addiction Yin, Henry Department of Psychology and Neuroscience, Duke University, Durham, NC27708. hy43@duke.eduAddiction as vulnerabilities in the decision process Redish, A. David, Jensen, Steve and Johnson, Adam Department of Neuroscience, University of Minnesota, Minneapolis, MN 55455; Graduate Program in Computer Science, University of Minnesota, Minneapolis, MN 55455; Graduate Program in Neuroscience and Center for Cognitive Sciences, University of Minnesota, Minneapolis, MN 55455. redish@umn.eduhttp://umn.edu/~redish/ jens0491@umn.edujohn5726@umn.edu