R1. Affective perception

The commentaries by LoBue, Todd & Thompson, Greening & Mather, and Vuilleumier discuss concepts related to affective perception, including the perennial question of automaticity.

The Cognitive-Emotional Brain (Pessoa Reference Pessoa2013) specifies multiple mechanisms for affective modulation of visual processing. LoBue also suggests that the field should investigate “multiple pathways” that imbue emotion-laden stimuli with their properties. A particularly compelling aspect of her work is that she seeks to devise experiments that can unravel diverse sources of bias for emotionally valenced stimuli, including both bottom-up-like and top-down-like contributions.

Despite the “pluralistic” account of affective vision in the book (Ch. 2–4 and 7), it missed an important component. Todd & Thompson corrected this omission by describing the contributions of the locus coeruleus to “affect-biased attention,” as recently developed in the “biased attention via norepinephrine” (BANE) model (Markovic et al. Reference Markovic, Anderson and Todd2014). Their point is important for a more general reason, too. The goal of describing multiple mechanisms of affective attention was to highlight that the field needs to move past the idea of “single structures” or even “single circuits.” The omission of an important mechanism demonstrates that the list is far from complete; it is likely that several additional mechanisms play important roles in affective vision, too.

Vuilleumier, a major contributor to our understanding of emotional perception, argues that it is time to move past general questions like “Is emotional perception automatic?” to specific, testable mechanistic questions, and that the proposals we offered remain too abstract. Although it is true that more mechanistic accounts are important for the field to advance further (for an exampleof a formal model, see Grossberg et al. Reference Grossberg, Chang and Cao2014), at this point of model development, my goal was to describe a general (“abstract”) framework that, if persuasive to others, would lend itself to further refinement. Somewhat ambitiously, the situation is analogous to the description of the biased competition model (Desimone & Duncan Reference Desimone and Duncan1995), which had to await a few years before more mechanistic notions of competition based on receptive fields were developed based on subsequent empirical data (Luck et al. Reference Luck, Chelazzi, Hillyard and Desimone1997; Reynolds et al. Reference Reynolds, Chelazzi and Desimone1999).

Greening & Mather discuss their arousal-biased competition model (Mather & Sutherland Reference Mather and Sutherland2011). The model describes how arousing stimuli enhance perceptual processing of other neutral stimuli. It thus clearly covers territory not addressed by the dual competition model (Ch. 7). But one of the original goals of my proposal was to describe how competition takes place when items have affective and/or motivational significance, including situations that may involve both negative and positive items (Fig. R1). This is something that is not addressed by the arousal-biased completion model. For example, in a recent study, we investigated the interactions between reward and threat on brain and behavior during a visual discrimination task (Hu et al. Reference Hu, Padmala and Pessoa2013). Reward was manipulated by linking the task-relevant stimulus categories (pictures of houses or high-rise buildings) to reward or non-reward, whereas threat was manipulated using task-irrelevant backgrounds (of different colors) that were previously paired or unpaired with aversive electrical stimulation. Behaviorally, an unspecific effect of reward was observed: responses were faster during reward versus non-reward conditions (say, detecting houses when they were reward associated). More important, a reward by threat interaction was detected: The slowing of reaction time by irrelevant threat stimuli (discriminating between house and building stimuli was slowed when a background was a CS+) that was observed during non-reward was eliminated during reward conditions. Within the dual competition framework, the reward-associated visual stimulus was thus able to more effectively compete with the background CS+ stimulus (which itself was able to more effectively compete with the no-reward stimulus category).

Figure R1.

The dual competition model. Visual competition incorporates both affective and motivational factors, such that perception will reflect the interplay of multiple “forces” that sculpt it. In the hypothetical examples here, both a negative-image distractor and a reward-associated target influence perception. (A) Emotional images interfere with perception when the target item is positive but relatively weak. (B) In contrast, when the target item is associated with high reward, it wins the competition and in this way may reduce (or even eliminate) the deleterious impact of the negative image.

R2. The status of cognition and emotion: Together or separate?

The Cognitive-Emotional Brain describes how cognition and emotion go together, a notion that was welcomed across several commentaries (see sect. R4 below). However, some commentaries discussed contrasting viewpoints that espouse greater autonomy for cognition and emotion. According to these views, it is best to conceptualize cognition and emotion as somewhat separate processes, or even involving stronger separation in the mind-brain.

Winkielman, Inzlicht, & Harmon-Jones (Winkielman et al.) endorse what they call a modern separatist approach. They provide examples of what they refer to as affect and motivation with minimal cognition. More interesting, to my mind, is their discussion of cognitive control – “often seen as the paragon of higher cognition,” as they say – and the tight link that it has with emotion. In particular, they discuss the framework of Inzlicht and Legault (Reference Inzlicht, Legault, Forgas and Harmon-Jones2014), who formulated the affect alarm model of self-control. In this framework, affect is not a mere moderator of control, nor a by-product of self-control. Instead, it is essential to self-control and signals when control is needed by amplifying the detection of conflict and producing the urgency to conflict resolution. Steps in the direction of uniting cognitive and affective aspects of control were previously taken by Botvinick (Reference Botvinick2007) in his attempt to link conflict monitoring and decision making. Noting the importance of the anterior cingulate cortex to conflict monitoring, on the one hand, and evaluating action outcomes and learning to avoid aversive events, on the other hand, he described the outline of an initial “integrative account” in which these processes are interrelated. More generally, one of the most elaborate integrative accounts of anterior cingulate cortex function was developed by Shackman and colleagues (Reference Shackman, Salomons, Slagter, Fox, Winter and Davidson2011), who propose that the integration of negative affect, pain, and cognitive control in this region follows from anterior cingulate contributions to adaptive control, as discussed by Shackman, Fox, & Seminowicz (Shackman et al.).

Whereas I am sympathetic to Winkielman et al.'s suggestion that there is a need to “restore the balance” in accounts of behavior that favor cognition to the exclusion of emotion, and vice versa, I am less certain about the need to separate (even without dichotomizing) emotion and cognition. But Winkielmanet al.'s position is certainly defensible and constitutes an alternative position to the mind-brain than the one described in The Cognitive-Emotional Brain. And as the field evolves to describing how multiple processes interact during complex behaviors, the question of whether they should be described as cognitive or emotional will largely fade.

In contrast, Pascual-Leone, Pascual-Leone, & Arsalidou (Pascual-Leone et al.) adopt a traditional separatist stance in conceptualizing emotion and cognition. One of their chief arguments is that emotion is subjectively different from cognition, an issue to which I turn next. I strongly disagree, however, that the distinction between cognition and emotion is one of “two sorts of value,” as they propose – “truth” value for cognition and “vital” value for emotion – a suggestion that is unlikely to move the field forward.

Regarding the subjective difference between cognition and emotion, it is instructive to consider how brain circuits are linked to bodily states. For example, the central amygdala is at times viewed as a “controller of the brainstem” (Cardinal et al. Reference Cardinal, Parkinson, Hall and Everitt2002) and uses its widespread projections to the hypothalamus and brainstem nuclei to coordinate behavioral, autonomic, and neuroendocrine responses. Given the effects of these structures on bodily states and the regulation of the internal milieu, a more direct link with emotional “felt states” is established. Furthermore, cortical regions, including medial and orbital frontal sectors, as well as the insula, are strongly interconnected with brain regions in the basal forebrain, midbrain, and hindbrain that can directly affect bodily states, and be affected by them (Pessoa Reference Pessoa2013, Ch. 9). If researchers choose to call processes more directly linked to bodily states “emotional,” this would seem reasonable, as long as they also emphasize that such processes are strongly coordinated with others that have a less direct impact on body states. Broadly, this “emotion-cognition” coordination is always present.

Montemayor also espouses a separatist view of emotion and cognition. He misquotes me as saying that emotion is often “dumb,” and suggests that cognition requires inferential and conceptual capacities. But if we consider the frameworks by Botvinick (Reference Botvinick2007) and by Shackman and colleagues (Reference Shackman, Salomons, Slagter, Fox, Winter and Davidson2011) summarized above, for example, such description is misleading. To put it plainly: How can emotion be dumb if it is part of cognition (say, cognitive control)? Even more troubling, Montemayor suggests that there is a “deep kind of dissociation between emotion and cognition,” venturing as far as stating that it is a “normative kind of dissociation.” But to argue in favor of a normative distinction is to embrace the very dichotomy that even Montemayor acknowledges can be simplistic!

R3. Evolution: Implications for the understanding of emotion and cognition

Dobzhansky famously titled a paper, “Nothing in biology makes sense except in the light of evolution” (Dobzhansky Reference Dobzhansky1973). This is a blessing and a curse, however, because ad hoc evolutionary “explanations” are all too frequent. Here, I will discuss some of the Commentaries that discuss evolution in a way that I found problematic.

Pascual-Leone et al. suggest that an important difference between emotion and cognition is that emotion (they use the term affect to refer to a concept that is dichotomous with cognition) has innate-instinctual, evolutionary roots. In contrast, cognitions are always situation-bound and hence cannot be innate. But this argument runs the risk of suggesting that emotion has deep evolutionary roots for survival, while cognition runs atop and refines the “lower” level depending on context. Instead, the entire brain must be understood in evolutionary terms, not just emotion/motivation. A more appealing and powerful view is described by Pezzulo, Barca, & Friston (Pezzulo et al.), whose evolutionary perspective is one that “implies that the very fabric of cognition – from its ancestral origins to the most modern and sophisticated skills – is inextricably linked to utility, adaptivity, and meaningfulness.”

Another problematic line of reasoning is described by Montemayor, who subscribes to antiquated notions of an old emotional system that is autonomous. The brain has, of course, old systems. However, these do not exist within a “layered” architecture, with newer structures added on top of old ones, such that the ones on top control the ones at the bottom (cf. the triune brain). I would argue that evolutionary changes to brain circuits are such that “new” systems are embedded within “old” ones. This interweaving creates a web of functional and structural coupling in a way that blurs “old” and “new” (Fig. R2; see also discussion in sect. R3.4). To motivate this perspective, it is relevant to consider a network perspective of the relationship between function and structure in the brain – structure-function mappings (Ch. 8; Pessoa Reference Pessoa2014 and associated commentary). More important, the architecture of the brain includes extensive avenues for signal communicating, in general, and cognitive-emotional interactions, in particular. Two particularly interesting examples are those of the hypothalamus and the amygdala (Ch. 9).

Figure R2.

Brain evolution. (A) Layered brain evolution where newer structures/circuits are added atop older ones. (B) Embedded brain evolution where newer structures/circuits are integrated with older ones, thus expanding the functional repertoire of older regions. Multiple types of integration are possible, including “looped circuits,” widespread influences with both “descending” and “ascending” components, as well as projection systems from older structures that modulate newer ones. Basal ganglia refer to regions at the base of the brain; pallium refers to developmental structures that are precursors to cortex. Panels adapted with permission from Butler (Reference Butler and Squire2009) and originally based on MacLean (Reference MacLean1990). (C) Hypothalamic ascending connectivity illustrating how old and new brain parts interact, thus integrating diverse types of signals. Summary of four major pathways from the hypothalamus to the cerebral cortex on a flattened representation of the rat brain. The “basal ganglia” refer to the basal forebrain and the amygdala complex. Note that one of the indirect connections first descends to the brainstem. Key: BG: basal ganglia; BS: brainstem; CTX: cortex; HY: hypothalamus; TH: thalamus. Reproduced with permission from Risold et al. (Reference Risold, Thompson and Swanson1997).

R3.4. Neuroevolutionary perspective

Bos, Brummelman, & Terburg (Bos et al.) correctly point out that a weaknesses of the book is that it lacks a neuroevolutionary approach in describing brain function. I hope to remedy this in future treatments of brain architecture, structure-function mappings, and cognitive-emotional interactions. But, I disagree with Boset al. when they propose that, from an evolutionary perspective, “cognition can be seen as the tip of the emotional iceberg.” This is an unfortunate metaphor because it, once again, perpetuates the “layered” view of brain evolution (MacLean Reference MacLean1990). A traditional view is that cortex is a late addition to the brain plan and that it controls subcortex. Indeed, the idea of cortical inhibition of subcortex has a long history dating to early researchers, such as Hughlings-Jackson (see Parvizi Reference Parvizi2009). But what is the basic plan of the vertebrate brain? It is now understood that both cortex and subcortex are part of the plan. Figure R3 shows a proposed brain “archetype” by Striedter (Reference Striedter2005).

Figure R3.

Basic plan of the vertebrate brain. Reproduced with permission (Striedter Reference Striedter2005).

What do we know about the evolution of some “emotional” regions? The amygdala of mammals is composed of more than a dozen subregions. Chareyron and colleagues (Reference Chareyron, Banta Lavenex, Amaral and Lavenex2011) found that the lateral, basal, and accessory basal subregions are dramatically more “developed” in monkeys than in rats (based on morphological characteristics, such as cell counts and the volume of subregions). One possibility, as described by the authors, is that the differences between rats and monkeys are linked to their degree of connectivity with other brain structures, in line with the proposal of correlated evolution between components of functional systems (Barton & Harvey Reference Barton and Harvey2000). The lateral, basal, and accessory basal nuclei are more developed in primates than in rodents, and parallel the greater development of the cortical areas with which these nuclei are interconnected in primates. Chareyron and colleagues (Reference Chareyron, Banta Lavenex, Amaral and Lavenex2011) propose that such correlated evolution may be responsible for a higher convergence and integration of information in the primate amygdala, and that the relative development of these amygdala nuclei might be influenced by their interconnections with other brain structures – namely, their afferent and efferent connections (Amaral et al. Reference Amaral, Price, Pitkanen, Carmichael and Aggleton1992).

To sum up, an evolutionary perspective to brain function is absolutely needed, as suggested by Bos et al. But I disagree with them when they suggest that, in terms of evolution, the network perspective that I adopt runs into problems when faced with amygdala heterogeneity (i.e., multiple subregions). This is far from being the case. For example, the lateral and central amygdala, while strongly interrelated territories, are parts of different brain circuits that have had different evolutionary trajectories. Thus, I describe in Chapter 3 how the amygdala mobilizes both brain (via the lateral amygdala) and the body (via the central amygdala).

R4. Manifold forms of cognitive-emotion integration

Several of the commentators were enthusiastic about a framework in which cognition and emotion are strongly interactive and provided particular examples of the explanatory power of the interaction/integration framework. Foster & Keane suggested that the emotion of surprise constitutes a good example of when emotion and cognition are interdependent. Manfrinati suggests that an integration stance leads to proposals of the brain bases of moral processing that are more closely aligned with those by Moll and colleagues (Reference Moll, de Oliveira-Souza, Moll, Ignacio, Bramati, Caparelli-Daquer and Eslinger2005; Reference Moll, de Oliveira-Souza and Zahn2008a) instead of the dichotomist formulation by Greene (Greene et al. Reference Greene, Sommerville, Nystrom, Darley and Cohen2001). According to Manfrinati, moral judgment is the product of complex interactions between emotional and cognitive mechanisms. Egidi discusses how integration plays a role in understanding the impact of happy and sad moods on discourse and sentence comprehension. Verweij & Senior discuss potential implications of cognitive-emotional integration for the social sciences, broadly defined, and in particular, implications for theorizing within economics, political science, sociology, and anthropology. Petrolini argues that the integration framework described in the book can be successfully applied to psychopathology and, in particular, to the reasoning of delusional subjects. Gardiner discusses cognition-emotion integration in the context of music. Kiverstein & Miller suggest that integration is important for the understanding of the human social brain. Olds & Marewski suggest that cognitive-emotional integration needs to be taken seriously by those building formal/computational models of “cognitive architectures,” which as the name implies, largely ignore affective components. Froese proposes that the framework of The Cognitive-Emotional Brain should inform enactive approaches in the cognitive sciences. Indeed, I found his suggestion of extending the notion of functional connectivity and context dependency to include bodily and environmental dynamics fascinating and very worthwhile of investigation.

R5. Dual process theories

Kiverstein & Miller suggest that the framework of The Cognitive-Emotional Brain challenges dual process theories of cognition in general. This is a theme that pervades much of the book. For example, Chapter 4 discusses the notions of automatic and controlled processes and argues instead for a gradient of processing efficiency.

Common to all dual process models is the strong assumption of the existence of two qualitatively different mental systems, for example, “intuition” and “reasoning” (for a lucid discussion, see Keren & Schul Reference Keren and Schul2009). A common practice is to call the two components “System 1” and “System 2,” where the first is automatic/heuristic/reflexive and the second is controlled/analytic/reflective (Evans Reference Evans2008). But as others have expressed in the past, the idea of a dual system model is both slippery and conceptually unclear (Keren & Schul Reference Keren and Schul2009). For one thing, nearly all dual process models have as a central component the automatic versus controlled dichotomy, which is not a viable distinction, as discussed in The Cognitive-Emotional Brain.

As with the question of automatic versus controlled processing of emotion-laden visual stimuli (Ch. 4), the question of whether there are two systems in dual process models is not an entirely empirical one. This is because no single critical experiment can provide a final, definitive answer. In the end, however irresistible dichotomies are to the human mind (Kelso & Engstrøm Reference Kelso and Engstrøm2006; Newell Reference Newell and Chase1973), dichotomizing implies oversimplifying (Keren & Schul Reference Keren and Schul2009; Kruglanski et al. Reference Kruglanski, Erbs, Pierro, Mannetti and Chun2006). A continuous framework is better, albeit more complex (Kruglanski et al. Reference Kruglanski, Erbs, Pierro, Mannetti and Chun2006).

R6. Functional diversity of brain regions and networks, and cognitive ontologies

Kiverstein & Miller suggest that the integration framework of The Cognitive-Emotional Brain has implications for understanding so-called “cognitive ontologies.” Indeed, this is a theme that I have briefly addressed in recent papers (see Pessoa Reference Pessoa2014).

If brain regions are engaged in many processes based on the networks they are affiliated with in particular contexts, they should be engaged by a range of tasks. As described in the Précis, we recently (Anderson et al. Reference Anderson, Kinnison and Pessoa2013) characterized the function of brain regions in a multidimensional manner via their functional fingerprint (Passingham et al. Reference Passingham, Stephan and Kotter2002). Activations were classified in terms of task domains chosen to represent a range of mental processes, including perception, action, emotion, and cognition. The functional fingerprint for a given region thus represented both the set of domains that systematically engaged the region and the relative degree of engagement (see Fig. 13 of target article). Based on fingerprints, we calculated a diversity index to summarize the degree of functional diversity across the brain (see Fig. 14 of target article). The fingerprint concept was extended to brain networks, providing a way to compare them and to advance our understanding of the properties of constituent nodes.

Our findings showed that brain regions – and, importantly, large-scale networks – are very diverse functionally (see also Poldrack Reference Poldrack2006; Reference Poldrack2011). Beyond the descriptive aspects of the approach, it outlines a framework in which a region's function is viewed as inherently multidimensional: a vector defines the fingerprint of a region in the context of a specific domain structure. Although the domain that we explored used a task classification scheme from an existing database, it was not the only one possible. How should one define the domain structure? One hope is that cognitive ontologies can be defined that meaningfully carve the “mental” into stable categories (Bilder et al. Reference Bilder, Sabb, Parker, Kalar, Chu, Fox, Freimer and Poldrack2009; Price & Friston Reference Price and Friston2005). However, I believe that no single ontology will be sufficient. Instead, it is better to conceive of several task domains that are useful and complementary in characterizing brain function and/or behavior. Thus, a region's functional fingerprint needs to be understood in terms of a family of (possibly related) domains.

R7. What form of cognitive-emotional brain is better?

Views of the framework advocated in The Cognitive-Emotional Brain were mixed. Most commentators praised the integration framework and suggested that they may have implications in many related domains – even to the social sciences more generally. But some questioned the proposed form of interaction/integration between cognition and emotion and, in some cases, argued against it. Perhaps such state of affairs is not surprising in the end. Emotion “feels” different from cognition. These mental states and associated processes also appear, at first blush, to be subserved by fairly independent brain regions and circuits. Yet, when we consider the available neuroscientific data, attempts to characterize regions as either “emotional” or “cognitive” quickly break down. An architecture of rich interconnectivity leads to a structure-function mapping that is both one-to-many and many-to-one. Ultimately, looking at the brain from the perspective of one brain region at a time is bound to produce a highly distorted and, more critically, impoverished description of the brain. What is required is a framework where cognition and emotion are highly interactive, as I have argued in The Cognitive-Emotional Brain.