COGNITIVE DERIVED MODELS WITH NEURAL CONSTRAINTS

Selectively spared or impaired task performance (dissociations) involving different groups of stimuli, conceptual knowledge, and/or classes of words have typically been referred to as category-specific deficits. The finding of a consistent group of stimuli that are selectively spared or impaired during performance of cognitive tasks has been offered as evidence for an organizational framework that is common across human brains. Most often such dissociations are observed in confrontation naming tasks for categories of items such as animals, fruit and vegetables, artifacts such as tools, or in what appear to be whole grammatical classes such as a deficit in naming verbs relative to nouns or nouns relative to verbs (Damasio & Tranel, 1993).

An early model that was motivated by observations of selective dissociations of category representations was the Sensory/Functional Theory (SFT), which was initially advanced by Warrington and Shallice (1984). This model proposed the existence of two main types of semantic knowledge within the domains of animate and inanimate items: (1) sensory attributes (color, smell, sound, and tactile qualities), and (2) functional attributes (usefulness, value, where it is typically found). These two types of semantic knowledge were subsequently referred to as visual and verbal semantics, respectively, which have been adopted elsewhere in the neuropsychological literature without the specific definition first proffered by Warrington, Shallice, and colleagues (Warrington & McCarthy, 1987; Warrington & Shallice, 1984). An animate/inanimate distinction was first observed in patients who demonstrated selective impairment in distinguishing items in the animate and inanimate categories, with respect to naming or comprehension of items in those categories. Thus, the SFT model was initially limited in scope by only accounting for the brain organizing semantic memory into the broad distinction of separate stores for animate and inanimate items.

As more selective deficits and dissociations were detected that were limited to one specific category (e.g., animals, fruits and vegetables, and so on), the model was extended. The extended model suggests that the semantic memory encoding in the brain consists of multiple channels of processing within both the motor and sensory input systems. Differential activation or weightings of these channels during acquisition of a semantic memory for an item, and then collectively of a group of related items in a category, are the basis for the finer-grained categorical organization of semantic memory (e.g., impairment of groups of channels common to items in a specific category result as what appears as a category specific deficit) (Crutch & Warrington, 2003; Warrington & McCarthy, 1987).

This model of conceptual knowledge, which is distributed in terms of functional divisions with separate anatomic locations in modality-specific semantic subsystems, has been modified in several iterations. Humphreys and Forde (2001) proposed the Hierarchical Interference Theory (HIT), which advocates that “stored structural descriptions” encode object shape. The structural descriptions are proposed as an analog to visual semantics of the SFT (however, it leaves unclear where other visual concepts such as color, orientation, patterns, visual texture, and so on, are represented in this framework). The HIT model suggests that, because animals are more visually similar to one another than are the broad class of inanimate objects, their structural descriptions (best described as a visual recognition system; Humphreys & Riddoch, 2003, p. 266) would be anatomically located in closer proximity relative to those of inanimate items, with focal damage to the structural description system differentially affecting animals compared with items from other categories. The other key components of the HIT model are that semantic memory is otherwise organized by modality-specific stores for features and that concepts engage an interactive process between the perceptual and semantic levels of representations.

The extended SFT and HIT models advocate that features that items share in common can form a set of key, shared features that designate these items as members of a group. It is these shared feature-sets that constitute categorical organization in these two models (see Table 2). Within this framework, one would predict that living things are more strongly associated with sensory features, typically visual features, making them more readily identifiable by their physical appearance. Thus, the prediction from these models would be that impairment in visual feature knowledge would lead to a relatively substantial impairment in the category of living things.

Organization of category, feature, and object memory for models derived primarily with neural constraints

Another model predicated upon a variation of the distributed featural approach to categorical organization is the Organized Unitary Content Hypothesis (OUCH), which was proposed by Caramazza and co-workers (Caramazza et al., 1990; Hillis et al., 1990; Rapp et al., 1993). This model proposes that item classes (animals, artifacts, and so on) are organized by conceptual features that are highly intercorrelated and that items that have multiple features in common would be represented together. The OUCH model further advocates that these items with highly correlated features have representations that are “neurally contiguous” and are more likely susceptible to selective damage as a group of items. For example, animals are a group of items with highly intercorrelated features in terms of both sensory and functional properties. If a lesion in the brain were to damage a region that encodes features shared by animals, then the category of animal items would be differentially impaired compared to other groups of items.

There are two key differences between the OUCH model and the extended SFT and HIT models: (1) the intercorrelated features that designate a category can be across multiple feature domains (visual, functional, nonvisual sensory features, and so on) and not just focused on one feature domain in the OUCH, and (2) there is the assertion in OUCH that items with highly correlated features are encoded in the brain in neurally contiguous regions. This hypothesis did not make strong assertions regarding the specific anatomic regions associated with these semantic representations.

In contrast to the correlated, distributed feature accounts, Caramazza and Shelton (1998) have proposed the Domain Specific Knowledge Hypothesis (DSKH). To account for the observed dissociations for animate and inanimate items, this model suggests that the key organizing principle of semantic memory is the “role that objects have played in our evolutionary history.” The model proposes that evolutionarily driven selection has resulted in “domain-specific neural circuits” that are dedicated to efficiently processing information that is essential to an individual's and to its species' survival. The key categories of stimuli deemed of evolutionary significance to be segregated into specific systems are animals/living things that represent both predators and food, and plants as potential reservoirs of food and medicine. These evolutionary specifications are imputed to have led to functionally and anatomically dissociable neural circuits that process both perceptually and conceptually the categories of animals, plants, and likely tools. This “special” domain-specific scheme is also imposed to provide an organizational framework over other levels (e.g., feature) of semantic memory. The model extends the possibility that features are organized by specific domains (e.g., animals) in both modality-specific semantic stores as well as presemantic constructs such as structural descriptions in the visual system.

The authors note that the domain-specific hypothesis is not inconsistent with the notion that conceptual knowledge of objects can be organized in modality specific stores. If this is the case, then within a given modality, there should be anatomically distinguishable regions for living things separate from nonliving things (Mahon & Caramazza, 2003). This has been shown to be the case in most neuroimaging studies in the visual modality. Further investigations in other modalities await such confirmation (also see Gainotti, 2000; Miceli et al., 2001; Sartori et al., 1993; Sartori & Job, 1988). Two of the key features in the DSKH model—categorical/domain organization and evolutionary significance as a governing factor to semantic organization—are pivotal points that need to be addressed in any model of semantic memory.

The clinical observations that informed the above models were based on case studies of individuals with specific lesions. As noted by Mahon and Caramazza (2003) and Capitani et al. (2003), there are clear advantages and limitations to the single-lesioned patient-report approach. One clear advantage is the use of data from these cases to catalog the presence of specific dissociations that suggest functional units for cognitive models, without necessarily accounting for the brain regions encoding for these units. However, as the occurrence of these patients with well-described behavioral delineations is rare and the location of the brain damage not consistent, multiple occurrences of the same deficit are not common and, thus, is a key limitation. In addition, the changing nature of cognitive deficits over time from the onset of the acute lesion, often associated with inflammation, resolution of the acute state, and potential reorganization/recovery processes, produce increased variability in the characterization of these patients. Nonetheless, the existence of behavioral delineations in the chronic postlesion state provides considerable insight into functional organization that reflects brain organization. As Capitani et al. (2003) noted, the best application of the findings of these cases should be that they are viewed as a pattern of results that converge to a common end. We are in agreement with Capitani et al. that by avoiding overinterpretation of an isolated finding, recognizing that there are individual variations in how knowledge was acquired, and thus potentially how it is stored, optimal insight can be gained from these lesion studies (see models and comments below). Another optimal use of these case studies is that a unique case better suggests that a specific organization may exist in semantic memory than it “disallows” an organizational distinction from existing. This is particularly relevant if that organizational principle derives from a collection of multiple sources of evidence.

COGNITIVE AND BRAIN ANATOMY DERIVED MODELS

It has been suggested that collecting multiple examples of cases with either a consistent anatomic lesion or consistent cognitive deficit can lead to building a consensus on dependable brain–behavior correlations that provide added strength to any model. On the basis of his own and other's observations in patients with lesion-related semantic deficits (Gainotti, 1990; Gainotti & Silveri, 1996; Gainotti et al., 1995; Martin, 1998; Martin et al., 2000; Martin & Chao, 2001; Saffran & Schwartz, 1994), Gainotti formulated what has been termed the Sensory-Motor Model of Semantic Knowledge (see Table 3). An organizing principle of this model is that different semantic categories are organized along the lines of the sensory and motor channels in which the information about those categories was acquired. Impairment for items in a given category is proposed to be related to the lesion impinging on brain regions that encode the sensorimotor functions associated with items from that category. This model is an extension in some sense of the SFT model initially proposed by Warrington and Shallice (1984).

Organization of category, feature, and object memory for models derived in conjunction with neuroanatomical correlates

In terms of the lexical system, information or access to information about verbs would be stored in or near regions that control the motoric instantiation of such verbs. In contrast, information or access to information about nouns, and perhaps more specifically nontool items, is not encoded in the motor (predominantly left frontal) regions, but more posteriorly and inferiorly, in the (more left than right) temporal and ventral temporal regions.

In terms of the broader scope of conceptual knowledge, the finding associated with the model is that selective deficits of artifacts in general were associated with the left frontoparietal region and that impairments of biological entities were associated with the anteromesial and inferior temporal lobes (see Table 3). Further breakdown of these item groups have suggested that information about items such as tools would be stored in or near the brain regions that would control the motor actions typically associated with the use of such tools (premotor cortex). While some have suggested that biological entities can be further divided into subgroups with distinct neuroanatomic correlates, Gainotti (2000) reviewed the cases in the literature and did not find selective impairment of categories within the biological domain when checking across all subjects (e.g., patients with selective impairment for animals with specific anatomic correlates that are different from the correlates for selective deficits in fruits and vegetables). Gainotti further suggests the intriguing notion that gender plays a large role in selective deficits within the biological domain. In the instances he reviewed, men showed a relative impairment for flowers, fruits, and vegetables, whereas women showed a relative impairment for animals (Albanese et al., 2000; Barbarotto et al., 2002; Capitani et al., 1999; McKenna & Parry, 1994). This provides evidence of a clear dissociation between animals and plants in the overarching living category.

The Sensory–Motor Model has limited support derived from functional neuroimaging studies. However, there is support for the model from neurophysiological studies (Pulvermuller, 2005; Pulvermuller et al., 2005). Pulvermuller et al. recorded brain activity using high-density magnetoencephalography when action words involving the face or leg were presented. The findings were that face-word stimuli activated inferior frontocentral areas more strongly than leg words, whereas the leg words activated the superior central sites more strongly than the frontocentral areas. This finding supports the Sensory–Motor Model assertion that action words are represented neurally near their associated sensorimotor cortices. Although these distinctions support the Sensory–Motor Model's claims, these findings are also applicable to several other models, suggesting featural contribution to semantic object representation.

Another multidomain feature correlation model, known as the Conceptual Structure Account (CSA), has been proposed by Tyler and her collaborators. This model posits that object memory is organized as a distributed network of features (Moss et al., 2002; Taylor et al., 2007; Tyler et al., 2000; Tyler & Moss, 2001). In this model, an object is represented by overlapping patterns of activation across features that constitute the object, and categories emerge from a distributed network containing the conceptual features of all items, without explicit category or domain neural distinctions (see Table 3). Specifically, the model proposes that the distinctiveness of features is important in identifying an individual object from among other objects in its category, while the correlation of co-occurring features is indicative of category membership. This model does not propose that categories or domains of knowledge are represented in neuroanatomically distinct stores. Object knowledge is stored in a network composed of units representing semantic properties, which provides the flexibility to account for the dissociations described in the literature of patients with semantic memory deficits. The model accounts for disproportionate impairments of one category relative to another after brain damage by the pattern of disruption to the features that are more distinctive or correlative of the objects they encode. The model proposes that all features are encoded in the semantic system in a distributed manner and are organized along two major dimensions: (1) correlation, which is the degree that features are shared by different items, and (2) distinctiveness, which represents the extent that a feature is unique to a particular item and distinguishes it from other entities. Thus, as opposed to a categorical organizational scheme, or a pseudo-categorical scheme which is based on categories being designated by a group of shared features of items, CSA proposes that the organizational distinctions are by feature type.

The concepts of distinctive and correlative dimensions to features are not novel in object memory in terms of cognitive constructs, but the proposal that the brain organizes semantic memory in some anatomical or physiological sense along these dimensions is novel. Proposing this type of organizational structure for the brain may appear counterintuitive and complicated for a neural system that is fast and efficient in object recognition. However, a framework like the one proposed in this model allows for great flexibility in incorporating knowledge about objects and their component features as they are encountered developmentally or experientially.

Tyler and Moss (2001) cite a lack of evidence of domain- or category-specific foci of functional magnetic resonance imaging (fMRI) signal changes in the literature as support of their connectionist model. This interpretation of fMRI and lesions studies of semantic memory is not necessarily shared by other investigators (for example, see Mahon & Caramazza, 2003). Based on findings from their fMRI studies, the CSA model does suggest that the entorhinal and perirhinal cortices are engaged in featural–object integration in semantic memory (Moss et al., 2005). It remains to be seen if lesion studies and other evidence will continue to support this proposed neural organization.

COGNITIVE, BRAIN ANATOMY, AND BRAIN PHYSIOLOGY DERIVED MODELS

The SFT (Warrington & Shallice, 1984) discussed previously did not focus extensively on designating anatomical regions associated with the cognitive components of the model. However, through neural simulations of SFT, semantic memory access deficits were attributed to impaired neuromodulatory systems, while semantic storage deficits were attributed to damage to neurons encoding semantic information (see Gotts & Plaut, 2002, for details). Neuromodulatory systems in general affect a variety of normal neuronal transmissions. For example, synaptic depression is a neuromodulatory function that refers to decreased response of a postsynaptic neuron to presynaptic neuronal input after the presynaptic neuron has been firing frequently. This down-regulation of postsynaptic responsivity following stimulus repetition is counterbalanced by neurotransmitters such as acetylcholine and noradrenaline, which reduce synaptic depression and block firing rate adaptation effects. While it is beyond the scope of this review to discuss the manipulations in these neuromodulatory circuits, the authors propose that these manipulations can account for the observed behavioral performance of patients with semantic memory access deficits in neural network models. Conversely, loss of neurons that encode memory in semantic stores can also account for the behavioral performance in patients with semantic memory storage deficits.

The modulation of neuronal responses to various synaptic influences provides unique cellular and molecular mechanistic approach to account for semantic memory components. While the general principles of neuromodulation hold merit, the specification at a synaptic level, although plausible, is presently not supported by experimentally derived evidence at the neural level. This promising approach requires evidence that connects the conceptual constructs directly to the neural elements being proposed.

Barsalou and colleagues (Simmons & Barsalou, 2003; Barsalou et al., 2003) have proposed an ambitious model that extends from the concept of convergence zones (Damasio, 1989) and is referred to as the Conceptual Topography Theory (CTT). Groups of neurons, referred to as convergence neurons, are co-activated to encode stimulus features or groups of features of increasing complexity from initial sensory perception through the highest levels of cognitive associations. The neural populations can then be coactivated again upon recall of a representation of the stimulus. There are two principles that are fundamental to the CTT model. The first is the Similarity in Topography principle, which specifies that the spatial proximity of neurons reflects the similarity of the features that they encode. The second is the Variable Dispersion principle, which suggests that the conjunctive neurons that encode for a category of objects are dispersed into loci that are separated in a manner that reflects the similarity of the objects to one another. Ensembles of neurons representing different categories or objects are admixed in this schema, and any given grouping may contain cells that contribute to the neural instantiation of more than one category or object.

According to the CTT, there are multiple components that are at work in encoding, storage, and retrieval of information in semantic memory. These components comprise feature maps, analytic models, holistic and modality convergence zones for each sensorimotor modality and for emotion. The feature maps encode sensory modality-specific information. Analytic models are developed and refined as multiple exemplars of a specific item are perceived over time, and presumably facilitate the storage of conceptual knowledge and the efficient matching of sensory inputs to the knowledge of such concepts. Holistic convergence zones facilitate extraction of information about rough outlines or overall properties of a stimulus, as opposed to the finer details comprising an analytic model. At a level above the analytic and holistic data, the CTT advances the notion of modality convergence zones that code for correlations across the component analytic and holistic properties of a stimulus and thus provide the basis for supraordinate categories.

Several elements that this model proposes are unique in their conceptual framework. As evidence to support the CTT model, the authors draw from data obtained in animals (Tanaka, 1997). Single-cell recording data predominantly from the visual system demonstrate cellular responsiveness to increasingly complex stimulus features or feature combinations as one moves from primary visual perceptual cortices to temporal association regions. The spatially distributed nature of the cellular assemblies that mediate access to feature, object, or category level knowledge is supported by fMRI data (Haxby et al., 2001), where membership of a visually presented object in a specific category was correlated robustly with specific patterns of spatially separated foci of signal change. Supporting evidence for convergence zones is drawn from intracortical electrophysiologic recordings in humans, which demonstrated object-category–sensitive firing rates of cells in the entorhinal, hippocampal, and amygdala structures (Kreiman et al., 2000a,b). While these lines of evidence suggest that certain principles proposed in CTT have some precedent in the nervous system, the support is mostly for the theoretical elements of the model (cellular constructs) thus leaving most constructs still largely unsupported by directly observable data. Such data will be critically important if this model is to gain credence.

Some indirect support for the CTT model derives from lesion studies showing that damage to relatively low-level sensorimotor feature maps can produce not only deficits at the rudimentary input processing level but also agnosias specific to the affected sensory or motor system. Damage to the systems that putatively mediate higher-level processing (analytic, holistic, and modality) affects processing only at and above the level of damage. For instance, damage to assemblies or networks of conjunctive neurons that encode information about shape at an analytic level would be expected to result in deficits for multiple categories of objects that share that aspect of shape. Unfortunately, it is unclear to what these higher-level processing constructs correspond in terms of neural elements. Hence, further specification of each component of this model is required, along with integration with human brain anatomy and physiology and corresponding semantic memory deficits by means of experimental findings before this model gains wider acceptance.

Martin and colleagues have proposed the Sensory Motor Property Model for Representing Domain-Specific Information (Martin & Chao, 2001), which is based on the concept of an object being composed of semantic primitives. These primitives represent properties of an object that allow for fast and efficient recognition of that object. The model proposes that humans possess neural circuitry for learning specific sensory- and motor-related features associated with an object's perceptual properties and functions. Such properties are purported to be universal, restricted in number, and accessed implicitly and automatically. They are thought to be stored in the system that is activated when the properties are initially encountered and learned such that primitives for perceiving an object's form and motion, for example, reside in the visual processing system while primitives related to visuomotor properties reside in the action systems (Martin, 2007). The theory further advances that this conceptual feature knowledge is distributed over modality-specific representation stores.

To support these claims, positron emission tomography (PET) experiments have helped reveal the brain regions subserving the feature activation in multiple modalities that are intrinsic to the encoding of objects from multiple categories: (1) animate items—lateral portion of the fusiform gyrus, superior temporal sulcus, amygdala (for survival; Yang et al., 2005), and medial prefrontal cortex; (2) tools—left medial fusiform gyrus, middle temporal gyrus, intraparietal sulcus, and premotor cortex; (3) places—parahippocampal cortex, (4) letter strings—the visual word form area located in the ventral temporal cortex (Cohen et al., 2000; Polk & Farah, 1998). As can be seen in these localizations, the ventral occipitotemporal cortex plays a significant role in encoding feature representations across a variety of categories of items and appears to have a distinct organization (see Figure 1). In addition, the model also proposes that there are specified regions that subserve semantic processes (e.g., semantic working memory; Martin & Chao, 2001; see Figure 2).

Group functional magnetic resonance imaging (fMRI) data showing category-related activations associated with silently naming pictures of animals and tools. Regions showing greater activity for naming animals than for tools are designated in white [lateral fusiform gyrus, right superior temporal sulcus]. Regions showing greater activity for naming tools than for animals are shown in black [medial fusiform gyrus, left middle temporal gyrus/inferior temporal sulcus, and left ventral premotor cortex]. Isolation of these regions supports the existence of category stores for objects in semantic memory. [Figure from Martin, A. & Chao, L.L. (2001). Semantic memory and the brain: Structure and processes. Current Opinion in Neurobiology, 11, 194–201. Copyright 2007, Elsevier Ltd. All rights reserved. Reprinted with permission.]

Schematic representation of ventral (left) and lateral (right) surfaces of the brain. The colored areas designate the approximate locations of regions where semantic information about object form, motion, and object-use–associated motor patterns may be stored. Semantic information may be integrated in the temporal lobes, with specificity increasing along the posterior to anterior axis. Specific regions of the left inferior parietal cortex and the temporal poles may be engaged in retrieving, selecting, and accessing semantic information. [Figure from Martin, A., & Chao, L.L. [2001]. Semantic memory and the brain: Structure and processes. Current Opinion in Neurobiology, 11, 194–201. Copyright 2007, Elsevier Ltd. All rights reserved. Reprinted with permission.]

As the Sensory Motor Property Model provides specific localizations for regions associated with certain categories, it also invokes anatomic and physiologic principles to account for relatively consistent localizations of these category-related regions. The model proposes that genetic mechanisms or connections between regions constrain the localization of stores of semantic knowledge. For example, the lateral regions of the fusiform gyrus may have developed a role in representing animate objects secondary to its access to information from the amygdala (see Freese & Amaral, 2005). The medial aspect of the fusiform gyrus receives information about object manipulation from intraparietal and premotor regions. Thus, the regularities in connectivity between regions may be predetermined by genetic mechanisms that dictate specific localizations, or by consistent white matter connections.

Martin et al. (1996) raise several issues that need to be resolved in the formulation of any model of semantic memory. An important aspect is to better understand how the nodes of the neural systems are bound together. In addition, it is imperative to understand how activity within the network is coordinated in the service of conceptual processing (Kraut et al., 2002a), and how category-related information links with lexical information and with other brain regions involved in supporting more general conceptual and semantic processes (Beauchamp et al., 2004).

Hart and co-workers and Kraut and colleagues (Hart et al., 2002; Kraut et al., 2003, 2004) have proposed the Neural Hybrid Model of Semantic Object Memory, which posits that there are cortical regions that encode representations associated with objects in sensorimotor and higher-order cognitive systems (e.g., lexical semantic, emotion, and so on). Based on our findings, and the work of others, we have proposed both feature-based (see Hart & Gordon, 1992; Haxby et al., 2001; Miceli et al., 2001 for further description of featural organization) and category-based neural representations for several of these sensorimotor/cognitive domains. One proposed mechanism of retrieval of an integrated object concept in semantic memory involves the activation of these representations in the neural systems, which are then integrated by means of synchronized neural firing modulated by the thalamus (Kraut et al., 2002a; Slotnick et al., 2002).

The model supports the findings of what appear to be clear functional–anatomic organizations for featural representations within modality-specific sensorimotor/cognitive domains that may encode for features of either items or groups of items/categories (e.g., visual–perceptual features for animals; Hart & Gordon, 1992; Haxby et al., 2001; Miceli et al., 2001; Sartori et al., 1993; Sartori & Job, 1988) or across groups of items/categories (e.g., the feature manipulable, hand-related in premotor regions for both tools and fruit and vegetables; Kraut et al., 2002b). The model also proposes a neurally distinct categorical organization, consistent with the domain-specific account. The evidence supported for both these sets of distinctions comes from both lesion studies and functional activation studies (Kraut et al., 2003) and will be discussed specifically for the auditory system below. We have reported on interactive and distributed relationships between the categorical and featural representations within specific modalities; however, the relationships between representations may be dependent on the specific modality.

The data that inform this model are derived from both lesion and functional neuroimaging studies (Segal et al., 2003). The critical question raised is; what do the neural regions detected in PET, fMRI, and lesion studies do that translates into the cognitive distinctions noted in semantic memory? The model proposes at least five plausible functions for these neuronal nodal populations: (1) encoding a representation at some level (premotor region encoding for how to position the hand/fingers to hold an item), (2) processing semantic information or performing some semantic memory operation, (3) integration of input from multiple representational levels or modalities (examples of regions engaged in processing or integration can be seen in Beauchamp et al., 2004; Hart & Gordon, 1990), (4) access to individual memory encodings that are represented by spatially or spatially/temporally distributed patterns of neuronal firing that is not located in that specific region, or (5) detection of a specific semantic category, trait, or feature. The latter four functions for these nodal populations subserve semantic memory operations and processes (Hart et al., 2002; Martin & Chao, 2001).

We have reported on nodes in the auditory semantic system, which we define as populations of neurons whose activity is associated with performing a specific cognitive operation. The nodes in the auditory system appear to function as detectors of whether a nonverbal sound represents an item with either (1) specific semantic attribute (threatening) or (2) being a member of a specific category (animal; Kraut et al., 2006a). Subjects were presented sounds associated with real objects and perceptually equivalent nonsounds during fMRI for a yes/no decision on whether the sounds were associated with a real object. Two distinct anatomic foci in the right superior temporal gyrus were detected—one for animal sounds (includes threatening and nonthreatening animals) and another for threatening items (includes threatening animals and objects) (see Figure 3). Both of these foci were outside of regions associated with perceptual processing such as the primary auditory cortex, but in relative close proximity to that region. Stimuli that represented threatening animals, which had both the animal categorical designation and threatening featural attribute, activated both areas of the right superior temporal gyrus foci, suggesting a distributed neural representation across both category and feature nodes. The object stimuli did not activate the animal node, and the nonthreatening stimuli did not activate the threatening node to a significant degree, suggesting that these nodes were specific to the stimuli reported. Given the specificity of these nodes and their proximity to primary sensory cortices, these category- and feature-specific nodes most likely engage in early extraction of semantic memory information from these sounds. The serial and transient nature by which auditory stimuli are momentarily present mandates rapid, on-line identification of the objects/items producing those sounds. Accomplishing this rapid processing can be best mediated with close proximity between regions that perform the early stages of auditory perceptual processing and those performing higher-order operations, such as evaluating for relevant semantic constructs in auditory semantic memory.

The right lateral projection of the brain shows signal changes in the right temporal foci that occur for threatening (designated by red) and animal (designated by green) sounds during a real/nonreal nonverbal sound decision task (see Kraut et al., 2006a, for task design and stimuli with controlled factors). Also drawn in are the approximate boundaries of primary auditory cortex (blue) from Rademacher et al. (2001). This representation shows these neural semantic memory detectors in proximity to the primary sensory cortex processing perceptual sound input. The green foci shows signal changes with an animal sound stimulus, the red foci with a threatening sound stimulus, and both the green and red show signal changes to a threatening animal stimulus. These findings support the assertion that there are both specific category and feature stores in semantic memory. [Figure from Kraut, M., Pitcock, J., Calhoun, V, Li, J., & Hart, J. (2006a). Neuroanatomic organization of sound memory in humans. Journal of Cognitive Neuroscience, 18, 1877–1888. Copyright 2007, MIT Press. All rights reserved. Reprinted with permission.]

The model thus proposes that these two right superior temporal loci of fMRI signal change represent nodes or neural detectors that are engaged in early auditory semantic memory processing, and which respond differentially to the animal category and the threatening feature of sound stimuli (Kronbichler et al., 2004). If several of these attributes (animal, threatening) are present in a sound, then several of these detectors will be activated simultaneously, and the summated and contemporaneous coactivation of the detectors will facilitate object recall. Thus, in this particular instance of evaluation of threatening animal sounds, there are interactions between nodes that access category-level (animal) and feature-level (threatening) level information (see Figure 3).

The most common process in semantic memory is retrieval of objects in memory. To investigate this retrieval mechanism from a neural perspective, we investigated with fMRI how objects are retrieved from featural input. The subjects were presented visually with two words (e.g., “desert” and “humps”) to decide if they resulted in the retrieval of a specific item [e.g., “camel”; referred to as SORT (Semantic Object Retrieval Test); Kraut et al., 2002a, 2006b]. This task has been performed several times with consistent activation in the bilateral medial Brodmann Area 6 (BA6)/ pre-SMA, bilateral ventral temporal lobe, and the left dorsomedial and pulvinar nuclei of the thalamus (Kraut et al., 2002a, 2003). These results motivated a study to determine how the thalamus mediates object retrieval on SORT. Lesion studies have also implicated the role of thalamus in semantic processing (Segal et al., 2003). Crosson et al. (1997) described a patient with a lesion involving the pulvinar, who exhibited category-specific deficit for medical items. The patient exhibited naming deficits for both man-made medical items and medical conditions relative to items from other categories and control subjects. Although it is not suggested that specific categories are stored in the thalamus, the thalamus is imputed to play a vital role in selective engagement of cortical regions involved in semantic processing.

The role of the thalamus was examined in a patient who was being implanted with bilateral thalamic depth electrodes for epilepsy treatment (Slotnick et al., 2002). Before starting the therapeutic stimulation, the visual word–word object retrieval task (SORT) and a control semantic association task were presented to the subject while recording an electroencephalogram (EEG) from both the thalamic electrodes and a limited number of surface electrodes. The findings demonstrated that, during all trials of the SORT, there was a decrease in the α-band EEG power globally throughout all surface electrodes, which was followed by a spatially selective increase in γ-band EEG power in the thalamus and occipital scalp electrodes for only those trials where object retrieval occurred. It was proposed that the generalized low-frequency (α-band) power rhythm decreases were likely driven by inhibitory (e.g., γ-aminobutyric acid-ergic) projections from the thalamic reticular nucleus to thalamocortical cells in other thalamic nuclei, which mediate cyclic activity (Klimesch, 1996, 1999; Klimesch et al., 1993, 1999; Schier, 2000). The high-frequency (γ) synchronizing rhythms were believed to likely reflect spatially specific, oligosynaptic excitatory (e.g., glutaminergic) cortico–thalamo–cortical pathways.

In support of this mechanism, the SORT experiment was repeated using event-related fMRI to obtain higher temporal resolution of the time course of brain activation. This study demonstrated the two thalamic activations—one initial fMRI signal peak in the dorsomedial (DM) nucleus and a focus of signal change that reached its peak amplitude later, in the pulvinar (Kraut et al., 2003). The region with the earliest activation that was roughly coincident with the DM nucleus was medial BA6. Brodmann area 6 is connected directly to the dorsomedial nucleus (Ilinsky et al., 1985; Inase et al., 1996), and we suggest that BA6, along with its dorsomedial nucleus connections, is most likely involved in the process of generating a target object concept from featural input stimuli. We posit that this is accomplished by initially engaging the network of brain regions associated with representing components of an object (including regions encoding featural representations) by means of the decrease in a-power in these cortical regions.

The late fMRI signal changes in the pulvinar suggest that this region engages late in the object retrieval process and is, thus, likely the mediator/modulator of the selective gamma burst rhythm. We propose that this resulting gamma rhythm facilitates either integrated object retrieval and/or object visualization. Medial portions of the pulvinar connect with inferotemporal visual cortex as well as somatosensory cortex of the insula and with the amygdala, where features and other elements of an object are encoded (for review, see Gutierrez et al., 2000). Thus, the pulvinar is ideally situated to be involved in modulating connections in the semantic object retrieval network and in integrating the semantic elements by means of 30 Hz. synchronized neural firing. These elements include but may not be limited to features and category assignment, with various weightings of the values for each of these elements. The overall strength of the semantic memory reflects the frequency and presumably the contexts in which the object, its constituent features, its defining features, potentially its category membership, and how knowledge about the item was acquired in its sensorimotor/cognitive channels. We further propose that this integrated 30 Hz synchronized firing of the elements that encode an object represents a plausible mechanism of how the integrated object memory is encoded (see caricature in Figure 4).

The caricature demonstrates how the integrated semantic object memory can be retrieved as the coactivation of the neural regions encoding for the attributes (features, category membership, and so on) of the object, with the synchronous coactivation being modulated by the thalamus with synchronizing rhythms. [Figure from Hart, J. & Kraut, M. (2007). Neural basis of semantic memory. London: Cambridge University Press. Copyright 2007, Cambridge University Press. All rights reserved. Reprinted with permission.]

The Neural Hybrid model proposes that, for some classes of stimuli, feature- and category-level nodes co-exist, and may be at least partly collocated in a sensorimotor or cognitive domain, as was noted in the category and feature distinct modules in the nonverbal sound object system. Within a given domain involving both category and featural nodes, one can obtain a category level impairment without a feature impairment, or a category and feature impairment depending on the following neural interaction between the two nodes: (1) whether there is an interactive, distributed, and/or hierarchical relationship between the nodes; (2) how nodes are connected to each other (by anatomically direct pathways, or indirectly); (3) electrophysiological synchrony and coherence between modalities and brain regions; (4) the presence, saliency, and strength of representations that each modality (sensorimotor/cognitive/emotion) contributes to the item representation; (5) the saliency and distinguishing nature that features may have in the overall integrated object concept in semantic memory (e.g., “moo” for cow); and (6) if the offending brain lesion results in complete or, more likely, partial degradation in any of these structures, leading to varying degrees of impairment.

Thus, the cognitive distinctions noted in behavioral assessment would reflect the degree to which the lesion affects these nodes and connections. We would propose that similar levels of nodes would be linked either anatomically, physiologically, or both across sensorimotor/cognitive domains and, thus, affect the nature of the behavioral deficit.

Recent studies (Gil-da-Costa et al., 2006; Kraut et al., 2006a) provide possible evidence for the influence of evolutionary significance upon categorical and/or featural organization. The findings of Kraut et al. are among the first to delineate subdivisions in the nonverbal sound domain, with most previous efforts in semantic memory focused mostly on the visual and verbal domains. This extension into the sound domain points up the need to account for information from other sensorimotor and cognitive domains. We agree with Caramazza and Shelton's (1998) claims for domain-specific representations for evolutionary significance, particularly with the emphasis on survival-based issues, adding that this extends to features such as “threat.” We believe that this organization is also compatible with the sensory motor model's organization and propose that as more neural systems are investigated, the category- and/or feature-based framework will be further identified, with the organizations dependent to some degree on the specific semantic memory system (lexical–semantic, sensorimotor, emotional, and so on).

In summary, the Neural Hybrid Model is so termed because the model subsumes or can potentially subsume several cognitive schemas, as well as several neural frameworks. Several main points are:

1. the existence of both category and featural representation stores (including modality-specific) in semantic object memory;
2. these categorical and featural stores can link with each other in a variety of manners as noted (e.g., hierarchical, distributed, interactive, conjunctive, independent), partially depending on modality of the stores;
3. brain regions have been are detected in both focal lesions in single case studies and functional activation studies to encode for neural representations and to medial semantic processes (e.g., semantic search, categorization, access, multimodal integration, and so on) in a variety of brain regions (e.g., dorsolateral prefrontal cortex, left inferior parietal–posterior temporal lobe (Hillis et al., 2001), (left fusiform gyrus, dorsomedial BA6, and left inferior frontal gyrus to name several; Hart et al., 2002); and
4. a neural mechanism by which components of an object (features, category membership) from multiple modalities represented at separated sites within the brain can be bound to form an integrated object memory. This synchronous coactivation is modulated by the thalamus via synchronizing rhythms.

CRITICAL FOCUS ON THE FUTURE

Upon review of investigations in semantic memory, there are several issues to consider in terms of the above models as well as potential targets to be considered as the field advances. The following issues are not in order of significance, time, or importance, but are considered distinct entities. We predict that the investigations outlined above will likely continue, with several new candidate organizations (e.g., Cognitive And Brain Anatomy And Brain Physiology And Neurotransmitter/Hormone/Protein And Genetic Derived Models) evolving. The following candidate issues need to be critically addressed:

1. Although there are more similarities than differences in the cognitive models, there has been considerable progress noted in this area. The specificity of the models of organization has increased over the past 20 years and will likely increase further with additional detail in the testing of the subjects to delineate dissociations in performance. While not obvious at present but in keeping with Mahon and Caramazza's suggestion, it may be possible to create consistently reproducible temporary lesions, which will afford the opportunity for repeat testing. Repetitive transcranial magnetic stimulation or comparable temporary lesion techniques may facilitate such progress.

1. Consideration needs to be given to issues such as how information about an item is acquired and the prior experiences of the individual in both learning and use of the items throughout the life of the person.

1. It should be acknowledged that, in some respects, functional imaging studies of intact individuals typically use circumstances that are not necessarily natural in context. Significant and definitive advances in the understanding of the mechanisms subserving semantic memory may have to await the development of tools that will facilitate observing these processes under more natural circumstances.
2. Although there are numerous studies assessing the neural underpinnings of categories such as animals, food, fruit and vegetables, and tools, leading to an evolution-based hypothesis for their existence, there are several other categories or features that have not been isolated but could be considered from an evolutionary perspective. For example, clothing, shelter, and weather are evolutionarily important issues that need to be investigated thoroughly and, if not detected as distinct representations, then should be accounted for by models. Attributes similar to the threatening/unpleasant features described in Kraut et al. (2006a) also need to be further investigated to determine whether there is evidence for their neural instantiation in other cognitive domains. Other potential features that are essential for survival that may be organized across individuals in a consistent manner should also be explored.
3. Consideration should be given to the choice of data analysis techniques used in functional neuroimaging studies when considering categories, features, and objects and their anatomic localization. It is common for data analyses to use random effects models to ascertain if the anatomic localization of a region of signal change can be generalized to the population under study, in addition to the subjects being investigated. This approach has numerous advantages, particularly when investigating potential consistent anatomic localization for a specific component in semantic memory. However, for components of memory systems that are organized along highly individualized patterns, there may not be a significant finding on a functional imaging study, not due to the absence of the semantic memory component, but due to the absence of a consistent location. In considering the semantic category and feature entities that have not been detected in functional imaging studies, as the techniques and imaging equipment continue to improve in terms of detection capabilities, we may have to consider evaluating both normal controls and patients on an individual basis. Discussion of how well these findings fit evolution/survival models can be debated; however, one could posit that consistent semantic dissociations across subjects that may not have a common neural localization still may be in keeping with these being core memory components needed for evolution/survival reasons.
4. There are at present numerous techniques to assess brain function while cognitive tasks are being performed. As these techniques continue to be refined and new ones emerge, the amount of information available to further refine models will expand. These findings can improve the information contained in each of the above groups of models. Furthermore, by combining data from multiple investigative platforms, one can get beyond the “where” data afforded by lesion/deficit and activation studies, the “when” gleaned from electrophysiologic studies (EEG and magnetoencephalography), and perhaps get closer to the mechanistic “how” by discerning the temporal relationships between these spatially separated neural events. Crosson et al. (2007) propose such a model of the basal ganglia and dopamine receptor's roles on the processes of intention and intentionally guided attention that support semantic functions. Additionally, integrating multiple domains of investigation could also aid in resolving some of the issues still noted in models of semantic organization.
5. When possible, confirmation of any of the findings in humans should be sought using animal models. Gil-da-Costa et al. (2006) have showed that species-specific vocalizations in rhesus monkeys activated homologues in perisylvian regions that appeared to represent higher-order rather than perceptual processing and supporting further the evolution-based hypothesis. Further similar studies could allow for the development of a broad range of perceptual and memory homologues in animals, which in turn could be used to clarify neural-computational operations that might plausibly subserve similar functions in humans.
6. The findings associated with these models should be linked with the clinical investigations of patients with semantic deficits. Several investigators have directly linked models with clinical conditions associated with semantic deficits (see Gainotti, 2000; Garrard et al., 2005; Grossman et al., 2003, as several examples). Given the prevalence of semantic impairments in dementing illnesses and the potential significant disruption to activities of daily living with some semantic deficits, there is a immediate need for neuropsychological tests that are specific and sensitive to semantic dysfunction [see Palm Trees and Pyramids Test (Howard & Patterson, 1992) and SORT (Kraut et al., 2006b) as examples]. These assessments will also need to be linked with clinical implications of semantic deficits associated not only with cognitive performance but activities of daily living. Our increasing understanding of the neural mechanisms underlying semantic memory and how these may be disrupted can greatly inform our evaluation of semantic deficits.
7. As techniques advance, we would propose that more fine-grained distinctions in biologic specification will greatly inform models of semantic memory. There have been some specifications at present, but we would expect that increased information available on genes, proteins, neurotransmitters, and synaptic physiology emerge, these findings will greatly inform models of semantic memory. For example, genetic alleles/polymorphisms may be associated with specific semantic localizations. Neurotransmitters associated with specific semantic processes may not only provide a clearer explanation for how these processes function, but potentially lead to treatment interventions.

In summary, there have been considerable advances made in delineating theories and models of semantic memory, with ever-increasing sophistication in all three groups of models discussed. The levels of sophistication of the models and advances all have been encouraging since an initial foray in the mid-1980s after the reports of category-specific deficits. The challenges that lie ahead will likely focus on using more sophisticated instruments and technologies to further refine the models to address a molecular–cellular–genetic basis of semantic memory and to export the findings of all of these groups of models into clinical neuropsychological assessments that can better inform diagnostic, prognostic, and therapeutic decisions in patient populations.