R1. “Neats” and “Scruffies”

Several commentators (e.g., Fields & Glazebrook; Davis, Altmann, & Yee (Davis et al.)) oppose the necessity for the distinctions we make between (supposedly) different representational entities. Such commentaries echo the classic debate in cognitive science between “Neats” and “Scruffies” (see Schank & Abelson, in Clark Reference Clark2013; Marcus Reference Marcus2009).

For example, Fields & Glazebook believe that it would be possible to find “a single computational architecture (that) works… at every useful level of analysis.” Such models would have “the advantage of being rigorously testable at every experimentally-accessible level of analysis, from those of basic physics, intracellular, and cellular processes up to the whole-organism scale and beyond.” As examples for such “master algorithms,” they mention Karl Friston's Active Inference Theory and their own “Chu space” model.

Undoubtedly, the idea of a “theory of everything” is attractive, if tenable. However, the suggestion that the same science can apply to every level of analysis – from physics to cellular processes, to whole organisms and societies – is an extraordinary claim that would require extraordinary evidence. If a universal model can be used to predict hurricanes and anticipate wars, then it could be argued that specific models in different sciences may become redundant. In the meantime, each of the sciences still requires its own models (e.g., political science models for prediction of war, biological models for prediction of cancer, and so on). All in all, we believe that sciences have progressed, and have done so without a theory of everything.

Furthermore, we believe that general and specific theories need to complement each other to provide humans with an understanding of their world. A good analogy is the Darwinian theory of evolution that provides a “neat” overarching framework for the understanding of life, and fully accommodates the “scruffier” world of genetics. It is the synthesis between these two lines of research that provided a real understanding of biological life. It appears that our position does not differ from that of Friston, whose Active Inference Theory is cited by Fields & Glazebook as the prime example of scale-free models. As Friston notes:

In other words, although broad “scale-free” theories such as Active Inference Theory (and like evolutionary theory) may describe universal principles of biological entities, they are still “unsaturated,” and their application to a given domain requires filling in many “scruffy” details. A full understanding of cognition will entail having an accurate picture of different generative models (i.e., representational entities) used by the brain, as well as its overarching organizing principles.

Importantly, understanding the universal principles that govern a system does not tell us all there is to know about the system. The system will be influenced by random, as well as systematic factors, and by forces that are external to it. For example, the theory of evolution does not entail that there should be lions and giraffes in the world. Nonetheless, these specific implementations of the evolutionary algorithm (e.g., the existence of giraffes) are indispensable to our understanding of that which exists in our world.

R2. Filling in the blanks

Ironically, our criticism that overly-broad models are insufficient to account for the richness of cognition has been directed against our own account.

Fiedler suggests that our model does not sufficiently discuss how behavior is determined by extra-cognitive factors, namely, how the structure of the world shapes psychological phenomena. For example, if in one's experience, people who are similar to us occupy more spatially proximal places than people who are different than us, one should see various effects on human behavior that reflect this association between social distance and spatial distance. An explanation of this association may not require a lengthy discussion of humans' motivations and neural mechanisms. Instead, this phenomenon may be well-explained by the analysis of statistical regularities of the environment to which they were exposed.

We agree that distance-to-distance associations can be found in one's environment. We think, however, that the association between distance and abstraction may have a different status. The variability and uncertainty that come with increased distance necessitate abstract mental representations in an individual's mind only if she seeks to travel the distance whereas still predicting events in the new environment. To use Fiedler's example, the fact that a map that represents a larger area (e.g., a continent rather than a neighborhood) would have cruder resolution is dictated by constraints on the usefulness of this map for a traveler, rather than being a physical necessity. Therefore, we agree that objects at a greater distance entail greater uncertainty, and we think that this regularity gives rise to a stable intrapsychic link between construal and distance.

Indeed, our account takes an intrapsychic perspective of human psychology. However, this does not mean that we (or the predictive cognition view more broadly) ignore the structure of the world. Instead, it means that biologically inspired accounts of human psychology choose to focus on ecologies within which we were embedded for the longer timescales. The long neck of the Giraffe is an imprint of its environment over evolutionary timescales. Similarly, the representational structures we discuss are a reflection of the ecology that has impinged upon our mind and brain.

This point is also highlighted in the commentary by Friston, which discusses how the “factorization” of the mind into different representational types is a consequence of the structure of the world. As Friston writes: “knowing what something is, does not tell you where it is and vice versa. This (conditional) independence is manifest beautifully, in terms of the functional anatomy of the dorsal and ventral streams in the brain.”

Indeed, the distinction between modality-specific, multi-modal, and categorical abstracta can be thought of in terms of the world rather than the mind.

Modality-specific visual representations are an adaptation to the fact that objects in the world reflect light at various frequencies. An organism living in a world of complete darkness for several millennia may lose the functionality of V1 and eventually may even lose the need for eyes altogether. Multi-modal representations are a function of the statistical regularities embedded in the world. The location of objects in space and time is not random, because things that are closer in time are also closer in space. An organism living in a world wherein light shines for only a few seconds every several minutes will not develop sophisticated mechanisms for binding visual sensations. Finally, a system for categorical representation probably would not have evolved if proto-humans did not have an anatomy that allowed them talk to each other.

Surely, aside from the stable characterizations of our ecology, there are aspects of the world that influence the way we represent it – without entailing any changes in neuroanatomy. When a psychologist is confronted with a specific behavior, it is undoubtedly essential to consider the degree to which this behavior is well-explained by factors that are external to humans' internal makeup.

Similarly to Fiedler, Demetriou suggests that our model misses out on some crucial details. He sees deficiencies in that that we do not provide: (i) a mechanistic description of the process of abstraction, (ii) a detailed developmental theory of the emergence of different mental travel capacities, and (iii) an account of individual differences in abstraction and mental travel.

We agree with Demetriou that the model we describe leaves these crucial questions open. However, we see this as a question of scope, rather than an inherent limitation. We think that our framework has the potential to consolidate further ideas and findings in different areas of investigation, such as those highlighted by Demetriou. For instance, although our account does not provide a detailed mechanistic model of the act of abstraction, it provides a framework for such details to be filled in. As we suggest in the manuscript, the broad principles we outline take different forms in different modalities.

Scharinger's comment fills some of these missing specifications by explaining how the hierarchical nature of representations is expressed in the domain of sound and speech. His analysis further highlights how a hierarchy of abstractness is an omnipresent regularity of cognition rather than a feature of a specific system (e.g., the visual system).

Furthermore, as noted by Demetriou, our account does not provide a detailed developmental trajectory of the emergence of different aspects of mental travel. The commentary by O'Brien, Rutt, & Atance begins to fill in these details by describing current evidence about children's ability to traverse different dimensions of psychological distance and highlights future efforts to connect these data with children's emerging representational capacities. As they note: “The results of such efforts may not fundamentally contradict Gilead et al.'s claims, but may instead add richness and precision to an account of how humans develop the remarkable capacity for mental travel.”

Finally, as noted by Demetriou, our account does not provide a discussion of individual differences. The commentary by Belmonte highlights how the model is consistent with observations concerning variability in autistic traits, and how the relevance of our model to this literature can be empirically tested.

Belmonte's compelling theory of autism coheres with our view of representational hierarchy and its role in prediction. This account explains how many of the cognitive impairments observed in autistic individuals (e.g., inability to understand the minds of others, to consider temporally distant outcomes of one's behaviors, and to deal with uncertainty) can be thought of as deficiencies in mental travel, which in turn, can result from autistic individuals' over-reliance on modality-specific representations, rather than multi-modal and categorical abstracta (which is evident, e.g., in autistic individuals' preference for processing modality-specific relationships, described in the research on the “systemizing” phenotype). Belmonte's account addresses the emotional burden of being unable to predict the world and the futility of trying to find structure in a world of constant change using low-level, concrete mental representations.

Belmonte also suggests new predictions of this account linking abstraction, prediction, and autism. One interesting ramification of this link between our model and Belmonte's may relate to a better understanding of the diversity of autism. It is often argued that autism is, in fact, a collection of disorders; however, the underlying endophenotypes that give rise to different manifestations of autism are difficult to identify without a generative theory of this disorder. If it is indeed the case that autism is related to over-reliance on concrete representation, it is possible to ask, for example, whether some autistic individuals overly rely on multi-modal abstracta (but have difficulties with forming and using categorical abstracta) and whether other autistic individuals cannot even rely on multi-modal abstracta in processing the world. Such research may be able to relate behavioral phenotyping to the neuroscientific phenotyping of modality-specific areas of the brain, the default network, and frontotemporal regions supposedly involved in categorical abstracta.

R3. Can Predictive Processing accommodate abstract cognition?

Our paper explores the potential role of abstract (including symbolic) mental representations in predictive cognition. As we discuss in the article, one of the most prominent perspectives in the predictive brain literature is the so-called Predictive Processing (PP) approach. Several authors have responded to the target article by assessing the extent to which the representational ontology we describe is (or can be) consistent with PP.

Litwin & Milkowski present a critical perspective on the PP approach to psychiatric illness. They suggest that our account – that highlights the importance of structured representations – is at odds with the connectionism-inspired PP paradigm.

The question of whether and how our approach can be made compatible with the PP approach is somewhat tricky because there is diversity in PP models. In many cases, the exact theoretical commitments of the different models (e.g., active inference/predictive coding/the Bayesian brain) diverge. Active Inference Theory, which is the predominant account in the PP world, is a broad framework that accommodates many concrete “process models” of cognitive architecture and neuroanatomical implementation. Some of these process models deny the existence of structural representations of the kind we assume, and as such, are radically inconsistent with our view. Others are probably entirely consistent with our model.

Surely, as noted by Litwin & Milkowski, many PP process models of psychiatric illness describe a world that does not provide a place for the representations we posit in our ontology. This argument is actually the main point we tried to make in our discussion of the relevance of our model to PP theories. We echoed the standard psychological view of psychopathology, which assumes that abstract, structured representations of the self and others have a critical role in mental illness. Furthermore, the standard account posits that altering these representations via language-mediated processes reduces distress. Such high-level processes are not part of PP process models of psychiatric illness. Nonetheless, we suggested that abstract cognition must play a role in disease, and if that is the case, then something is missing from the PP literature on the topic.

PP models of psychiatric illness can, in principle, integrate abstract cognition into their “factorized” system of generative models. Indeed, as noted in the response by Friston, whereas the simplest PP process models may not be able to account for the brain's representational diversity, more sophisticated schemes of variational message passing may be fully congruent with a symbolic representational view. In our discussion of the Robot Herbie, we tried to appeal to such a formulation wherein PP models and symbolic architectures can co-exist.

More broadly, unlike Litwin & Milkowski, we do think that our model coheres with (what we see as) the fundamental tenets of Active Inference Theory. Specifically, our account agrees with PP models regarding the crucial role of generative models/representations, the componential and hierarchical nature of these models (i.e., their “factorization”), and the idea that these models serve one ultimate purpose: to anticipate events – from perceptions to complex social behaviors – before they occur.

The potential ability of PP-inspired process models to accommodate higher cognition is also addressed in the response of Vilas & Melloni. However, Vilas and Melloni's account, similarly to Litwin & Milkowski's response, presents a critical analysis of PP, and specifically, criticizes the ability of this framework to explain the phenomenon of consciousness. Villas and Melloni argue that aside from its representational diversity, the mind also exhibits phenomenological diversity (e.g., the sense of consciousness differs from that of unconsciousness); they suggest that this diversity is not captured by explicating different types of mental representations, or by describing the process of active inference.

However, the response of Friston to our target article does address the emergence of phenomenology, as the consequence of active inference processes upon complex representations. As he notes:

In other words, according to Friston's theory, the sense of agency and the act of mental travel are intertwined; moreover, this account implies that the extent of a sense of agency may relate to the extent of mental travel. Namely, asking “what is likely to be out there” is associated with a lower sense of agency than asking “what is likely out there given what is likely that I will do,” and “what is likely to be out there given what is likely that I will do, given what is likely to be out there …” and so on.

This conceptualization may address a crucial requirement for a scientific theory of consciousness, namely, an explanation of the mechanisms that should give rise to the phenomenology of being an agent in the world (or not). According to Friston, the demarcation of a sense of consciousness may be treated as a continuous variable that pertains to the “temporal thickness” of the generated model. Moreover, the neural correlates of consciousness may be evaluated once some measure of the temporal extent of acts of mental travel is considered. This reasoning suggests that studying the process of mental travel will be crucial for studying the biological basis of consciousness. As noted, we argue that understanding the biological basis of mental travel relies on a clear understanding of different representational bases that subserve it.

Undoubtedly, much further research is needed to identify whether the principles of PP indeed extend to the domain of higher-order cognition and conceptual thought. Petro & Muckli describe possible routes by which to achieve this goal. Specifically, they highlight how high-resolution laminar fMRI, used in conjunction with machine-learning methods, can be used to go beyond the coarse anatomical characterization we describe, and provide a higher-resolution analysis of predictive processes at the cortical levels. They suggest focusing on the process of mental imagery as a potential central paradigm for studying conceptual processes and reconciling them with what is already known regarding perceptual processes of predictive cognition. The methodological advances highlighted in Petro and Muckli's response provide an exciting avenue for future research.

Another example of the potential usefulness of the integration we proposed relates to the commentary of Deręgowski & Tatler. As noted in the manuscript, predictive cognition approaches suppose that before we engage with the world, we already have a representation of how we expect it to be. Bayesian formulations of PP posit that these prior representations are empirical priors, namely, grounded on what we have previously learned. However, Deręgowski & Tatler ask how can such a claim be reconciled with research on perceptual illusions. For example, people who move around in the Ames room illusion seem to be changing their size. The prediction that people can spontaneously shrink does not seem to reflect our empirical priors, which appears to be a problem for predictive cognition theories of perception.

We think that our model of abstraction and prediction can help explain the supposed contradiction raised by Deręgowski and Tatler. Our account highlights the plurality of mental representations/models that are used by predictive cognition; moreover, it is a constructivist account, in the sense that it highlights that prediction (or more broadly, mental travel) entails forming a specific target representation from a multitude of source representations – which is an act of abstraction in and of itself. Abstraction, as we conceive it, is a judgment about which dimensions are relevant (and which are not) in a specific context.

When a constructivist perspective is applied to the Ames room illusion, it becomes more apparent that there are likely multiple dimensions and possible interpretations in the visual scene. Indeed, the hypothesis that individuals can shrink and grow is an unlikely one, but similarly, the hypothesis that the room is a trapezoid (as is the case in this illusion) is likewise unsupported by past experiences. Although humans don't shrink, things can shrink (after all, some objects, such as balloons and Antman do indeed shrink). In other words, it may be the case that when two unlikely hypotheses are entertained, individuals try to find meaning by generating a more abstract construal (i.e., “things shrink”) that can accommodate the data. Although people haven't encountered shrinking people and trapezoid rooms, they may have encountered more shrinking things than trapezoid things.

R4. Are multi-modal abstracta indeed distinct from modality-specific abstracta?

A common criticism leveled against our model pertains to our claim that multi-modal abstracta and categories are functionally and neuroanatomically distinct from modality-specific abstracta. Namely, criticism that comes from proponents of embodied cognition. This view was echoed, at least to some extent, in the response of Schilling, Chang, Rohlfing, & Spranger (Schilling et al.); Borghi & Tummolini; Davis et al.; and Pazzaglia & Leemhuis.

The relevant question here is that: Are there mental representations that do not necessarily involve modality-specific properties?

We argued that the answer to this question is a clear yes. We do not negate the existence of modality-specific representations and their importance – we only claim that there are mental representations that are abstracted away from their modality-specific origins. As such, an affirmative answer to the question above is quite easy to defend.

This conclusion can be justified based on conceptual analyses that date back to Kant, but psychologists such as us are better equipped to illustrate the empirical case.

As we note in the manuscript, there is ample evidence that: (i) a large part of the brain (i.e., the default mode network, including the left inferior frontal gyrus, angular gyrus, lateral temporal cortex, and anterior temporal lobe) plays a critical role in conceptual processing; (ii) these regions treat modality-specific representations of a given concept as substitutable (e.g., they respond similarly to the concept “dog” regardless of whether the participant is presented with the sound of a bark or the image of a dog's tail). Namely, they do not subserve a specific modality such as sight or sound.

To the best of our knowledge, these empirical findings are not disputed by those who endorse the embodied cognition view. A potential point of contention is whether these regions should be considered as subserving multi-modal representations. Alternatively, they could be thought of as a “hub” that bind together modality-specific representation (e.g., Rogers & McClelland Reference Rogers and McClelland2004), or, perhaps more precisely – merely as a “switchboard” between the modality-specific cortex, where the “real action” of semantic cognition takes place.

We construe multi-modal abstracta as entities whose concreta are modality-specific abstracta. As such, these abstracta indeed have “switchboard” capacities. However, we argue that they constitute a distinct representational layer – upon which categorical abstracta are instantiated. This distinction may seem to some as being irrelevant, but it is not. The difference between our view and the “switchboard” view is that we contend that the processing of non-perceptual information (e.g., “what is the capital of Norway?”) does not require the involvement of the modality-specific cortex. A “widely-distributed, modality-specific view” of semantic cognition will predict the mandatory involvement of the modality-specific representations.

This nuance is critical. If an individual subscribes to the view that modality-specific information is sometimes activated during conceptual processing, this does not make him/her a proponent of the strong “widely-distributed, modality-specific view.” Thus, the question that we need to ask is – What is the evidence that modality-specific representations are necessarily activated during the processing of non-perceptual concepts?

As noted by the commentators, there is ample evidence that processing concrete language activates modality-specific regions of the brain, which is consistent with the embodiment view. However, this does not provide evidence for necessity. In contrast, we think that the evidence against the necessity view is strong:

1. (1) As we note in the manuscript, whereas specific studies often find involvement of sensorimotor areas of the brain in language comprehension, meta-analyses of language comprehension suggest that the processing of meaning occurs outside of the sensorimotor cortex.

   We think that such a finding is difficult to reconcile with the view that sensorimotor representations should always take part in semantic processing. However, several commentators (e.g., Davis et al.; Borghi & Tummolini) argue that different concepts predominantly rely on different modalities (e.g., visual, auditory, and motor). In such a case, averaging neural activity across different concepts will “wash away” modality-specific activations. We agree that this is a viable criticism. Better evidence against “strong embodiment” can come from studies that examined the processing of concepts that involve specific modalities, for example, from research on processing of action verbs that involve motor and visual components.

2. (2) Even when one carefully looks at the literature that examined language processing of a specific modality, there is strong evidence against the idea of necessity. For example, in Gilead et al. (Reference Gilead, Liberman and Maril2013), we asked participants to read concrete and abstract sentences in the past, present, and future tense. The concrete sentences described manual actions that provided participants with both visual and motor information (e.g., “He opened the door”), whereas the abstract sentences did not involve such specific motor and visual representations (e.g., “She respected the decision”). The comparison of concrete and abstract sentences yielded significant activity in the regions identified in meta-analyses of concrete versus abstract sentence processing (Wang et al. Reference Wang, Conder, Blitzer and Shinkareva2010), namely, only in areas associated with the visual imagery such as the fusiform gyrus. More importantly, this activation was evident only in past- and present-tense sentences. For future tense sentences, there was no evidence of increased activity in the visual cortex. Our interpretation of this finding (and other similar findings, e.g., Aravena et al. Reference Aravena, Delevoye-Turrell, Deprez, Cheylus, Paulignan, Frak and Nazir2012; Reference Aravena, Courson, Frak, Cheylus, Paulignan, Deprez and Nazir2014; Tomasino & Rumiati Reference Tomasino and Rumiati2013; Tomasino et al. Reference Tomasino, Fabbro and Brambilla2014) is that the involvement of visual representations in the semantic task is ancillary, rather than obligatory.

3. (3) If meta-analyses “wash away” modality-specific activations because of their heterogeneity, we should expect that a greater focus on a specific modality should allow for the detection of modality-specific activations. However, the meta-analysis of “action verb” processing (Watson et al. Reference Watson, Cardillo, Ianni and Chatterjee2013) did not find evidence for activation in motor or premotor regions, as predicted by the embodiment view.

4. (4) One could argue that the lack of evidence for motor involvement in Watson et al.'s meta-analysis stems, again, from “washing away” the effect. For example, maybe some action verbs rely more on the hands, feet, or mouth, and a meta-analysis misses this. However, those studies that conducted the most anatomically precise analyses (e.g., used functional localizers to precisely pin-point motor areas) often failed to support the embodiment hypothesis (e.g., Postle et al. Reference Postle, McMahon, Ashton, Meredith and de Zubicaray2008).

5. (5) Although much of the evidence supportive of the embodied cognition view comes from functional neuroimaging studies, it could be argued that such studies are nonetheless limited, for example, because of their low temporal resolution and correlative nature. Indeed, there is causal evidence that suggests that individuals that suffered motor cortex lesions exhibit impairments in action semantics (e.g., Kemmerer et al. Reference Kemmerer, Rudrauf, Manzel and Tranel2012). However, as noted by Kemmerer (Reference Kemmerer2015) in studies such as Kemmerer et al. (Reference Kemmerer, Rudrauf, Manzel and Tranel2012), it is possible that the participants who had motor lesions failed action semantics tasks because they also had lesions in other regions of the left hemisphere crucial for semantic cognition (e.g., LIFG). A comprehensive study that examined the correlation between lesion location and impairment (Tarhan et al. Reference Tarhan, Watson and Buxbaum2015) did not find an association between motor areas and action semantics (but rather, localized such impairments to posterior temporal areas).

Thus, although there is compelling evidence that the modality-specific cortex is sometimes involved in semantic processing (e.g., Huth et al. Reference Huth, De Heer, Griffiths, Theunissen and Gallant2016), there is similarly compelling evidence that the sensorimotor cortex is not always involved in semantic processing. As such, we think that the strong embodiment view has been convincingly falsified. Instead, the activity of the default network likely reflects multi-modal processing that does not necessarily require modality-specific instantiation.

R5. Are categorical abstracta indeed distinct from multi-modal abstracta?

Another criticism concerning our view of conceptual cognition came from proponents of distributional semantics (i.e., Davis et al.; Montefinese, Ambrosini, Visalli, & Vinson (Montefinese et al.)). Namely, the idea that the meaning of words can be captured by keeping track of the spatiotemporal proximity of different lemmas. For example, the word “equality” is correlated in natural language with the words “liberty” and “inclusiveness.” It has long been suggested that statistical learning algorithms should be able to use this ecological co-occurrence to capture the meaning of words, including highly intangible concepts such as equality.

Indeed, research into natural language processing using artificial neural networks has been able to use such spatiotemporal associations between lemmas to devise useful representations that allow computers to perform many linguistic tasks (e.g., respond to our commands and simulate conversations, identify taxonomic relationships, and even solve word analogies; e.g., Mikolov et al. Reference Mikolov, Chen, Corrado and Dean2013).

The success of these advances in machine learning requires that we seriously consider the possibility that the acquisition of word meaning in humans is likewise merely a matter of keeping track of statistical co-occurrence. Namely, that concepts such as “executive branch” and “ball” both rely on simple associative learning, and can be thought of as multi-modal abstracta, or, in the terminology of Peirce, indexical relationships. As suggested by Montefinese et al., concepts such as “ball” may rely on spatiotemporal association between multi-modal representations (e.g., the sight of a ball and the sound of the word “ball”), and expressions such as “executive branch” may (mostly) rely on associations between different lemmas (e.g., “government” and “legislature”).

Although this account is attractive, we think that there is clear evidence that this example of machine learning is not a good description of how humans learn the meaning of (at least some) concepts. For example, I can teach you the following concept: A Kapara is a ceremony where an individual takes a live chicken by their shoulder blades and swings it above one's head exactly three times; during this process, one's sins are transferred to the chicken, such that the person performing the ceremony is absolved.

Even if you were unaware of what a Kapara is, you now understand the meaning of this word. This acquisition of meaning did not rely on a lengthy process of teaching you a model of the co-occurrence of the word Kapara with the rest of the words in the English language. In fact, you only saw one instance of this word. In our terminology, you received the criterion of substitutability of this abstractum via symbol-based interaction (i.e., reading). This criterion is the algorithm that allows you to identify the Kapara ritual next time you see it, and to perform it yourself.

We believe that such “experiments” provide clear evidence that some abstracta do not rely on statistical co-occurrence, and do not rely on innate criteria of substitutability. We term these categorical abstracta. As we review in the manuscript, there is evidence that indeed, such categorical representations rely on neural activity in areas that extend beyond the multi-modal cortex.

R6. Neglected entities?

Whereas some commentators argue that there is redundancy in our representational ontology, others have suggested that there are representational entities we may have missed out on.

One potentially neglected construct is Grafman's “Structured event complexes,” which are schematic representations of events. To us, Grafman's structured event complexes resemble the construct of script, which we discuss in our ontology, and we did not manage to find any distinguishing features between the two constructs (indeed, in Knutson et al. Reference Knutson, Wood and Grafman2004, the concept of structured event complex and script are discussed as being identical).

According to our account, scripts are amalgams of abstracta that (unlike memory for episodes) remain invariant across different particular situations. As such, scripts should rely on modality-specific and multi-modal abstracta to a lesser extent; instead, they should rely more on categorical abstracta. As we review in the manuscript, research that compares memory of specific events and scripts indeed shows that the latter is associated with activation in frontotemporal regions that supposedly subserve categorical abstracta.

Based on his extant empirical work, Grafman suggests that structured event complexes (which, as we note, may be thought of as scripts) rely on the prefrontal cortex (PFC). Although we agree that scripts rely on the PFC, the difference between our perspectives is in the interpretation of these findings. We suggest that the association between the PFC and scripts stems from the PFC's role in subserving categorical abstracta. Grafman argues that scripts are the basic representational building block that subserves PFC activity.

Evidence for our view comes from research showing that processing of relatively abstract objects (rather than events) is associated with PFC activity. For example, as we note in the manuscript the processing of superordinate objects (vs. basic level) concepts, and intangible (vs. tangible) categories recruits the PFC. Future research on the theory of structured event complexes may be able to accommodate such findings regarding the processing of non-event representations. Such attempts at theoretical arbitrations between our account and Grafman's account may help scientists' understanding of the functionality of the PFC.

Popescu & Fitch's fascinating commentary opens up a crucial question regarding the representational capacities that subserve human generativity – and the dynamic generation of complex embedded structures – evident in human language and music.

We suggested the notion of predicator as the representational entity that subserves much of the generativity of cognition. Our discussion of the construct of predicator is grounded in linguistics; within our account, predicators endow us with “compositional semantics” in that one predicator modifies its argument in a predictable manner. Popescu & Fitch suggest that linguistic predicators are a special case of a broader representational group of “partially unsaturated” entities that they call “Treelets”: “These are subunits with open ‘slots’ that allow them to flexibly combine with other subunits in constrained ways.”

These purported subunits can be observed, for example, in music: “just as a transitive verb has ‘slots’ demanding an animate subject and an object, a tonic–subdominant–dominant chord sequence allows a constrained set of continuations.” Indeed, the tonic–subdominant–dominant (henceforth, TS…) subunit bears much resemblance to verbs that require an argument. Just like how the verb “John opened…” begs you to continue it with an object, the chord progression C major, followed by F major begs you for an answer of what comes next (typically, G major), and as such it is at least in some sense “unsaturated.”

If the TS… subunit resembles the notion of predicator, what makes it distinct? Popescu & Fitch suggest that the TS… subunits are not predicators because, unlike predicators, they “make no semantic reference to the world, and entail no truth values.” Indeed, according to our conceptualization, predicators refer – in the sense that they have concreta. Furthermore, they have a truth value – in the sense that they, similar to other types of abstracta, are an embodiment of beliefs about substitutability (and as such, similar to any belief, have a “mind-to-world direction of fit,” and therefore, a truth value). However, in our account of abstraction, all of the representational building blocks of cognition (i.e., abstracta) can be thought of as having both sense and reference. Thus, we feel that it may be warranted to consider the TS… entity as a predicator nonetheless.

What we see as the main difference between TS… and predicators such as “…opened…” is that the latter are categorical abstracta (i.e., their criteria of substitutability are based on symbolic interaction) and the TS… predicator is a multi-modal abstractum (i.e., typically acquired via spatiotemporal association).

In the target article, we discuss predicators as a unique type of categorical abstractum. Indeed, those predicators that seem to be crucial for higher-order cognition and for a “language of thought” (e.g., logical predicators, mental state predicators, and mathematical predicators) are categorical abstracta, as their criteria of substitutability are not spatiotemporal. The case of musical subunits such as TS… reveals the possible existence of predicators on the level of spatiotemporal associations and even at the level of modality-specific abstractions. For example, an incomplete visual Gestalt can be thought of as a modality-specific predicator. The distinction between these types of predicators clearly deserves much attention in future behavioral and neurological studies.

R7. Constructivist critiques of our account of abstraction

We described prototypical acts of abstraction as forming the basis for mental travel, and as such, as preceding it. Several commentators have suggested that it is likely that mental travel often serves as the basis for acts of abstraction, and thus precedes it. This critique may be best summarized as such: we describe abstraction as a judgment, rather than a creative act. We relegate human creativity to the domain of mental travel.

Specifically, Fedyk & Xu suggest that human creativity does not find its proper place in our ontology and argue that it should be considered as one of the bases for acts of abstraction. In a similar vein, Faber hypothesizes that spontaneous processes of mental travel (i.e., mind wandering) may be functional in that they generate various creative permutations of events that enrich the storehouse of mental content. This synthesis of observations (which has proven to be a useful method to increase generalization of machine-learning models) can then be used as the basis for abstraction. The importance of “housekeeping” operations that occur whenever the mind is decoupled from the “here and now” is also highlighted by Visalli & Cellini, who discuss the role of sleep in the formation and updating of abstracta.

We agree with these comments that the act of mental travel (whether occurring during wake, sleep, or absent-mindedness) provides essential building blocks for acts of abstraction. As such, the relationship between abstraction and mental travel is likely a two-way street.

We discussed three routes through which we attain criteria of substitutability: innate, based on personal experiences, and based on symbol-based interaction with other humans. Fedyk & Xu argue that the contents of one's own mental travels should be considered a fourth origin. We think that this is a legitimate way to construe things. Nonetheless, it is worthwhile to recognize that there is a logical asymmetry between the three origins we discuss and this fourth one. Mental travel necessarily relies on abstraction, and abstraction relies on criteria of substitutability. Criteria of substitutability indeed sometimes originate from our mental travels – but are sometimes given to us (by nature or by our social environment) “for free.”

Just like Fedyk & Xu, so do Burman and Shillcock worry that we have neglected the generative nature of human cognition in our model of abstraction. Specifically, Burman sees an affinity between our account and Piaget's early theorizing, but not with his later work. Piaget's better-known theory construes abstraction through the perspective of extensional logic, which means that representations are strictly defined by their referents or, in our terminology, their concreta. In his later studies, Piaget saw this formulation as a mistake and reconstrued mental representations as instantiating intensions/senses. Likewise, Shillcock perceives our description of abstraction as relying on purely inductive processes: “The authors’ ontology reveals the conventional positivist notion of stepwise building up, from supposedly assumptionless ‘atomic’ foundations, verifying each move, until the highest processing is achieved.” As such, he suggests that our account may apply to recognition tasks carried out by deep neural networks, but does not capture the productive agency of human cognition.

We think that this characterization of our view of abstraction as a bottom-up process that is based on the logic of extensions is a misperception of our perspective. Indeed, our account is described from the bottom up, reflecting the ontogenetic and phylogenetic emergence of higher-order cognition. However, we explicate that the abstractum is not merely a “grouping” of exemplars. Instead, the abstractum is a mental device that instantiates “criteria of substitutability,” which is de facto, an intension. Moreover, the abstractum we describe is generative and is akin to speculative theory building, rather than a positivist collection of absolute facts. As we note: “criteria of substitutability can take a form such as ‘things that are tasty are often made of chocolate’; this means that they can implement a theory (Murphy & Medin Reference Murphy and Medin1985). Theories allow us to generate predictions of (or imagine) future members of a set, rather than just assign a probability of class membership given a list of features (i.e., they implement a generative model, Ng & Jordan Reference Ng, Jordan, Dietterich, Becker and Ghahramani2002; Helmholtz Reference Helmholtz and Shipley1856).”

Indeed, as suggested by Shillcock, feed-forward neural networks are only able to recognize patterns, and do not capture the productivity of human thought; however, bi-directional neural networks, as assumed in the predictive cognition world, implement generative models – and begin to instantiate such productivity.

R8. Theory versus simulation and abstract versus concrete representation

We suggested two distinct routes by which mental travel occurs – simulation and theory-based inference. Furthermore, we suggested an affinity between simulation and relatively concrete representation and inference and abstract representation. Several commentators suggested some qualifications to this characterization.

Dohrn provides an in-depth discussion of the notion of simulation, relying on the rich philosophical history on the topic. He disagrees with our view that mental simulation entails an “autonoetic” component, and suggests that a mental image of a boulder rolling down a hill is also a simulation, but does not involve the presence of an experiencing self.

We believe that Dohrn's response highlights that we have not adequately explicated our view of simulation. We think of the rolling boulder example highlighted by Dohrn is an example of vivid mental imagery. Our view is that one can engage in concrete imagery without engaging in a simulation – because, per (our) definition (following a similar definition by Goldman Reference Goldman2005) simulation is the process wherein a self-projected agent experiences the imagined world, and “reacts” to (simulated) occurrences.

Consider the example of a quarterback who is contemplating his next play. This quarterback may imagine the play from the perspective of a person sitting in the bleachers, or simulate his perspective as the actor holding the ball in his hands. Only if the quarterback “self-projects” into the field and allows his reality-oriented processes react to the event (i.e., “this huge guy is running at me! Ahh! I have to get rid of the ball!”), then we will say he engaged in simulation. The vivid representation of the field from the bleachers can highlight relationships between different states; the field becomes an abstract map of the situation – which is maintained in mind in a modality-specific manner. This ‘bleachers view’ is identical to how mathematical equations represent abstract ideas. The specifics of the font by which the mathematical formula is represented is irrelevant and only serves as a stand-in to convey highly intangible ideas; thus, such processes of mathematical reasoning in front of an imaginary blackboard are nonetheless theory-based inferences, supported by imagery (rather than simulation) processes.

According to our account, this vividness that simulation entails creates a difficulty in that it raises the danger of actual confusion between simulated and real events. For example, a person who vividly simulates taking a prescription pill might be confused and believe that the pill has already been taken. Reyna & Broniatowski argue that the opposite is true and that “concreteness and imageability are negatively related to false memory, per theory (Brainerd et al. Reference Brainerd, Yang, Reyna, Howe and Mills2008).” The findings Reyna & Broniatowski refer to are of higher levels of correct recall for concrete words in the DRM paradigm. Indeed, the concreteness of a word can improve its ability to be remembered. However, this does not entail that the concreteness of a simulation cannot lead to a confusion wherein a simulated event is perceived as real. Much research in the “imagination inflation” literature conclusively shows that this is indeed the case, and that simulation of an event can cause a blending between actually experienced events and imagined ones.

Whereas the imagination inflation literature highlights how simulation can permeate into one's representation of reality, Mrkva, Cian, & Van Boven (Mrkva et al.) suggest that the permeable boundaries of simulation and reality are bidirectional. Specifically, they highlight how the contents of one's simulations can be invaded by direct experience, limiting adaptive decision making. When I am contemplating whether to go on a trip to Italy, I may simulate walking the streets of Rome, eating pasta and drinking wine; the question of whether I have just eaten dinner or not should not matter for my decision processes. However, as explicated by Mrkva, Cian, and Van Boven, such fleeting sensations have substantial effects on one's decision making.

A prediction of our account, which is entailed by Mrkva et al.'s commentary, is that decisions that are based on theory-based inference should be less susceptible to one's experience in the here-and-now as compared with decisions that rely on simulation. This prediction can be readily tested. For example, such research can use the finding that prompting participants to simulate a future event in a concrete manner increases their propensity to save money (Peters & Büchel Reference Peters and Büchel2010); we predict that manipulations of one's current mood will have a greater effect on saving in the condition where participants are prompted to concretely simulate the future.

As noted, the concreteness of the simulation is an integral part of its usefulness in that it confuses us and react to the simulation as if it were the reality. However, we may not have been explicit enough in qualifying that this affinity between simulation-and-concrete, and theory-and-abstract is statistical rather than absolute. As noted by Dohrn, and discussed above, theory-based inference can sometimes be assisted by concrete representations (e.g., mathematical symbols on a blackboard). Furthermore, as highlighted by Zhao, Mildner, & Tamir (Zhao et al.), simulation often involves a complex interaction between abstract and concrete representations.

Zhao et al. establish their argument based on a fascinating study by Meyer et al. (Reference Meyer, Hershfield, Waytz, Mildner and Tamir2019). This study found that creative experts can construct vivid representations of psychologically distal scenarios by relying on activity in brain areas associated with high-level abstracta (e.g., dmPFC, angular gyrus, LIFG, and the temporal pole). At face value, the fact that the greater vividness of the representations created by creative experts relies on areas associated with categorical abstracta may seem inconsistent with our model.

However, as we discuss, one of the most important functionalities of highly abstract mental representations is in their ability to facilitate the creative generation of novel models via permutation and analogical reasoning. Our account specifically highlights the role of predicators in the creative process – as these entities subserve the ability for conceptual combination and analogical reasoning. Indeed, consistent with this view, in the research by Meyer et al., regions that supposedly subserve predication were those found to be essential for creative outputs.

Predicators are somewhat confusing – they are high-level abstracta that concretize a mental representation. When analyzing neural activity associated with predication, one should first and foremost expect to find the involvement of regions that are involved in high-level representation (e.g., angular gyrus and LIFG); however, as a consequence of this activity, we should also expect to see the involvement of lower-level representations.

Thus, we predict that when participants in Meyer et al.'s study produced vivid construals, activity in areas associated with high-level abstracta must have been correlated with activity in areas associated with vivid imagery. Namely, that the relationship between condition (creatives/controls) and activity in the medial temporal subsystem should be mediated by angular gyrus activity and moderated by vividness ratings. Such a prediction of our model could be readily examined in further analysis of the study by Meyer et al.

R9. The pragmatic perspectives

Our paper argues that conceptualizations within the predictive cognition world often endorse overly simple models of mental representations. We suggested that research into “model-based” reinforcement learning neglects the diversity of models/representations employed by cognition. Because of this limited view of representation, we think that this literature also neglects the role of discursive/deductive processes in decision making.

Dayan's commentary describes how Bayesian Decision Theory (BDT) may view the topics addressed in the paper. Dayan proposes that Bayesian decision theorists such as himself are “simple-minded sophisticates.” Namely, scholars who use technically complex models but try to explain the brain and behavior by adhering to several straight-forward, common-sense principles. Their view is that individuals survive by making good decisions and developed brain mechanisms that support such decisions. To make good decisions, individuals characterize their current predicament, evaluate the consequences of the potential choices that are available in this predicament, and choose based on these evaluations.

According to Dayan, adopting such a view entails that the process of mental simulation should not be seen as fundamentally distinct from theory-based inference. Simulation does not entail reliance on life-like representations because the Bayesian Decision Maker does not need to predict actual future events. All that needs to be predicted is the value of such events. In the words of Dayan: “BDTs only need to predict evaluations – predicting in more detail what will happen is at most a means to this particular end”; “we only really need to predict evaluations rather than actual future outcomes.”

We perceive an insufficiency in this focus on valuation as the driver of decision making. Surely, valuation is crucial, but we see it as the simplest aspect of the decision, one that merely entails crunching the numbers in the Bayesian computer. When predicting the value of actions, the challenge is to predict outcomes and imagine hypothetical realities. This part of cognition that BDTs may see as “at most a means toward an end” is what we see as the central preoccupation of the human mind.

Consider the example of an investment firm that tries to predict the value of its current portfolio in 3 years. This value may depend on the price of metal; the price of metal may depend on the question of who will be the American president in the next years; this question depends on whether Senators decide to remove the current president from office or not; this will be decided by factoring the senators' value system, political calculations, and so on.

These intermediary steps all rely on various models/representations and different processes, including processes of abstract deduction and vivid simulation. Surely, it is not the case that Bayesian Decision Models cannot integrate such complexities under their framework. However, the BDT perspective on human cognition often attempt to skirt around the messiness of such complex “inferential cranks” that are associated with model-based decisions.

Most notably, BDT typically construes model-based decision making as relying on variants of state-space search algorithms that operate upon episodic or semantic representations. The problem with this approach is that it assumes away the central challenge of the decision-maker. The state-space itself (or, in our words, the target representation) is typically not given to the problem solver and must be construed from a plethora of potential source representations.

Similarly to Dayan, Spurrett argues for the centrality of processes of evaluation in the predictive brain. Given the supposed importance of utility calculations for organisms' survival, Spurrett is surprised that we have not explicated how utility is represented in the brain and cognition. Specifically, we have not addressed a central question in Decision Theory regarding whether the brain employs an abstract “common currency” for utility calculations – or whether it uses concrete utility representations for different types of pleasures and pains.

Indeed, our paper does not address this question. However, the same principles we outlined with regard to the representation of actions, objects, and relationships translate easily to the representation of utility. Specifically, in our terminology, the question of common-currency can be recast as whether utility representations are modality-specific, multi-modal, or categorical.

We defined modality-specific abstracta as those that rely on “sensory impressions.” However, a more accurate definition of “sensory” is – “that for which organisms have some innate detection mechanism.” Organisms detect the sensory impression of “thermal,” “chemical,” and “mechanical” pain using different receptors. Some receptors are “polymodal” and respond to all three types of pain perturbations. According to our account, such polymodal receptors are still modality-specific, because they do not require any sort of association learning. Thus, pain may be effectively discussed as a single modality. Similarly, the sense of thirst stems from receptors that detect levels of osmolyte concentration and receptors that detect water volume, and are all part of a single system, and thereby, a modality.

It is possible that organisms have innate, polymodal “aversion” detectors that similarly respond to pain and thirst (without reliance on association learning). If such a system exists, then this representation of (negative) value serves as a modality-specific common currency representation and should exist outside of the multi-modal brain, potentially in sub-cortical areas.

Importantly, even if such modality-specific common currency representations indeed exist, they may live side-by-side with multi-modal and categorical common-currency representations. For example, representations of aversion can be associated with conditioned stimuli (e.g., the light that precedes the electric shock). This will give rise to a multi-modal abstractum that allows the organism to treat both the light and the shock as equivalent. Furthermore, some common currency representations of negative utility rely on categorical abstracta, acquired from society (e.g., the valuation of death as being a better outcome than the betrayal of one's sacred values). Such common-currency representations will likely be supported by frontotemporal areas that subserve categorical abstracta (rather than subcortical and modality-specific representations).

Thus, an implication of our view of abstraction and our pluralistic account of representations is that one should not expect to find a single “common currency” representation in the brain. Instead, one should expect to find various types of common currency representations, whose physiological bases depends on the extent to which they rely on modality-specific, multi-modal, or categorical abstracta.

The idea that representations of value can be modality-specific, multi-modal, or categorical has important ramifications for choice behavior. Ledgerwood, Eastwick, & Gawronski's commentary explicates some of these consequences concerning the choices people make in social settings. They suggest that contextual factors that influence the level of construal determine whether individuals rely on categorical representations of value (which they term “ideas regarding liking”) rather than on the concrete/embodied experience of value.

Importantly, there may be many cases where representations of value at different levels diverge. For example, a person eating at an over-hyped, expensive restaurant may have the abstract idea that she is having the time of her life, whereas her embodied representations may signal to her that this is an aversive experience. Thus, a representational diversity account of value may entail that one's construal of pain and pleasure can sometimes be quite fragmented. Such a multi-level account of value can have significant ramifications for people's construal of their (and others') choices.

R10. The social construction perspectives

The complexity of the human condition, entailed by our ability to generate and transfer categorical abstracta, extends well beyond the realm of valuation processes. This point is clearly articulated in the commentary by Rossignac-Milon, Pinelli, & Higgins concerning the social nature of abstraction and mental travel.

Their response highlights several aspects of the intersubjective aspects of mental travel, which we have not sufficiently highlighted. We discussed how meanings are transmitted from mind to mind; however, an important, understudied process is how two or more individuals jointly construct meaning. Such processes can sometimes be reduced to the workings of each mind, but sometimes may have emergent properties that may require their own explanatory mechanisms.

As we note, the creation of collective meaning is not merely a palatable experience but may have had actual survival benefits. We highlighted how intangible concepts could facilitate cohesion by generating facts that cannot be verified, and as such, can be used to generate myths that support social structure. Montefinese et al. extend this idea. They suggest that intangible constructs create interdependence on others – once we realize that we cannot figure out their meaning on our own. The uncertainty created by newly “discovered” intangibles pulls people together into communities where such puzzles are generated (i.e., intangible constructs are created) and then jointly solved. In other words, Montefinese et al. suggest that humans' fascination and preoccupation with abstract ideas is adaptive, in that it provides people with mysteries that bind them together.

As evident by the numerous fascinating responses to our paper, the sense of mystery surrounding concepts such as “abstraction” and “mental travel” can indeed prompt people to engage in constructive, joint efforts to comprehend the intangibles of the mind.