The puzzle of ideography can be broadly described as the near absence or rarity of self-sufficient generalist graphic codes for use in human communication. Morin proposes a resolution to this puzzle in terms of a “specialization hypothesis” (target article, sect. 4), which is subsequently explained in terms of a “standardization account” (target article, sect. 6). The latter does well in predicting empirical features in communication practice. Morin also pays due diligence to the alternative “learning account” (target article, sect. 5.4) explanation, which helps the discussion. However, both these explanations have primarily been cast in behavioral terms. We argue that there also exists a “cognitive account” which complements the behavioral explanation.

The cognitive account we propose is anchored on two well-studied cognitive phenomenon: (i) Modality effect, and (ii) Chunking. We show that (i) and (ii) offer a complementary explanation to the specialization hypothesis. Furthermore, combining the above two phenomena with two design features pertaining to cognitive processing: Multisensory integration, and the serialized nature of attention, explains what enables human languages to become self-standardizing, and also the precise role of learning.

The modality effect has extensively been studied in experimental psychology (Murdock, Reference Murdock1968; Penney, Reference Penney1989). It refers to improved recall of items on a list when presented verbally in contrast to a purely visual representation. Subsequent studies, investigating this advantage of auditory over visual text modality during learning, have attributed the effect to factors including reduction in extraneous cognitive load, effective expansion of working memory, and early-sensory processes (Rummer, Schweppe, Fürstenberg, Scheiter, & Zindler, Reference Rummer, Schweppe, Fürstenberg, Scheiter and Zindler2011; Tabbers, Martens, & Van Merrienboer, Reference Tabbers, Martens and Van Merrienboer2001). Additionally, the instructional benefits of presenting information across modalities found support in a meta-analysis involving 43 independent studies (Ginns, Reference Ginns2005).

Prima facie, the modality effect concerns learning and memory. However, one could argue that what can be learned or recalled better is also easier to repair during the process of turn-taking, and hence easier to standardize within a population. With respect to the specialization hypothesis, what this means is that at the population level, spoken forms of information exchange will dominate those that are purely visual. This comprises one part of our account.

The other part relevant to the cognitive account is chunking. In cognitive psychology, this refers to the process by which individual pieces of an information set are bound together into a meaningful whole (de Groot, Reference de Groot, Warson and Johnson-Laird1968; Miller, Reference Miller1956). Evidence for perceptual chunking is found in how primitive stimuli are grouped into larger conceptual groups, such as the manner by which letters are grouped into words, sentences, or paragraphs (Johnson, Reference Johnson1970). Chunking is an effective strategy for overcoming capacity limits of working and long-term memory via coherent grouping of information (Laird, Rosenbloom, & Newell, Reference Laird, Rosenbloom and Newell1984). It has been observed in paradigms involving verbal learning, learning of perceptual-motor skills, expert memory, language acquisition, and learning multiple representations (Gobet et al., Reference Gobet, Lane, Croker, Cheng, Jones, Oliver and Pine2001).

What is relevant here is that chunking facilitates the creation of higher-order cognitive representations specific to the individual's perceptions and past experiences. Thus, allowing for greater abstraction and generality of cognitive representations. One could argue that this is not only about learning or memory, but underlies the core cognitive mechanism for generating flexible higher-order conceptualizations.

Why is this so important vis-à-vis the specialization hypothesis? The answer is that ideographs and other graphic codes that are not languages, allow for very limited abstraction and higher-order representation via compositionality of symbols. In language, compositions from only a limited set of letters allow for creating an astoundingly large number of new concepts and higher-order representations. The same is not true for graphic codes (where base symbols already represent a specific concept). Take for example, the case of Bliss: Composing symbols does not allow for reduction in complexity of expressions or creation of higher-order chunks. In contrast, a study presenting stimuli generated by an artificial grammar showed that subjects unintentionally learned to respond efficiently to the underlying structure (Servan-Schreiber & Anderson, Reference Servan-Schreiber and Anderson1990). Moreover, it was demonstrated that the learning process was chunking and that grammatical knowledge was implicitly encoded in a hierarchical network of chunks.

The point we want to emphasize is that the mechanism of chunking is precisely the brain's operationalization of compositionality of cognitive representations. That the latter happens to be what languages are designed for, explains why the specialization hypothesis holds at all. Compositionality is the reason languages beat graphic codes for general-purpose communication. This also explains why writing is the only exception of a graphic code that is general-purpose: Because it simply tags upon language.

Finally, we discuss two cognitive design features relevant to the puzzle. First, multisensory integration. The effectiveness of a stimulus of one modality in eliciting attentive behaviors and faster reaction times was dramatically affected by the presence of a stimulus from another modality (Hershenson, Reference Hershenson1962; Stein, Meredith, Huneycutt, & McDade, Reference Stein, Meredith, Huneycutt and McDade1989). The brain reinforces saliency of representations that use congruent evidence from multiple modalities, thus, making the case for enhanced learnability of concepts involving unambiguous auditory and visual representation. This is true for words of a spoken language but not so for graphic codes that one may verbalizable in different ways and hence be difficult to standardize.

The second feature is the serialized nature of attention. This matters at the decoding/read-out stage. The brain's attention system seems designed to (consciously) process one task at a time (Baars, Reference Baars1998). Studies have shown divided attention impedes performance (Pashler, Johnston, & Ruthruff, Reference Pashler, Johnston and Ruthruff2001). Spoken language is strictly temporal and hence serialized. Written language tags along that temporality. However, complex ideographs may have layouts as two-dimensional patterns, such that different decoders may assume a different read-out order. At the population level, this directly speaks to the difficulty in standardizing such codes.