R.2.1. On distinguishing codes from noncodes

The target article was focused on the evolution of graphic codes, rather than graphic communication in general. The key property that marks out graphic codes from other forms of visual communication is the importance of conventional (or standardized) mappings between symbols and meanings. Writing systems, heraldic emblems, musical notations, and so on are highly codified in this sense. Visual art forms like paintings, graffiti, comic books, and so on, are not. This does not mean visual art does not carry information: As Sueur & Pelé's commentary notices, even simple abstract doodles or abstract paintings produced by apes possess informational content, in the sense that they form shapes that are both visually complex and predictable to a degree. But graphic codes carry information in a way that is quite different, and much more powerful. Graphic codes possess two important properties: They allow their users to compress a great deal of information into a few simple signs (Garrod, Fay, Lee, Oberlander, & MacLeod, Reference Garrod, Fay, Lee, Oberlander and MacLeod2007; Tamariz & Kirby, Reference Tamariz and Kirby2015; Winters & Morin, Reference Winters and Morin2019), but they require users to learn the code. This capacity to compress information is both a defining property and a key advantage of graphic codes, as Moldoveanu notices when he aptly defines graphic codes as a form of “source coding.” Being compressed, codified signals tend to be simpler, cheaper to produce, and less cumbersome to store. In contrast, noncodified representations convey a much smaller quantity of information. They tend to be more complex, thus more expensive to produce; but they can be interpreted immediately without the need to master a code. How do we recognize that a mode of expression relies on codification? Two simple cues are the amount of time or effort needed to learn the code, and the need for translation. Using this yardstick, we can easily see, for instance, that comic books are clearly less codified than natural language, and that Chinese characters are as codified as the vocabulary of a spoken language.

These points did not persuade all commentators. Zhang, Hu, Li, & Chen (Zhang et al.) argue that Chinese writing is iconic, not symbolic: The characters' meaning is immediately accessible without knowing the underlying code. Cohn & Schilperoord argue that comic books are a visual language on a par with spoken languages. Other commentaries (e.g., Straffon, Papa, Øhrn, & Bender [Straffon et al.]; Wisher & Tylén) broadly agree with the target article but underscore the importance of visual arts and iconic resources in graphic communication.

R.2.2. On iconicity

Signs are iconic, according to the target article's definition, if their meaning can be transparently derived from their form alone, without being acquainted with a specific conventional code. Following a broad scholarly consensus, I claimed that writing, like language, is not iconic in this way. Not all commentators agree. Zhang et al. (and perhaps Yan & Kliegl, more cautiously) argue that Chinese writing is iconic to a certain extent, whereas Wisher & Tylén cast doubt on the general idea that language is an arbitrary code. Zhang et al. argue that most Chinese characters directly describe the shape of things, citing characters such as 日 (rì, sun), 月 (yuè, moon), and 山 (shān, mountain). Yan & Kliegl echo this view in a more watered-down version, arguing that “…some [Chinese] characters are…largely recognizable even by untrained eyes (e.g., 山 for mountain and 田 for farmland).” Is this true?

The studies that investigated the iconicity of Chinese characters (Koriat & Levy, Reference Koriat and Levy1979; Luk & Bialystok, Reference Luk and Bialystok2005; Xiao & Treiman, Reference Xiao and Treiman2012) generally use a two-alternative forced choice (2AFC) paradigm. Participants are shown a character (e.g., 山) and asked to choose between two associated meanings, the correct one (mountain) and an incorrect distractor (e.g., lake). A 2AFC experiment is considered conclusive when a statistically significant proportion of guesses reaches above the level of chance (50%). This sets a rather low bar for a symbol to count as iconic. True iconicity would be obtained if participants spontaneously guessed a symbol's accurate meaning on seeing the symbol, without being primed with two meanings that include the right one. (This happens less than 2% of the time in Luk & Bialystok, Reference Luk and Bialystok2005.) Even using the 2AFC task, for cherry-picked sets of iconic characters, performances are mediocre. In the most recent study on the topic (Xiao & Treiman, Reference Xiao and Treiman2012), the characters cited by the commentaries, 山, 田, 月, and 日 are recognized at rates of 65, 85, 55, and 35%, respectively. And are such characters the majority? No, as Yan & Kliegl recognize. Out of the 213 (highly common) characters studied by Xiao and Treiman (Reference Xiao and Treiman2012), only 15 are guessed above chance in the 2AFC.

The example above underlines the perils of relying on 2AFC tasks to show that symbols or wordforms are iconic as opposed to arbitrary. Such studies show that symbols provide some information about their referents by iconicity alone, and they usually show that this amount is small. They do not demonstrate that the symbols derive most of their informative power from iconicity alone. More generally, studies showing nonrandom associations between (some) wordforms and their meaning are intriguing (Dingemanse, Blasi, Lupyan, Christiansen, & Monaghan, Reference Dingemanse, Blasi, Lupyan, Christiansen and Monaghan2015; Monaghan, Shillcock, Christiansen, & Kirby, Reference Monaghan, Shillcock, Christiansen and Kirby2014); but they do not claim to challenge the view that language mostly rests on conventional mappings between symbols and referents (Lewis, Reference Lewis1969; Skyrms, Reference Skyrms2010). They actually show the opposite: For instance, Monaghan et al. (Reference Monaghan, Shillcock, Christiansen and Kirby2014) estimate at 0.02% the share of variance in one-syllable English wordform properties that can be predicted by iconicity. Such differences in degree are massive enough to justify drawing a sharp boundary between spoken or signed languages, on the one hand, and on the other, means of expression that rely overwhelmingly on iconicity, such as visual arts.

R.2.3. Comics are not a visual language

In this spirit, I presented two simple arguments against the view that comic books rely on a visual language (Cohn, Reference Cohn2013). First, when exported to a different country, comic book drawings require no translation, whereas their written dialogues do. Second, the conventions of a particular genre (manga, for instance) can be assimilated in a few hours, compared to the years required to attain fluency in spoken language. More fine-grained arguments could be given. Wisher & Tylén, for instance, note that “the comics principle … has not evolved into conventionalized ideographic codes,” in particular when it comes to encoding anaphoric relations. In their commentary, Cohn & Schilperoord address none of these objections.

Most of the disagreements between me and Cohn & Schilperoord spring from the way we understand conventionality, or standardization (two words that I use interchangeably). The topic of the target article is the evolution of codes, understood as standardized (or conventional) mappings between symbols and their meanings (de Saussure, Reference de Saussure2011; Lewis, Reference Lewis1969; Millikan, Reference Millikan1998; Scott-Phillips, Reference Scott-Phillips2014). I took the view that the kind of standardization that matters is the standardization of the code that pairs meanings and symbols – in other words, codification. Linguistic codes, I argue, are far more standardized than many visual codes (including comics). Thanks to this, they can carry and store much more information, compared to merely iconic means of expression. Cohn & Schilperoord's reply stems from a different view of conventionality, or standardization. In their definition, contrary to mine, standardization has nothing to do with the way images map to meanings. Thus, purely iconic signs can be highly standardized. They provide a number of examples for iconic signs that are conventional, in the sense that artists in different traditions depict eyes or fists in the same way, quite different from the way that is taught in other traditions.

So far, we simply have different ways of using the same words; but the disagreement becomes substantial when Cohn & Schilperoord go on to claim that standardization has nothing to do with the distinction between iconic or noniconic signs. This, in my view, is tantamount to dismissing the contribution that standardized codes make to communication. Indeed, they go on to show that comic books, being standardized, are a full-blown language on a par with spoken languages. The fact that comic books overwhelmingly rely on iconicity is irrelevant in their view, because spoken languages (they claim) also rely on iconicity to some extent. (A point I addressed in the previous section.)

Suppose we agree, and the formal standardization of styles is all that matters for comic book styles to count as full-blown languages. That would compel us to put any form of graphic expression showing variations in style on the same footing as spoken language. This includes cave art (Guthrie, Reference Guthrie2006), culturally transmitted pottery decorations (Crema, Kandler, & Shennan, Reference Crema, Kandler and Shennan2016); beyond this, cultural patterns have been found, or claimed, for nest building in some bird species (Madden, Reference Madden2008), for primate tool making, and so on. All these things may be called “visual languages” – and why not? They are, after all, visual forms of expression, forming cultural patterns. Yet this simplification comes at a cost. It renders us incapable of putting a name on something that makes spoken and signed languages uniquely powerful and informative. That something is a shared code. A shared code allows language users to convey information cheaply and efficiently, saving them the effort of depicting or explaining meanings that are already encoded. Having a shared code requires standardizing mappings between symbols and meanings – mere standardization of forms does not suffice.

R.2.4. Visual arts cannot fulfill all the functions of graphic codes

I welcome Straffon et al.'s eloquent plea for taking seriously the variety of visual communication systems, and I fully agree with them that writing is very far from being the only efficient way of transmitting information with images. I am less persuaded by the claim that visual arts alone can enable rich forms of communication comparable to those that graphic codes make possible. I worry that we may be falling into an old trap: The temptation to underestimate the degree to which visual communication is codified. This temptation has been a recurrent problem in the study of American graphic codes – the region on which Straffon et al. focus their commentary. Cuna shamans' pictographs were dismissed as mere drawings before anthropologists like Severi (Reference Severi2019) showed how they worked; the very idea of a writing system native to America was dismissed until quite late (e.g., Gelb, Reference Gelb1963); and it took a long time before the sophisticated encoding system of khipus – not a writing system but a complex graphic–haptic code – was taken seriously (Urton, Reference Urton2017). Precedents like this make me wary of embracing the highly controversial view that Teotihuacan had no writing (Helmke & Nielsen, Reference Helmke and Nielsen2021, make a good case for a native glottographic writing system at Teotihuacan). Likewise, I doubt the Inka could have held together a vast empire using visual arts alone: They could hardly have done it without their khipus. These minor disagreements aside, I agree on Straffon et al.'s most important point: The evolution of writing was highly contingent and unpredictable, due in part to the availability of graphic codes that could fulfill some of its functions.

To summarize, the puzzle of ideography is a puzzle about the evolution of graphic codes. Graphic codes are mostly conventional: They cannot be read or produced fluently by someone who does not know the underlying standard for pairing symbols with meanings. As a corollary, many ideographic codes are not iconic at all (algebraic signs, logical symbols, monetary symbols, etc.). Some are only residually iconic, like musical notations. The relative height of notes on the staff score is iconically linked to their pitch, but this does not get us very far if we want to know a note's exact pitch, because the clef symbol is not encoded iconically, and neither are the alterations, and many aspects of the code have no iconic meaning – for example, the fact that white notes are longer than black ones. This is why codes like this one interest me: Because they can only evolve if their users are taught a series of conventions. Forms of expression that are not strongly codified, like visual arts, were not my primary concern, because they lack the crucial power of graphic codes: The power to compress information, thus allowing us to store and transport it.

R.3.1. Why literacy requires linguistic competence

Three commentaries that do not object to the specialization hypothesis in general nonetheless wish to nuance the claim that writing encodes language. Two of them (Yan & Kliegl; Zhang et al.) are specific to Chinese writing. Sterelny's remarks are much more general. He makes threeFootnote ¹ objections against the view that writing encodes language.

First, holding that view forces me to deny the possibility that someone achieves literacy in a language they cannot speak. This happens (in Sterelny's view) to academics who learnt a language through books alone. This case is very close to situations of literate diglossia (which the target article does mention). Diglossia typically occurs when a language spawns a literate variant that is overwhelmingly used by clerks for literate use. Classical Chinese, literate Arabic, or Renaissance Latin are cases in point. Such literate languages can develop on their own and become quite distinct from their spoken counterpart. The people who can read and write these languages can also read them aloud, and are trained to do so in specific settings (e.g., classical poetry recitations). This, to me, counts as a kind of linguistic competence, albeit one that lacks fluency. Today's classicists are able to speak Latin – their own literate kind of Latin. Sterelny seems to disagree, but holding this stance would force him to say that linguistic competence in Sumerian, ancient Egyptian, Hittite, and so on has entirely disappeared, turning most paleographers into strange impostors.

Sterelny turns to semantics for his last objection. If it were true that writing represents spoken language, he notes, then written sentences would be representations of spoken sentences, not of states of the world. Thus, the written sentence “Berlin is the capital of Germany” would have the corresponding spoken sentence as its truth condition: It would not be a statement of fact. This would make standard semantics inapplicable to written sentences. Sterelny reads a lot into my use of the verb “represent”; I more often wrote that writing systems encode spoken languages – probably a more adequate verb. Inscriptions that encode sentences should not be confounded with the truth-bearing proposition that the encoded sentence may express, as Frege (for instance) made clear:

Written sentences, in this Fregean view, can be analyzed on two levels. On the first level, the printed inscription “Berlin is the capital of Germany” is an encoding of a possible spoken sentence, that is to say, a set of instructions or a recipe for forming a sentence (a set of possible spoken sentences, to be precise, because accentuation, prosody, etc. are not usually encoded). As an encoding, the inscribed sentence may be more or less accurate (for instance, it may contain typos); but it lacks full-blown truth-conditional meaning. On a second level, the sentence that the inscription encodes may express a proposition with truth conditions, if some additional conditions are fulfilled: The proposition must be expressed with the right assertoric force, and at the right moment (the proposition was neither true nor false in 1960 or 1750, but it is true today). Sterelny's challenge dissolves when we distinguish propositions, which are truth-bearers, from inscriptions, which are not.

R.3.2. Chinese writing

There was on the whole, relatively little push-back against the glottographic view of Chinese writing advocated in the target article. Zhang et al. dispute it, but only concerning its earliest manifestation, the writing on Shang oracle bones. Even though the use of phono-semantic compounds seems attested on oracle bones (Boltz, Reference Boltz1993), and even though Old Chinese phonology has been successfully reconstructed, I agree that the writing system was highly ambiguous (Demattè, Reference Demattè2022). In this it is similar to other pristine inventions of writing, where the mapping between syllables and symbols took a long time to become systematic and standardized (see, e.g., Hermalin & Regier, Reference Hermalin and Regier2019). Yan & Kliegl broadly agree that “Chinese encodes a natural spoken language just as alphabetic scripts do,” but also point out the rather unique aspects of the relation between Chinese writing and language. Their insightful commentary contains claims that I doubt – how easy it is for Japanese readers to read Chinese, or how iconic some Chinese characters may be. There are also claims that I agree with: It is true that “phonetic radical[s] merely provides an unreliable clue to [their] pronunciation.” However the same claim can be made (to varying degrees) about most other writing systems except the most regular ones (e.g., Finnish or Hungarian writing), because many writing systems are highly irregular (including English). Overall, I take Yan & Kliegl's welcome qualifications as matters of nuance.

R.3.3. Strengthening the specialization hypothesis: The centrality of language

Thus, only a minority of commentaries challenge the specialization hypothesis, or the glottographic view of language defended in the target article, and they do not provide strong arguments to reject it. Other commentaries seem generally to endorse it, and three commentaries go much further than the target article. Overmann sees a more watertight boundary between writing and other graphic codes than I do; Harbour argues that graphic codes rely on the language faculty even when they do not encode the language that their users speak; Winters argues for a strong role of language not just in the emergence of writing, but in the evolution of all graphic codes.

Overmann's commentary emphasizes the uniqueness of writing (which she calls “visible language”) vis-à-vis other graphic codes. She argues that writing is entirely distinct from other graphic codes (e.g., musical or numerical notations), functionally, psychologically, and historically. I agree with her that writing was a rather singular invention, which needs to be distinguished starkly from other graphic codes. That being said, I am not convinced that writing has “no direct material precursors” (because all pristine inventions of writing had clear precursors, I assume the truth of Overmann's claim hinges on what we mean by “direct”). Nor do I think that the use of symbol recombination is unique to writing (see sect. R.7.3). The specialization hypothesis does not need to overplay the uniqueness of writing.

Harbour's commentary highlights the many surprising ways in which graphic codes, including writing, may rely on the language faculty, or at least on cognitive mechanisms that are shared with the language faculty broadly construed (such as recursion). Harbour deftly shows that some of the linguistic structure in writing systems is not actually derived from the language that they encode. Writing systems have a degree of autonomy from their target language, and may form structures that have no counterpart in it, like the determinatives found in Egyptian hieroglyphs. These structures can be analyzed using the tools of linguistics, because they behave according to the same rules as similar structures found in other languages – but not in the encoded language. Thus, Egyptian hieroglyphs reinvented determinatives, even though the language they encode lacks them. With examples like this one, Harbour makes a convincing case for the research program that studies writing systems as linguistic objects without reducing them to mere reflections of the structure of their target language.

Harbour goes on to suggest that, because linguistic structure pervades graphic codes, ideography was always dead on arrival (so to speak). If ideography is defined as a language-independent graphic code, and if language-like structures always find ways to sneak into graphic codes, even the codes not made to encode them, how could a language-free ideography ever evolve? This argument rests on an ambiguity in how we define “language.” The way the target article defines them, ideographies are graphic codes that do not encode elements of a specific spoken (or signed) language. This leaves open the possibility that ideographies may themselves possess language-like features such as recursion, duality of patterning, or compositionality (see sect. R.7.3). Put differently, what differentiates a writing system from an ideography is the fact that writing systems encode a specific natural language that preexists in them – not the fact that it possesses general language-like features like recursion, syntax, and so on. Taking Harbour's argument to its extreme limit would lead us to a familiar position in the debates over ideography: Ideography cannot logically exist, because it is conceptually impossible to assign meanings to symbols without in some way using language (Boltz, Reference Boltz1993; du Ponceau Reference du Ponceau1838). If this position were true, there would be no such thing as an ideograph, and we would be completely unable to describe the difference between symbols like 2, 3, , and the written words “one,” “two,” “love” (Edgerton, Reference Edgerton1941Footnote ²). Ideography is simply the fact that some symbols encode ideas directly, bypassing words. These symbols may still possess rich language-like properties.

Winters's commentary further stresses the centrality of language to all graphic codes, not simply writing. He makes many important points, all of which I endorse. Most important perhaps is his remark that self-sufficient specialized graphic codes, like musical or mathematical notations, seem to evolve much more readily in literate societies. This does not mean societies without writing do not produce rich and sophisticated graphic codes – they do – but it does mean these codes are unlikely to be self-sufficient (i.e., they should rely on oral glosses). I agree with Winters that this is a natural consequence of the standardization account, because writing, itself a product of successful coordination, in turn becomes a powerful coordination device.

Winters raises a question also broached by other commentaries (Adiego & Valério; Gainotti): To what extent can a graphic code ever be independent of spoken language? Adiego & Valério argue that the creation and first acquisition of codes (if not their use) must involve linguistic communication. Gainotti shows, on the basis of clinical evidence with aphasic patients, that learning to communicate ideographically is not a domain-general process, but rather relies on language-specific cognition. The difficulties that aphasics experience in learning Bliss or other pictorial codes do suggest this, because their language faculty is damaged but their capacities for perception or memorization are preserved. For Winters, creating and learning a graphic code from scratch without using language is possible in theory, but unlikely in practice. One intriguing argument that he gives is the apparent lack of codified permanent visual marks in nonhuman animals, in contrast to the importance of acoustic codes in several species of birds and primates. Here again, I am tempted to agree; bower-bird nest decorations are the closest thing to a counterexample that comes to my mind, but they do not have codified meanings in the way that, for instance, vervet monkey calls do.

R.4.1. Alternatives to Liberman's learning account

Harris et al. object to my critique of the learning account, because it focuses on Liberman and his motor theory (which they quickly dismiss). In their view, I should have criticized the learning account in a much more general way, going beyond Liberman's specific theory to include other possible versions of the learning account. This discussion gives me an occasion to do exactly that.

Harris et al. present an intriguing argument to defend the general idea of a learning account. Dyslexic as well as congenitally deaf individuals cannot easily link letters to sounds, but this would not be an obstacle to literacy if they simply could treat writing like an ideography. In other words, if humans had no problem learning arbitrary pairings between signs and meanings, dyslexics and deaf people could simply treat written words as ideographs – that is, take them as pointing directly to ideas, bypassing language. This, in their view, does not happen. Only a tiny minority of profoundly deaf people learn to read in this way (and then again, not very successfully). As for dyslexia, attempts to cure it by training learners to process words as ideographic symbols aggravate the condition instead of improving it. Harris et al. take this as a strong indication that ideography raises an insurmountable learning problem.

The learning difficulties that Harris et al. highlight are real and significant. The fact that they are also found for the Chinese script is an important reason to resist nonglottographic accounts of Chinese characters. But do they show that ideographies in general are unlearnable? No. The fact that deaf people or dyslexics usually fail to recognize written words by their visual shape alone does not make this point. Writing it is not an ideography, so attempts to learn writing as if it were ideographic are bound to fail. The people Harris et al. refer to all have some mastery of the spoken language that writing encodes; they cannot simply ignore their linguistic knowledge – as Harris et al. acknowledge. Thus, I fail to see how Harris et al.'s argument could make a case against the learnability of ideographies. Writing is the very opposite of an ideography; dyslexics and deaf people struggle with writing precisely because writing encodes language.

A counterpoint to Harris et al.'s position is found in Sterelny's commentary. Sterelny considers that a glottographic view of writing cannot account for the success of profoundly deaf people in learning to read. Harris et al. blame the target article for taking deaf literacy too seriously; Sterelny criticizes it for failing to take deaf literacy seriously enough. I attempted to explain literacy in profoundly deaf people by noting that phonemes, syllables, or morphemes are not sounds but contrastive categories, making it theoretically possible to become literate in a language when one's main contact with that language is visual, not aural. Sterelny says this account is not “natural,” suggesting that he subscribes to the view that profoundly deaf people may learn a spoken language from print only, by mapping printed words directly onto sign language words (Hoffmeister & Caldwell-Harris, Reference Hoffmeister and Caldwell-Harris2014). This view can be disputed, however, because most deaf people who learn to read know the encoded spoken language from partial hearing, lip-reading, or signed versions of the spoken language. Whether this phonemical awareness acts as a help, a hindrance, or a mere byproduct, is not entirely clear, a point that my article should have paid more attention to. In any case, Harris et al.'s commentary makes evident the drastic limitations that profoundly deaf people face in learning to read (Hirshorn & Harris, Reference Hirshorn and Harris2022).

Arsiwalla's thoughtful version of the learning account puts forward two mechanisms that enhance the learnability of spoken language as opposed to visual ones. Arsiwalla's first argument is that ideographic messages are difficult to decompose into discrete chunks, making ideographic codes harder to memorize. To make this point, Arsiwalla relies on the assumption that ideographic codes lack compositionality – an assumption that I refute in section R.7.3. His second argument is based on the view that multimodal learning, combining visual and aural information, is more efficient compared to learning through one modality alone. It rests on a literature showing, in classroom contexts, that students are better at retaining things taught using both visual material and an oral gloss, compared to things taught with exclusively visual material (usually a diagram and a written gloss). I do not quite understand how these intriguing studies would make a case against the learnability of ideography. To make such a case, Arsiwalla would have needed to show that spoken and signed languages are learnt multimodally, and that domain-specific ideographies are not. The acquisition of codes generally relies on several modalities, and this includes ideographic codes like musical notations. Arsiwalla does not explain what would make speech acquisition special in this respect. More importantly, he does not mention sign languages, which are perfectly learnable for deaf people who have at best limited access to the aural modality.

Overall, the two commentaries written in defense of the learning account do not succeed in producing a cognitive mechanism that makes it difficult to learn ideographic codes, without raising the same problem for spoken languages. If one wants to claim that a learnability problem is what prevents us from acquiring a generalist ideography, one has to explain where exactly the problem resides. And the answer cannot be modality. The reason generalist ideographies do not take off is not because they are visual – sign language linguistics has taught us that. One could perhaps imagine other versions of the learning account that do not face this problem, and that would be an exciting research program; but the commentators have not done this yet.

R.4.2. Strengthening the case against learning account

Nephew et al. provide an intriguing argument against some versions of the learning account by highlighting human proficiency with face recognition. The fact that we routinely process and correctly recognize thousands of faces shows how far human visual memory can go. Although this argument would not affect Liberman's version of the learning account (because in Liberman's view we learn codes through a motor route not a visual one), it would clearly be a problem for anyone who claims that ideographies are unlearnable because of visual memory constraints (as Harris et al. seem to do). I take Nephew et al.'s point, with a handful of qualifications. The typical inventory of faces a person can identify ranges in the thousands – 5,000 on average and up to 10,000 for “super-recognizers” (Jenkins, Dowsett, & Burton, Reference Jenkins, Dowsett and Burton2018). This is as high as the size of the most complex-documented graphic codes.Footnote ³ It is not, however, extremely high if compared to the vocabulary size of English (to take a well-studied example), which was estimated at 42,000 lemmas on average, with other estimates ranging from 10,000 to hundreds of thousands (Brysbaert, Stevens, Mandera, & Keuleers, Reference Brysbaert, Stevens, Mandera and Keuleers2016). Another important caveat, pointed out by Nephew et al. themselves, is the domain-specific nature of face recognition, which is cognitively distinct from neighboring skills such as voice recognition (Young, Frühholz, & Schweinberger, Reference Young, Frühholz and Schweinberger2020). With these limitations in mind, I agree with Nephew et al.: Our face recognition abilities suggest that visual memory per se does not stand in the way of ideography.

Another commentary that redoubles my critique of the learning account is Howard's. He concentrates his commentary on Bliss symbols that, he claims, are easily taught and learnt. He blames me for not citing the abundant literature that (he thinks) makes this point. I disagree. The authors of these studies did not actually try to teach Bliss symbolics to their participants. Their goal was different: To compare Bliss with other ideographic systems like Carrier symbols, or pictographic systems like Rebus, Picsyms, or the Picture Communication System, based on transparent pictures. To perform these comparisons, they familiarized their participants with a small number of symbols, in the short term.Footnote ⁴ Only one of the studies cited comes close to a genuine attempt at teaching Bliss: Funnell and Allport (Reference Funnell and Allport1989). Finding the results disappointing, they gave up on using Bliss as an alternative communication tool with their two aphasic patients. Gainotti points to several other studies that reached the same conclusion. There is, thus, no clear-cut evidence that Bliss symbolics is readily learnable. By itself, this lack of evidence proves little – only that seriously trying to teach an ideographic language to normal adults or children (let alone patients) would be a costly and complex enterprise that researchers are yet to tackle.

R.7.1. Standardization of communication in the digital world

The target article argued that codes based on cheap, fast, and transient signals are easier to standardize, because of two independent mechanisms. The first is cheap and fast production leading to repair and alignment: Cheap and fast signals can be modified or repaired multiple times, allowing interlocutors to converge on shared meanings. The second is face-to-face communication with transient signals. Transient signals constrain interlocutors to face-to-face interactions, where the advantages of common ground are maximized. The target article argued that digital communication shared some (not all) of the characteristics of the cheap, fast, and transient signals of spoken or signed languages. This should ease the standardization problem somewhat. Most of the commentaries that broach this issue agree with the target article on these points: Digital communication is closer to spoken or signed communication than to legacy graphic communication, and this should ease the standardization problem to a degree (Clark; Gandolfi & Pickering; Veit & Browning).

The target article, however, was unclear on the exact nature of the mechanism that could further standardization in digital communication. I mentioned two mechanisms: Cheap and fast repair and alignment and face-to-face communication with transient signals. The two are independent; they can be dissociated, and in the case of digital communication they clearly are. The commentaries by Feldman and Gandolfi & Pickering allow me to clarify my views on this. What, in my view, makes digital communication different is cheap and fast production leading to repair and alignment. It is not face-to-face communication with transient signals. Gandolfi & Pickering's commentary make this point much better than I did. Digital communication can be synchronic, and symbols can be exchanged at a fast rate, close to the pace of spoken conversation, because they require little effort to produce. Thanks to this, ideographic symbols can play the same role as the signals used for repair and feedback in face-to-face conversation.

R.7.2. How standardized are the meanings of emojis?

Commentators (Clark; Feldman; Gandolfi & Pickering; Veit & Browning) all appear to agree that digital communication is a favorable environment for solving the standardization problem. The question then becomes: Has this standardization happened yet, and what kind of symbols would it apply to? The target article took a cautious stance on the matter – too cautious for several commentators. I argued that emojis, gifs, and other digital pictographs may acquire increasingly precise and standardized meanings; but this stage may not have been reached yet, possibly because digital communication is not yet fast enough compared to the pace of speech, or because we are only seeing the beginning of its evolution. Applying this view to emojis, I made two points. First, emojis are not yet standardized enough to function as a self-sufficient ideography; second, some emojis may reach a sufficient level of standardization in the future, starting with emojis that encode paraverbal cues.

Regarding the first point, two commentaries (Feldman; Veit & Browning) argue that I underestimate the level of standardization that emojis have already achieved (a point also suggested by Gandolfi & Pickering's commentary). Veit & Browning cite the successful standardization of available emojis across platforms in support of their view; I would reply that the standardization of emoji keyboards should not be confounded with the degree to which their meanings are standardized between users. Feldman criticizes my focus on face-expression emojis, which she sees as much less standardized than other emojis. I chose to focus on face-expression emojis because, as Feldman acknowledges, they are by far the most commonly used emojis (see, e.g., Daniel, Reference Daniel2021). Their poor standardization tells us something important about emoji users' capacities to converge on a shared meaning when that meaning is not immediately given through iconicity. In the studies cited by Feldman (Barach, Feldman, & Sheridan, Reference Barach, Feldman and Sheridan2021; Częstochowska et al., Reference Częstochowska, Gligorić, Peyrard, Mentha, Bień, Grütter and West2022), the nonface emojis that are highly standardized stand for objects, activities, food, in a straightforwardly iconic fashion, so that associating with “beer” or with “key” requires little in the way of a shared code. There are, of course, nonface emojis that have acquired a standardized meaning, often quite remote from their figurative referent: Feldman mentions the or emojis. But how standardized are these figurative meanings? Quantitative data would be needed to estimate this – an intriguing topic for future studies.

Gandolfi & Pickering broadly agree that emojis are poorly standardized, but they make an exception for the ones used in backchannel interactions such as repair or other kinds of feedback (e.g., , ), especially in SMS interactions, which allow for quick exchanges. The point is well taken. I see their commentary as a more specific and more accurate version of the point I attempted to make in section 6.4 in the target article – namely, that digital technologies should foster the standardization of emojis if (1) they are used as paraverbal cues and (2) the interactions are rapid and synchronous, leaving a lot of room for repair and alignment. Gandolfi & Pickering's views also provide an interesting contrast with Feldman, who explicitly ignores single emojis that stand on their own, and focuses her commentary on emojis as embedded in sentences or messages. Feldman makes a point that the target article acknowledges, and which does not contradict my theory: Emojis do not require much standardization as long as they are surrounded by linguistic information that helps readers narrow down an emoji's meaning. Gandolfi & Pickering make the same point: Emojis are often used as complements to writing rather than replacements for it, playing the same part that paraverbal cues (cospeech gestures, nods, and facial expressions) play for speech; this limits their potential to evolve into a complete ideography.

In summary, there is no unanimity on what level of standardization emojis may have reached, but whatever we take it to be, it is not yet sufficient to allow them to function as a self-sufficient ideography. Still, the evolution of emojis highlights the new opportunities for standardization that digital communication opens up.

R.7.3. What would a generalist ideography look like?

If the target article is right and ideographies are both learnable, and capable of becoming standardized in the future, thanks to digital communication, what would such a future ideography look like? The commentaries lay down prerequisites and highlight important challenges that this ideography would need to face. Historically, inventors of ideography (like inventors of universal languages in general) tend to fall into one of two camps. There are those who think their system should improve upon natural language, by being closer to the true structure of concepts. Frege's ideography is a successful example of ideography in this sense (even though it is highly specialized) (Frege, Reference Frege1883). Then there are those who think they would be lucky enough if they could produce a tool for communication that simply works. Cheng and Moldoveanu keep the first tradition alive: A good ideography should be a language of ideas that carves concepts at their joints; it should avoid ambiguity; it should express only intrinsic properties of things, not contingent ones. Most other commentaries have more modest ambitions: When they consider the prospects for a functioning graphic code, they simply ask whether it will be good enough for communication. Mazur & Plontke go further, denouncing the quest for a language of ideas devoid of ambiguity as a positivistic myth. Chrisomalis agrees: It is doubtful whether ideography in that sense was ever possible or even needed. These points are well taken.Footnote ⁶ Another thing the target article did not consider was the use of ideography as a universal language.Footnote ⁷ As Adiego & Valério rightly note, we cannot expect a generalist ideography to escape the laws of language evolution: Languages change, diverge, fragment, to form mutually incomprehensible dialects. I wrote that a generalist ideography would cut across language barriers, allowing its users to communicate even when they have no spoken language in common; but I never said that it could stand as a universal language or, as Charles Bliss put it, “overcome Babel.”

Even with these relatively modest ambitions, a generalist ideography would still have many challenges to overcome. Compositionality and the capacity to express abstract ideas are two things that natural languages excel in, but graphic codes may struggle with – according to the commentaries. Cheng sees abstract and intangible concepts as the greatest hurdle, whereas Chrisomalis points at the difficulty of using graphic codes componentially – fusing several symbols to communicate one idea. In line with this, Overmann sees the capacity to combine and recombine elements (be they phonetic units or morphemes) as something that spoken or written langue are uniquely good at – a point echoed by Cohn & Schilperoord and Arsiwalla. The specialization hypothesis implies that ideographies can and do express abstract concepts as well as spoken languages do, but only for a restricted range of meanings. It makes the same prediction about combinatoriality: Ideographies are capable of it, as long as the number of basic symbols to be recombined is not too high and their meanings are narrow enough. Both predictions, I think, are confirmed by existing ideographies. Abstraction is no issue for formal logic or mathematical notations, whereas combinatoriality at multiple levels is an obvious feature of many graphic codes, from musical notations to cattle branding (Youngblood, Miton, & Morin, Reference Youngblood, Miton and Morin2023). There are rules for the combination of meaningless elements (duality of patterning) in heraldry (Morin & Miton, Reference Morin and Miton2018), and rules to combine meaningful elements into meaningful compounds in musical or mathematical notations, road signs, and so on. The degree of compositionality at play in many graphic codes is arguably equal or superior to that of natural language: The meaning of compound expressions depends unambiguously on the meaning of the parts and can be derived from systematic rules. This is always true, by construction, for logical notations. But we may well ask, following Chrisomalis, whether this property can scale up for a generalist ideography. If the standardization hypothesis is on the right track, there should be no special difficulty in designing a generalist graphic code capable of high levels of abstraction and obeying compositional combination rules. The problem would lie in getting most users to align on a shared standard, one that cannot be artificially decreed but has to emerge from the back-and-forth of communicative interactions.