Goldin-Meadow & Brentari (G-M&B) argue that we can learn about cognition and language by examining interactions between sign or speech and gesture and highlight the changing view of these representational formats as independent and distinct. Conditions under which gesture takes on properties of sign or speech are revealing about the integral coordination of gesture and language. In this commentary, we focus on the written modality and incorporate the contrast between the mimetic, imagistic representational format characteristically applied to gesture and the discrete, categorical format characteristically applied to signed or spoken language. Our goal is to designate ways in which emoticons in textual communication invite comparisons with gesture. We propose that recent evidence on the comprehension and production of emoticons suggests that it parallels the functions of gesture. That observation contributes to a broader conceptualization of gesture, one that anticipates a nuanced interdependence with language whether signed, spoken, or texted.

Why might emoticons play a gestural role in written language? Recent studies of bimodal bilinguals, individuals who speak or read a written language in addition to a signed language, provide compelling support for an abstract level at which information across modalities is integrated during both language comprehension and production (e.g., Emmorey et al. Reference Emmorey, Giezen and Gollan2015; Morford et al. Reference Morford, Wilkinson, Villwock, Piñar and Kroll2011). Research on spoken bilingualism demonstrates that bilinguals activate the two languages in parallel even when only a single language is required (e.g., Kroll et al. Reference Kroll, Dussias, Bice, Perrotti, Liberman and Partee2015). The surprising finding is that bimodal bilinguals reveal many of the same cross-language interactions that characterize spoken bilingualism, suggesting that the principle of disparate forms that converge at an abstract level not only governs interactions within written or spoken language alone, but also applies to bimodal communication.

Emoticons can obviously function simply as “emblems” with stereotyped and context invariant meanings, but this should not distract us from the evidence that emoticons can function as “illustrators” and be integral to the intentional act: social and linguistic coordination. Vandergriff (Reference Vandergriff2014) and Yus (Reference Yus2011; Reference Yus2014) described the way that emoticons in text extend beyond their iconic characteristics so that a smiley face can indicate a request for acceptance or agreement and not simply that the writer is happy. Likewise, patterns of emoticon use can be revealing about the producer and their social standing, similar to dialectal variation in speech (Schnoebelen Reference Schnoebelen2012).

In a recent study, Aragon et al. (Reference Aragon, Chen, Kroll, Feldman, Kennedy, Agarwal and Yang2014) analyzed the presence of emoticons in a spontaneous written corpus of an online work environment derived from the texting behavior of an international group of French and American scientists who communicated in English, the first language (L1) of the Americans and the second language (L2) of the French. Over time, the L2 speakers altered their repertoire of nonverbal behavior (viz., emoticon vocabulary size) depending on the ratio of L1 and L2 speakers in the chat room audience. Like spoken communication, where there is evidence for a progression from the use of iconic signs for grounded shared meanings to a form that is more abstract and limited to a particular context and set of interlocutors (Galantucci & Garrod Reference Galantucci and Garrod2011; Garrod et al. Reference Garrod, Fay, Lee, Oberlander and MacLeod2007), the data on texting revealed alignment between interlocutors. The mutual influence of interlocutors has been variously termed coordination (Clark Reference Clark1996), alignment (Pickering & Garrod Reference Pickering and Garrod2004), or accommodation (Shepard et al. Reference Shepard, Giles, Le Poire, Robinson and Giles2001). This pattern of increasing similarity has been documented at the levels of the syntax (Branigan et al. Reference Branigan, Pickering and Cleland2000), the acoustics of speech (Giles Reference Giles1973), and the lexicon (Niederhoffer & Pennebaker Reference Niederhoffer and Pennebaker2002; Wilkes-Gibbs Clark Reference Wilkes-Gibbs and Clark1992) as well as gesture (Mol et al. Reference Mol, Krahmer, Maes and Swerts2012). Convergence is often influenced by social factors, including the relative dominance or perceived prestige of each speaker (Gregory et al. Reference Gregory, Dagan and Webster1997; Gregory & Webster Reference Gregory and Webster1996). The study by Aragon et al. demonstrates that nonverbal emoticon vocabulary, like many other aspects of language and gesture, is subject to social and linguistic interlocutor alignment.

In a smaller-scale experimental study, Feldman et al. (Reference Feldman, Cho, Aragon, Kroll, Noelle, Dale, Warlaumont, Yoshimi, Matlock, Jennings and Maglio2015) examined the mutual influences of words and emoticons. They discovered an asymmetric interaction between emoticons and text in visual word recognition. Whereas emoticon valence had little influence on the time to recognize a word, word valence (BALMY, DUSTY) influenced decisions as to whether an emoticon reflected an emotion that was positive, negative, or neutral. Positive emoticon targets were facilitated by word valence, while negative emoticon targets were marginally slowed. Experimental demonstrations of how text influences the interpretation of emoticons are more robust than demonstrations of the effect of emoticons on text. A striking aspect of the Feldman et al. results was that a manipulation of timing had no effect on the observed asymmetry, suggesting that the pattern was not contingent on the time required to comprehend the emoticons.

Although research on processing emoticons in text is at an early stage of development, the preliminary findings are promising in suggesting that emoticons interact with text in ways that resemble other reported interactions of nonverbal gesture with language. There are many questions that have not yet been investigated, including those related to relative contributions such as how proficiency in emoticon production is likely to modulate the emoticon-text interactions in online communication. Like gesture in sign and spoken language, emoticon use in text may come to reveal the subtle interplay between linguistic and social aspects of communication. But critically, like research on bimodal bilingualism, these results show that gestural features of language are not bound by input modality. The interactions between emoticons with text, and between gesture with sign and speech, suggest that gesture, in its different forms, is indeed an integral component of language.