R2.1. Distinguishing gesture from sign

The first feature that most of the commentaries focused on, and that we continue to think is essential in identifying a gesture, is gradience (as opposed to discreteness). But gradience is not an easy heuristic to implement simply because, as Berent & Dupuis note, although analog nonlinguistic computations (i.e., gestures) give rise to gradience, gradience can also result from the realization of discrete grammatical categories (see also Occhino & Wilcox). So gradience per se is not a surefire marker of a gesture. Although gradience is not a sufficient marker of gesture, we suggest that it is a necessary marker. Gestures are analog and not discrete.

Second, gestures are imagistic: The form of a gesture is transparently related to its meaning. But here again the heuristic is not straightforwardly applied, particularly in sign languages where many (grammatical) forms are transparently related to their meanings. Iconicity is found in both signed and spoken grammatical categories and thus cannot, on its own, mark a form as gesture. We suggest that, here again, being imagistic is necessary but not sufficient to signal that a form is gestural.

A third feature, one that we did not stress in our target article, is that gesture typically falls outside of awareness for both producers and receivers. Gestures (or at least the kinds of gestures that we want to focus on; see sect. 3) are produced along with linguistic communicative acts and thus take on the intentionality of those acts. But they are not themselves the focus of attention for either the producer or the receiver, which may be what gives gesture its cognitive traction. We suggest that gesture accesses implicit, nondeclarative knowledge in learning that can ultimately be integrated with declarative knowledge.

Fourth, following from the third feature, gesture (again, the kind we want to focus on; see sect. 3) is parasitic on the language it accompanies. It is meaningful, but the meaning is framed by the speech or sign with which it occurs. Language can stand on its own without gesture (for the most part, although not always, which is why these features are heuristics rather than defining properties), but gesture cannot stand alone without language. As an important corollary, when spoken language is removed and the manual modality is left to take on the full burden of communication (as in silent gesture or homesign), we do not consider the resulting system to be gesture. As we note in the target article, silent gesture is not gesture in the way we use the term here; it is not co-language gesture and, at least in terms of discreteness, it is closer to language than it is to gesture (see Goldin-Meadow et al. Reference Goldin-Meadow, McNeill and Singleton1996; Singleton et al. Reference Singleton, Goldin-Meadow, McNeill, Emmorey and Reilly1995).

Finally, although gestures do combine with one another (as Tversky notes), the combinations are characterized by a flat structure rather than a hierarchical structure. The combinatorial system involving different levels of description (i.e., phonological, morphological, syntactic, semantic, pragmatic) that characterize language is not needed to characterize co-speech or co-sign gesture.

R2.2. What variability signals in gesture and sign

Two commentaries raise the issue of variation as it pertains to distinguishing gesture from sign (Berent & Dupuis, Emmorey). There are two types of variability that need to be considered, and both types are found within an individual (i.e., the individual varies within herself) and across individuals (i.e., individuals within a group vary). In the first type, there is little consistency in the target form that is used for a particular meaning. Duncan (Reference Duncan2005) points to this type of variability in Taiwanese Sign Language. The signers she studied all used the same handshape to refer to a cat, but the ways in which the handshape was modified to indicate that the cat was crawling up a tight drainpipe varied across signers, and in just the same ways that hearing speakers vary their manual gestures when describing the same event. Sandler (Reference Sandler2009) uses the same type of variability across signers to distinguish gestural from linguistic facial expressions in Israeli Sign Language (ISL). Different signers produced different iconic mouth gestures to describe the cat climbing up the inside of the narrow drainpipe. One signer indicated the bulge of the cat seen from outside with puffed cheeks; another indicated the cat's tight fit with narrowed eyes and a pursed mouth gesture. Importantly, the facial expressions used in grammatical constructions in ISL (e.g., wh-questions and conditionals) do not display this type of variation.

It is worth pointing out that this type of variability is, once again, not a surefire marker of gesture. Variability has been found to characterize manual movements that are not used along with a codified linguistic system (and thus are not co-language gesture), but are instead used in place of such a system. For example, Goldin-Meadow et al. (2015) examined variability in the handshapes used to refer to a particular object in adult homesigners in Nicaragua whose hearing losses prevented them from acquiring Spanish and who had not been exposed to Nicaraguan Sign Language (NSL). The homesigners displayed more variability in the handshapes they used for an object than did NSL signers, even signers who were members of the earliest cohort of the newly emerging language. This result is not surprising given that each homesigner was inventing his or her own homesign system and was not part of a linguistic community. More surprising is the fact that the homesigners also displayed more variability within themselves; that is, an individual homesigner was less likely than was an individual NSL signer to use the same handshape every time she referred to an object. By not repeatedly tapping into a single form (an invariant form-meaning mapping suggestive of langue) homesign lacks one central feature of a linguistic system. Importantly, however, homesign contains the essential feature of discreteness, as well as many other linguistic properties (e.g., morphological marking on the verb, Goldin-Meadow 2003a; Goldin-Meadow et al. Reference Goldin-Meadow, Namboodiripad, Mylander, Özyürek and Sancar2015b), and thus is not gestural in the way we use the term.

Having chosen a target form, there may be variability, both within and across speakers, in how that form is produced, and this is the second type of variability that may help us distinguish gesture from sign. This type of variability can be based on many factors (speaker, gender, age, speech register, etc.). For example, in the production of English /r/, the properties of being “bunched” or “retroflexed” vary both by speaker and by the surrounding segmental context (Archangeli et al. Reference Archangeli, Baker, Mielke, Ridouane and Clements2011). This type of variability is also found in fingerspelled letters of the ASL manual alphabet. The letters N, R, U, and V all have selected index and middle fingers, and closed (i.e., flexed) unselected ring and pinky fingers; the degree of flexion of the ring and pinky fingers varies by signer and by the surrounding segmental context (Keane Reference Keane2014). In terms of distinguishing between gesture and language, one interesting question is whether there is more of this type of (phonetic) variation in the handshapes produced in gesture than in the handshapes produced in sign language. To address this question, we would, of course, need to control for the number of gestures in the sequence, which can be difficult because co-speech gestures often come only one to a clause (McNeill Reference McNeill1992).

R2.3. How to treat properties that gesture and sign have in common

In our target article, we highlighted differences between language and gesture. However, we recognize that there are also commonalities between the two. Barca & Pezzulo point to an embodied framework for language that unites gesture and sign. We did not focus on embodiment in our target article simply because it does not highlight the difference between gesture and sign, as it applies to both. The fact that gesture (like sign) is an act of the body may be critical in determining some of its effects. But it is important to recognize that the impact that gesture has on thinking does not stem entirely from its being an act of the body. For example, Novack et al. (Reference Novack, Congdon, Hemani-Lopez and Goldin-Meadow2014) explored the impact of instruction in which learners acted on a math problem during a lesson and compared it to instruction in which learners gestured about the problem. They found that children who acted on the problems learned how to solve the problem just as well as children who gestured about the problem. But gesturing was much more effective than acting when it came time to generalize what they had learned to new problem types. Gesture and action both involve the body; the different impact they have on learning cannot be explained by embodiment. Similarly, gesture and sign both involve the body. They may, or may not, have the same effects on cognition, an empirical question that takes us beyond the fact that the body is representing information and forces us to consider the format by which that information is represented.

What is more pressing in terms of commonalities, however, is that we need to ask whether a form that elicits the same response in signers and speakers is necessarily a gesture. This question arises with respect to the study of telicity mentioned in Strickland et al. Strickland et al. (Reference Strickland, Geraci, Chemla, Schlenker, Kelepir and Pfau2015) conducted a perception study on American English speakers without sign language experience, using signs from Italian Sign Language (LIS). They found that speakers reliably paired LIS signs for events with and without an endpoint with videos of events that did and did not have endpoints (telic vs. atelic events); the telic/atelic distinction is found in many sign languages (Wilbur Reference Wilbur and Brentari2010).

On the basis of these findings, we might be tempted to argue that these forms are gestural not only for non-signers, but also for signers. But the fact that a form is transparently related to its meaning does not make it gestural. Iconicity runs throughout linguistic systems, particularly sign systems, but that iconicity is constrained. For example, the temporal affixes for telic (punctate movement, with an endpoint) and atelic (continuous movement, with no endpoint) cannot be applied to all verbs, and the particular set of signs to which the affixes apply varies from sign language to sign language. Knowledge of the lexical semantics of the verbs and their syntactic frames is not explored in Strickland et al.’s (2015) fixed, binary-choice task. As a result, the findings show only that both groups use iconicity to interpret the meaning of a form, not whether the form has other grammatical or lexical constraints. We suggest that these restrictions will be present in signers but not non-signers. Strickland et al.’s lovely results make it clear that signs for telicity have iconic roots, but the findings should not be used to argue that this aspect of sign is gestural.

To push the point even further, imagine two handshapes – a straight index finger versus a straight index+ middle finger – presented with the picture of a pen. Hypothetically, if non-signers and signers both preferentially associate the single index finger handshape with the pen, we would not want to conclude that non-signers have a phonological contrast for the number of fingers, nor would we want to conclude that signers lack a phonological system for handshape. Participants would simply be using iconicity to make the best of a fixed-choice scenario, with no information provided about the way the form fits into a phonological system. Sign languages work with iconicity most of the time, not against it, and so it is not surprising that there are some situations where signers and non-signers behave in precisely the same way.

R2.4. When gesture and sign differ

There are many ways in which gesture and sign are similar as semiotic systems, a point made in the target article and emphasized by Kendon. But we argue that the differences are too fundamental to ignore. Both contribute different kinds of information that are largely complementary (when gesture adds or reinforces information found in sign or speech), but sometimes gesture conveys information that is different from the information conveyed in sign or speech (a so-called mismatch), and, when it does, it offers a window onto cognition. Pouw et al. suggest that we can't have it both ways: that gesture can't mismatch speech and, at the same time, be part of an integrated system. But mismatches in morphology and syntax (Sadock Reference Sadock1991) and in phonology and morphology (see Fig. 3 in the target article for mismatches in number of syllables, a phonological unit, and number of morphemes, a morphological unit) happen relatively frequently, yet all of these components function as a unified system. Along the same lines, the allosemantic enrichment examples that Schlenker describes fit the definition of a mismatch. In allosemantic enrichment, gesture provides information that is not conveyed within the linguistic unit that it accompanies, the classic definition of gesture-speech mismatch: for example, producing a hit gesture while talking (or signing) about punishment without explicitly mentioning how the punishment was administered. In autosemantic enrichment, by contrast, gesture provides information that is expressed within the linguistic unit: for example, producing the hit gesture while talking (or signing) about hitting. As Schlenker makes clear, these examples are not anomalies but rather work within the constraints of the semantic system and can be helpful in revealing the properties of that system.

There are two points that we need to stress about gesture-speech mismatches. First, the information conveyed in gesture is different from, but not contradictory to, the information conveyed in speech, as Pouw et al. suggest. In fact, in all instances of true mismatches, the information conveyed in the two modalities has the potential to be integrated; it just hasn't been integrated by the speaker or signer. Importantly, the brain treats speech and gesture as if they belong together when the two convey different but potentially integrable information, but not when the two modalities convey contradictory information. For example, in a study of the brain's electrophysiological response (ERPs, event-related potentials) to different types of stimuli, Kelly et al. (Reference Kelly, Kravitz and Hopkins2004) found that gestures conveying information that is truly incongruent with the information conveyed in speech (gesturing short while saying “tall”) produce a large negativity at 400 ms (the N400 is known to be sensitive to incongruent semantic information; Kutas & Hillyard Reference Kutas and Hillyard1984).

Importantly, gestures conveying information that is different from, but integrable with, information conveyed in speech (gesturing thin while saying “tall” to describe a tall, thin container, a true gesture-speech mismatch) are processed no differently at this stage from gestures that convey the same information as speech (gesturing tall while saying “tall”; Kelly et al. Reference Kelly, Kravitz and Hopkins2004). Neither one produces a large negativity at 400 ms; that is, neither one is recognized as a semantic anomaly. It is important to note, however, that at early stages of sensory/phonological processing (P1-N1 and P2), speech accompanied by gestures conveying different but potentially integrable information (e.g., gesturing thin while saying “tall”) is processed differently from speech accompanied by gestures conveying overlapping information (gesturing tall while saying “tall”). Thus, potentially integrable differences between gesture and speech are noted at early stages of processing but not at later, higher-level stages.

The second point about mismatches is that, although detecting gesture-speech mismatch in an experimental situation requires elaborate controls, as Koschmann points out, mismatches are noticed and, in fact, acted on by ordinary communication partners. For example, Goldin-Meadow and Singer (Reference Goldin-Meadow and Singer2003) videotaped teachers using whatever techniques they wanted to instruct children in mathematical equivalence. The one-on-one tutorials were videotaped and the children later classified according to whether or not each child produced gesture-speech mismatches; the teacher's instruction to each child was then analyzed. The teachers were found to instruct children who produced mismatches differently from children who did not (they taught mismatchers more different types of problem-solving strategies in speech and in gesture). Teachers seem to know when learners produce gesture-speech mismatches (albeit not consciously; see Alibali et al. Reference Alibali, Flevares and Goldin-Meadow1997), and they alter their instruction accordingly. Along the same lines, mothers interacting with their toddlers at home, responded to the child's gesture+speech utterance with longer utterances of their own when gesture and speech conveyed different information (and thus mismatched: e.g., “mommy”+point at hat, to indicate that the hat belonged to mom) than when they conveyed overlapping information (e.g., “hat”+point at hat; Goldin-Meadow et al. Reference Goldin-Meadow, Goodrich, Sauer and Iverson2007a). The procedures developed to ensure methodological rigor in identifying gesture-speech mismatch in the lab do not render the concept useless in the real world, as both teachers and parents seem to know (and act on) a mismatch when they see one.

But Pouw et al. are not questioning whether mismatch can be identified; they are asking about the mechanism by which gesture introduces new ideas into a learner's repertoire. Cook et al. (Reference Cook, Duff and Goldin-Meadowunder review) have hypothesized that gesture's power to change thought stems from its ability to introduce nondeclarative knowledge into a learner's repertoire, knowledge that is often different from, but produced at the same time as, declarative knowledge expressed in speech. Co-speech gesture and, we would argue, co-sign gesture resemble nondeclarative knowledge in that both are implicit and (as Pouw et al. suggest) based in action. Co-language gesture is thus ideal for integrating declarative and nondeclarative knowledge not only because it is itself a vehicle for conveying nondeclarative knowledge, but also because it is seamlessly integrated with speech or sign, the gold-standard vehicle for conveying declarative knowledge.

R2.5. Multiple levels of representation in sign, but not gesture

The idea that linguistic forms are subjected to multiple levels of representation during processing has been addressed directly for spoken (Levelt Reference Levelt1989) and signed (Pfau et al. Reference Pfau, Steinbach and Woll2012) languages. If a form is linguistic, it will tap into multiple levels of representation. Coppola & Senghas suggest that, in addition to categorizing a form as gradient or discrete (categorical in their terms), it should be identified by whether or not it combines with other forms (which they refer to as “systematic” vs. “non-combining”). Being systematic may capture some (but not all) aspects of the fact that some forms are part of a linguistic system; however, we are not convinced that the four cells fully capture distinctions between gesture and sign. Combination per se isn't the issue; the combinations need to reflect a system that engages phonological, morphological, syntactic, semantic, and pragmatic levels. Hearing speakers frequently string pointing gestures together, and the particular combinations of gestures they produce are meaningful (e.g., strings of pointing gestures that convey different problem-solving strategies for solving mathematical equivalence problems, Alibali & Goldin-Meadow Reference Alibali and Goldin-Meadow1993). But these gestures are not part of a hierarchical system of multiple levels.

Supalla (Reference Supalla1982), Benedicto and Brentari (Reference Benedicto and Brentari2004), Brentari et al. (2012), and Brentari et al. (in press) illustrate the notion of multiple levels at work in sign language phonology, morphology, and syntax for two types of handshapes in classifier constructions. These constructions are polymorphemic, iconic, motion, and location predicates that may lie at the boundary between language and gesture. The two types are (1) handling classifiers, expressed via handling handshapes, which have “hand-as-hand” iconicity because they represent how hands handle objects; and (2) entity/semantic (object) classifiers, expressed via object handshapes, which have “hand-as object” iconicity because they represent properties of the object on the hand.

These constructions are, first of all, motor schemas, the first level of representation. The second level at which the constructions can be analyzed is conceptual. The iconic relation between form and meaning in the two types of handshape forms is clear. However, there is a difference in how these forms are used by signers and non-signers. Non-signers, when asked to gesture without speech (i.e., to produce silent gesture), have a strong bias toward one of the two types of iconicity: hand-as-hand iconicity. Padden et al. (Reference Padden, Meir, Hwang, Lepic, Seegers and Sampson2013) have shown this effect in a naming task, where they find that silent gesturers display a bias toward using handling handshapes (as opposed to object handshapes) to represent instrument-like objects. Non-signers understand the idea that the hand can represent the hand, but when the hand is called upon to represent something else (in this case, an object), they are more likely than signers to use a neutral handshape not displaying any properties of the object, or to trace the shape of the object with an index finger. Signers, in contrast, will represent properties of the object in the handshape itself.

The third level is morphological. Both object and handling forms combine with a wide range of movements in classifier predicates to achieve agentive (handling handshapes) and non-agentive (object handshapes) meanings, so the handshapes are meaningful and productive. Indeed, even homesigners, who are inventing their gestures without benefit of a linguistic model, use handling and object handshapes systematically for agentive and non-agentive meanings (e.g., Rissman & Goldin-Meadow, Reference Rissman and Goldin-Meadowin press); importantly, the same structures are typically not found in silent gesture (Brentari et al. Reference Brentari, Di Renzo, Keane and Volterra2015a), nor in the co-speech gestures that the homesigners’ hearing parents use with them (Goldin-Meadow et al. Reference Goldin-Meadow, Mylander and Butcher1995; Reference Goldin-Meadow, Goodrich, Sauer and Iverson2007a).

The fourth level is phonological. There are differences between signers and silent gesturers with respect to the internal structure of handshape (Brentari et al. Reference Brentari, Coppola, Cho and Senghas2016). Signers balance the complexity of their handshapes between joint complexity and finger complexity, showing a double dissociation between handling handshapes (high joint complexity–low selected finger complexity) and object handshapes (low joint complexity–high finger complexity). Silent gesturers also use more joint complexity in their handling handshapes (almost veridically representing how the hand holds the object) than in their object handshapes. But, unlike signers, they show the same pattern with respect to finger complexity: more finger complexity in handling handshapes than in object handshapes (Brentari et al. 2012). Building this kind of division of labor in a system is the work of a phonological level of representation; signers have it and non-signers do not.

The final level of representation is syntactic. Diagnostic tests have been applied to these forms in several sign languages (ASL, Hong Kong Sign Language, LIS, British Sign Language) to show that handling handshapes are sensitive to both the grammatical subject and object arguments in a sentence, whereas object handshapes are sensitive only to the grammatical object argument. To date, grammaticality judgments of this type have not been obtained from non-signers (nor from homesigners).

The fact that an iconic form is part of a system for signers but not for non-signers suggests that iconic forms may be processed differently in signers and in non-signers. Davidson notes that when bimodal bilinguals (individuals equally fluent in a signed and spoken language) are speaking English, they often produce classifier constructions comparable to those found in sign language alongside speech; in this context, the classifier constructions are serving the function of co-speech gesture. In other words, their status changes from sign to gesture when spoken language shoulders the linguistic burden of communication.

The converse – that is, gesture (albeit in this case, silent gesture) is treated as sign – has been shown in recent work by Newman et al. (Reference Newman, Supalla, Fernandez, Newport and Bavelier2015), who collected manual event descriptions of objects moving in space from signers and from non-signers. The signers produced classifier constructions, which (as noted earlier) are transparently related to their meanings and thus have the potential to be gestural rather than linguistic. The non-signers’ descriptions were produced without speech and thus were silent gestures transparently related to their meanings. These productions were then used as stimuli in a functional magnetic resonance imaging (fMRI) study and shown to signers and non-signers. Once low-level stimulus features were taken into account, signers were found to process the classifier signs within the left-lateralized frontal-temporal network used for spoken languages. But they also processed the silent gestures within the same network, demonstrating an influence of lifelong experience with sign on the perception of silent gesture. In contrast, non-signers processed both the classifier signs and the silent gestures within regions involved in human action perception, not within linguistic areas. In other words, both types of forms were processed linguistically in signers, and non-linguistically in non-signers. This work highlights the fact that the iconicity of a form is not sufficient to make the form gestural. When silent gestures (which are iconic) are processed by signers, they activate language areas; not so when they are processed by non-signers. The interesting question from the point of the framework we have proposed here is – Will signers process co-speech gesture within language areas? We suspect that they will not. Although silent gesture has enough of the discreteness found in sign language to activate language areas in individuals who have had lifelong experience with sign, co-speech gesture does not. We therefore hypothesize that co-speech gesture will activate regions involved in human action perception not only in non-signers, but also in signers.