Goldin-Meadow & Brentari's (G-M&B's) cogent and timely article reviews how the study of gesture and the study of sign language proceed synergistically, and inform our understanding of the nature of language. We agree wholeheartedly that the classical division of language by the physical channel of production, that is, “speech” versus “manual gesture,” is not the best means of assigning representational format. As their examples illustrate, speech can take on properties of gesture, gesture can take on properties of sign language, and sign languages show aspects of both categorical and gradient language in a single manual-visual channel. Kendon (Reference Kendon2014) resolved this issue by putting speech and gesture together into one superordinate category of representational format. G-M&B propose that we should retain the divide, but that it should not be drawn strictly according to the channel of production, that is, of spoken versus manual communicative behavior. We agree with this point and suggest that G-M&B have not gone far enough in reconceptualizing the boundaries. Fortunately, once we cast off the dichotomy based on channel, we are no longer restricted to a bipartite system, and we can consider multiple factors to determine the subsystems of language.

In forcing representational format into two types, G-M&B conflate multiple dimensions into a single contrast. On one side are forms that are categorical, conventionalized, and listable, while on the other side are forms that are gradient, imagistic, and spontaneously generated. This division results in the somewhat awkward assignment of signed spatial-relation expressions (which have internal structure) with holistic and imagistic expressions – and it leaves nowhere for emblem gestures, which are inconveniently highly categorical and highly conventionalized, yet have no internal structure, and cannot combine with other elements.

We propose that there is more than a single cut across the space of multimodal expressions. Figure 1 illustrates the divisions resulting from separation according to what we believe are the two most relevant dimensions: (1) whether a form is categorical, or gradient, and (2) whether it participates in a combinatorial system or is holistic and noncombining. The first dimension characterizes not the physical makeup of an expression but rather how the expression maps to its meaning or referent, whether in a discrete or analog way. The second dimension characterizes whether an expression includes any elements that combine grammatically in the construction of words and phrases. The examples in the four quadrants of Figure 1 demonstrate that aspects of vocal and manual communication, in both signed and spoken language, appear in every permutation of these two dimensions.

Figure 1.

A schematic providing examples of unimodal and multimodal phenomena in sign and spoken language according to the dimensions of Systematic-Noncombining and Categorical-Gradient. Importantly, each quadrant contains examples attested in both sign and spoken language. Some quadrants also present examples of phenomena attested in spoken language only or in sign language only.

One might be tempted to define “gesture” as everything except the upper left quadrant, or as only the lower right quadrant. One end of each of these dimensions feels like the “gestural” end of the scale. Some forms feel gestural because they are gradient; others, because they don't combine. These are not the only dimensions that exhibit this contrast between prototypical nongesture and gesture, and different researchers have emphasized different factors when making their defining cut. Relevant dimensions can also include whether a form is conventionalized or produced ad hoc, whether it iconically references the real world and models of real world space, whether it is informed by semantic presuppositions, whether it is listable or infinite, and whether it is highly imagistic (e.g., de Vos Reference de Vos2015; Duncan Reference Duncan2005; Emmorey & Herzig Reference Emmorey, Herzig and Emmorey2003; Liddell Reference Liddell2003; Lillo-Martin & Meier Reference Lillo-Martin and Meier2011; Okrent Reference Okrent, Meier, Quinto-Pozos and Cormier2002; Schlenker et al. Reference Schlenker, Lamberton and Santoro2013; Wilcox & Xavier Reference Wilcox and Xavier2013). Yet these dimensions can vary independently; a form can be both highly imagistic and highly conventionalized. Gesture is not a single phenomenon.

Utterances can be the product of multiple productive systems, bundled together into a composite stream to communicate a complex meaning (Enfield Reference Enfield2009). Defining gesture requires more than selecting the right thread from the composite. Inviting both extremes of multiple dimensions into linguistic analyses by characterizing the whole as “language plus gesture” does not resolve the problem. Much of the core grammar of sign language would inevitably be slotted into the “gesture” part, along with material very different in nature.

G-M&B's proposed next step of developing new technologies to measure utterances more finely will not clarify the distinction. As even they mention, the same form can be generated by either a gradient or categorical system (sect. 4.2, para. 3). Take their example of wiggling fingers moving along a path to indicate a person running (sect. 5, para. 3). Nothing in the form itself determines whether it has internal combinatorial structure; what matters is whether pieces of that form map holistically and veridically to the world (where legs, internal movement, and path of movement all occur together) or according to a system used to generate this and other utterances, using recombinable handshape, manner, and path elements. Figure 2 illustrates that the same manual utterance can be iconic and holistic in one linguistic context, and morphological and combinatorial in another.

Figure 2.

Two descriptions of the same rolling-down motion event: a holistic co-speech gesture produced by a Nicaraguan Spanish speaker (left) and a combinatorial sign produced by a deaf signer of Spanish Sign Language (Lengua de Signos Española, LSE). We can characterize these forms as differing in format only by considering how they relate to other forms within their respective systems, and whether they have subcomponents that map to meaning.

We agree with the importance of creating a unified account of language that includes all aspects of its production, whether manual or vocal. We suggest that the study of spoken and signed languages at moments of change – particularly language acquisition, emergence, and change – offer a better view of the sources of language structure. The dimensions of discreteness and combinatoriality are of interest not because they help define gesture, but because they represent an abstraction and reconfiguration of information from how it is organized in the world. Accordingly, these dimensions are sites of qualitative shifts as language is created and changed. Forms appearing in new contexts constitute evidence of corresponding changes along these dimensions. For example, at some point learners transformed the onomatopoeic verbal gesture “mooo,” allowing it to participate in combinatorial expressions like “The cow moo-ed all day.” The path that elements follow as they become linguistic reveals human language-making capacity at individual, community, and multigenerational timescales. The world offers a continuous and image-rich stream of experience; representations that derive their structure directly from the world will be correspondingly gradient and holistic. Our own work demonstrates that gradient, context-dependent, and imagistic forms are reshaped by learners into discrete, recombinable elements (Coppola & Senghas Reference Coppola, Senghas and Brentari2010; Senghas et al. Reference Senghas, Kita and Özyürek2004). Investigating transformation over time along these dimensions will reveal how the nature of language reflects human minds, rather than the world to which language refers.