Goldin-Meadow & Brentari (G-M&B) provide a concise summary of how our thinking about sign and gesture has shifted over the past century. The latest installment in this history involves evidence gathered by multiple researchers of gesture production within sign-based interaction (Duncan Reference Duncan2005; Emmorey 1999; Goldin-Meadow et al. Reference Goldin-Meadow, Shield, Lenzen, Herzog and Padden2012; Liddell & Metzger Reference Liddell and Metzger1998). G-M&B, however, caution in their target article against blurring the “categorical division between gesture and sign,” warning that there are “phenomena that depend on the divide, for example, predicting who is ready to learn a particular task” (sect. 7.2, para. 1). The particular phenomenon at issue is “gesture-speech mismatch” (e.g., Alibali & Goldin-Meadow Reference Alibali and Goldin-Meadow1993; Goldin-Meadow et al. Reference Goldin-Meadow, Shield, Lenzen, Herzog and Padden2012). In this commentary, I will explore why research on gesture-speech mismatch might be challenged by changing views of what constitutes a gesture.
Mismatch is a phenomenon that arises when there is a perceived discrepancy between produced gestures and co-occurring speech. In the example provided by G-M&B, a subject has been asked to provide an explanation for her solution to the equivalence problem, 6+3+4=___+4. The subject reports that she added the three numbers on the left-hand side and produced their sum as a solution, but as she does so she points not only to the three numbers on the left side of the equation, but also to the number 4 on the right. This is taken by the authors as evidence that the child is in a “discordant” (Alibali & Goldin-Meadow Reference Alibali and Goldin-Meadow1993, p. 496) cognitive state, that she is on “the verge of learning” (p. 470). The child, in such circumstances, is thought to entertain “two different hypotheses at the same moment” (p. 477). The hypotheses constitute different “representations” of the problem, some being theorized to be “accessible to gesture but not to speech” (p. 510).
Over the years some extravagant claims have been made regarding gesture-speech mismatch – that learners convey information through gesture of which they may not be aware: that mismatch provides a window into the learner's mind. The detection of mismatch, however, faces a large and difficult challenge. Calbris (Reference Calbris and Copple2011, pp. 23–30) reported that a single gestural form can represent different ideas (polysemy), while a single concept can be evoked using different forms (polysign). Gestural forms, therefore, are inherently “inscrutable” (Quine Reference Quine1968, p. 196), there being no one-to-one correspondence between signifier and signified.
Experimenters seeking evidence of mismatch have attempted to overcome this by cataloguing the gestures commonly seen with regard to a particular task and using coders familiar with those catalogued gestures. The experimental protocol employs a double-coding technique (Alibali & Goldin-Meadow Reference Alibali and Goldin-Meadow1993). Videotapes of the subject producing an explanation at the board are first coded, working solely off the audio track, by someone familiar with the strategies children commonly employ for the problem in question. The recording is then recoded, now with the sound off, by a different coder familiar with the gestures associated with these same strategies. When the codes generated by the two methods are discrepant, the subject's explanation for that problem is classified as exhibiting a mismatch. In more recent work with signing subjects (Goldin-Meadow et al. Reference Goldin-Meadow, Shield, Lenzen, Herzog and Padden2012), the videos are viewed first by an American Sign Language (ASL) coder and again by a coder familiar with the problem-specific gestural “formulations” (Koschmann et al. Reference Koschmann, LeBaron, Goodwin, Zemel and Dunnington2007).
There are several concerns with this way of operationalizing mismatch. The experimental protocol does not really resolve the inscrutability problem, but merely seeks to control it by tightly limiting the range of gestural formulations that might arise. This strategy rests on a number of assumptions: that subjects have finite gestural repertoires, that the ways of understanding (or misunderstanding) the problem are fixed, and so forth. The interview is rigidly structured, and there is no built in “proof procedure” for meaning construction. When interaction is constrained in this way, there is no opportunity to pursue intersubjectivity (Interviewer: “I notice that you have pointed at four numbers, but your sum only includes three. How does that 4 to the right change your answer?”). The errant gesture produced by the subject in the example performs the work of “disclosing the world in sight” (Streeck Reference Streeck2009, pp. 8–9). Gestures that function in this way do so through tight temporal coordination with co-occurring talk (Hindmarsh & Heath Reference Hindmarsh and Heath2000; Streeck Reference Streeck2009). The experimental procedure for producing mismatch, however, is blind to this coordination owing to its method of coding gesture and speech/sign independently.
So, the returns are still out on how the discovery of gesture production by signers will impact further research on learning and gesture. Educational psychology has, from its inception, treated learning as an “occult” (Koschmann Reference Koschmann2002, p. 2) matter. The experimental finding of gesture-speech mismatch is provocative in that it takes up learning as something witnessable. What we need, however, is to find new ways of studying gesture in learning, ways that treat gesture as an interactional matter and that honor what G-M&B identify as “the principle that speech and gesture form an integrated system” (sect. 5, para. 3).
Goldin-Meadow & Brentari (G-M&B) provide a concise summary of how our thinking about sign and gesture has shifted over the past century. The latest installment in this history involves evidence gathered by multiple researchers of gesture production within sign-based interaction (Duncan Reference Duncan2005; Emmorey 1999; Goldin-Meadow et al. Reference Goldin-Meadow, Shield, Lenzen, Herzog and Padden2012; Liddell & Metzger Reference Liddell and Metzger1998). G-M&B, however, caution in their target article against blurring the “categorical division between gesture and sign,” warning that there are “phenomena that depend on the divide, for example, predicting who is ready to learn a particular task” (sect. 7.2, para. 1). The particular phenomenon at issue is “gesture-speech mismatch” (e.g., Alibali & Goldin-Meadow Reference Alibali and Goldin-Meadow1993; Goldin-Meadow et al. Reference Goldin-Meadow, Shield, Lenzen, Herzog and Padden2012). In this commentary, I will explore why research on gesture-speech mismatch might be challenged by changing views of what constitutes a gesture.
Mismatch is a phenomenon that arises when there is a perceived discrepancy between produced gestures and co-occurring speech. In the example provided by G-M&B, a subject has been asked to provide an explanation for her solution to the equivalence problem, 6+3+4=___+4. The subject reports that she added the three numbers on the left-hand side and produced their sum as a solution, but as she does so she points not only to the three numbers on the left side of the equation, but also to the number 4 on the right. This is taken by the authors as evidence that the child is in a “discordant” (Alibali & Goldin-Meadow Reference Alibali and Goldin-Meadow1993, p. 496) cognitive state, that she is on “the verge of learning” (p. 470). The child, in such circumstances, is thought to entertain “two different hypotheses at the same moment” (p. 477). The hypotheses constitute different “representations” of the problem, some being theorized to be “accessible to gesture but not to speech” (p. 510).
Over the years some extravagant claims have been made regarding gesture-speech mismatch – that learners convey information through gesture of which they may not be aware: that mismatch provides a window into the learner's mind. The detection of mismatch, however, faces a large and difficult challenge. Calbris (Reference Calbris and Copple2011, pp. 23–30) reported that a single gestural form can represent different ideas (polysemy), while a single concept can be evoked using different forms (polysign). Gestural forms, therefore, are inherently “inscrutable” (Quine Reference Quine1968, p. 196), there being no one-to-one correspondence between signifier and signified.
Experimenters seeking evidence of mismatch have attempted to overcome this by cataloguing the gestures commonly seen with regard to a particular task and using coders familiar with those catalogued gestures. The experimental protocol employs a double-coding technique (Alibali & Goldin-Meadow Reference Alibali and Goldin-Meadow1993). Videotapes of the subject producing an explanation at the board are first coded, working solely off the audio track, by someone familiar with the strategies children commonly employ for the problem in question. The recording is then recoded, now with the sound off, by a different coder familiar with the gestures associated with these same strategies. When the codes generated by the two methods are discrepant, the subject's explanation for that problem is classified as exhibiting a mismatch. In more recent work with signing subjects (Goldin-Meadow et al. Reference Goldin-Meadow, Shield, Lenzen, Herzog and Padden2012), the videos are viewed first by an American Sign Language (ASL) coder and again by a coder familiar with the problem-specific gestural “formulations” (Koschmann et al. Reference Koschmann, LeBaron, Goodwin, Zemel and Dunnington2007).
There are several concerns with this way of operationalizing mismatch. The experimental protocol does not really resolve the inscrutability problem, but merely seeks to control it by tightly limiting the range of gestural formulations that might arise. This strategy rests on a number of assumptions: that subjects have finite gestural repertoires, that the ways of understanding (or misunderstanding) the problem are fixed, and so forth. The interview is rigidly structured, and there is no built in “proof procedure” for meaning construction. When interaction is constrained in this way, there is no opportunity to pursue intersubjectivity (Interviewer: “I notice that you have pointed at four numbers, but your sum only includes three. How does that 4 to the right change your answer?”). The errant gesture produced by the subject in the example performs the work of “disclosing the world in sight” (Streeck Reference Streeck2009, pp. 8–9). Gestures that function in this way do so through tight temporal coordination with co-occurring talk (Hindmarsh & Heath Reference Hindmarsh and Heath2000; Streeck Reference Streeck2009). The experimental procedure for producing mismatch, however, is blind to this coordination owing to its method of coding gesture and speech/sign independently.
So, the returns are still out on how the discovery of gesture production by signers will impact further research on learning and gesture. Educational psychology has, from its inception, treated learning as an “occult” (Koschmann Reference Koschmann2002, p. 2) matter. The experimental finding of gesture-speech mismatch is provocative in that it takes up learning as something witnessable. What we need, however, is to find new ways of studying gesture in learning, ways that treat gesture as an interactional matter and that honor what G-M&B identify as “the principle that speech and gesture form an integrated system” (sect. 5, para. 3).