The complexity of demarcating speech, sign, and gesture is elegantly surveyed in the target article. The analysis promises to be a valuable roadmap for research in multimodal communication. However, we doubt whether the analysis – as currently presented – achieves one of its other goals, that is, to enhance our ability to make “predictions about learning” (para. 5).
Goldin-Meadow & Brentari (G-M&B) argue that regardless of whether information is expressed via the manual or vocal system, a distinction should be made between speech/sign and gesture on the basis of whether categorical or imagistic representations are underlying their expression. This distinction should help explain gesture-related learning, such that mismatches between gesture and speech or sign (and their correlation with learning) are driven by “distinct representational formats – a mimetic, imagistic format underlying gesture versus a discrete, categorical format underlying language, sign, or speech.” (sect. 6, para. 14).
Yet we observe that there is a tension in the target article in that it also stresses an “integrated,” “single,” and “unified” gesture-speech system (sect. 5 & 6). In the case of learners who are producing mismatches in gesture and speech, it is argued “that [the] mismatch is generated by a single gesture-speech system” (sect. 5, para. 15). G-M&B argue that, although learners are unaware of the mismatches they produce, the fact that they are more receptive to learning after they produced mismatches suggests a unified system: “if gesture and speech were two independent systems, the match or mismatch between the information conveyed in these systems should have no bearing on the child's cognitive state” (sect. 5, para. 12).
Unfortunately, in their overview we see no clear arguments (other than stating the case) for resolving the apparent logical contradiction of positing two representational devices (categorical vs. imagistic) that differ and contradict in their informational content (as reflected by gesture and speech mismatches) but are nevertheless part of an integrated system.
Beyond problems of coherence, this contradiction is potentially problematic for understanding learning. Note that learning fundamentally involves a change in the cognitive system. Further note that G-M&B make no attempt to specify how the imagistic information that is supposedly accessed by gesture (and not speech/or sign) is potentially transformed and fed back into the system (cf. Goldin-Meadow Reference Goldin-Meadow2003a; Pouw et al. Reference Pouw, De Nooijer, Van Gog, Zwaan and Paas2014). If gestures do not transform the cognitive system but are only reflective of its underlying imagistic representation, then mismatches reflect that the gesture-speech system is dis-integrated (hence the contradiction). Moreover, G-M&B see the fact that mismatches have bearing on the child's cognitive state as evidence for a unified system, but they fail to account for how the gesture producing the mismatch has any causal force in changing the cognitive system (i.e., how it predicts learning). In other words, the current account begs the question: Why do gesture and speech mismatches have a bearing on the child's cognitive state if gestures reflect information that is already integrated?
What is the alternative? Insights from embedded and embodied cognition challenge the idea that action should be regarded as the mere output of the cognitive system (e.g., Hurley Reference Hurley2001). Such insights have been applied to gesticulation (Cappuccio et al. Reference Cappuccio, Chu and Kita2013; Clark Reference Clark and Radman2013; Pouw et al. Reference Pouw, De Nooijer, Van Gog, Zwaan and Paas2014). If these accounts are on track, the cognitive system is distributed over brain and body, wherein any state that this distributed brain-gesture system enjoys is brought about by loops of circular causation of perception and action (Clark Reference Clark and Radman2013).
Such approaches can be brought in line with G-M&B's proposal that gesture can access distinct information that is not available to speech. Yet it requires rethinking in which way this distinct information is “accessed” and believed to be “present” in an underlying “representation,” and relatedly to which degree this information is integrated with the speech system. As mentioned, G-M&B's current presentation fosters a static understanding of gesture wherein mismatching gestures merely access and output imagistic information. From a more dynamically embodied perspective, gesturing may bring forth imagistic information that is not in any cognitively potent way present in an underlying representation before the act of gesturing. From this perspective, gestures add something to the neural precursors from which they emerge. Namely, gesturing adds kinematic information that is being fed back through the visual and proprioceptive system (Pouw et al. Reference Pouw, De Nooijer, Van Gog, Zwaan and Paas2014).
In sum, we think a more complete account of gesture-related learning requires the specification of how a gesture-speech system integrates incongruent information that is brought forth by the act of gesturing rather than assuming that this information is already integrated. In pursuit of such an account, we support G-M&B's call to develop more sophisticated measures to assess kinematic regularities expressed in gesture, as this allows researchers to further pinpoint what, in the act of gesturing, it is that is cognitively potent for learning. For example, problem solvers have difficulty in judging verbally when cups of different sizes spill water, but they drastically improve when they are allowed to gesture (Schwartz & Black Reference Schwartz and Black1999). It is likely that this performance is dependent on the ability to correctly physically enact the laws that govern the task (which involves being sensitive in gesticulation to relevant properties of the objects gestured about, such as the size of the cups, and rotational inertia). Possibly, the kinematic invariants that are present in such gestures may become more stable over time as expertise develops, and it may be the case that such increasing kinematic regularities are predictive for the susceptibility for categorical integration in speech (e.g., Chu & Kita Reference Chu and Kita2008). We thereby submit that understanding learning from gesture-speech mismatches at least requires specifying how gesture's emergent kinematic regularities (i.e., embodied information) related to the learning task becomes categorizable (and thus transformed) through time, as well as understanding how this affects the potentiality of integration with speech.
The complexity of demarcating speech, sign, and gesture is elegantly surveyed in the target article. The analysis promises to be a valuable roadmap for research in multimodal communication. However, we doubt whether the analysis – as currently presented – achieves one of its other goals, that is, to enhance our ability to make “predictions about learning” (para. 5).
Goldin-Meadow & Brentari (G-M&B) argue that regardless of whether information is expressed via the manual or vocal system, a distinction should be made between speech/sign and gesture on the basis of whether categorical or imagistic representations are underlying their expression. This distinction should help explain gesture-related learning, such that mismatches between gesture and speech or sign (and their correlation with learning) are driven by “distinct representational formats – a mimetic, imagistic format underlying gesture versus a discrete, categorical format underlying language, sign, or speech.” (sect. 6, para. 14).
Yet we observe that there is a tension in the target article in that it also stresses an “integrated,” “single,” and “unified” gesture-speech system (sect. 5 & 6). In the case of learners who are producing mismatches in gesture and speech, it is argued “that [the] mismatch is generated by a single gesture-speech system” (sect. 5, para. 15). G-M&B argue that, although learners are unaware of the mismatches they produce, the fact that they are more receptive to learning after they produced mismatches suggests a unified system: “if gesture and speech were two independent systems, the match or mismatch between the information conveyed in these systems should have no bearing on the child's cognitive state” (sect. 5, para. 12).
Unfortunately, in their overview we see no clear arguments (other than stating the case) for resolving the apparent logical contradiction of positing two representational devices (categorical vs. imagistic) that differ and contradict in their informational content (as reflected by gesture and speech mismatches) but are nevertheless part of an integrated system.
Beyond problems of coherence, this contradiction is potentially problematic for understanding learning. Note that learning fundamentally involves a change in the cognitive system. Further note that G-M&B make no attempt to specify how the imagistic information that is supposedly accessed by gesture (and not speech/or sign) is potentially transformed and fed back into the system (cf. Goldin-Meadow Reference Goldin-Meadow2003a; Pouw et al. Reference Pouw, De Nooijer, Van Gog, Zwaan and Paas2014). If gestures do not transform the cognitive system but are only reflective of its underlying imagistic representation, then mismatches reflect that the gesture-speech system is dis-integrated (hence the contradiction). Moreover, G-M&B see the fact that mismatches have bearing on the child's cognitive state as evidence for a unified system, but they fail to account for how the gesture producing the mismatch has any causal force in changing the cognitive system (i.e., how it predicts learning). In other words, the current account begs the question: Why do gesture and speech mismatches have a bearing on the child's cognitive state if gestures reflect information that is already integrated?
What is the alternative? Insights from embedded and embodied cognition challenge the idea that action should be regarded as the mere output of the cognitive system (e.g., Hurley Reference Hurley2001). Such insights have been applied to gesticulation (Cappuccio et al. Reference Cappuccio, Chu and Kita2013; Clark Reference Clark and Radman2013; Pouw et al. Reference Pouw, De Nooijer, Van Gog, Zwaan and Paas2014). If these accounts are on track, the cognitive system is distributed over brain and body, wherein any state that this distributed brain-gesture system enjoys is brought about by loops of circular causation of perception and action (Clark Reference Clark and Radman2013).
Such approaches can be brought in line with G-M&B's proposal that gesture can access distinct information that is not available to speech. Yet it requires rethinking in which way this distinct information is “accessed” and believed to be “present” in an underlying “representation,” and relatedly to which degree this information is integrated with the speech system. As mentioned, G-M&B's current presentation fosters a static understanding of gesture wherein mismatching gestures merely access and output imagistic information. From a more dynamically embodied perspective, gesturing may bring forth imagistic information that is not in any cognitively potent way present in an underlying representation before the act of gesturing. From this perspective, gestures add something to the neural precursors from which they emerge. Namely, gesturing adds kinematic information that is being fed back through the visual and proprioceptive system (Pouw et al. Reference Pouw, De Nooijer, Van Gog, Zwaan and Paas2014).
In sum, we think a more complete account of gesture-related learning requires the specification of how a gesture-speech system integrates incongruent information that is brought forth by the act of gesturing rather than assuming that this information is already integrated. In pursuit of such an account, we support G-M&B's call to develop more sophisticated measures to assess kinematic regularities expressed in gesture, as this allows researchers to further pinpoint what, in the act of gesturing, it is that is cognitively potent for learning. For example, problem solvers have difficulty in judging verbally when cups of different sizes spill water, but they drastically improve when they are allowed to gesture (Schwartz & Black Reference Schwartz and Black1999). It is likely that this performance is dependent on the ability to correctly physically enact the laws that govern the task (which involves being sensitive in gesticulation to relevant properties of the objects gestured about, such as the size of the cups, and rotational inertia). Possibly, the kinematic invariants that are present in such gestures may become more stable over time as expertise develops, and it may be the case that such increasing kinematic regularities are predictive for the susceptibility for categorical integration in speech (e.g., Chu & Kita Reference Chu and Kita2008). We thereby submit that understanding learning from gesture-speech mismatches at least requires specifying how gesture's emergent kinematic regularities (i.e., embodied information) related to the learning task becomes categorizable (and thus transformed) through time, as well as understanding how this affects the potentiality of integration with speech.