Goldin-Meadow & Brentari (G-M&B) highlight the relationship between imagistic properties and categorical properties in communication, focusing on how they emerge from and interact in speech, sign, and gesture. Indeed, it does seem clear that the appropriate comparison in forthcoming research on this topic should be the combinations of (a) speech and gesture with (b) sign and gesture, given increasingly sophisticated theoretical and experimental tools able to distinguish gesture and sign, and because imagistic gestures and categorical properties of speech and sign form complementary aspects of the communicative signal.
One important piece of evidence in favor of this view comes from a combination of sign and speech that was not discussed in the target article: sign-speech (“bimodal”) bilingualism. Communication between individuals who are native users of both a signed and spoken language frequently involves natural “code blended” utterances (simultaneous signed and spoken analogs of unimodal bilingual “code switches”) that exhibit aspects of the grammar of both languages. Studying code blends can provide unique insight into the ways that combinatorial (“linguistic”) and imagistic (“gestural”) signals can combine through oral and manual articulators, because in bimodal bilingual utterances each set of articulators has the potential to carry a full linguistic signal.
The flexible relationship between language and gesture is perhaps most clearly highlighted in code blends involving sign language classifier predicates, which are signs that involve a categorical/linguistic handshape that bears a grammatical/linguistic relation to the sentence's subject pronounced with a movement and location that have imagistic properties of gesture (see sect. 4 of the target article). When combined in code blends with a spoken language, these classifier predicates can either serve the place of a typical co-speech gesture (when the overall structure is based on the spoken language), or they can serve as a main predicate (when the overall structure is based on the sign language) with an accompanying gesture in the spoken language.
Consider example 1 below: in that example, English is the language providing the structure of the utterance and the mouth produces most of the words, while the hands merely provide an accompanying gesture but one that includes categorical, linguistic components from ASL (the classifier handshape for legs using the “V” handshape, CL-V). In another kind of blend, ASL provides the dominant underlying structure in example 2, and (whispered) English provides a sound effect – a verbal gesture of sorts – to accompany classifier constructions (contrast this with the English word golf that accompanies the nonclassifiers sign GOLF).
-
(1)
English speech: And my mom's you know walking down.
ASL sign: CL-V(walking down stairs)
(Emmorey et al. Reference Emmorey, Borinstein, Thompson and Gollan2008)
-
(2)
ASL Sign: GOLF CL-1(path of ball going up) BALL CL-1(path of ball going up)
English Whisper: golf (sound-effect) ball
(soundeffect)
“In golf the ball goes high up, the ball goes like this … ”
(Petroj et al. Reference Petroj, Guerrera and Davidson2014)
Both (1) and (2) are examples of co-opting articulators typically used for combinatorial information in each language, now for imagistic, gestural purposes. Both blends using classifiers support the view that the oral and manual modalities are each capable of providing either imagistic or combinatorial components; that the manual modality is sometimes considered to be primarily gestural is a result of the traditional focus only on spoken language.
Bimodal bilingual code blends also support a second claim from the target article: that multimodal utterances convey a single proposition. Although unimodal (sign or speech only) bilingualism leaves open the question whether with, for example, two mouths, a bilingual could or would simultaneously produce two separate propositions in two languages, in the case of bimodal bilinguals the answer is evident: Despite separate articulators, many studies of bimodal bilingualism have shown that the two channels combine to produce a single proposition (de Quadros et al. Reference de Quadros, Lillo-Martin, Pichler, Marschark and Spencer2015). It is crucial, then, to understand how a compositional semantic system should handle all of the components of such a proposition, both the imagistic and discrete, in sign and/or in speech.
It is unfortunate that in the target article the authors discuss the difficulties of accounting for imagistic components of sign (and speech) at various levels of linguistic analysis: the phonological, morphological, and syntactic, but have no dedicated discussion about meaning (semantics). However, very recent theoretical linguistic advances within formal semantics and pragmatics have provided tools to address precisely this question of how to incorporate both the gestural and linguistic aspects of meaning in many areas of semantics, including binding, scales, anaphoric reference, speech reports, and sign language classifier predicates.
Classifier predicates have especially been the focus of one account that directly compares the sign and gestural components of classifiers with the combination of speech and co-speech gesture, in an implementation of one of the primary suggestions of the target article. This is accomplished within a formal framework by modeling the gestural component as a demonstration that functions as a manner adverbial: Just like someone can run quickly or happily (which are categorical linguistic descriptions), they can also run in a manner consistent with an imagistic gesture that is provided (essentially, a gestural description). In speech, the gesture can be part of asserted content through such demonstrational adverbials, dependent on pragmatics; in sign language classifier constructions, the handshape provides the combinatorial/linguistic structure of a verb with minimal meaning on its own, which takes as a modifier the location and movement provided by the classifier (Davidson Reference Davidson2015; Zucchi Cecchetto & Geraci Reference Zucchi, Cecchetto and Geraci2012). A similar event semantic analysis may be able to be extended to ideophones, which convey both imagistic and combinatorial information completely within the oral mode (Henderson Reference Henderson2016).
It is important to include bimodal bilingual “code blends” as a unique source of further evidence for the viewpoint advocated by G-M&B that sign, speech, and gesture all work together to create a single proposition, and to illustrate how imagistic and categorical components are both possible in both sets of articulators. Furthermore, recent advances in semantics provide a framework for incorporating gestural components meaningfully into a model of linguistic communication, as long as the analysis begins with the appropriate comparison of speech+gesture and sign+gesture.
You have
Access
Goldin-Meadow & Brentari (G-M&B) highlight the relationship between imagistic properties and categorical properties in communication, focusing on how they emerge from and interact in speech, sign, and gesture. Indeed, it does seem clear that the appropriate comparison in forthcoming research on this topic should be the combinations of (a) speech and gesture with (b) sign and gesture, given increasingly sophisticated theoretical and experimental tools able to distinguish gesture and sign, and because imagistic gestures and categorical properties of speech and sign form complementary aspects of the communicative signal.
One important piece of evidence in favor of this view comes from a combination of sign and speech that was not discussed in the target article: sign-speech (“bimodal”) bilingualism. Communication between individuals who are native users of both a signed and spoken language frequently involves natural “code blended” utterances (simultaneous signed and spoken analogs of unimodal bilingual “code switches”) that exhibit aspects of the grammar of both languages. Studying code blends can provide unique insight into the ways that combinatorial (“linguistic”) and imagistic (“gestural”) signals can combine through oral and manual articulators, because in bimodal bilingual utterances each set of articulators has the potential to carry a full linguistic signal.
The flexible relationship between language and gesture is perhaps most clearly highlighted in code blends involving sign language classifier predicates, which are signs that involve a categorical/linguistic handshape that bears a grammatical/linguistic relation to the sentence's subject pronounced with a movement and location that have imagistic properties of gesture (see sect. 4 of the target article). When combined in code blends with a spoken language, these classifier predicates can either serve the place of a typical co-speech gesture (when the overall structure is based on the spoken language), or they can serve as a main predicate (when the overall structure is based on the sign language) with an accompanying gesture in the spoken language.
Consider example 1 below: in that example, English is the language providing the structure of the utterance and the mouth produces most of the words, while the hands merely provide an accompanying gesture but one that includes categorical, linguistic components from ASL (the classifier handshape for legs using the “V” handshape, CL-V). In another kind of blend, ASL provides the dominant underlying structure in example 2, and (whispered) English provides a sound effect – a verbal gesture of sorts – to accompany classifier constructions (contrast this with the English word golf that accompanies the nonclassifiers sign GOLF).
(1) English speech: And my mom's you know walking down.
ASL sign: CL-V(walking down stairs)
(Emmorey et al. Reference Emmorey, Borinstein, Thompson and Gollan2008)
(2) ASL Sign: GOLF CL-1(path of ball going up) BALL CL-1(path of ball going up)
English Whisper: golf (sound-effect) ball
(soundeffect)
“In golf the ball goes high up, the ball goes like this … ”
(Petroj et al. Reference Petroj, Guerrera and Davidson2014)
Both (1) and (2) are examples of co-opting articulators typically used for combinatorial information in each language, now for imagistic, gestural purposes. Both blends using classifiers support the view that the oral and manual modalities are each capable of providing either imagistic or combinatorial components; that the manual modality is sometimes considered to be primarily gestural is a result of the traditional focus only on spoken language.
Bimodal bilingual code blends also support a second claim from the target article: that multimodal utterances convey a single proposition. Although unimodal (sign or speech only) bilingualism leaves open the question whether with, for example, two mouths, a bilingual could or would simultaneously produce two separate propositions in two languages, in the case of bimodal bilinguals the answer is evident: Despite separate articulators, many studies of bimodal bilingualism have shown that the two channels combine to produce a single proposition (de Quadros et al. Reference de Quadros, Lillo-Martin, Pichler, Marschark and Spencer2015). It is crucial, then, to understand how a compositional semantic system should handle all of the components of such a proposition, both the imagistic and discrete, in sign and/or in speech.
It is unfortunate that in the target article the authors discuss the difficulties of accounting for imagistic components of sign (and speech) at various levels of linguistic analysis: the phonological, morphological, and syntactic, but have no dedicated discussion about meaning (semantics). However, very recent theoretical linguistic advances within formal semantics and pragmatics have provided tools to address precisely this question of how to incorporate both the gestural and linguistic aspects of meaning in many areas of semantics, including binding, scales, anaphoric reference, speech reports, and sign language classifier predicates.
Classifier predicates have especially been the focus of one account that directly compares the sign and gestural components of classifiers with the combination of speech and co-speech gesture, in an implementation of one of the primary suggestions of the target article. This is accomplished within a formal framework by modeling the gestural component as a demonstration that functions as a manner adverbial: Just like someone can run quickly or happily (which are categorical linguistic descriptions), they can also run in a manner consistent with an imagistic gesture that is provided (essentially, a gestural description). In speech, the gesture can be part of asserted content through such demonstrational adverbials, dependent on pragmatics; in sign language classifier constructions, the handshape provides the combinatorial/linguistic structure of a verb with minimal meaning on its own, which takes as a modifier the location and movement provided by the classifier (Davidson Reference Davidson2015; Zucchi Cecchetto & Geraci Reference Zucchi, Cecchetto and Geraci2012). A similar event semantic analysis may be able to be extended to ideophones, which convey both imagistic and combinatorial information completely within the oral mode (Henderson Reference Henderson2016).
It is important to include bimodal bilingual “code blends” as a unique source of further evidence for the viewpoint advocated by G-M&B that sign, speech, and gesture all work together to create a single proposition, and to illustrate how imagistic and categorical components are both possible in both sets of articulators. Furthermore, recent advances in semantics provide a framework for incorporating gestural components meaningfully into a model of linguistic communication, as long as the analysis begins with the appropriate comparison of speech+gesture and sign+gesture.