Emmorey, Giezen and Gollan (Emmorey, Giezen & Gollan) in their Key Note article review data bearing on language control in bimodal bilinguals and provoke questions critical to theoretical advance. I consider here two interrelated questions: one on multimodal synchrony and one on the control of serial order.
By way of background: how might we characterise the nature of language control in bilingual individuals? Language control enables speakers to access different language control states so they can achieve their communicative goals. Consider two types of conversation. When conversing with a non-signing addressee, the control state is competitive - the use of speech requires inhibition of signing and so activated signs must be gated from production. When conversing with another bimodal bilingual, the control state is cooperative as both speech and sign can be recruited to achieve the communicative intention. A parallel contrast applies to bilingual speakers who may use just one of their languages when addressing a monolingual speaker but may code-switch between their languages within a conversational turn when licensed by the norms of their bilingual community (Green & Abutalebi, Reference Green and Abutalebi2013; see also Green, Reference Green1998). Although we can draw a parallel, the precise demands for language control in bimodal bilinguals will shape adaptive response in line with the adaptive control hypothesis (Green & Abutalebi, Reference Green and Abutalebi2013).
Consideration of the control demands in bimodal bilinguals spurs us to incorporate a fundamental property of human communication in our accounts – its multimodality. Conversational partners not only align their use of language across different levels of linguistic representation to coordinate actions (e.g., Clark, Reference Clark1996; Pickering & Garrod, Reference Pickering and Garrod2004), they also synchronise their behaviour along multiple dimensions (such as head nodding, face touching, smiling and hand gestures). Alignment may render joint-action more efficient and effective and synchrony facilitate the resolution of coordination problems (e.g., Louwerse, Dale, Bard & Jeuniaux, Reference Louwerse, Dale, Bard and Jeuniaux2012). Emmorey et al. report that bimodal bilinguals do occasionally code-blend even when addressing non-signers. Such occurrences indicate a loss of competitive control – a failure of suppression as they note. My first question is: how does inadvertent code-blending affect conversational synchrony?
Using sign inappropriately with a non-signer might seem innocuous but code-blending risks misconstrual. Bimodal bilinguals appear sensitive to such a possibility. Reference Emmorey, Giezen and GollanEmmorey et al. (Reference Emmorey, Giezen and GollanEmmorey et al) observe that intrusions involving facial features signalling a grammatical marker that conflicts with conventional facial gestures (e.g., a furrowed brow that co-occurs with a WH-question) occur much less frequently than those that do not so conflict (e.g., the eye-brow raise and head tilt linked to the onset of a conditional). In the end, Reference Emmorey, Giezen and GollanEmmorey et al. (Reference Emmorey, Giezen and GollanEmmorey et al) are of the view that code-blends do not disrupt communication with non-signers “because the relevant information is present from speech and gestures accompanying speech are common.” However, mightn't inappropriate code-blending (albeit dependent on its frequency) be more subtly and widely disruptive to the multimodal synchrony of normal conversation? It would be interesting to know, via experimental research, the extent to which inadvertent use of signs (especially those that conflict with conventional gestures) disrupts such synchrony and so impairs joint-action. If it does not, does this mean that the addressee successfully discounts the code-blend or do they begin to imitate it? Furthermore, is it really the case that the successful suppression of code-blending re-establishes multimodal synchrony? Or is it rather the case that suppressing code-blends (a competitive control state) impacts on normal gesture use suggesting that sign and gesture compete even if sign and speech (bar mouthings) do not? If that is the case, then we need to give greater consideration to the possibility that bimodal bilinguals must also control interference but within the non-verbal channel.
Turning now to my second question. In contrast, to an interactional context, where a bimodal bilingual is addressing a non-signing speaker, code-blending between two bimodal bilinguals is communicatively licensed. Reference Emmorey, Giezen and GollanEmmorey et al. (Reference Emmorey, Giezen and GollanEmmorey et al) discuss some interesting examples of naturally occurring and simultaneously-produced code-blends. In all cases, there is cooperative control state in which the resources of either language are open for use. My second question concerns the process of controlling serial order that permits such code-blending to occur.
Speech and sign are planned in advance of execution and so there must be a mechanism that yields serial order from the parallel representation of the sentence plan (Lashley, Reference Lashley and Jeffress1951). In the case of code-switching in hearing bilinguals, code-switches meet the incremental syntactic constraints of the sentence so far. Because simultaneity is possible for code-blends, the incremental constraints are different but the serial order problem remains. One general, and neurally plausible solution to the serial order problem involves use of a competitive queuing mechanism (Green & Li, Reference Green and Li2014). This mechanism comprises two layers: a planning layer and a competitive choice layer (Bohland, Bullock & Guenther, Reference Bohland, Bullock and Guenther2009; Grossberg, Reference Grossberg, Rosen and Snell1978; Houghton, Reference Houghton, Dale, Mellish and Zock1990). Lexical items and constructions have different levels of activation in the planning layer that reflects the intended order of production. The competitive choice layer suppresses all items other than the item with the highest level of activation and releases that item for further processing (e.g., for phonological encoding or sign production) at the same time as removing it from the planning layer so that the item with next highest level of activation can be selected via the competitive choice layer. The question is this: at the level of items or constructions, is it the case that speech and sign use distinct competitive queuing mechanisms to resolve the problem of serial order in each of these channels? If the answer is yes, then given the different dynamics of production their output must be synchronised perhaps via a timing signal from the speech channel. If there is a single competitive queuing mechanism at this level then it must sample two parallel, linked but not necessarily identical, sentence plans. Where speech and a code-blend capture a different aspect of an event being described, the pertinent constructions (speech or sign) must be allocated prior to submission to the planning layer. Such allocation reflects the speaker's sensitivity to the opportunities offered by multimodal nature of human conversation. Non-signers may similarly recruit gesture for explicit communicative purposes.
In conclusion, multimodality is the tiger's tail for theories of language control and the study of bimodal bilingualism shows that it is time to grasp it. Inappropriate code-blending arises when a bimodal speaker converses with a non-signer. It is a temporary loss of competitive language control and risks multimodal synchrony, typical of conversational interactions. Appropriate code-blending is an outcome of a cooperative control process. But it poses a computational challenge. How does the brain serialise and coordinate sentence plans to do with speech and sign? A competitive queuing mechanism offers a neurally-plausible solution.
You have
Access
Emmorey, Giezen and Gollan (Emmorey, Giezen & Gollan) in their Key Note article review data bearing on language control in bimodal bilinguals and provoke questions critical to theoretical advance. I consider here two interrelated questions: one on multimodal synchrony and one on the control of serial order.
By way of background: how might we characterise the nature of language control in bilingual individuals? Language control enables speakers to access different language control states so they can achieve their communicative goals. Consider two types of conversation. When conversing with a non-signing addressee, the control state is competitive - the use of speech requires inhibition of signing and so activated signs must be gated from production. When conversing with another bimodal bilingual, the control state is cooperative as both speech and sign can be recruited to achieve the communicative intention. A parallel contrast applies to bilingual speakers who may use just one of their languages when addressing a monolingual speaker but may code-switch between their languages within a conversational turn when licensed by the norms of their bilingual community (Green & Abutalebi, Reference Green and Abutalebi2013; see also Green, Reference Green1998). Although we can draw a parallel, the precise demands for language control in bimodal bilinguals will shape adaptive response in line with the adaptive control hypothesis (Green & Abutalebi, Reference Green and Abutalebi2013).
Consideration of the control demands in bimodal bilinguals spurs us to incorporate a fundamental property of human communication in our accounts – its multimodality. Conversational partners not only align their use of language across different levels of linguistic representation to coordinate actions (e.g., Clark, Reference Clark1996; Pickering & Garrod, Reference Pickering and Garrod2004), they also synchronise their behaviour along multiple dimensions (such as head nodding, face touching, smiling and hand gestures). Alignment may render joint-action more efficient and effective and synchrony facilitate the resolution of coordination problems (e.g., Louwerse, Dale, Bard & Jeuniaux, Reference Louwerse, Dale, Bard and Jeuniaux2012). Emmorey et al. report that bimodal bilinguals do occasionally code-blend even when addressing non-signers. Such occurrences indicate a loss of competitive control – a failure of suppression as they note. My first question is: how does inadvertent code-blending affect conversational synchrony?
Using sign inappropriately with a non-signer might seem innocuous but code-blending risks misconstrual. Bimodal bilinguals appear sensitive to such a possibility. Reference Emmorey, Giezen and GollanEmmorey et al. (Reference Emmorey, Giezen and GollanEmmorey et al) observe that intrusions involving facial features signalling a grammatical marker that conflicts with conventional facial gestures (e.g., a furrowed brow that co-occurs with a WH-question) occur much less frequently than those that do not so conflict (e.g., the eye-brow raise and head tilt linked to the onset of a conditional). In the end, Reference Emmorey, Giezen and GollanEmmorey et al. (Reference Emmorey, Giezen and GollanEmmorey et al) are of the view that code-blends do not disrupt communication with non-signers “because the relevant information is present from speech and gestures accompanying speech are common.” However, mightn't inappropriate code-blending (albeit dependent on its frequency) be more subtly and widely disruptive to the multimodal synchrony of normal conversation? It would be interesting to know, via experimental research, the extent to which inadvertent use of signs (especially those that conflict with conventional gestures) disrupts such synchrony and so impairs joint-action. If it does not, does this mean that the addressee successfully discounts the code-blend or do they begin to imitate it? Furthermore, is it really the case that the successful suppression of code-blending re-establishes multimodal synchrony? Or is it rather the case that suppressing code-blends (a competitive control state) impacts on normal gesture use suggesting that sign and gesture compete even if sign and speech (bar mouthings) do not? If that is the case, then we need to give greater consideration to the possibility that bimodal bilinguals must also control interference but within the non-verbal channel.
Turning now to my second question. In contrast, to an interactional context, where a bimodal bilingual is addressing a non-signing speaker, code-blending between two bimodal bilinguals is communicatively licensed. Reference Emmorey, Giezen and GollanEmmorey et al. (Reference Emmorey, Giezen and GollanEmmorey et al) discuss some interesting examples of naturally occurring and simultaneously-produced code-blends. In all cases, there is cooperative control state in which the resources of either language are open for use. My second question concerns the process of controlling serial order that permits such code-blending to occur.
Speech and sign are planned in advance of execution and so there must be a mechanism that yields serial order from the parallel representation of the sentence plan (Lashley, Reference Lashley and Jeffress1951). In the case of code-switching in hearing bilinguals, code-switches meet the incremental syntactic constraints of the sentence so far. Because simultaneity is possible for code-blends, the incremental constraints are different but the serial order problem remains. One general, and neurally plausible solution to the serial order problem involves use of a competitive queuing mechanism (Green & Li, Reference Green and Li2014). This mechanism comprises two layers: a planning layer and a competitive choice layer (Bohland, Bullock & Guenther, Reference Bohland, Bullock and Guenther2009; Grossberg, Reference Grossberg, Rosen and Snell1978; Houghton, Reference Houghton, Dale, Mellish and Zock1990). Lexical items and constructions have different levels of activation in the planning layer that reflects the intended order of production. The competitive choice layer suppresses all items other than the item with the highest level of activation and releases that item for further processing (e.g., for phonological encoding or sign production) at the same time as removing it from the planning layer so that the item with next highest level of activation can be selected via the competitive choice layer. The question is this: at the level of items or constructions, is it the case that speech and sign use distinct competitive queuing mechanisms to resolve the problem of serial order in each of these channels? If the answer is yes, then given the different dynamics of production their output must be synchronised perhaps via a timing signal from the speech channel. If there is a single competitive queuing mechanism at this level then it must sample two parallel, linked but not necessarily identical, sentence plans. Where speech and a code-blend capture a different aspect of an event being described, the pertinent constructions (speech or sign) must be allocated prior to submission to the planning layer. Such allocation reflects the speaker's sensitivity to the opportunities offered by multimodal nature of human conversation. Non-signers may similarly recruit gesture for explicit communicative purposes.
In conclusion, multimodality is the tiger's tail for theories of language control and the study of bimodal bilingualism shows that it is time to grasp it. Inappropriate code-blending arises when a bimodal speaker converses with a non-signer. It is a temporary loss of competitive language control and risks multimodal synchrony, typical of conversational interactions. Appropriate code-blending is an outcome of a cooperative control process. But it poses a computational challenge. How does the brain serialise and coordinate sentence plans to do with speech and sign? A competitive queuing mechanism offers a neurally-plausible solution.