Introduction
Most people that can be considered bilingual speak two spoken languages, such as English, German or French. In contrast, bimodal bilinguals speak a spoken language and a sign language. It is well known that spoken languages and sign languages are equally complex and very much distinct from each other, even within one country. For example, spoken German and German Sign Language (DGS) are as distinct from each other as English and DGS.
One important difference between spoken languages and sign languages is the primary modality that is used for language production: While spoken languages make use of auditory input and vocal output (at least in speaking, though not in writing), sign languages make use of visual input and manual output. This difference becomes critical when bilinguals use or switch between two languages. A person using two spoken languages in the same conversation must switch between them because both languages are of the same modality (i.e., the oral-auditory modality) and thus are both produced using the vocal tract. This switching behavior is often called “code-switching” (cf. Milroy & Muysken, Reference Milroy and Muysken1995). If the two languages are of different modalities, the two languages can be produced simultaneously, as in bimodal language blending (for a review, see Emmorey, Giezen & Gollan, Reference Emmorey, Giezen and Gollan2016). Sequential bimodal language switching is rare, but it is possible and it does occur in offline data (Emmorey, Borinstein, Thompson & Gollan, Reference Emmorey, Borinstein, Thompson and Gollan2008a).
The present study aimed at exploring unimodal and bimodal language switching. We reasoned that the use of the same output modality (as in unimodal language switching) vs. two different output modalities (as in bimodal language switching) might influence language switching and bilingual language control.
Bilingual language switching
The language-switching paradigm has been used widely in the psychological study of bilingual language production, which has focused on unimodal (spoken-spoken) production (e.g., Costa & Santesteban, Reference Costa and Santesteban2004; Meuter & Allport, Reference Meuter and Allport1999; Philipp, Gade & Koch, Reference Philipp, Gade and Koch2007). In these experiments, participants are instructed to produce words in the target language for each trial, with the target language indicated by a cue and the response word indicated by a stimulus such as a digit or a picture. The trial sequence is designed so that the target language is either the same as the previous trial (a repeat trial) or different (a switch trial). Previous studies testing unimodal language switching have found longer reaction times and higher error rates in switch trials than repeat trials, and this difference, called “switch costs”, has been found often, particularly in unbalanced bilinguals (e.g., Heikoop, Declerck, Los & Koch, Reference Heikoop, Declerck, Los and Koch2016; for reviews, see e.g., Bobb & Wodniecka, Reference Bobb and Wodniecka2013; Declerck & Philipp, Reference Declerck and Philipp2015; Kroll, Bobb, Misra & Guo, Reference Kroll, Bobb, Misra and Guo2008).
The study of bilingual speech production is divided into two primary theoretical approaches: (1) language-specific selection models and (2) inhibition models. In selection models, language selection is closely tied to lexical selection. It is assumed that lexemes from the non-target language cannot be selected and therefore do not compete with lexemes from the target language in later stages of production (e.g., Costa & Caramazza, Reference Costa and Caramazza1999; Finkbeiner & Caramazza, Reference Finkbeiner and Caramazza2006). Inhibition models, in contrast, assume that bilingual parallel lexical activation occurs first, followed by the inhibition of one lexeme or language in order to allow the correct lexeme in the target language to be selected for production (e.g., Green, Reference Green1986, Reference Green1998; Meuter & Allport, Reference Meuter and Allport1999). These models were developed on the basis of studies of unimodal language production, in which language selection is forced due to the physiological restrictions associated with sharing a primary output channel (the vocal tract). However, in the present case of bimodal language production, a spoken word and a sign from the sign language can be and often are produced simultaneously; this observation is consistent with inhibition models because parallel production indicates that the lexemes from both languages can be selected and produced in parallel (Emmorey et al., Reference Emmorey, Borinstein, Thompson and Gollan2008a).
In inhibition models, language switch costs are presumed to result from overcoming the inhibition of the language from the previous trial. On a repeat trial, the same language is needed as in the previous trial, so the active language remains active and the inhibited language remains inhibited. On a switch trial, the relevant language of the current trial is still inhibited because it was inhibited in the previous trial; this inhibition must be overcome, and the result is that responses are slower for switch trials (e.g., Green, Reference Green1986, Reference Green1998; Meuter & Allport, Reference Meuter and Allport1999; Kroll et al., Reference Kroll, Bobb, Misra and Guo2008; Philipp & Koch, Reference Philipp and Koch2009; see also Koch, Gade, Schuch & Philipp, Reference Koch, Gade, Schuch and Philipp2010).
Bimodal bilingualism
There has been little previous work on bimodal language switching; and bimodal language switch costs, along with the inhibitory control mechanisms which may underlie these costs, have not yet been thoroughly examined (for a recent review of bimodal bilingualism, see Emmorey et al., Reference Emmorey, Giezen and Gollan2016). The present study is aimed at filling this gap, as we examine bimodal language switching (switching between German and DGS) and directly compare it to unimodal language switching (switching between German and English) in the same multilingual participants. The motivation for looking at switch costs in bimodal language switching, and what makes it different from unimodal switching, is that the two relevant languages are produced with two distinct primary output channels (i.e., with the hands and the mouth). Emmorey and colleagues (Reference Emmorey, Borinstein, Thompson and Gollan2008a) suggest that switch costs arising from competition for the mouth, as present in unimodal switching, will not be found in bimodal switching. The two distinct output channels might result in a different form of language control than that of unimodal language switching, and language control mechanisms might differ within the same participants based on switching context (unimodal vs. bimodal).
A first indication that language control might differ in unimodal and bimodal language switching was provided by a study (Emmorey, Luk, Pyers & Bialystok, Reference Emmorey, Luk, Pyers and Bialystok2008b) that compared executive control in monolinguals, unimodal bilinguals and bimodal bilinguals using a flanker task. This study found advantages in executive control for unimodal bilinguals, as compared to both monolinguals and bimodal bilinguals. The authors suggest that the unimodal bilingual advantage stems not just from being bilingual, but specifically from switching between their spoken languages, which results in increased inhibitory control. They suggest that bimodal bilinguals do not enjoy this advantage because they are not forced to switch between their languages sequentially and instead blend them simultaneously; this pattern of language blending may lead to a different form of language control with different advantages and/or disadvantages.
The present study was designed to examine unimodal vs. bimodal language control. However, our goal was not to compare unimodal bilinguals with bimodal bilinguals, as previous studies have done (e.g., Emmorey et al., Reference Emmorey, Luk, Pyers and Bialystok2008b). Our goal was to compare unimodal and bimodal conditions within the same participants. Consequently, our focus is not identifying differences between unimodal and bimodal bilinguals, but rather on examining potential differences in language control between unimodal and bimodal language switching conditions.
Materials and methods
Participants
Eighteen participants (age 23–26; 17 women, 1 man) took part the experiment. All participants were hearing native speakers of German who learned English in school (starting in elementary school) and had completed 7–9 semesters of DGS instruction in the Deaf Education program at the University of Cologne. For all participants, English proficiency was unambiguously greater than DGS proficiency because they started learning English as children, whereas they started learning DGS as adults. Thus, they had many more years of English instruction than of DGS instruction.
Task and procedure
The experiment was programmed in Presentation on a 15.4” Lenovo laptop with a screen resolution of 1280 × 800. Only native hardware was used (i.e., the microphone was laptop-internal).
Language cues were squares in one of three solid colors (red, blue, yellow), measuring 400 × 400 pixels, for the three languages. The mapping of cue color to language was counterbalanced across participants. Stimuli were 10 line drawings of common objects: chair, dolphin, door, egg, garbage, mountain, pitcher, scissors, suitcase, and worm, taken from the International Picture-Naming Project database (Szekely, Jacobsen, D'Amico, Devescovi, Andonova, Herron, Lu, Pechmann, Pléh, Wicha, Federmeier, Gerdjikova, Gutierrez, Hung, Hsu, Iyer, Kohnert, Mehotcheva, Orozco-Figueroa, Tzeng, Tzeng, Arévalo, Vargha, Butler, Buffington & Bates, Reference Szekely, Jacobsen, D'Amico, Devescovi, Andonova, Herron, Lu, Pechmann, Pléh, Wicha, Federmeier, Gerdjikova, Gutierrez, Hung, Hsu, Iyer, Kohnert, Mehotcheva, Orozco-Figueroa, Tzeng, Tzeng, Arévalo, Vargha, Butler, Buffington and Bates2004). The stimuli were selected so that the sign's place of articulation was the neutral signing space. The stimuli measured 300 × 300 pixels.
The task throughout the experiment was picture naming in one of three languages (German, English, DGS). Participants performed in two different conditions: unimodal language switching (German/English) and bimodal language switching (German/DGS). Reaction times for vocal responses in German and English were registered using a voice key (voice onset), recorded by the experiment software, with response registration triggered by the voice surpassing a sound threshold. Reaction times for manual responses in DGS were registered using a homekey (motion onset). Participants were instructed to use their dominant hand to keep the homekey pressed during all bimodal blocks and to release the homekey only to produce a DGS sign. Errors for both vocal and manual responses were recorded by the experimenter.
The experiment lasted approximately 30 minutes, with 5–10 minutes of instruction and training, 20 minutes for the experiment itself, and about 5 minutes for a short questionnaire and debriefing afterwards. The instruction consisted of the participant reading an instruction sheet and, if necessary, asking clarifying questions; the experimenter also informed the participant as to the stimuli words/signs in the three languages. The training was a short mock experiment with 4 blocks of 10 trials, two for each condition (German/English, German/DGS), in which participants were familiarized with the cue and stimuli images, the pace of the experiment, and the procedure for both vocal and manual responses.
There were four experimental blocks of 100 trials each, with two consecutive blocks for each condition. The sequence of conditions was counterbalanced across participants. In each trial, the language cue was presented for 500 ms. The stimulus then appeared within the cue square, and both the cue as a frame and the stimulus were shown for 1500 ms, followed by 1000 ms of black screen. The participant was supposed to respond after the appearance of the stimulus but before the end of the trial. If the participant did not respond quickly enough, or if the response was too quiet for the microphone to register it, they were shown the message “schneller/lauter!!” (‘faster/louder!!’) for 500 ms, followed by a black screen for 1000 ms before the beginning of the next trial. The sequence of trials was controlled to produce an equal number of trials in each language in both conditions (i.e., German vs. English in the unimodal condition and German vs. DGS in the bimodal condition).
Design
In a first analysis, the unimodal blocks were examined in isolation; the within-subject independent variables were Language (German vs. English) and Shift (repeat vs. switch). The dependent variables were reaction time and error rate.
In a second set of analyses, unimodal blocks were compared to bimodal blocks. The RT analysis was restricted to German responses across conditions as these were the only responses that occurred in both conditions and because (due to timing differences) vocal and manual responses should not be compared directly. In this analysis, the within-subject independent variables were Condition (unimodal vs. bimodal) and Shift (repeat vs. switch). For error rates, all three languages were included in an analysis in which the within-subject independent variables were Condition (unimodal vs. bimodal) and Shift (repeat vs. switch).
Results
The first trial of each block was excluded from the analysis for both reaction times and error rates. Furthermore, RTs in all trials were z-transformed per participant and trials with a z-score of -2/+2 were discarded as outliers. Also, a trial was excluded if a technical problem occurred (e.g., if the voicekey did not work properly; 2.7%). For RT analysis, trials in which the participant made an error and the subsequent trial were excluded. Due to a problem in converting the Presentation output file, the last trial of each block was excluded from the RT analysis (1%). These criteria resulted in an exclusion of 11.6% of all trials in the RT analysis and 5.1% in the error analysis. Mean reaction times and error rates across conditions, by language, are presented in Table 1.
Mean reaction times, error rates (standard errors) across Conditions (unimodal and bimodal), by Language (German vs. English vs. German Sign Language [DGS]) as (A) subject-based values and (B) item-based values.
Unimodal language switching
In a first analysis, we tested the unimodal switching blocks in isolation. For RTs, we conducted subject-based and item-based Language × Shift analyses of variance (ANOVAs). The main effect of Language was not significant, F1(1, 17)=2.6; p > .05; ηp 2=.132; F2(1, 9)=0.9; p > .05; ηp 2=.093, but there was a significant main effect of Shift, F1(1, 17)=25.6; p < .001; ηp 2=.601; F2(1, 9)=144.3; p < .001; ηp 2=.941, indicating higher RTs in switch than in repeat trials. Importantly, the interaction between Language and Shift was significant in the subject-based analysis, F1(1, 17)=30.1; p < .001; ηp 2=.638, but not in the item-based analysis, F2(1, 9)=2.1; p > .05; ηp 2=.189. For the subject-based analysis, this means that switch costs were significantly larger for L1 (German: 57 ms; t(17)=5.9; p < .001, for a post-hoc two-tailed, paired t-test comparing repeat and switch trials in German) than for L2 (English: 27 ms; t(17)=3.6; p < .01).
For error rates, we conducted the same analyses. There was a significant main effect of Language, F1(1, 17)=12.2; p < .01: ηp 2=.418, indicating higher error rates in German than in English in the subject-based analyses but not in the item-based analysis, F2(1, 9)=4.2; p > .05; ηp 2=.316. Importantly, there was a significant main effect of Shift, F1(1, 17)=6.7; p < .05; ηp 2=.282; F2(1, 9)=11.0; p < .01; ηp 2=.549, demonstrating more errors in switch trials than in repeat trials. Although switch costs were numerically larger for L1 (2.9% subject-based) than for L2 (0.8% subject-based), the interaction between Language and Shift did not reach significance, F1(1, 17)=3.0; p > .05; ηp 2=.150; F2(1, 9)=5.0; p > .05; ηp 2=.355.
Unimodal vs. bimodal language switching
In the next set of analyses, we compared unimodal language switching directly to bimodal language switching. These data are presented in Figure 1. We restricted the RT analysis across modality conditions to German vocal responses as a function of whether they were performed in the context of a unimodal language-switching block (German/English) or a bimodal switching block (German/DGS).
Subject-based mean reaction times (German responses only), error rates (all responses) as a function of Condition (unimodal vs. bimodal) and Shift (repeat vs. switch). Vertical lines represent standard errors.
We conducted subject-based and item-based Condition × Shift ANOVAs. There were significant main effects of Condition, F1(1, 17)=71.6; p < .001; ηp 2=.808; F2(1, 9)=73.4; p < .01; ηp 2=.891, and Shift, F1(1, 17)=15.6; p < .001; ηp 2=.479; F2(1, 9)=67.7; p < .001; ηp 2=.883, and importantly, the interaction between Condition and Shift was also significant in the subject-based analysis, F1(1, 17)=5.1; p < .05; ηp 2=.232, though not in the item-based analysis F2(1, 9)=3.0; p > .05; ηp 2=.249. The results indicate that, in general, German vocal responses were faster in bimodal switching blocks than in unimodal blocks. Further, and even more important, switch costs were (significantly in the subject-based analysis and numerically in the item-based analysis) smaller for bimodal blocks (21 ms; t(17)=1.4; p > .05; for a post-hoc two-tailed, paired t-test comparing German repeat and switch trials in bimodal blocks, subject-based) than for unimodal blocks (57 ms; t(17)=5.9; p < .001).Footnote 1
For error rates, we conducted Condition × Shift ANOVAs on all responses since errors can be compared directly for manual and vocal responses. There were significant main effects of Condition, F1(1, 17)=10.3; p < .01; ηp 2=.378; F2(1, 9)=16.2; p < .01; ηp 2=.643, and Shift, F1(1, 17)=7.4; p < .05; ηp 2=.304; F2(1, 9)=24.2; p < .01; ηp 2=.729, with a lower percentage of errors in the bimodal blocks compared to unimodal blocks and in repeat trials compared to switch trials (see Figure 1). Switch costs were numerically smaller for bimodal blocks (1.2% subject-based) than unimodal blocks (1.9% subject-based), but the interaction between Condition and Shift did not reach significance, F1(1, 17)=1.4; p > .05; ηp 2=.076; F2(1, 9)=1.0; p > .05; ηp 2=.099. Excluding English and DGS trials from the error analysis did not change the pattern of results, with bimodal switch costs still numerically although not significantly smaller than unimodal switch costs (see Table 1).
Discussion
The aim of the present study was to examine unimodal and bimodal language switching within the same participants. When comparing unimodal and bimodal language switching, the most important findings were shorter overall RTs, lower error rates, and smaller switch costs in bimodal blocks than in unimodal blocks, which provide evidence of an advantage for bimodal language switching over unimodal language switching – or put differently, a bimodal advantage in language switching.Footnote 2
Language control in unimodal and bimodal language switching
In the present study, participants were instructed to switch between languages, producing one lexeme from one language per trial, rather than blending (for an examination of switch costs in bimodal blending, see Kaufmann & Philipp, Reference Kaufmann and Philipp2015). Unimodal bilinguals produce code-switches in natural language production, which indicates that in the unimodal production mode, only one lexeme remains active, and the other must be inhibited (e.g., Kroll et al., Reference Kroll, Bobb, Misra and Guo2008; Philipp et al., Reference Philipp, Gade and Koch2007; Philipp & Koch, Reference Philipp and Koch2009). In contrast, bimodal bilinguals can and do produce code-blends in natural language production, which indicates that in the bimodal production mode, the two lexemes can remain active (i.e., uninhibited) through production (Emmorey et al., Reference Emmorey, Borinstein, Thompson and Gollan2008a; Emmorey et al., Reference Emmorey, Giezen and Gollan2016).
To account for the bimodal advantage in language switching, we assume that inhibition plays a crucial role in language switching (cf. Green, Reference Green1986, Reference Green1998). For unimodal language switching, we assume that lexical inhibition takes place (e.g., Declerck & Philipp, Reference Declerck and Philipp2015; Kroll et al., Reference Kroll, Bobb, Misra and Guo2008; Philipp & Koch, Reference Philipp and Koch2009). In contrast, for bimodal language switching, both lexemes can remain uninhibited, and the output channel or articulator must be inhibited at a later stage of production in order to prevent the non-target lexeme from being uttered. The difference in the size of switch costs might thus indicate that lexical inhibition is costlier than output channel inhibition or articulatory inhibition. Put differently, the larger overlap in response-related processes (i.e., phonological encoding and articulation) between two spoken languages, which share both (a part of) the phoneme inventory as well as the output modality, as compared to a spoken and a sign language which share neither, might have led to greater between-language interference. To resolve this greater between-language interference, a higher amount of inhibitory control might have been necessary, resulting in higher switch costs in unimodal versus bimodal language switching.
It is important to note that we do not suggest a completely different form of language control in unimodal and bimodal language switching. In each case, language control presumably refers to the same mechanisms: inhibition. Put differently, we suppose that inhibitory language control plays a crucial role in both unimodal and bimodal language control. Similarly, a recent study also demonstrated inhibitory language control in bimodal bilinguals in language comprehension (Giezen, Blumenfeld, Shook, Marian & Emmorey, Reference Giezen, Blumenfeld, Shook, Marian and Emmorey2015). However, we also suppose that the target of this inhibitory control is influenced by the specific switching condition – even within the same participant. Thus, we suggest that language control can have different loci and is not restricted to one stage in language processing (for similar ideas, see Bobb & Wodniecka, Reference Bobb and Wodniecka2013; Declerck & Philipp, Reference Declerck and Philipp2015; Gollan, Schotter, Gomez, Murillo & Rayner, Reference Gollan, Schotter, Gomez, Murillo and Rayner2014; Kroll, Bobb & Wodniecka, Reference Kroll, Bobb and Wodniecka2006).
When interpreting the switch cost difference between unimodal and bimodal language switching, we acknowledge that this difference was significant in the subject-based RT analysis but not in the item-based RT analysis. To account for this divergence, it is important to note that the descriptive data pattern is comparable across both analyses (switch costs were 57 ms in the unimodal vs. 21 in the bimodal condition in the subject-based analysis and 60 vs. 34 ms in the item-based analysis; see Table 1). Thus, we suppose that the divergence in terms of statistics is due to the relatively small number of ten items and that future studies with an increased item set might be able to resolve the divergence between subject- and item-based analyses. Based on the comparability of the data patterns and the reproducibility in a different study (see Footnote 2), we are confident that the present study demonstrates a difference between unimodal and bimodal language switching.Footnote 3
Yet, one might also argue that the different size in switch costs between the unimodal and bimodal condition is caused by the language proficiency in the non-dominant language rather than a difference in inhibitory language control. As the participants in the present study were clearly more proficient in English than in DGS, the smaller switch costs for German in the bimodal condition could be explained by a lower confusability between German and DGS due to the larger proficiency difference. A similar argument was made by Philipp and colleagues (Reference Philipp, Gade and Koch2007), who observed larger overall switch costs for switching between L1 and L2 or L2 and L3 as compared to switching between L1 and L3 (the condition with the largest proficiency difference). However, it is important to note that such a pattern of results was not observed in other studies. For example, in the study of Costa and Santesteban (Reference Costa and Santesteban2004) the overall switch cost was numerically higher for unbalanced bilinguals (Experiment 1) than for balanced bilinguals (Experiment 2; the size of switch costs in this study was, however, not directly compared between experiments). Furthermore, Costa and Santesteban also observed that the overall size of switch costs for balanced bilinguals was relatively similar when switching between the two dominant languages (Experiment 2) and L1 and L3 (Experiment 4). Therefore, although we cannot rule out that a more similar proficiency between the languages in each condition might have had an effect, we are confident that the data pattern cannot be explained by proficiency difference alone. Furthermore, in the more general domain of task switching, a higher similarity between tasks led to a reduction of rather than an increase in switch costs (cf. Arrington, Altmann & Carr, Reference Arrington, Altmann and Carr2003).
The study by Arrington and her colleagues is interesting with respect to the present study because task similarity was manipulated in terms of an overlap in response modalities in their second experiment. More specifically, participants switched among four different stimulus categorization tasks. For two of these tasks, the response had to be given vocally, and for the other two, manually. The results demonstrated larger switch costs when both the stimulus categorization and the response modality had to be switched as compared to switching the stimulus categorization while keeping the response modality the same. This result is also in accordance with observations that a switch of the response modality in isolation (e.g., from a vocal to a manual response) leads to switch costs (Philipp & Koch, Reference Philipp and Koch2011). In the present study, switching the language and the response modality (i.e., the bimodal switching condition) led to smaller switch costs than switching the language only (i.e., the unimodal switching condition). Therefore, we suppose that switching the response modality in our language switching study did not imply an (additional) cost of a modality switch, but rather led to an advantage in terms of reduced between-language interference.
Asymmetric switch costs in unimodal language switching
Looking at the unimodal blocks in isolation, our results demonstrate asymmetric switch costs, with larger switch costs for the dominant language, German, than for the non-dominant language, English. Previous studies also often found a switch-cost asymmetry, with larger switch costs for the dominant language than for the non-dominant language in unbalanced bilinguals (for a review, see Bobb & Wodniecka, Reference Bobb and Wodniecka2013). This result does not necessarily hold for balanced bilinguals, who have no dominant language (see, e.g., Costa & Santesteban, Reference Costa and Santesteban2004). The switch-cost asymmetry can be accounted for assuming that greater inhibition is required to suppress the dominant language when producing the non-dominant language than is required to suppress the non-dominant language when producing the dominant language (reactive inhibition; Green, Reference Green1998). When switching back to the dominant language, overcoming this greater inhibition results in shorter reaction times and higher error rates as compared to switching back to the less dominant, second language (e.g., Kroll et al., Reference Kroll, Bobb, Misra and Guo2008; Meuter & Allport, Reference Meuter and Allport1999).
The results of the present study are in line with this argumentation and, thus, provide further evidence for the role of reactive inhibitory control in language switching (although asymmetric switch costs are not an unequivocal marker for inhibition; see Declerck & Philipp, Reference Declerck and Philipp2015; Koch et al., Reference Koch, Gade, Schuch and Philipp2010; but see Declerck, Thoma, Koch & Philipp, Reference Declerck, Thoma, Koch and Philipp2015; Philipp et al., Reference Philipp, Gade and Koch2007 for evidence of inhibition in language switching).
Conclusion
In summary, this study found shorter reaction times, lower error rates and smaller switch costs in bimodal language switching as compared to unimodal language switching, indicating that for language tasks, an additional language modality leads to a reduction in costs. Our result suggests that inhibitory language control is used in both unimodal and bimodal language switching, but that there are different inhibitory mechanisms, or a different degree of inhibition, at work in unimodal and bimodal language switching – with lexical inhibition, at work in unimodal switching, being costlier than the output channel inhibition that is involved in bimodal switching.
