Introduction
Bilinguals are able to communicate in two different languages, and they can use both languages in the same conversation, which they commonly do. Whereas most bilinguals indeed speak two languages, for example German and English, bimodal bilinguals are competent in a spoken language and a signed language. Sign languages are as complex as spoken languages, and they are distinct from the spoken language(s) used in the same country or region in terms of lexicon, phonology, syntax, etc. It follows that a switch from spoken German to German Sign Language (DGS), for example, is as much of a language switch as one between any two languages, be they spoken or signed.
However, using two languages in a conversation is very much different in unimodal (spoken-spoken) and bimodal (signed-spoken) bilinguals. This is because in unimodal switching, the two languages are of the same modality (i.e., the vocal-auditory modality) and as such, they share a single primary output channel: the vocal tract. As a consequence, unimodal bilinguals can switch between their languages in the course of a conversation, a behavior that is often referred to as code-switching (cf. Milroy & Muysken, Reference Milroy and Muysken1995), but they cannot produce two words from the two different languages at the same time.
In contrast, the languages used by a bimodal bilingual are of two different modalities and as such, they do not share a primary output channel, which in sign languages (of the visual-spatial modality) is composed of the hands. For this reason, simultaneous natural language production (i.e., language blending) is possible for bimodal bilinguals, while this simultaneous form of bilingual production is not possible for unimodal bilinguals (e.g., Bishop & Hicks, Reference Bishop and Hicks2005; Emmorey, Borinstein, Thompson & Gollan, Reference Emmorey, Borinstein, Thompson and Gollan2008a). At the same time, the possibility of simultaneous bimodal language production does not preclude sequential bimodal language alternation in the form of language switching and, while rare, this does occur in naturalistic data (ibid.).
In the present study, we aimed at exploring whether this difference between bimodal and unimodal language production has implications for language control in bimodal bilinguals. We used three different measures of bilingual language control (i.e., mixing costs, switch costs, and dual-response costs) to assess bimodal bilingual language control. More specifically, we were interested as to whether a Blend can be considered simply as a combination of two independent responses or whether a Blend rather represents a joint response which has specific characteristics (different from the sum of the two single responses).
Empirical measures of language control
Bilingual language control can be measured in (at least) three different forms: Mixing costs, switch costs, and dual-response costs. In each of the respective paradigms, participants are instructed to produce words in the correct language for each trial, with the correct response language usually indicated by a cue and the response word indicated by a stimulus such as a digit or a picture. The individual empirical measures are then derived from different conditions or trial sequences (for a review, see Declerck & Philipp, Reference Declerck and Philipp2015).
First, mixing costs represent the performance difference (in terms of RT and accuracy) between pure language blocks, in which only one language is relevant, and mixed language blocks, in which two languages are relevant. To differentiate between mixing costs and switch costs (see below), pure language blocks are compared to repeat trials of the mixed language blocks only – that is, to trials in which the relevant language repeats from one trial to the next (cf. Los, Reference Los1996). Mixing costs are observed due to longer reaction times in mixed blocks than in pure blocks and can be interpreted as reflecting a relatively global, sustained consequence of between-language interference and language control in blocks in which more than one language is potentially relevant (cf. Philipp, Kalinich, Koch & Schubotz, Reference Philipp, Kalinich, Koch and Schubotz2008; Rubin & Meiran, Reference Rubin and Meiran2005, for a similar argument with respect to task switching). Mixing costs as a marker of language control have been observed in several studies (e.g., Christoffels, Firk & Schiller, Reference Christoffels, Firk and Schiller2007; Wang, Kuhl, Chen & Dong, Reference Wang, Kuhl, Chen and Dong2009). For example, Christoffels and colleagues (Reference Christoffels, Firk and Schiller2007) reported mixing costs, which were larger for the dominant language (L1). The authors attributed this finding to a sustained mechanism of language control that reduces the overall performance of the dominant language and that comes into play whenever participants switch between one dominant and one non-dominant language.
Second, language-switch costs in mixed language blocks are measured as the performance difference between repeat trials, in which the relevant language is the same as in the previous trial, and switch trials, in which the relevant language is different from the one in the previous trial. Previous studies have found longer reaction times and higher error rates in switch trials than repeat trials, and these “switch costs” have been replicated often, particularly for unbalanced bilinguals (for reviews, see e.g., Bobb & Wodniecka, Reference Bobb and Wodniecka2013; Kiesel, Steinhauser, Wendt, Falkenstein, Jost, Philipp & Koch, Reference Kiesel, Steinhauser, Wendt, Falkenstein, Jost, Philipp and Koch2010; Kroll, Bobb, Misra & Guo, Reference Kroll, Bobb, Misra and Guo2008). Switch costs are often interpreted as the result of overcoming the inhibition from the previous trial (e.g., Green, Reference Green1986, Reference Green1998; Meuter & Allport, Reference Meuter and Allport1999; Kroll et al., Reference Kroll, Bobb, Misra and Guo2008; Philipp & Koch, Reference Philipp and Koch2009; see also Koch, Gade, Schuch & Philipp, Reference Koch, Gade, Schuch and Philipp2010). Inhibition models (see Green, Reference Green1986, Reference Green1998) assume that in any given trial, parallel bilingual lexical activation occurs first and is followed by the inhibition of the incorrect lexeme or language in order to allow the production of the correct lexeme in the correct language. In a repeat trial, the relevant language of the current trial (n) is the same as that of the previous trial (n-1): the active language remains active and the inhibited language remains inhibited. In a switch trial (n), the correct language is different from the correct language of the previous trial (n-1): the active language from the previous trial must be inhibited, and the inhibited language must be activated, in order for the current trial to be produced correctly.
Finally, and most important for the present study, a third empirical measure of language control is dual-response costs. Dual-response costs can be measured as the performance difference between dual-response trials, in which two responses are performed at the same time, and single-response trials, in which each of the responses is performed in isolation (cf. dual-task costs, Pashler, Reference Pashler1994). Importantly, dual-response costs can be assessed in pure-language blocks by comparing pure blocks of single-response trials to pure blocks of dual-response trials, as has been done in previous studies (e.g., Emmorey, Petrich & Gollan, Reference Emmorey, Petrich and Gollan2012). However, it is also possible to assess dual-response costs within mixed-language blocks by comparing single-response trials and dual-response trials, as we have done in the present study. In pure-language blocks, dual-response costs represent mainly resource limitations and the adverse effects of crosstalk when processing two things at the same time (e.g., Pashler, Reference Pashler1994), whereas dual-response costs in mixed language blocks are additionally affected by inhibition acting on single-response trials and dual-response trials.
Language blending: Two combined responses or one joint response?
Many previous studies of dual-responses have used the dual-task paradigm and found dual-task costs for various contexts other than bimodal bilingual designs (for a review, see Pashler, Reference Pashler1994; Welford, Reference Welford1952). In the dual-task paradigm, responses from single-task blocks are compared to responses from dual-task blocks. For example, in the study by Strobach, Liepelt, Pashler, Frensch and Schubert (Reference Strobach, Liepelt, Pashler, Frensch and Schubert2013), participants had to respond either vocally to the frequency of a tone or else manually to the location of a visual stimulus in two separate single-task blocks. Both tasks were then combined in dual-task blocks so that – in each trial – participants had to respond to the auditory and the visual stimulus. For both component tasks (i.e., the auditory-vocal one and the visual-manual one), performance was worse in dual-task blocks than in single-task blocks, indicating dual-task costs. The extent of the dual-task costs, i.e., their reduction, depends on the temporal distance between the two stimuli in the dual-task blocks (stimulus-onset asynchrony, Pashler, Reference Pashler1994) but also on the degree of interference between tasks due to, for example, response-code conflict (e.g., Logan & Schulkind, Reference Logan and Schulkind2000; Navon & Miller, Reference Navon and Miller1987). Dual-task costs have also been reduced or eliminated through setting task priorities or extensive practice (e.g., Hazeltine, Teague & Ivry, Reference Hazeltine, Teague and Ivry2002; Schumacher, Seymour, Glass, Fencsik, Lauber, Kieras & Meyer, Reference Schumacher, Seymour, Glass, Fencsik, Lauber, Kieras and Meyer2001; Strobach et al., Reference Strobach, Liepelt, Pashler, Frensch and Schubert2013).
In contrast to dual-task experiments, in which two distinct responses are given to two distinct stimuli, there are also experiments in which one stimulus requires two compatible, redundant responses; these can be referred to as dual-response experiments (Fagot & Pashler, Reference Fagot and Pashler1992; Holender, Reference Holender, Stelmach and Requin1980; Huestegge & Koch, Reference Huestegge and Koch2009, Reference Bobb and Wodniecka2013, Reference Huestegge and Koch2014). Dual-response costs are calculated in the same way as dual-task costs, that is, as the performance difference between dual-response trials and single-response trials. For example, in the study by Huestegge and Koch (Reference Huestegge and Koch2014), participants had to respond to a visual stimulus with either a saccade to the right vs. left or else with a right vs. left key press in single-response trials. In dual-response trials, both a saccade and a key press had to be performed in response to a single visual stimulus. So the key design difference between dual-task and dual-response experiments is the stimulus and the redundancy of the responses (i.e., both responses have the same meaning, like a leftwards saccade and a left key press). The distinction may seem minor, but Fagot and Pashler (Reference Fagot and Pashler1992) show that in order to produce a redundant response, conjoint response selection occurs, and they rule out two separate response selections, as would be the case for dual-tasks.
For a study of bimodal bilingual Blends, in which the word and the sign are semantically redundant, a dual-response design is appropriate rather than a dual-task design. A recent study (Emmorey et al., Reference Emmorey, Petrich and Gollan2012) examined bimodal bilinguals using a dual-response paradigm, which they described as dual-task. In this picture-naming study, single-response pure blocks of English and American Sign Language (ASL) were compared to dual-response pure blocks of word-sign Blends. The data pattern showed a lack of dual-response costs across pure blocks for manual ASL production for RTs and error rates. Also, examining the results for frequency effects, the authors found a dual-response advantage in error rates for low-frequency signs. For English vocal responses, they found a dual-response cost. However, the authors conclude that this cost is a result of timing spoken words to sign production – that is, a coordination cost or response grouping, rather than a reflection of processing costs. These conclusions will be reexamined in the context of previous dual-response studies in a non-language context.
A typical finding of these studies is asymmetric dual-task or dual-response costs, with the slower response incurring larger costs. The theoretical interpretation of this robust finding is that the faster response must be processed before the slower response. However, on the one hand, this interpretation is at odds with the results reported by Emmorey et al. (Reference Emmorey, Petrich and Gollan2012) because vocal responses (showing dual-response costs) were faster than manual responses (showing no dual-task costs). On the other hand, this interpretation was also challenged by a recent study from Huestegge and Koch (Reference Huestegge and Koch2013). The authors examined dual-response costs by using vocal, manual, and saccade responses in both single-response trials and dual-response trials (with all three possible pairings tested). The authors found that rather than the slower response per se incurring larger costs, task-set context or modality dominance is the key. The authors observed that the very same vocal responses – although the slower modality in both combinations – showed a dual-response cost when combined with a saccade, but not when combined with a manual response (i.e., saccades dominate vocal responses and vocal responses dominate manual responses).
Additionally, a recent paper (Huestegge & Koch, Reference Huestegge and Koch2014) challenged even the assumption that there are always dual-response costs. The authors hypothesized that a situation in which one response is highly automatic and the other is not could lead to the highly automatic response being produced erroneously on single-response trials for the less automatic response. They showed that saccades, a highly automatic response, are performed erroneously on single-task manual trials, leading to very high error rates. In this case, manual response performance suffers more interference as a single-response than as a dual-response. This is a rare case of a dual-response benefit, with lower error rates (though not faster RTs) for dual-responses as compared to single-responses. Thus, in terms of bimodal bilingual dual-responses, the question is whether either of the components of a Blend can qualify as highly automatic in relation to the other and whether there is a modality dominance in a Blend.
With respect to bimodal bilinguals, producing a sign in a sign language and speaking a word can each be considered a single-response, whereas a Blend represents a dual-response in the dual-response paradigm. The two responses are both triggered by one stimulus, and in terms of semantics, they are redundant, as are the vocal and manual responses used in previous studies (e.g., Huestegge & Koch, Reference Huestegge and Koch2013; Reference Huestegge and Koch2014). However, the non-language manual task of pressing a button labeled with a response that is redundant to a spoken word (as in Fagot & Pashler, Reference Fagot and Pashler1992; Holender, Reference Holender, Stelmach and Requin1980; Huestegge & Koch, Reference Huestegge and Koch2013) is very clearly different from the task of producing a sign since sign production requires lexical retrieval. With responses semantically redundant but drawn from two separate lexica in two different languages, it is unclear whether the Blend will produce the same pattern of results as other dual-response tasks across modalities. To this end, we will present the data from the present study and reassess the data from the previous bimodal dual-response study (Emmorey et al., Reference Emmorey, Petrich and Gollan2012).
Aim of the present study
The present study was designed to assess the nature of the Blend: whether a Blend is a combination of two responses or rather just one response which is different from an additive collection of two single responses which each have their own functions and which are timed together in production (cf. Ulrich & Miller, Reference Ulrich and Miller2008). To this end, we examined the language control mechanisms underlying costs in sequential and simultaneous bimodal language blending. We examined mixing costs, switch costs, and dual-response costs (in both pure and mixed blocks) within the same participants. More specifically, we were interested in how these very specific responses are controlled in bimodal bilingual language production.
The present study examines both pure blocks (in which either German vocal responses, DGS manual responses, or Blends were required throughout each block) and mixed blocks, in which participants had to switch between these three Target-responses. Therefore, for the first time, we can measure mixing costs for both single-response trials (i.e., German responses and DGS responses) and dual-response trials (i.e., Blends). Further, we can systematically explore bimodal language switching by comparing Target-response repeat trials with Target-response switch trials, with switches between single-responses (e.g., from German to DGS) and from or into dual-response Blends (producing two languages at the same time). Finally, we are also able to examine dual-response costs in both pure language blocks (cf. Emmorey et al., Reference Emmorey, Petrich and Gollan2012) and mixed language blocks.
Moreover, by combining language switching and a dual-response design, we extend the results of the previous bimodal dual-response study (Emmorey et al., Reference Emmorey, Petrich and Gollan2012) by 1) addressing the nature of a Blend beyond response grouping, and 2) including a mixed block, which allows for the assessment of mixing costs and switch costs in addition to dual-response costs. Additionally, we extend the results of the non-language study which found a dual-response benefit for responses in which one component was highly automatic (Huestegge & Koch, Reference Huestegge and Koch2014). That study used mixed blocks with only switch trials for the three responses (saccade, manual button press, both). The present study goes beyond this by including both pure blocks and repeat trials, which allow for the assessment of mixing costs and switch costs.
Our specific hypotheses are as follows.
-
1. Mixing costs: Comparing our pure blocks with our mixed blocks, we anticipated mixing costs for German responses, DGS responses and Blends, i.e., we anticipated that performance would be better in pure blocks than in mixed blocks.
-
2. Language switching with dual-response trials: Examining the mixed blocks, we anticipated language-switch costs for both vocal and manual responses. If a Blend is the same as any of the single-responses, switching from and into a Blend should not be different than switching from German to DGS or vice versa. If a Blend is a simple combination of two responses (while keeping their characteristics), we should observe no (or very small) switch costs for switching from or into a Blend since one of the response modalities is repeated.
-
3. Dual-response costs in pure and mixed blocks: We anticipated that for vocal responses, there would be a dual-response cost, that is, RTs would be longer in pure dual-response (Blend) blocks than in pure single-response (German) blocks. For manual responses, we anticipated no dual-response costs (cf. Emmorey et al., Reference Emmorey, Petrich and Gollan2012). Observing dual-response costs would clearly support the assumption that a Blend is a dual-response trial and, thus, different from a single (joint) response.
Method
Participants
Twelve participants took part in this experiment, all hearing native speakers of German (age 22–26; 11 women, 1 man) who had completed 7–9 semesters of DGS instruction in the Deaf Education program at the University of Cologne. No participants were native signers. For these participants, DGS is very clearly the non-dominant language since all of them acquired spoken language, and no sign language, naturally as their first language. Thus, all participants can be considered as unbalanced bilinguals (for differences in language control between highly proficient and unbalanced bilinguals, see the review in Declerck & Philipp, Reference Declerck and Philipp2015).
Unfortunately, no standard measures of German Sign Language L2 competence analogous to the widely-available English or German competence tests exist, so we are unable to report standard measures. However, participants’ competence level enabled them to conduct conversations about familiar topics and engage in everyday communication with Deaf interlocutors but not to participate in academic or scientific discussions in DGS. This competence level was assessed by one of the authors, who is fluent in DGS, as well as the sign language program coordinator.
Task and procedure
The experiment was programmed in Presentation on a 15.4″ Lenovo laptop with a screen resolution of 1280 × 800 pixels. An external microphone was used.
The task throughout the experiment was picture naming. Stimuli were ten line drawings of common objects: chair, dolphin, door, egg, trash,Footnote 1 mountain, pitcher, scissors, suitcase, worm. We used the German and German Sign Language translations of these words as target words/signs. The stimuli images measured 300 × 300 pixels and were taken from the International Picture-Naming Project database (Szekely, Jacobsen, D'Amico, Devescovi, Andonova, Herron, Lu, Pechmann, Pléh, Wicha, Federmeier, Gerdjikova, Gutierrez, Hung, Hsu, Iyer, Kohnert, Mehotcheva, Orozco-Figueroa, Tzeng, Tzeng, Arévalo, Vargha, Butler, Buffington & Bates, Reference Szekely, Jacobsen, D'Amico, Devescovi, Andonova, Herron, Lu, Pechmann, Pléh, Wicha, Federmeier, Gerdjikova, Gutierrez, Hung, Hsu, Iyer, Kohnert, Mehotcheva, Orozco-Figueroa, Tzeng, Tzeng, Arévalo, Vargha, Butler, Buffington and Bates2004). In stimulus selection for this study, we took measures to ensure that all manual responses could be directly compared to each other. This was done by choosing as stimuli DGS signs that all have the same place of articulation: the neutral signing space, which is where the homekey was located relative to the seated participant. This measure minimized and standardized the lag between homekey release and sign onset for all signs and ensured that no individual sign led to RTs that skew significantly shorter or longer than RTs for other signs.
There were three possible responses to each stimulus: German, DGS and Blend, which we label as Target-response. German and DGS trials were both single-responses, while Blend trials were dual-responses and included both the German word and the DGS sign. For a schematic of the blocks in this study, see Figure 1. There were two categories of blocks in this experiment, which we label Block-type: pure blocks and mixed blocks. There were three pure blocks, in which one Target-response (German, DGS, or Blend) was produced throughout, with one block for each Target-response. In the other Block-type, i.e., mixed blocks, participants switched between the three Target-responses (German, DGS, or Blend), with all trials either being a repeat trial (same Target-response as the previous trial) or a switch trial (different Target-response from the previous trial).
Design of the experiment. The abbreviations indicate RT = response time, %Error = Percentage of errors, v = Vocal response, m = manual response, Ger = German, DGS = German Sign Language, Bl = Blend.
Target-response cues for single-response trials were squares in solid primary colors (red, blue, yellow) measuring 400 × 400 pixels. Task cues for dual-response trials were a 2 × 2 checkerboard of the two relevant language cue colors, measuring 400 × 400 pixels in total. Two of the three colors were selected for each participant, and the selection of colors and the mapping of cue color to language were counterbalanced across participants.Footnote 2
Reaction times for vocal responses in German and Blends were registered using a voicekey (voice onset), recorded by the software, with response registration triggered by the voice surpassing a sound threshold. Reaction times for manual responses for DGS and Blends were registered using a homekey (motion onset). Participants were instructed to keep the homekey pressed during pure DGS, pure Blend, and mixed blocks and to release it only in order to produce a DGS sign. This setup has been used to measure RT in various other studies involving sign language (e.g., Emmorey et al., Reference Emmorey, Petrich and Gollan2012; Myers, Lee & Tsay, Reference Myers, Lee and Tsay2005), as well as other manual responses (e.g., Barratt, Reference Barratt1967; Massen & Prinz, Reference Massen and Prinz2007). Errors for vocal and manual responses were recorded by the experimenter.
The experiment lasted approximately 45–50 minutes, with 5–10 minutes of instruction and training, 30–35 minutes for the experiment itself, and 5–10 minutes for a brief questionnaire and debriefing afterwards. The training was a short mock experiment with 4 blocks of 10 trials (one for each of the three pure blocks and a mixed block), in which participants were familiarized with the cue and stimuli images, the pace of the experiment, and the response procedure for vocal, manual and Blend responses. There were a total of 8 experimental blocks in this study. For pure blocks, there was one block of 70 trials for each Target-response. For the mixed blocks, there were five consecutive blocks of 90 trials. Each mixed block had 30 trials with each Target-response, so there were 150 total instances of each Target-response in mixed blocks. Approximately half (~75 trials) were switch trials and half were repeat trials. The pure blocks should not lead to errors, as they consisted of repeating the same 10 responses a total of 7 times each with the same Target-response. However, we anticipated errors in the mixed blocks. Excluding the errors from the RT analysis would leave perhaps 70 repeat trials and 70 switch trials for each Target-response. Thus the number of trials for each Target-response is comparable for pure blocks (70 trials) and repeat trials in mixed blocks (~75 trials minus errors), which are the relevant trials to calculate mixing costs, as well as for repeat and switch trials in mixed blocks, which are the relevant trials to calculate switch costs.
The trial order was such that each stimulus item was named an equal number of times with each Target-response (German, DGS, Blend) for all participants. The trial sequence was designed so that there were no more than 4 consecutive trials of the same Target-response (i.e., up to three consecutive repeats of the same Target-response) and a roughly equal number of repeat and switch trials. The sequence of blocks was counterbalanced across participants using half of a Latin square, and we controlled the sequences so that each block appeared equally often at each sequence position (see Appendix A).
Analyzing vocal and manual responses separately
In analyzing our RT data, vocal and manual responses were analyzed separately in order to avoid confounds. Due to timing differences in language production between languages of different modalities, it may not be possible to accurately compare RTs between signed and spoken languages directly. This issue is not only methodological; there are foundational theoretical questions involved in the issue of whether vocal and manual language responses are temporally compatible and therefore whether they can be directly compared in an RT analysis, or whether doing so would compromise the internal validity of the RTs. This question remains even if one can be entirely sure that in a particular experiment there is no lag between homekey release and sign onset, which is not the case in the present study despite the measures taken in stimuli selection, or if post-hoc measures are taken to compensate for this lag.
There are great differences in language production across language modalities, with the production of individual signs slower across the board than the production of individual words (Bellugi & Fischer, Reference Bellugi and Fischer1979; see also Brentari, Reference Brentari, Meier, Cormier and Quinto-Pozos2002). This disparity has to do with physical differences between the primary articulators, with the vocal tract moving at a much faster rate than the hands, and it is tied to the greater degree of simultaneity in signed languages as compared to the sequentiality of spoken languages (Bellugi & Fischer, Reference Bellugi and Fischer1979). More importantly here, the preparatory stage, in which the articulators are brought into position for word or sign production, is much longer for sign language than spoken language: it takes the hands much longer to reach onset position than the vocal articulators (Myers et al., Reference Myers, Lee and Tsay2005). So even if both sign and word onset coincide with the respective RT measures, it is not clear that sign onset is temporally comparable to word onset in vocal language production (ibid.). For this reason, it is prudent to perform separate analyses of manual and vocal responses, at least until more is known about word and sign onset timing. As a final note, it is also clear that the vocal response registration of a volume threshold makes vocal RTs compatible with each other, but it should be noted that the sound threshold usually does not coincide with word onset.
Design
An overview of the different blocks is provided in Table 1 and Figure 1.
Schematic of the blocks in the present study. This schematic presents an example of how one participant would perform: first the pure German Sign Language (DGS) block, then the pure German block, then the pure Blend block, and finally the mixed blocks. All uppercase letters indicates a DGS sign, while all lowercase letters indicates a German word. Blends are noted with the sign and the word. It is important to note that the stimuli used in the experiment were images, so the written representations in this schematic are not related to participants’ performance.
In a first analysis, the RTs in pure blocks were examined in isolation in order to examine dual-response costs across pure blocks. This analysis determined whether our results from the pure blocks replicate the results of the previous bimodal dual-response study (Emmorey et al., Reference Emmorey, Petrich and Gollan2012), which included only pure blocks. The within-subject independent variable was Response-type (single-response vs. dual-response), and the dependent variables were vocal RT and manual RT. There was no error analysis for pure blocks because the task was the same throughout each block and consequently hardly any errors occurred.
In a second analysis, we assessed mixing costs; we again analyzed only RTs since there were no meaningful errors in the pure blocks. The within-subject independent variables were Block-type (pure vs. mixed) and Response-type (single-response vs. dual-response), and the dependent variables were vocal RT and manual RT.
A third analysis examined the mixed blocks in isolation in order to explore switch costs. For the vocal and manual RT analyses, the within-subject independent variables were Response-type (single-response vs. dual-response) and Shift (repeat vs. switch). However, in this analysis, “switch” is actually a pooling of two transitions; for example, a switch into a manual DGS trial pools DGS trials in which the previous trial was German with those in which the previous trial was a Blend (see Table 2). In order to examine individual effects between these Transitions (from-German, from-DGS, from-Blend), we performed additional analyses.
Schematic of Transitions within the mixed blocks and their relation to the labels corresponding to Response-type for the RT analyses. All uppercase letters indicates a German Sign Language (DGS) sign, while all lowercase letters indicates a German word. Blends are noted with the sign and the word. The Transition indicates the Target-response from the previous trial, with nine possible constellations (labeled as T1-T9). The differences between Transitions and the labels corresponding to Response-type reflect the separate analyses of vocal and manual responses required for Response-type, which are not relevant for Transitions. For example, in Trial 5, the Target-response is Blend, Transition is from-German because the previous trial was German, and the corresponding Response-type labels are vBl_switch & mBl_switch, which include the vocal and manual responses of the Blend and do not distinguish between from-German and from-DGS.
Finally, we also analyzed error rates within the mixed blocks. For the error analysis, vocal and manual responses can be compared directly and both components of a Blend can be assessed together. The within-subject independent variables were therefore Target-response (German vs. DGS vs. Blend) and Shift (repeat vs. switch). Following the interaction analysis, further analyses were again carried out in order to examine individual effects between Transitions for each Target-response.
Results
The first two trials of each block were excluded from the analysis for both reaction times and error rates. In order to remove outliers from the analyses, approximately 1% of all trials were excluded, half on either end of the data spread, separately for vocal and manual responses, pure and mixed blocks, single-response and dual-response trials, and within the mixed blocks, repeat and switch trials. For the RT analysis, trials in which the participant made an error and the subsequent trial were excluded. Also, trials in which a technical problem occurred were excluded. Examples of technical problems are the voice key not working properly, the participant saying “um” before the target word, or the homekey registering two responses, presumably due to hesitation on a manual response. Technical errors made up 1.3% of all trials, with 0.3% in DGS trials, 0.8% in German trials, and 0.2% in Blend trials. Mean reaction times are presented in Table 3.
Mean vocal and manual RTs | SDs (ms) across Block-types (pure vs. mixed), by Response-type (single-response vs. dual-response). For vocal RTs, single-response means German responses and dual-response means vocal responses in Blends. For manual RTs, single-response means DGS responses and dual-response means manual responses in Blends.
Reaction times
Dual-response costs in pure blocks
In a first analysis, in order to determine whether our data patterned with the results of the previous bimodal dual-response study (Emmorey et al., Reference Emmorey, Petrich and Gollan2012), which found a dual-response cost across pure blocks for vocal responses and no dual-response cost across pure blocks for manual responses, we tested the pure blocks in isolation. RTs for vocal and manual responses were analyzed as a function of whether the response was made in the context of a dual-response or single-response pure block.
For vocal responses, we conducted a two-tailed paired t-test. The t-test revealed significantly longer RTs for dual-response blocks than single-response blocks, t(11) = 7.908; p < .001. For manual responses, we conducted a two-tailed paired t-test. RTs in dual-response blocks were numerically shorter than in single-response blocks, but, as anticipated, the difference was not statistically significant, t(11) = 1.228; p > .05. So, for pure blocks in isolation, we found a dual-response cost for vocal responses and none for manual responses. Thus, our results pattern with those of the previous bimodal dual-response study (Emmorey et al., Reference Emmorey, Petrich and Gollan2012).
Mixing costs
In a second analysis, in order to assess mixing costs, we compared RTs across Block-types, comparing responses from our pure blocks to repeat trials from our mixed blocks. We expected shorter RTs in pure blocks than in (repeat trials of) mixed blocks (i.e., mixing costs). The data are presented in Figure 2.
RTs (mixed blocks: repeat trials only), with vocal and manual responses separate, as a function of Response-type (single-response vs. dual-response) and Block-type (pure vs. mixed). Error bars represent standard errors.
For vocal responses, we conducted a Block-type (pure vs. mixed) × Response-type (single-response vs. dual-response) analysis of variance (ANOVA). The main effect of Block-type was marginally significant, F(1, 11) = 4.112; p = .067; ηp 2 = .272. RTs in pure blocks were numerically shorter than RTs in mixed blocks. The interaction between Block-type and Response-type was not significant, F(1, 11) = .445; p > .05; ηp 2 = .039, though mixing costs were numerically smaller for single-response trials (20 ms) than for dual-response trials (30 ms). Finally, as in the first analysis, there was a significant main effect of Response-type, F(1, 11) = 97.554; p < .001; ηp 2 = .899, indicating dual-response costs.
For manual responses, we also conducted a Block-type (pure vs. mixed) × Response-type (single-response vs. dual-response) ANOVA. There was a significant main effect of Block-type, F(1, 11) = 8.941; p < .05; ηp 2 = .448, with RTs in the pure blocks shorter than those in the mixed blocks (i.e., mixing costs). Interestingly, the interaction between Response-type and Block-type was also significant, F(1, 11) = 5.496; p < .05; ηp 2 =.333, with smaller mixing costs for single-response trials (20 ms) than for dual-response trials (48 ms) for manual RTs. As in the first analysis, there was no significant main effect of Response-type, F(1, 11) =.363; p > .05; ηp 2 = .032 (i.e., no indication of overall dual-response costs for manual responses).
As regards mixing costs, RTs in the pure blocks were numerically shorter than RTs in the mixed blocks for both vocal and manual responses (for manual response the difference was significant), which is the expected pattern of mixing costs. Mixing costs are generally a robust finding (cf. Los, Reference Los1996; for a review, see Kiesel et al., Reference Kiesel, Steinhauser, Wendt, Falkenstein, Jost, Philipp and Koch2010), and the results from the present study show that in a bimodal design, mixing costs can be observed. Additionally, mixing costs were larger (at least numerically for vocal responses, and significantly for manual responses) for dual-response trials than for single-response trials. This theoretically interesting result will be discussed below.
Finally, across pure and mixed blocks, vocal RTs in single-response trials were shorter than in dual-response trials, while there was no difference for manual RTs. This result mirrors that of the first analysis; that is, we found dual-response costs for vocal responses but not for manual responses.
Language switching with dual-response trials
In a third set of analyses, we examined the mixed blocks in isolation. The RT data are presented in Figure 2. For vocal responses, we conducted a Response-type (single-response vs. dual-response) × Shift (repeat vs. switch) ANOVA. There were significant main effects of Response-type, F(1, 11) = 122.32; p < .001; ηp 2 = .917, indicating dual-response costs, and Shift, F(1, 11) = 10.275; p < .01; ηp 2 = .483, indicating switch costs. However, there was basically no difference in switch costs between single-response trials (35 ms) and dual-response trials (36 ms), and the interaction between Response-type and Shift was accordingly not significant, F(1, 11) = .011; p > .05; ηp 2 = .001.
For manual responses, we conducted the same Response-type (single-response vs. dual-response) × Shift (repeat vs. switch) ANOVA. There was no significant main effect of Response-type, F(1, 11) = .476; p > .05; ηp 2 = .041, i.e., no dual-response costs. But there was a significant main effect of Shift, F(1, 11) = 14.8; p < .01; ηp 2 = .574, demonstrating switch costs. Importantly, the interaction between Response-type and Shift was also significant, F(1, 11) = 5.883; p < .05; ηp 2 = .348, with larger switch costs for single-response trials (56 ms) than for dual-response trials (33 ms).
So, within the mixed blocks, for vocal responses, we found longer RTs for dual-response trials than single-response trials; this pattern fits in with the previous analyses of the pure blocks and mixing costs. Additionally, RTs for repeat trials were shorter than RTs for switch trials, meaning that switch costs were found. However, there was no difference in switch costs between single-response and dual-response trials, meaning that switch costs were the same for German responses alone and German responses in a Blend, so there was no dual-response advantage and no disadvantage. For manual responses, we found no overall difference in RTs between single-response and dual-response trials, meaning there were no overall dual-response costs; this pattern also fits in with the previous analyses. RTs for repeat trials were shorter than RTs for switch trials, i.e., there were switch costs. Switch costs for manual responses in dual-response Blend trials were significantly smaller than in single-response DGS trials. In this case, the smaller switch cost for dual-response trials represents a dual-response advantage.
In the previous set of analyses, switch trials were always averaged across two different types of switch trials (i.e., two different transitions; see Table 2). In order to examine individual effects of each transition in RTs, taking single-responses and dual-responses separately, we compared the three transitions (from-German, from-DGS, from-Blend) for both vocal German and manual DGS responses separately; see Figure 3.
Left: RTs, with vocal and manual responses separate, as a function of Response-type (single-response vs. dual-response) and Shift (repeat vs. switch) in mixed blocks only. Top right: All Transitions (from-German, from-DGS, from-Blend) for vocal responses, separated into single-response and dual-response trials. Bottom right: All Transitions for manual responses, separated into single-response and dual-response trials. Error bars represent standard errors. Note: *p < .05; **p < .01; ***p < .001.
For single-response responses, we expected shorter RTs in repetition trials than in switch trials (i.e., switch costs) as well as shorter RTs when switching from a single-response trial into a single-response trial than when switching from a dual-response trial into a single-response trial due to the inhibition of the relevant response component. Thus, we conducted 3 one-tailed paired t-tests each for vocal responses (German from-German vs. German from-DGS; German from-German vs. German from-Blend; German from-DGS vs. German from-Blend) and manual responses (DGS from-DGS vs. DGS from-German; DGS from-DGS vs. DGS from-Blend; DGS from-German vs. DGS from-Blend).
All differences between all measures were statistically significant, ts > 1.8; ps < .05. More specifically, repeat trials incurred shorter RTs than all other trial types. Single-response to single-response switch trials (DGS from-German, German from-DGS) showed longer RTs than the corresponding repeat trials and shorter RTs than the corresponding from-Blend trials. Most importantly, from-Blend trials incurred the longest RTs. Taken together, the data support the assumption that repeat trials are easiest, while switching from a Blend to a single-response is most difficult. It seems that inhibiting a Blend from the previous trial in order to produce one of its component responses as a single-response is more costly than inhibiting one single-response in order to produce a different single-response.
For dual-responses, we expected the shortest RTs for repeat trials (i.e., Blend from-Blend) and longer RTs for Blend from-DGS and Blend from-German, but no difference between these trials since we see no theoretical basis for a difference between Blend from-DGS trials and Blend from-German trials. Thus, we conducted 2 one-tailed paired t-tests (Blend from-Blend vs. Blend from-German; Blend from-Blend vs. Blend from-DGS) as well as 1 two-tailed t-test (Blend from-German vs. Blend from-DGS) for vocal and manual responses separately.
For dual-responses, the test results support our hypotheses for both vocal and manual RTs. Repeat trials incurred shorter RTs than all other trials, ts(11) > 2; ps < .05. Also, we found no difference between Blend from-German and Blend from-DGS for dual-response trials for either vocal or manual responses, ts ≤ 1.3; ps > .2. So, for dual-response trials, it matters only whether or not the previous trial is also dual-response, and not, in the case of switch trials, which single-response was performed in the previous trial.
Error rates from the mixed blocks
Finally, we examined error rates from the mixed blocks. Error rates are presented in Table 4. The error rate analysis is advantageous in that it avoids confounds due to issues of timing and differences in language production processes across language modalities as exist for RT analyses. Since the three Target-responses can be compared to each other directly on the measure of error rates, we conducted a Target-response (German vs. DGS vs. Blend) × Shift (repeat vs. switch) ANOVA. As Target-response is a variable with three levels, we report ε values when different from 1.0 and use the Huynh-Feldt test to report p values based on corrected degrees of freedom. However, we still report non-corrected degrees of freedom. The data are presented in Figure 4.
Mean error rates | SDs (%) by Target-response (German, DGS, Blend) and Shift (repeat vs. switch)
Top: Error rates in mixed blocks by Target-response (German vs. DGS vs. Blend) as a function of Shift (repeat vs. switch). Bottom: Error rates in mixed blocks split by Target-response in the current trial (German vs. DGS vs. Blend) as a function of the previous trial (Transition: from-German vs. from-DGS vs. from-Blend). Error bars represent standard errors. Note: *p < .05; **p < .01; ***p < .001.
There was a significant main effect of Target-response, F(2, 22) = 14.266; p < .01; ηp 2 = .698, and Shift, F(2, 22) = 41.620; p < .001; ηp 2 = .791, and importantly, the interaction between Target-response and Shift was also significant, F(2, 22) = 8.279; p < .05; ε = .737; ηp 2 = .580. In order to assess switch costs pairwise, we conducted three additional 2 × 2 ANOVAs. For Blend and German trials, switch costs in Blend trials (0.5%) were significantly smaller than in German trials (4.9%; F(1, 11) = 11.528; p < .01; ηp 2 = .512). The difference in switch costs between Blend trials (0.5%) and DGS trials (2.3%) was also significant (F(1, 11) = 8.901; p < .05; ηp 2 = .447 for the corresponding interaction). The difference in switch costs between German and DGS trials was only marginally significant (F(1, 11) = 4.471; p = .058; ηp 2 = .289 for the corresponding interaction). This result demonstrates a clear advantage for dual-response Blends for error rates in the mixed blocks. In the results of all three of the post-hoc 2 × 2 ANOVAs, there were significant main effects of both Target-response and Shift as well (Fs > 4.9; ps < .05), indicating that error rates were significantly smaller in Blend trials as compared to single-response DGS or German trials and that error rates were higher in switch than in repeat trials.
In order to examine individual effects of each transition, we compared the three transitions within each Target-response (see Figure 4). That is, for German, DGS and Blend Target-responses, we conducted 3 two-tailed paired t-tests each.
For the Target-response German, the transition with the numerically highest error rate was German from-Blend (7.0%), followed by German from-DGS (3.7%), and the lowest error rate was for the repeat German from-German (0.5%). All three tests between the three transitions revealed a significant difference in error rate, ts(11) > 2.6; ps < .05.
For the Target-response DGS, the numerical data pattern was the same, with the highest error rate in the Transition DGS from-Blend (4.8%), followed by DGS from-German (1.5%) and DGS from-DGS (0.7%). There was a significant difference between DGS from-Blend and DGS from-DGS, t(11) = 5.742; p < .001, as well as between DGS from-Blend and DGS from-German, t(11) = 2.980; p < .05. However, there was no significant difference between DGS from-German and DGS from-DGS, t(11) = 1.131; p > .05.
For the Target-response Blend, all error rates were low. There were no significant differences between any of the Transitions (two-tailed: t < 1.6; p > .13).
Taking both the single-response Target-responses (i.e., German and DGS) together, the data pattern demonstrates a significantly higher error rate when switching from a Blend to a single-response trial than when switching from one single-response trial to another or when repeating the same Target-response. This clearly shows a specific effect due to performing both a German vocal response and a DGS manual response simultaneously in the previous trial. Additionally, in terms of error rates, for all Target-responses taken together, we found significantly lower error rates and significantly smaller switch costs for dual-response Blend trials than for single-response trials, indicating a clear dual-response advantage for errors in this bimodal switching design.
Discussion
In the present study, we were mainly interested in potential differences in language control between unimodal and bimodal bilingualism. The reason for assuming differences in language control is the fact that simultaneous production in two languages is possible in bimodal but not in unimodal bilingualism. This specific feature and the experience that bimodal bilinguals have in the simultaneous combination of their two languages might result in a different form of language control as compared to unimodal bilinguals.
This argument is similar to the reasoning that explains a general bilingual advantage (i.e., better performance in bilinguals than in monolinguals). Some studies have found an advantage for unimodal bilinguals as compared to monolinguals in tasks utilizing non-language executive control, particularly in adults in middle age or older (e.g., Bialystok, Craik, Klein & Viswanathan, Reference Bialystok, Craik, Klein and Viswanathan2004; Costa, Hernández & Sebastián-Gallés, Reference Costa, Hernández and Sebastián-Gallés2008; Prior & MacWhinney, Reference Prior and MacWhinney2010; for reviews, see Bialystok, Craik, Green & Gollan, Reference Bialystok, Craik, Green and Gollan2009; Bialystok, Reference Bialystok2011). This bilingual advantage is also attributed to the life-long experience that unimodal bilinguals have with handling two languages and, thus, provides evidence that experience in one specific situation might transfer to language control or even cognitive control in general.
Particularly relevant in the context of the present study is a previous study that examined bimodal bilingualism, comparing executive control in monolinguals, unimodal bilinguals and bimodal bilinguals using a flanker task (Emmorey, Luk, Pyers & Bialystok, Reference Emmorey, Luk, Pyers and Bialystok2008b). This study found advantages in executive control for unimodal bilinguals, but not for bimodal bilinguals. The authors suggest that the performance advantage for unimodal bilinguals stems not from bilingualism itself, but from experience in switching between languages, which leads to increased inhibitory control. They suggest that the reason this advantage is not present in bimodal bilinguals is that they are not forced to switch between their languages sequentially, preferring to blend simultaneously – which might lead to a different form of language control, resulting in different advantages and/or disadvantages.
Although bimodal bilinguals can perform sequential language switching, the simultaneous execution of both a spoken word and a sign (i.e., a Blend) is much more common in natural production (Bishop & Hicks, Reference Bishop and Hicks2005; Emmorey et al., Reference Emmorey, Borinstein, Thompson and Gollan2008a). In order to examine language control, in the present study, such Blends were included in the language-switching paradigm in order to examine dual-response costs, mixing costs, and switch-costs, in one experiment. The outcome is a complex pattern of results that includes both bimodal advantages and bimodal disadvantages:
-
1) Dual-response costs were found in both pure and mixed blocks for vocal but not manual responses.
-
2) Significant mixing costs were found for manual responses, while mixing costs were only marginally significant for vocal responses. These mixing costs were (numerically for vocal responses and significantly for manual responses) larger for dual-response Blend trials than for single-response trials (a bimodal disadvantage for mixing costs).
-
3) Language switch costs were found for both spoken (vocal) and signed (manual) languages.
-
4) For manual (but not vocal) RTs, language-switch costs were smaller for dual-response Blend trials than for single-response trials (a bimodal advantage for switch costs).
-
5) Language-switch costs in terms of RT and error rate were larger when switching from a dual-response Blend trial to any single-response trial (German or DGS) as compared to switching from one single-response trial to the other (a bimodal disadvantage for switch costs).
These findings will be discussed in turn. First, we will discuss dual-response costs in pure and mixed blocks. Then we will focus on mixing costs and language-switch costs in a situation in which dual-response Blends are sometimes produced. Finally, we turn to potential mechanisms in bimodal bilingual language switching and discuss the specific nature of the Blend.
Dual-response costs
In both pure blocks and mixed blocks, the results showed dual-response costs for vocal but not for manual responses. In other words, a German vocal response was performed more slowly in a Blend than as a single-response in German. However, the source of the observed dual-response cost is not clear, i.e., there is no reason to assume based on the data from the present study that this is a case of a processing cost resulting from, for example, cognitive interference or inadequate short-term memory. As in the previous bimodal dual-response study (Emmorey et al., Reference Emmorey, Petrich and Gollan2012), participants in the present study seem to have timed their vocal responses to their manual responses on dual-response trialsFootnote 3 , making the vocal responses slower overall, which reflects a coordination cost or a response grouping for vocal responses (cf. Lien & Ruthruff, Reference Lien and Ruthruff2004).
If the vocal RTs in dual-response Blend trials are mainly influenced by their coordination with manual responses, it is difficult to interpret their pattern of results. This coordination cost might also be the reason that the pattern of mixing costs (see below) was numerically the same for vocal and manual responses but significant only for manual responses. The coordination cost incurred by vocal responses in Blend trials certainly influenced the dual-response cost and may have obscured other processing costs and/or advantages that may result from Blend production.
For the present study, we did not analyze dual-response costs for error rates in pure blocks due to the low number of errors in pure blocks in general. However, the data pattern for error rates in mixed blocks supports the notion that the dual-response costs were not caused by a pure processing cost. This is because the data pattern shows fewer errors in dual-response trials than in single-response trials – which indicates a dual-response advantage rather than a dual-response cost. This finding corresponds Huestegge and Koch (Reference Huestegge and Koch2014), who observed a comparable dual-response advantage in error rates (but not in RT as in the present study) in a non-language context. Huestegge and Koch (Reference Huestegge and Koch2014) interpreted their finding in terms of the high automaticity of saccadic responses such that it is harder to withhold the response than to execute it together with a manual response. For the present study, we propose that – at least when frequently switching between single-responses and dual-responses – it was easier (or even automatic) for participants to perform both responses than to execute only one response while inhibiting the other one. Inhibitory control processes and mechanisms of bimodal language control will be further discussed below. However, this observation is already first evidence for a specific nature of a Blend that goes beyond the occurrence of coordination costs or response grouping.
Mixing costs and switch costs with dual-responses
In the present study, we observed mixing costs. That is, RTs in the pure blocks were overall shorter than RTs in the mixed blocks for both vocal (marginally significant) and manual responses (cf. Los, Reference Los1996; for a review, see Kiesel et al., Reference Kiesel, Steinhauser, Wendt, Falkenstein, Jost, Philipp and Koch2010). A further interesting finding was that mixing costs differed between dual-response trials and single-response trials. For vocal responses, mixing costs were numerically, though not significantly, larger for dual-response trials than for single-response trials; it is possible that there is a significant difference and that the present study lacks the power to uncover it. For manual responses, mixing costs were significantly larger for dual-response trials (Blends) than for single-response trials (DGS). So at least for manual responses, we found a significant dual-response disadvantage (i.e., larger costs) in mixing costs.
Beyond mixing costs, we also found language-switch costs for both vocal and manual responses, with shorter RTs and lower error rates incurred by repeat trials than by switch trials. In the RT data, repeat trials were faster than any kind of switch trial (i.e., single-response from single-response, single-response from dual response, and dual-response from single-response). The error rates show the same effect, although for DGS, the difference between DGS from-German switch trials and DGS from-DGS repeat trials was not significant. However, this may reflect a ceiling effect due to the relatively low error rates in manual responses, and it is possible that a future study with more power or a shorter inter-trial interval, which presumably would lead to more errors across the board, would find a statistically significant difference here.
When comparing the size of switch costs between dual-responses and single-responses, no difference was found for vocal RTs. Yet, for manual RTs, language-switch costs were smaller in dual-responses than in single-responses (i.e., dual-response advantage for switch costs). Here, it is also interesting to note that in addition to the advantage in switch-costs in dual-response manual RTs, we also observed larger mixing costs for manual RTs in dual-response trials than in single-response trials. Thus, there was a dual-response processing disadvantage in mixing costs but a dual-response processing advantage in switch costs.
Mixing costs and switch costs are different markers of cognitive control: Mixing costs can be interpreted as reflecting a relatively global consequence of between-task interference (cf. Los, Reference Los1996; Philipp et al., Reference Philipp, Kalinich, Koch and Schubotz2008; Rubin & Meiran, Reference Rubin and Meiran2005). In contrast to such global effects, switch costs are mainly influenced by trial-to-trial modulation due to inhibition and activation (cf., Kiesel et al., Reference Kiesel, Steinhauser, Wendt, Falkenstein, Jost, Philipp and Koch2010; Declerck & Philipp, Reference Declerck and Philipp2015). Thus, it is not surprising to observe differential influences of one and the same manipulation on mixing costs and switch costs. Such a data pattern of differential effects in mixing costs and switch costs does occur in other language-related contexts as well, for example in asymmetric effects due to language dominance (cf. Christoffels et al., Reference Christoffels, Firk and Schiller2007; Declerck, Philipp & Koch, Reference Declerck, Philipp and Koch2013). For the present study, the differential effects might shed some light on the specific nature of the Blend in bimodal language switching. That is, the larger mixing costs for manual responses in dual-response trials than in single-response trials might indicate a general inhibition of blending, further supporting the notion that dual-responses (i.e., Blends) might have been easier and thus the dominant Target-response for participants rather than single-responses (see dual-response advantage in the error rate).
However, in language-switching studies, the more dominant language is usually also associated with larger switch costs than the less dominant language (see Bobb & Wodniecka, Reference Bobb and Wodniecka2013). Thus, when a Blend is the dominant Target-response, we should expect a relatively larger switch cost for dual-responses than for single-responses – which is the opposite of the data pattern from the present study. As noted above, other studies (Christoffels et al., Reference Christoffels, Firk and Schiller2007; Declerck et al., Reference Declerck, Philipp and Koch2013) also observed differential effects on mixing costs and switch costs due to language dominance. However, in none of the studies was the effect reversed between mixing costs and switch costs as in the present study. This reversal, leading to a dual-response advantage with respect to switch costs, could be accounted for by specific trial-to-trial language control mechanisms in bimodal language switching, which will be discussed in the next section.
With respect to language control mechanisms concerning Blend trials in language switching, it is most interesting to compare the different kinds of switch trials: A switch to a single-response from a single-response (i.e., German from-DGS or DGS from-German), a switch to a single-response from a dual-response (i.e., German from-Blend or DGS from-Blend), and a switch to a dual-response from a single-response (i.e., Blend from-German or Blend from-DGS). Yet, only the first two can be compared directly in RT data as the response in the current trial is a single-response in both cases.
In our data, single-response from dual-response switch trials incurred longer RTs and higher error rates than single-response from single-response switch trials. Put differently, switching from a Blend trial to a German or DGS trial results in higher RTs and error rates than switching from DGS to German or from German to DGS.
To account for this data pattern in terms of inhibitory language control (cf. Green, Reference Green1998), one would assume that inhibiting one single-response from the previous trial in order to produce a different single-response is relatively easy, while inhibiting a Blend from the previous trial in order to produce one of its component tasks as a single-response is more difficult. This assumption, however, includes two theoretically interesting aspects with respect to language control in bimodal bilingual language switching.
First, it points toward a critical distinction between persisting activation and persisting inhibition. In the inhibitory control model (Green, Reference Green1998), inhibition is executed to allow response execution in the target language. When switching to the previously irrelevant language in the subsequent trial, this language is still inhibited (i.e., persisting inhibition), and overcoming this persisting inhibition results in switch costs. However, switch costs can equally well be explained by persisting activation (for a discussion see Koch et al., Reference Koch, Gade, Schuch and Philipp2010; Philipp, Gade & Koch, Reference Philipp, Gade and Koch2007): The language that was previously relevant is still activated in a switch trial (i.e., persisting activation) and needs to be deactivated or inhibited in order to perform in the now-relevant language. The present data pattern clearly indicates that especially overcoming the persisting activation of a Blend increases RTs and error rates, indicating a crucial role for persisting activation in task switching. This is not in contradiction to the inhibitory control model (Green, Reference Green1998) as inhibition in the current trial still is an important aspect of language control, and presumably, language switching and the occurrence of switch costs is an interplay of both persisting activation and persisting inhibition. However, our results demonstrate that inhibition alone is not sufficient to explain bimodal bilingual language control.
The second theoretically important aspect refers to the question of whether a Blend must be inhibited in its entirety, or whether the two aspects of the Blend can be inhibited separately. If a Blend must be inhibited entirely, then one could speculate that there is a mechanism involved in lexicalization that locks these two representations together. The data pattern of the present experiment clearly indicates that a Blend is inhibited as one specific Target-response rather than separately as its two components. More specifically, the data pattern indicates that there is at least no persisting activation of the relevant language modality (e.g., the vocal response in a Blend and the German response) when switching from a Blend to a single-response trial. This can be taken as evidence that a Blend should be seen as a unit which is more complex than just a collection or response grouping of two single-responses. If the Blend were simply a collection of two single-responses, it would be easy to switch from a Blend to either single-response because the relevant single-response would simply remain activated and the non-relevant single-response would be inhibited. The results rather indicate that switching from a Blend leads to the inhibition of a blending response and that the reactivation of the relevant language modality of the single-response trial is necessary.
Furthermore, the even longer RTs and higher error rates in these switch trials as compared to single-response from single-response switch trials could be explained by an additional inhibition of the Response-type (single-response vs. dual-response). In this case, what must be suppressed is the execution of two simultaneous responses generally due to the fact that only a single response (as a general type of response) is relevant.
Such a switch in Response-type also takes place when switching to a dual-response from a single-response (i.e., Blend from-German and Blend from-DGS trials). For manual RTs (but not for vocal RTs), switch costs were smaller in dual-response trials than in single-response trials, and for all responses taken together, error rates were lowest and switch costs smallest for Blends. The finding of smaller switch costs in dual-response trials than in single-response trials represents a previously untested context in which we observed a bimodal advantage in task switching.
This observation is mainly interesting with respect to the question of persisting inhibition and persisting activation in language switching as it indicates that when switching from a single-response into a Blend, the activated language modality from the previous trial can remain activated and an additional modality is added. In this instance, persisting activation of the language modality from a previous single-response trial does not seem to interfere (much) with the formation of the Blend, whereas the persisting activation of a Blend from a previous trial does interfere with the production of one of its components as a single-response. In other words, it seems that you do not have to inhibit the relevant language modality from the previous trial in order to produce a Blend in the current trial. Rather, only the language modality which was not activated in the previous trial must be reactivated. Thus, in a bimodal context, it seems that the Blend is advantageous in additive contexts (in which the dual-response Blend response is formed by the addition of one language modality) and disadvantageous in subtractive contexts (in which the single-response response is formed by the subtraction of one language modality from the Blend). Thus, we assume that a different language control mechanism is at play in switch trials that include switching into or from a Blend. This would also mean that experience in the simultaneous production of a vocal and a manual response can alter language control in corresponding conditions. However, more research is needed in order to better understand language control mechanisms in language switching that includes dual-response trials.
As noted above, the data pattern of the present study is basically in line with an inhibitory control model (Green, Reference Green1998) – though it adds the importance of persisting activation and the distinction between inhibitory control processes in the current trial and persisting inhibition. Yet, other models of unimodal language switching cannot account for the present data. For example, the response exclusion hypothesis (Mahon, Costa, Peterson, Vargas & Caramazza, Reference Mahon, Costa, Peterson, Vargas and Caramazza2007) assumes that competing lexemes within one language (modality) must be eliminated in order for the correct word to be produced. In bilinguals, non-target-language distractors must be discarded in order to produce the target word in the correct language (Finkbeiner, Gollan & Caramazza, Reference Finkbeiner, Gollan and Caramazza2006). Most importantly, in this account, language control is located at a relatively late processing stage, namely at the level of the phonological output buffer (for spoken languages). Consequently, there should be no between-language interference between a spoken language using the output buffer and a signed language (cf. Hall, Reference Hall2011). However, the present study clearly demonstrates that switch costs occur when switching from one single-response to the other single-response (i.e., from a manual to vocal response or vice versa).
Despite the fact that we observed switch costs when switching from one single-response to the other, in our data, it does not matter which single response is switched into or out of. Previous reports of naturalistic data (e.g., Bishop & Hicks, Reference Bishop and Hicks2005; Emmorey et al., Reference Emmorey, Borinstein, Thompson and Gollan2008a) show that it is common for bimodal bilinguals to switch from speaking to blending and back, but not from signing to blending. It is possible that further studies with more power will uncover a difference, with switching between vocal and Blend responses incurring smaller costs than switching between manual and Blend responses. But it is equally possible that future studies will describe contexts in which the sign language is the base language and the bimodal bilinguals switch from signing to blending and back, e.g., when off-duty interpreters are chatting with each other and they want to provide some auditory information for non-signing hearing people nearby, as has been observed in Germany.
To summarize, the present pattern of results clearly indicates that a Blend is a response which is different from the mere combination of two independent responses and is more than a response grouping. Put differently, switching away from and into a Blend trial seems to incur specific costs which reflect mechanisms specific to bimodal language production.
Conclusion
The results of the present study show a complex pattern of dual-response processing advantages and disadvantages for bimodal language production, and this pattern is distinct from those found in previous studies in areas other than language modality. Within our pure blocks, there were dual-response costs for vocal responses and no dual-response costs for manual responses, replicating previous results. Additionally, this finding was extended to mixed blocks, in which we also observed dual-response costs for vocal responses but not for manual responses. However, error rates showed no dual-response cost in mixed blocks, but rather a dual-response advantage. In terms of mixing costs, there was a dual-response processing disadvantage, showing larger mixing costs for dual-response trials than single-response trials. In contrast, within the mixed blocks, we found a clear dual-response processing advantage. This advantage is reflected in lower switch costs for dual-response trials as compared to single-response trials in overall error rates and for manual RTs.
These results lead us to suggest that a Blend forms a unit which is different from the mere combination of two independent responses and more than simply a response grouping. The unit formed by the Blend leads to a disadvantage in switching from dual-response to single-response trials, and it leads to an advantage in switching from single-response to dual-response trials.
Appendix A: Sequences of blocks in partial counterbalancing
Sequences of blocks used in the present study. “DGS” means pure DGS (i.e., German Sign Language) blocks, “Ger.” means pure German blocks, “Bl.” means pure Blend blocks, and “Mix” means mixed blocks.
