1. Introduction
There exists considerable empirical evidence indicating language systems influence one another during the process of second language acquisition (SLA). For instance, results from numerous psycholinguistic studies suggest learned features of a second language (L2) are stored in tandem with pre-existing first language (L1) knowledge as part of an integrated mental lexicon (Kroll, Bogulski, & McClain, Reference Kroll, Bogulski and McClain2012; Tokowicz, Reference Tokowicz2014). From this perspective, initial L2 performance is influenced and mediated by existing L1 knowledge (Jiang, Reference Jiang2000; Dijkstra & van Heuven, Reference Dijkstra and van Heuven2002; Kroll, Van Hell, Tokowicz, & Green, Reference Kroll, Van Hell, Tokowicz and Green2010). Moreover, as L2 proficiency increases, the influence from L1 knowledge lessens, and increased L2 knowledge and proficiency may begin to influence L1 performance. This cross-linguistic influence can occur during both comprehension and production (Odlin, Reference Odlin, Doughty and Long2003; Alonso, Reference Alonso and Chapelle2020). Among the many ways to operationalise cross-linguistic influence, the Competition Model has proven to be a constructive framework to explain how different linguistic strategies associated with form-meaning mappings compete with one another during language acquisition, resulting in different levels of cross-linguistic influence (Bates & MacWhinney, Reference Bates, MacWhinney, Wanner and Gleitman1982, Reference Bates, MacWhinney and MacWhinney1987; MacWhinney, Reference MacWhinney2005). To date, there have been a large number of studies applying the Competition Model framework to compare sentence comprehension behaviour among monolingual and multilingual speakers of many different languages (MacWhinney & Bates, Reference MacWhinney and Bates1989; Year, Reference Year2003; Li & MacWhinney, Reference Li, MacWhinney and Chapelle2012).
This paper presents an argument for the replication of two of those studies, Liu, Bates, and Li (Reference Liu, Bates and Li1992) and Su (Reference Su2001), both focused on cross-linguistic influence between English and Mandarin Chinese. These two studies were both designed within the purview of the Competition Model to compare sentence comprehension behaviour among monolingual and bilingual English and Mandarin speakers. Specifically, the construct of interest was forward and backward transfer of strategies for mapping noun phrases to semantic roles (agent and patient). To do so, the two studies employed similar sentence stimuli, participants, and methods, and they also considered the influence of L2 proficiency. The results from these two studies have provided foundational, empirical evidence for (1) clear contrasts between the syntactic and lexical cues Mandarin and English speakers rely upon to identify noun phrase semantic roles during sentence processing, (2) forward and backward transfer of these cues among bilinguals of both languages, moderated by L2 proficiency, and (3) stronger forward and backward transfer effects for L1 Mandarin-L2 English speakers compared to L1 English-L2 Mandarin speakers. Both studies were published in Applied Psycholinguistics and have currently been cited over 100 times each, according to Google Scholar. One of the reasons why these studies are highly cited is because they are taken as evidence for qualitative differences in sentence processing strategies among speakers of Mandarin and English, which contemporary work has built upon to study different, more advanced syntactic contexts. Another reason is because these studies form part of the cohort of earlier Competition Model studies which depict forward or backward cross-linguistic transfer of specific sentence processing cues. As such, these studies are still relevant many years after they were conducted, but at the same time may carry a large share of the burden of proof for these phenomena.
In light of the ongoing call for replication in the broader scientific arena (Open Science Collaboration, 2015) as well as more specifically within the field of applied linguistics (Porte, Reference Porte and Porte2012), we suggest replication of Liu et al. (Reference Liu, Bates and Li1992) and Su (Reference Su2001). A further reason to suggest replication is because both studies were subject to similar limitations in sample size of stimuli and participants, operationalisations of L2 proficiency, and statistical analysis. Despite these limitations, these studies are highly cited in the literature as support for both the general predictions of the Competition Model as well as specific predictions regarding cross-linguistic influence of sentence processing strategies among L1 English-L2 Mandarin and L1 Mandarin-L2 English bilinguals. In Section 2, we discuss the background of the Competition Model. Next, we summarise relevant theoretical and linguistic differences between Mandarin and English and then describe the two studies suggested for replication in greater detail before offering suggestions for how they might be replicated.
2. Background
The Competition Model was introduced by Elizabeth Bates and Brian MacWhinney. The authors designed the model within the theoretical context of functional grammar (Bates & MacWhinney, Reference Bates and MacWhinney1981). A functionalist approach to grammar is one where grammar is explained and investigated in terms of the communicative and pragmatic goals different linguistic structures are associated with (Bates & MacWhinney, Reference Bates, MacWhinney, Wanner and Gleitman1982; Nichols, Reference Nichols1984). This approach thus contrasts with structural theories of language and grammar, such as Universal Grammar, which view syntactic structure as independent or unconstrained by meaning and/or communicative function (Chomsky, Reference Chomsky2006).
In terms of language development, the Competition Model is an emergentist model in that it assumes language knowledge results from dynamic interactions among linguistic form, cognitive processes, and lived experiences (Ellis, Reference Ellis1998; MacWhinney, Reference MacWhinney1998). In more concrete wording, the Competition Model framework is designed to measure and catalogue the manner in which language users choose from among competing lexical and morpho-syntactic patterns to understand larger, functional meanings of a sentence. These competing patterns are called cues and can be operationalised in a number of ways, although this is most commonly done at the lexical and morpho-syntactic layer of language. For example, L1 English speakers have been shown to reliably use word order to identify semantic roles of noun phrases in a sentence (e.g., agent, patient), whereas L1 Italian speakers rely on morphological markers to signal those same roles (Bates & MacWhinney, Reference Bates, MacWhinney and MacWhinney1987). Accordingly, an L1 English speaker learning Italian will find that their pre-existing L1 word order strategy to identify semantic roles has little productivity in Italian and will thus need to develop new, L2-specific strategies to fully understand semantic roles in Italian. During this learning process, the different L1 and L2 cues will compete with one another, hence the name the Competition Model.
As the previous example suggests, much of the research employing the Competition Model as an empirical framework has focused on the different cues L1 speakers used to understand the thematic roles of noun phrases in a sentence (Li & MacWhinney, Reference Li, MacWhinney and Chapelle2012). These studies typically present research participants with sentence stimuli including at least two noun phrases and a verb and ask participants to name the actor in each sentence. In this manner, different cues can be manipulated (e.g., word order in English) to determine which cues are the most reliable predictors of agent identification in a particular language (MacWhinney, Reference MacWhinney2005). For example, an English speaker can reliably look to the first noun in a transitive sentence to identify the agent (The fox ate the cake), but is also able to adjust this reliance in light of different competing syntactic constructions, such as passives (The cake was eaten by the fox). In this manner, canonical SVO (subject, verb, object) word order is a salient and reliable cue for English speakers, and the relative strength of this cue in English can be compared to other languages which use other cues. One of the languages which has attracted attention in this area is Mandarin Chinese.
The linguistic differences between English and Mandarin Chinese are vast. Mandarin uses a logographic writing system, lacks affixal morphology, and exhibits flexible word order whose interpretation relies heavily on context (Chao, Reference Chao1965; Li & Thompson, Reference Li and Thompson1989; Li, Bates, & MacWhinney, Reference Li, Bates and MacWhinney1993; Yip & Rimmington, Reference Yip and Rimmington2006). In contrast, English has a phonographic writing system and employs inflectional morphology for tense and person/number. That being said, the two languages share the same canonical word order (SVO) in simple transitive sentences. Yet, unlike English, which employs a strict word order pattern, Mandarin Chinese allows for greater word order flexibility and more lenient pro-dropping (i.e., allowing pronoun omission). Given this flexibility, word order has been claimed to be a weaker cue for thematic roles in Mandarin Chinese when compared to English (Li, Bates, Liu, & MacWhinney, Reference Li, Bates, Liu, MacWhinney, Chen and Tzeng1992). Mandarin speakers have been shown to rely on other lexical and semantic features as cues for the perception of thematic roles. As evidence, psycholinguistic research has shown that Mandarin speakers rely more on animacy (e.g., a person is animate whereas a rock is not) and the presence of functional word markers (e.g., applicative marker bǎ 把, passive marker bèi 被) more than word order when identifying the agent in a sentence (Li et al., Reference Li, Bates and MacWhinney1993). This difference between syntactic cues (word order) and semantic cues (animacy) is representative of a purported broader division between syntactic and semantic processing cues to assign thematic roles (Gass, Reference Gass1987). For these reasons, the differences between Mandarin Chinese and English create a fruitful opportunity to test the predictions of the Competition Model.
3. The original studies and approach to replication
Using English word order and Mandarin Chinese animacy cues as motivation, the two studies suggested for replication in the current paper, Liu et al. (Reference Liu, Bates and Li1992) and Su (Reference Su2001), both applied the standard sentence comprehension task commonly used in studies following a Competition Model approach. Participants in both studies were presented with simple transitive sentences, with the word order randomised to appear in three different variations: verb final (NP1 NP2 V), verb medial (NP1 V NP2), or verb initial (V NP1 NP2). The noun phrases (NPs) in each sentence were either two animate nouns or a combination of animate and inanimate nouns in different orders. As such, participants were asked to comprehend sentences from nine possible different combinations, summarised in Table 1.
The nine possible word order X animacy sentence conditions, with English examples
In each study, monolingual and bilingual participants listened to recordings of the sentence stimuli from all nine conditions in Table 1 in random orders. Their task was to determine which noun phrase was the agent (also called the doer) in each sentence. Monolingual participants heard sentences in only English or Mandarin. Bilingual participants listened to sentences in one session for each language, with the order of presentation counterbalanced (in Su Reference Su2001, participants waited 5–7 days between sessions, the interval in Liu et al., Reference Liu, Bates and Li1992 is not stated).
The dependent variable in both studies adhered to common practice with prior Competition Model studies. As such, performance on the sentence comprehension tasks was measured by calculating a ratio of the percentage of times a participant chose the first noun as the agent. In this manner, a higher percentage means the first noun was chosen more often, whereas a lower percentage meant the second noun was chosen more often. In the context of English and Mandarin, choosing the first noun would reflect canonical SVO word order, whereas choosing the second noun would reflect violations of this order.
3.1 Liu et al. (Reference Liu, Bates and Li1992): Summary
At the time of this study, assumptions associated with the Critical Period Hypothesis for L2 learning were emerging as fundamental topics of interest in SLA research. As such, these researchers chose to use age of exposure to the L2 as their primary measure of L2 proficiency when investigating forward and backward transfer of sentence processing cues. Fifty participants were categorised into seven groups: monolinguals of either language, English L1 speakers who learned Mandarin after the age of 20, Mandarin L1 speakers who learned English after the age of 20, and Mandarin L1 speakers exposed to English before they were 4 years old, between 6 and 10 years old, and between 12 and 16 years old.
Sentence stimuli were created by randomly sampling from lists of nouns and verbs to create a total of 54 different sentences (6 sentences of each type listed in Table 1). This was done twice: once for the English sentences and once for the Mandarin sentences. Because the nouns and verbs used were not the same in the two languages, the sentence stimuli were not translations of one another (see Appendix 2 in Liu et al., Reference Liu, Bates and Li1992). During the task, participants verbally reported their answers to the researchers.
Monolingual speaker behaviour aligned with the predictions of the Competition Model, in that English speakers relied on word order cues whereas Mandarin speakers relied more on noun animacy cues. For the bilingual participants, there were significant differences between the late English L2 learners (exposed to English after 20 years of age) and English monolinguals, whereby the late learners were significantly more likely to rely on L1 strategies when faced with competition between animacy and word order. Conversely, there were very few differences between this group and the Mandarin monolinguals. This was taken as strong evidence of forward transfer (Mandarin to English) in the late English L2 learners, with no suggestion of backward transfer. The corresponding comparison for the late English L1 Mandarin L2 participants found similar, yet weaker forward transfer (from English to Mandarin), again with no evidence of backward transfer.
The remaining comparisons were of the three Mandarin L1 English L2 groups at different ages of exposure to English: infant (0–4 years old), child (6–10 years old), and teen (12–16 years old). These comparisons suggested no evidence of forward transfer (Mandarin to English) for any group, some evidence of backward transfer (English to Mandarin) among the infant and teen groups, and no backward transfer for the child group. Liu et al. (Reference Liu, Bates and Li1992) also conducted exploratory post-hoc analyses of the data by calculating forward and backward transfer scores for each individual participant. They then correlated these scores with answers from a language background survey and found significant and strong correlations between measures of age of exposure and transfer.
3.2 Liu et al. (Reference Liu, Bates and Li1992): Approaches to replication
For an approximate replication, the procedural aspects of the study should remain the same. This means that participants would listen to the sentence stimuli and indicate which of the two nouns was the doer (i.e., the agent) in each sentence. Monolingual participants would listen to stimuli in their L1, whereas bilingual participants would perform the task twice, once in their L1 and once in their L2, in a random order. Scoring of the responses would also be the same, in that each item would be assigned a score based on whether the first noun was or was not chosen as the doer (a score of yes or no). Moreover, participants’ age of L2 exposure and language background should be collected using the same methods as in Liu et al. (Reference Liu, Bates and Li1992).
Then, a number of options are available for an approximate replication. Because the primary goal of the original study was to investigate interactions between L2 proficiency and transfer of syntactic processing cues, a replication may also want to adopt a standardised, continuous measure of Mandarin and English L2 proficiency. For English proficiency, there are a number of well-known standardised tests which could be used, such as TOEFL, IELTS, and the Duolingo English Test. One option would be to collect these scores. However, ensuring all participants take the same test and at the same times creates logistical challenges, as does the associated financial cost of many of these tests. The same constraints hold true for the HSK (汉语水平考试; hànyŭ shuǐpíng kǎoshì) test, a standardised test of Mandarin proficiency (www.chinesetest.cn). English L1 participants could report their HSK level, but again this poses similar logistical challenges.
A more efficient option would be to use less comprehensive but quicker tests of proficiency after participants have completed the task. For example, LexTALE is available online (www.lextale.com) and can be used as a quick, reliable assessment of English proficiency through a lexical decision task (where participants see single words and indicate whether or not the word is a valid English word). LexTALE is attractive because it is validated, free, and can be deployed quickly (Lemhöfer & Broersma, Reference Lemhöfer and Broersma2012). Moreover, there is a Mandarin version which has been developed but not officially launched as part of the LexTALE suite (Chan & Chang, Reference Chan and Chang2018). However, the test items are freely available for download and could be deployed using paper and pencil or using a computer (see https://github.com/gasparl/lextale for a working online version and see https://osf.io/qdy4n/ for the test items and answers). Because LexTALE and LexTALE_CH primarily measure L2 vocabulary knowledge, this information would allow for greater statistical comparison by capturing the same type of L2 knowledge, as contrasted with the variation associated with the measures of initial age of exposure. Researchers may also want to explore the use of other similar quick, validated measures of L2 proficiency in these languages.
Alternatively, instead of grouping participants into discrete categories based on age of exposure boundaries, a replication could include age of exposure as a continuous (rather than categorical) predictor variable in the statistical analysis. This would differ from the categorical groupings used in the original study. However, doing so would afford conceptual and statistical benefits and still allow for a direct comparison to the original results. The primary benefit is that using a continuous measure of L2 proficiency avoids the assumption that all participants in each group are similar. Although Liu et al. (Reference Liu, Bates and Li1992) made informed decisions based on age of exposure, the discrete categories participants were grouped into assumed homogeneity among the participants in each group, which may have obscured variation among members of the same group. As such, there was likely variance that remained among participants in these different groups, but nonetheless they were categorised as approximately similar in proficiency. Even if a researcher were to use age of exposure as a continuous variable as suggested, care should be taken in order to ensure the range of age of exposure values (age 4 to age 20+) is the same as the ranges described in Liu et al. (Reference Liu, Bates and Li1992).
In Liu et al. (Reference Liu, Bates and Li1992) the English L2 participants were all in an English as a second language (ESL) setting as they were all recruited in the United States. The Mandarin L2 participants were all Chinese as a foreign language (CFL) learners, as they were all English native speakers residing in the United States. Accordingly, any replication should recruit a similar population of ESL and CFL learners. A related limitation is that the Liu et al. (Reference Liu, Bates and Li1992) study included a relatively small number of participants in each group. Specifically, fifty subjects were separated into seven different groups, which in turn likely reduced the statistical power and generalisability of the results. As such, an approximate replication should endeavour to recruit a greater number of participants from within the age of exposure ranges originally gathered in Liu et al. (Reference Liu, Bates and Li1992).
Finally, it is suggested that the statistical analysis of the scores be performed differently. In the original study, the percentage of first noun choice agreement was calculated for each participant, and these scores were pooled into group means and compared using analyses of variance. A more appropriate method would be to model the response behaviour using a binary logistic regression (Jaeger, Reference Jaeger2008). With this approach, each sentence could be assigned a score of 1 (first noun chosen) or 0 (second noun chosen). Then, a binary logistic regression could be fit to predict the probability of a score of 1 or 0, with a three-way interaction fit among L2 proficiency, word order, and animacy condition of the sentence stimuli. The results would be represented as log odds which could be transformed into odds ratios, providing measures of likelihood towards choosing the first noun in a sentence. Moreover, using a mixed effects model would be the most ideal approach because participants and items could be assigned individual intercepts and the effects of the different predictor variables could be assigned individual slopes (Barr, Reference Barr2013; Bates et al., Reference Bates, Mächler, Bolker and Walker2015; Meteyard & Davies, Reference Meteyard and Davies2020). Effectively, this approach would better capture individual variance associated with the different participants as well as the effects of predictor variables on participants’ behaviour. Even with the use of these more advanced statistical techniques, the raw percentages of first noun choice could still be calculated and directly compared to the results from the original study.
3.3 Su (Reference Su2001): Summary
The Su (Reference Su2001) study was similar to Liu et al. (Reference Liu, Bates and Li1992) in regard to stimuli presentation and scoring, except the participants used pen and paper to indicate their answers. The main differences were related to participant selection, the measure of L2 proficiency used, and the stimuli creation. In Su (Reference Su2001), a total of 122 English or Chinese as a foreign language (EFL/CFL) participants were recruited from Taiwan and the USA. These participants were categorised based on the length of L2 instruction (e.g., number of years studying the L2) into three different stages of L2 learning: beginning learners, intermediate learners, and advanced learners, as well as monolinguals of each language. The Mandarin L1 speakers were all recruited in Taiwan and included students enrolled in vocational school or private language learning centres as well as professors and instructors of English who had earned postgraduate degrees in the USA. The English L1 speakers were all students or professors enrolled or working at The University of Hawaii, Manoa.
The sentence stimuli for Su (Reference Su2001) were created from a pool of twelve animate nouns, six inanimate nouns, and nine transitive verbs. Each noun and verb were used one time in each word order condition, with no noun-noun or noun-verb combinations repeating. This resulted in 27 different sentences. Unlike Liu et al. (Reference Liu, Bates and Li1992), the English sentences were formed by directly translating the Mandarin sentences (see Appendix, Su, Reference Su2001).
The results indicated evidence of forward transfer of L1 processing strategies for Mandarin L1 speakers, and the strength of this transfer interacted with proficiency in that higher L2 proficiency attenuated the amount of transfer. There was also some evidence of backward transfer, also influenced by proficiency, but also not as strong as the forward transfer. The analyses of the English L1 CFL learners also demonstrated forward transfer (English strategies applied to Mandarin) and to an even greater extent than observed in the EFL learners. There was little to no evidence of backward transfer for these participants.
3.4 Su (Reference Su2001): Approaches to replication
The majority of the suggestions for replication we provided above for the Liu et al. (Reference Liu, Bates and Li1992) study hold true for the Su (Reference Su2001) study. A similar population of participants should be recruited, the participants should perform the same task with the same scoring method, and more advanced statistical analyses should be employed. Additionally, Su (Reference Su2001) recruited foreign language learners sampled from both L1s at three approximately equivalent proficiency levels (beginning, intermediate, and advanced learners). A replication should take care to recruit a similar grouping of participants who are homogenous in regard to their L1 background. Ideally, Mandarin speakers from Taiwan and English speakers from the USA would be recruited. However, if this is not possible another homogenous group of Mandarin and English L1 speakers could suffice (because the dialectical differences between variations of Mandarin and English should not influence the processing cues).
An additional crucial factor for a replication would be to recruit participants with relatively homogenous L2 training. This was a strength in Su (Reference Su2001), who recruited L2 English learners from language learning centres in Taiwan, with the reasonable assumption that these students would have experienced similar standardised English curriculum through their educational experience in Taiwan. The Mandarin L2 speakers were also recruited from the same language training courses at the University of Hawaii. At the same time, participants should be relatively naïve regarding the purpose and goals of the study. For example, Su (Reference Su2001) also included lecturers of Mandarin Chinese in the advanced proficiency group, which a replication should avoid because participants with advanced knowledge of language and linguistics may not process the sentence stimuli in a similar manner to more naïve language users. One ideal scenario would be a collaboration among researchers located in Mandarin L1 and English L1 contexts who also have access to learners of either language as an L2. The same set of stimuli could be administered in different sites, allowing for more rapid collection of data from a greater number of participants and yet still allowing for a direct comparison to the original study.
In terms of the sentence stimuli, a researcher conducting a replication should make some modifications regarding the nature of the stimuli presentation. This is because Su (Reference Su2001) included direct translations of the English and Mandarin sentences. The best approach that a replication could take would likely be to use direct translations but also to use them in a counterbalanced design. This is because bilinguals should respond to both versions of the stimuli, and using direct translations raises the potential for priming and repetition effects to occur, with participants effectively responding to the same stimuli twice. Employing a counterbalanced design would require at least doubling the amount of stimuli sentence so that participants would only respond to one version of a sentence in any particular language (e.g., each participant responds only to half the English and half the Mandarin sentences). One suggestion for doing so is to sample nouns and verbs with similar semantic properties and frequency ranges as those used in Su (Reference Su2001).
4. Conclusion
The Liu et al. (Reference Liu, Bates and Li1992) and Su (Reference Su2001) studies have important theoretical implications because they provide foundational information regarding the cues Mandarin and English speakers use in determining the thematic roles within a sentence. This information has been used as a baseline for studies defining the standard order of L1 sentence processing cues in Mandarin (Li et al., Reference Li, Bates and MacWhinney1993) as well as examining forward and backward transfer of cues involved in more advanced syntactic constructions, such as passives (Wang & Xu, Reference Wang and Xu2015). Pedagogically, results from studies like these can be applied to meaning-focused approaches to instructed second language acquisition which represent a functional approach to grammar (Keck & Kim, Reference Keck and Kim2014). Although recent studies have begun analysing the transfer of more complex syntactic structures and cues between Mandarin Chinese and English, a replication of foundational studies such as the two suggested for replication here would serve to further bolster (or call into question) claims made about basic cue transfer.
Because of these theoretical and pedagogical applications and a number of limitations present in both studies, replication would benefit these respective theoretical and pedagogical arenas. These limitations are mostly methodological, and replications would allow for these limitations to be addressed while still providing a direct comparison to the results of the original studies. As such, an approximate replication of both studies is suggested. This is because variables and procedures would remain mostly the same, with a focus placed on standardising the participant selection and independent variables as well as expanding the lists of original stimuli (Porte, Reference Porte and Porte2012).
Finally, there are a number of similar studies comparing sentence processing cues between other languages. A return to the two studies discussed for the purpose of replication may invite new insights into the original assumptions of the Competition Model as well as spark energy for new studies based on results from those replications. The Competition Model approach is still being refined and developed into a universal explanation for language acquisition (MacWhinney, Reference MacWhinney, Gass and Mackey2009) and being extended to other language modes such as production (Duguine & Köpke, Reference Duguine and Köpke2019). Replications of the foundational studies using more sophisticated techniques would make an important contribution to this endeavour.
Victoria Chen is a Lecturer in the School of Linguistics and Applied Language Studies at Victoria University of Wellington in Wellington, New Zealand. Her primary research interest lies in comparative syntax, diachronic syntax, and linguistic typology, with a particular focus on Austronesian languages.
Stephen Skalicky is a Lecturer in the School of Linguistics and Applied Language Studies at Victoria University of Wellington in Wellington, New Zealand. He is broadly interested in research investigating associations between language and cognition, with a particular focus on lexical knowledge, figurative language, and humour in L1 and L2 contexts.
