INTRODUCTION
Mandarin immersion (MI) education, an alternative choice-based educational program, is characterized by subject matter schooling in Mandarin Chinese for no less than half of the elementary school day. In 2006, the Center for Applied Linguistics (CAL) self-report directory listed fewer than 20 such programs in the United States (CAL, 2017). However, the past decade has witnessed a surge in MI programs, and today there are more than 225 programs nationally (Mandarin Immersion Parents Council, 2016). Of these programs, more than 80% are publically funded, world language immersion programs that teach English-proficient children in Mandarin using simplified characters. Approximately one in four of Mandarin world language immersion programs implement the more intensive early total MI model, in which subject instruction in the primary grades and initial literacy take place either 90% or 80% of the time in Mandarin. Program goals include (a) academic achievement, (b) bilingualism and biliteracy, and (c) cultural competence (for a more in-depth overview, see Fortune, Reference Fortune2012).
A robust body of research has documented the efficacy of language immersion education in alphabetic languages (Genesee & Lindholm-Leary, Reference Genesee and Lindholm-Leary2013), particularly regarding children's ability to learn academic content in a second language (L2). Research-based evidence in support of language immersion in character-based languages such as Mandarin, in contrast, is limited and in need of further study (Fortune & Song, Reference Fortune and Song2016; Lindholm-Leary, Reference Lindholm-Leary, Tedick, Christian and Fortune2011; Padilla, Fan, Xu, & Silva, Reference Padilla, Fan, Xu and Silva2013; Valentino & Reardon, Reference Valentino and Reardon2014). There are several reasons why immersion in a character-based language might produce different outcomes than immersion in an alphabetic language. Because Mandarin uses a logographic script, the way in which MI learners need to visually process characters for meaning and sound information differs greatly from the more familiar processes used to map letters to sounds in alphabetic languages (Perfetti, Cao, & Booth, Reference Perfetti, Cao and Booth2013). There is also evidence that the cross-linguistic transfer potential between two distant languages is substantially less than between languages with similar writing systems (Koda, Reference Koda, Gass and Mackey2013). Such realities may lead to MI teachers focusing extensive amounts of time on character knowledge and literacy development with a concomitant reduction in creating opportunities for oral language development in Mandarin. Monitoring and ensuring growth in oral language is critical because research indicates a strong positive relationship between L2 oral skills and L2 reading comprehension (Geva, Reference Geva, August and Shanahan2006), which is key to learning subject matter in L2.
However, studies examining MI students’ L2 language outcomes, particularly related to speaking, are scarce. To date, oral proficiency reports on English home language MI students who are taught in Mandarin for half of the K–5 instructional day showed that scores increased steadily across the elementary grades (Bacon, Reference Bacon2007). In addition, Fortune and Song (Reference Fortune and Song2016) found that the majority of Grade 5 students participating in the more language-intensive early total MI program attained speaking proficiency in the intermediate-low range based on certified ratings given for the STAndards-based Measurement of Proficiency 4 Skills elementary (STAMP 4Se) assessment (Avant Assessment, 2012). As described by the STAMP 4Se Benchmarks and Rubric Guide (Avant Assessment, 2013), the intermediate-low level is distinguished by an ability to create strings of independent statements, questions, and commands. Speech typically displays good accuracy; however, language use may sound choppy due to a limited use of connectives. Against this backdrop of research, the current article asks the general questions: What more can we learn about early total MI students’ L2 oral language development by measuring subskills within speaking? How might this inform curriculum, instruction, and assessment?”
STUDY PURPOSE AND RESEARCH QUESTIONS
The study reported on here is part of a larger research project on the academic achievement and Mandarin language skills of English home language students from three elementary immersion programs (Fortune & Song, Reference Fortune and Song2016). In the current article, we report new findings on students’ Mandarin oral proficiency by first statistically analyzing scores from 277 administrations of two Mandarin oral proficiency assessments developed by CAL: (a) Student Oral Proficiency Assessment (SOPA) and (b) CAL Oral Proficiency Exam (COPE) (Rhodes & Thompson, Reference Rhodes, Thompson, Padilla, Fairchild and Valadez1990; Thompson, Boyson, & Rhodes, Reference Thompson, Boyson and Rhodes2006).We began with exploratory data analysis to determine the median score by proficiency domain for each grade level and to plot score distribution. Nonparametric follow-up analyses then tested for between-grade-level differences for kindergarten and Grade 2, and for Grade 2 and Grade 5. These test results also informed student selection for an in-depth exploration of three students’ speech samples using finely grained measures of linguistic complexity. We were interested in examining complexity as a means of documenting the elaborateness and variety of grammatical and lexical features produced by three learners at different grades. The three students, one kindergartener, one second grader, and one fifth grader, were among the 277 students who participated in the SOPA and COPE assessments. In each case, their proficiency ratings were representative of the median scores for their corresponding grade level.
The research questions are as follows:
1. How do median scores for oral fluency, grammar, vocabulary, and listening comprehension compare across kindergarten, Grade 2, and Grade 5?
2. To what extent do median oral proficiency scores differ between kindergarten and Grade 2, and Grades 2 and 5?
3. What changes occur in the linguistic complexity of Mandarin oral language produced by one kindergarten, one Grade 2, and one Grade 5 immersion learner during a proficiency assessment interview?
ASSESSMENT STUDY
Study Context and Participants
The 277 English-proficient MI students who participated in the SOPA and COPE assessments in the spring of 2010, 2011, 2012, and 2013 were kindergarteners, second graders, and fifth graders from three MI programs in two high-performing suburban school districts in the Midwest (Table 1). We intentionally selected students from these grade levels for three main reasons: (a) SOPA and COPE assessment is costly and labor intensive, so assessing all students in every grade was not feasible; (b) as a means of providing a programwide overview of K–5 oral proficiency outcomes using a coarse descriptive rating scale, we were interested in information from the beginning, middle, and end of the elementary years; and (c) we wanted to build on existing research and were aware of earlier studies of immersion students that had reported SOPA and COPE assessment findings for these same grades (Fortune & Tedick, Reference Fortune and Tedick2015).
Number of MI Students Assessed by School Year (Spring of 2010 – 2013)
The majority of students were European (69%) or Asian American (23%), with much smaller numbers representing American Indian (1%), African American (4%), and Hispanic American (3%) backgrounds. No students were identified as English learners, 7% of students received special education services, and an estimated 3%–4% received free or reduced price lunch. All participating students entered the program as kindergarteners.
Assessment Instruments and Procedures
Assessment Instruments
SOPA and COPE assessments are task-based, criterion-referenced tools designed to elicit a speech sample that represents a student's highest level of proficiency. Interaction between an adult interviewer and two students occurs in one of two formats: SOPA uses an adult-moderated interview; COPE employs adult-facilitated student–student role-plays. Tasks steadily increase in oral language demands until the administrator and rating team determine that the students have demonstrated their highest level of proficiency.
Teams made up of one adult interviewer and two raters administered and rated all assessments for one grade level only (either kindergarten, Grade 2, or Grade 5) in all three programs. To prepare, assessment team members participated in online and face-to-face trainings facilitated by or in collaboration with CAL. All team members were native speakers of Mandarin, and most were experienced immersion teachers. Raters used a 9-point analytic rating scale to assess each student's speech sample. A Chinese-specific rating scale (CAL, 2010) provided raters with qualitative descriptions of oral proficiency in four categories: oral fluency, grammar, vocabulary, and listening comprehension.
Proficiency levels ranged from 1(Junior [Jr.] Novice Low) to 9 (Jr. Advanced High) following functional proficiency levels established by the American Council on the Teaching of Foreign Languages (ACTFL) Proficiency Guidelines (ACTFL, 1986, 1999). “Jr.” levels were first introduced with the Educational Testing Service adaptation of the ACTFL Proficiency Guidelines (Rabiteau & Taft, Reference Rabiteau and Taft1985) and reflected an effort to better align proficiency descriptors with the language of junior high school learners. The Jr. levels differ from the ACTFL speaking guidelines by providing more detailed descriptive information about four subcomponents of oral proficiency: oral fluency, grammar, vocabulary, and listening comprehension. For the remainder of this article, we will use the acronyms presented in Table 2 that correspond to the nine Jr. levels of the SOPA/COPE rating scale.
CAL SOPA/COPE Rating Scale Levels and Abbreviations
Data Collection
Each spring during 2010–2012, 12 kindergarten and 12 Grade 2 students were selected by the assessment team coordinator at each program site, using a stratified random sampling technique. Prior to selection, grade-level immersion teachers had assessed each student's oral proficiency as “low,” “mid,” or “high” based on focused observation of their classroom language production. A sampling fraction was then calculated for each grade level by program so that a proportionally representative student sample could be randomly identified for each proficiency level. For example, if all kindergarten MI teachers in Program A rated 50% of all kindergarteners at high, 25% at mid, and 25% at low, then six kindergarteners with high teacher ratings, three with mid and another three with low were randomly selected from the entire kindergarten pool to participate in the SOPA assessment. These 12 students then represented the proportion of Program A kindergarteners in each proficiency category. Because three programs participated in this study, there were a total of 36 kindergarten and 36 Grade 2 students selected in 2010, 2011, and 2012; however, absenteeism and other administrative challenges reduced the actual total of assessed students, as illustrated in Table 1. In 2013, the final year of the data collection phase of the research, all 41 Grade 5 students were assessed, along with a final stratified random sample of 36 Grade 2 students.
Student pairs were interviewed outside the classroom by a three-member assessment team. Directly after each interview, team members discussed each speech sample while referencing the SOPA/COPE rating scale and notes taken during the interview, before reaching consensus on assigned scores. All interviews were video- and audiorecorded. After all interviews were rated, the assessment team coordinator sent the data (student pairings and demographics, video- and audiorecordings, rater notes, and SOPA/COPE ratings) to researchers for analysis.
Data Analysis
Prior to statistical analysis, we determined interrater reliability for 13% of all kindergarten and Grade 2 SOPA and 100% of Grade 5 COPE interviews. Ratings for all fifth graders’ interviews were verified because 2013 was the pilot year for COPE. This process began by sending interview videotapes and rater notes from assessments administrated in 2011 and 2013 to a highly experienced CAL rater for independent rating verification. Interrater reliability between the CAL rater and the assessment team raters was calculated using Cohen's kappa (.72 in kindergarten and Grade 2; .16 for Grade 5; Cohen, Reference Cohen1960). The inexperience of the COPE raters, who were rating Grade 5 speech production in the more complex and diffuse intermediate range for the first time, produced very low interrater agreements. In contrast, the kindergarten and Grade 2 SOPA raters had participated in multiple trainings and received feedback from the more experienced CAL rater in earlier years. Because of the subpar (below .70) kappa for Grade 5, we ran all statistical tests for Grade 5 using ratings assigned by CAL's rater.
Statistical analysis of the SOPA/COPE scores was conducted using the SPSS data management and analysis program. We first employed Tukey's exploratory data analysis (Tukey, Reference Tukey1977) to identify median scores and plot their distribution by grade level. This nonparametric analysis also graphs interquartile ranges (IQRs) and flags outliers for each criterion of the rating scale. Kruskal-Wallis (a nonparametric equivalent to the parametric ANOVA) and Mann Whitney U, a nonparametric follow-up test, were used next to identify significant differences in assessment scores across grades.
Assessment Results
RQ1. How do median scores for oral fluency, grammar, vocabulary and listening comprehension compare across kindergarten, Grade 2, and Grade 5?
Each grade level's median scores remained the same across all speaking criteria (oral fluency, grammar, vocabulary) and increased by one sublevel for listening comprehension (Figures 1–4). Kindergarteners’ median scores were level 2–JNM in oral fluency, grammar, and vocabulary, and 3–JIL for listening comprehension. There was a three-sublevel change in the median scores of Grade 2 students (5–JIM for speaking proficiency; 6–JIH for listening comprehension). Grade 5 median scores remained the same as Grade 2 in all four oral proficiency domains.
RQ2. To what extent do median oral proficiency scores differ between kindergarten and Grade 2, and Grades 2 and 5?
Tukey's Exploratory Data Analysis for Oral Fluency
Tukey's Exploratory Data Analysis for Grammar
Tukey's Exploratory Data Analysis for Vocabulary
Tukey's Exploratory Data Analysis for Listening Comprehension
Results of the Mann Whitney U tests (Table 3) indicate statistically significant differences between scores achieved in kindergarten and both other grades (p < .001). However, tests showed no significant differences between ratings for second- and fifth-grade students.
Mann Whitney U Test Results for Between-Grade-Level Differences
Note. **p < .01. ***p < .001. List Comp = listening comprehension.
Discussion
The three sublevel difference between the oral proficiency levels of kindergarten and Grade 2 MI students over the course of two academic years is striking. Equally striking, however, are the median scores earned by Grade 5 MI students after three additional years of immersion schooling, which indicate no change in proficiency levels since Grade 2 in any area. These findings resemble recent documentation of changes in oral proficiency for early total Spanish immersion students at these same grade levels. In both cross-sectional studies of oral proficiency—Fortune and Tedick (Reference Fortune and Tedick2015) and the one reported here—English-proficient children who entered the program with little to no proficiency in the immersion language achieved mid- to high-intermediate Jr. levels in speaking and high-intermediate to advanced-low Jr. levels in listening within 3 years of program participation. Beyond Grade 2, median score changes were limited to one sublevel or nonexistent, lending additional empirical support to earlier immersion research documentation of a certain “plateau effect” in students’ proficiency development (Harley, Reference Harley1992; Harley, Cummins, Swain, & Allen, Reference Harley, Cummins, Swain, Allen, Harley, Allen, Cummins and Swain1990; Hart, Lapkin, & Swain, Reference Hart, Lapkin, Swain, Malave and Duquette1991).
Is there really a proficiency plateau, or are these results an artifact of the proficiency framework and global rating scales used in SOPA/COPE assessment? Might a more finely grained analysis of the children's language reveal differences in the language produced by individual students at these same grade levels? Assessment outcomes are influenced by many factors, including (a) individual and group learner differences, (b) increased instructional time in English in Grades 3–5, (c) immersion students’ decreased willingness to communicate in L2 (MacIntyre, Burns & Jessome, Reference MacIntyre, Burns and Jessome2011), (d) assessment administration and rater differences, and (e) assessment task demand variation, among others. That said, the observed lack of change in proficiency between Grades 2 and 5 MI students, coupled with the inability of any broad rating scale to adequately detail the L2 production that is possible at a given proficiency level, motivated a follow-up case study to carry out a complexity analysis of three students’ production. We were interested in determining (a) if and how the oral language production of individual MI students representing different grades might vary linguistically and (b) how such information might inform MI curriculum, instruction, and assessment.
LINGUISTIC COMPLEXITY ANALYSIS
To gauge the multicomponential nature of L2 proficiency, second language acquisition (SLA) researchers have examined three dimensions of language: complexity, accuracy, and fluency (CAF; Housen, Kuiken, & Vedder, Reference Housen, Kuiken and Vedder2012). These interrelated constructs provide SLA researchers and educators the means to measure and particularize aspects of learner language during attempts to communicate (Ellis & Barkhuizen, Reference Ellis and Barkhuizen2005; Tarone & Swierzbin, Reference Tarone and Swierzbin2009). A thorough evaluation of each of these CAF constructs and their interconnectedness is beyond the scope of this article. Here, we will report findings from several measures of one of the three components, linguistic complexity. We focus on linguistic complexity in this article because (a) it includes both “system” and “structure” complexity measures that evaluate not only the global L2 system's breadth but also a local structure's depth (Bulté & Housen, Reference Bulté, Housen, Housen, Kuiken and Vedder2012), (b) it allows for measurement of both grammatical and lexical aspects of language and these correspond to two of the four domains (grammar and vocabulary) that were rated separately on the CAL SOPA/COPE rating scale, and (c) higher levels of complexity are potential indicators of higher overall proficiency (Bygate, Reference Bygate1999).
Research that measures L2 Mandarin child learners’ oral proficiency using quantifiable measures of complexity is scarce (Center for Advanced Research on Language Acquisition [CARLA], 2015; Yates, Reference Yates2012). A key SLA challenge for assessing and evaluating L2 Chinese proficiency exists, that is, the operationalization of linguistic measures that have primarily been used with Indo-European languages for Mandarin. This study responds to this challenge by exploring the oral proficiency of three young MI learners. We will begin by discussing how we adapted the analysis of speech unit (AS-unit) of measurement (Foster, Tonkyn, & Wigglesworth, Reference Foster, Tonkyn and Wigglesworth2000) and operationalized complexity measurement for Mandarin Chinese. We will then present findings from several global and specific complexity metrics. Our goal is to find out how more finely grained linguistic analyses can complement less progress-sensitive global assessment findings to better inform MI program stakeholders and help pinpoint instructional and assessment needs of MI learners.
Speech Sample Collection and Student Selection
Audio- and videorecorded speech samples were elicited by a trained assessment team member during SOPA/COPE assessments administered in a quiet location during the school day. Students were interviewed for 20–32 minutes in pairs, alongside a same-grade peer whose proficiency was estimated to be at a generally similar level based on MI teacher judgment. Speech sample selection was determined by how closely each sample's ratings mirrored the median proficiency scores for SOPA/COPE in their respective grade level. For example, individual ratings of the fifth grader, Dana, were the same as median scores for all fifth graders in every category but grammar. In grammar, there was a one sublevel difference between the Grade 5 median score (5–JIM) and CAL's rating of Dana (6–JIH). Because CAL ratings were used for Grade 5 and to be consistent across samples, we chose kindergarten and Grade 2 students from a subset who also had CAL-verified ratings. We also considered the quality of assessment audiorecordings for ease and accuracy of data transcription. Table 4 displays focal students’ CAL ratings by proficiency domain. All three students spoke English at home. Two students were female; one male. The kindergartener, Rose, and second grader, Connor, were European Americans; the fifth grader, Dana, was African American. None of the students received services in special education.
CAL Ratings by Proficiency Domain for Speech Samples of Three Student Representatives
Note. *K= kindergarten; 2 = Grade 2; 5 = Grade 5. List Comp = listening comprehension
Analytical Procedures
Data Preparation
Speech samples were transcribed and then segmented into AS-units. According to Foster et al. (Reference Foster, Tonkyn and Wigglesworth2000), “an AS-unit is made up of an independent clausal unit or a sub-clausal unit, and any associated subordinate clauses that are not separated by a .5 second or greater pause” (p. 367). Although consensus as to the most effective unit of measurement for analysis of Mandarin is lacking (Yuan, Reference Yuan2009), a research project analyzing the oral language of university students studying Mandarin as a foreign language modeled use of the AS-unit based on a recommendation from an experienced SLA researcher whose first language (L1) is Chinese (CARLA, 2017). We chose to follow this example in our analysis. Therefore, following Foster et al. (Reference Foster, Tonkyn and Wigglesworth2000), we next pruned each speech sample based on guidelines for Level 2 oral production data, which Foster et al. described as “highly interactional” (p. 370). This involved excluding false starts, repetitions that were not used for rhetorical effect, self-corrections, interruptions and scaffolding, one-word minor utterances (uh, 是 (yes), okay, etc.), and word-for-word echo responses. Pruned speech was placed in brackets. We also added an apostrophe to mark an error and a triangle to mark “not-preferred” language.
Adaptations for Mandarin
At times, we experienced difficulty when identifying the boundary of an AS-unit in Mandarin. As a topic-prominent language, the syntactic boundary that marks an independent clause in Mandarin is the end of the topic or event (Xu, Reference Xu and Ma1991). Consequently, we identified the boundary of an AS-unit by topic/event and occasionally included multiple independent clauses that were topically and/or logically chained with an overt or covert conjunction. For example, in the sentence 你要用杠杆和滑轮 :: 我们在做的 :: 是用杠杆// (Dana, Lines 36–37) (you need to use lever and pulley :: what we are doing :: is using a lever //), we marked the two underlined independent clauses as one AS-unit because (a) there was a clear continuation of the same topic; (b) even though no conjunction was used to connect the two clauses, it was implied; and (c) pause time between the two clauses was less than 0.5 second (a critical feature of AS-unit identification). Together, these factors suggested that the two independent clauses above constituted one AS-unit in Mandarin.
There were other occasions in our data where the conjunction 和 (and) was used at the beginning of a new independent clause to string multiple independent clauses together (Table 5). However, in Mandarin 和 (and) is only used as a conjunction between words and phrases (Li & Thompson, Reference Li and Thompson1981). These MI students appeared to use it more as a filler (similar to the informal English use of “and”) rather than as a conjunction connecting two independent clauses. When the two clauses were topically independent, we separated them into two AS-units.
Example of Independent Clauses as Separate AS-Units in Mandarin
For clause identification, we used Foster et al.’s (2000) definitions, which state that “an independent clause will be minimally a clause including a finite verb” (p. 365) and “a subordinate clause will consist minimally of a finite or non-finite verb element plus at least one other clause element” (p. 367). In addition, as Foster et al. (Reference Foster, Tonkyn and Wigglesworth2000) recommended, no pauses greater than 0.5 second existed between clauses. There were cases in our data in which the subject was omitted but implied. When this occurred, the single verb was counted as a clause. For example, 她回{那个}*去家里 :: 想 ::{我-} 我以后不会*^到别人^的家 (she went {well} at home :: she thought :: {I-} I won't arrive to another's house after this) (Connor, Line 326). In this portion of the AS-unit, 想 (thought) appeared without another element, but the subject 她 (she) was implied, so it was counted as a clause.
Because characters in modern Chinese combine to form words and serve as meaningful units that enter into grammatical relationships within the AS-unit (Duff et al., Reference Duff, Anderson, Ilnyckyj, VanGaya, Wang and Yates2013), we selected words in lieu of characters as our basic unit. While we acknowledge the inherent complexity of identifying Chinese word boundaries in Chinese linguistics (Zhu, Reference Zhu1982), we made use of Packard's (2000) criteria for morphological analysis of Chinese words to help us identify a word and distinguish word boundaries: (a) Is the character a “function” (grammatical) or “content” (lexical) morpheme? (b) Is the morpheme free (can stand alone as a word), or bound (must be paired with other morphemes to form specific words)?
To determine interrater reliability for these basic units, 10% of analyzed speech data was independently coded by two native Mandarin speakers. Interrater reliability for AS-unit and clause identification was 91% and 92%, respectively; for identifying words, 97%.
Linguistic Complexity Measures
In this study, linguistic complexity was examined at the operational level using global and local measures to quantify the degree of grammatical and lexical complexity (Bulté & Housen, Reference Bulté, Housen, Housen, Kuiken and Vedder2012). Calculations of grammatical complexity occurred at sentential, subsentential, and phrasal levels and included (a) mean length of the AS-unit (Tavakoli & Foster, Reference Tavakoli and Foster2008), (b) mean clause per AS-unit, (c) number of clauses per AS-unit, and (d) types and proportional use of modified noun phrases. Lexical complexity measures targeted word diversity and sophistication. The Guiraud index (Guiraud, Reference Guiraud1954), a type-token ratio measure of lexical diversity that was adapted to account for differences in the length of text produced, was calculated for all word types and content word types (Bulté, Housen, Pierrard, & Van Daele, Reference Bulté, Housen, Pierrard and Van Daele2008). An additional lexical complexity measure involved calculations of the proportion of Tier 1 versus Tier 2 versus Tier 3 words (sophistication) (Beck & McKeown, Reference Beck and McKeown1985). This three-tier framework is used to describe a mature literate person's word knowledge. Tier 1 words are basic, commonly used words such as 书 (book) and can readily be associated with a concrete visual referent. Tier 2 words reflect more precision and conceptual maturity. For example, 村子 (village) is a more precise way of stating where someone lives. Tier 3 words are less frequent and are discipline-specific, such as 滑轮 (pulley) and 杠杆 (lever).
Beck and McKeown's (1985) explanations of words composing each tier were further refined by cross-referencing two Mandarin-specific sources: (a) word lists for the Young Learners Chinese (YCT) tests, Levels I–IV, used to measure proficiency for learners of Chinese as a foreign language (Hanban, n.d.), and (b) a detailed frequency dictionary for Mandarin Chinese based on an extensive 50-million-word corpus of spoken and written texts (Xiao, Rayson, & McEnery, Reference Xiao, Rayson and McEnery2009). For example, our Tier 1 words included the most basic words such as 书 (book), 水 (water), 喜欢 (to like), 长 (long), were identified in YCT Levels I–III word lists, and were found in the Mandarin Chinese frequency dictionary's first 1,200 most commonly used words.
Findings
RQ3. What changes occur in the linguistic complexity of Mandarin oral language produced by one kindergarten, one Grade 2, and one Grade 5 immersion learner during a proficiency assessment interview?
Overall, complexity measures revealed steadily increasing levels of grammatical complexity across all three students, with Dana's (Grade 5) language sample showing descriptively more complex language as evidenced by longer AS-units and greater use of embedded clauses. Her speech also demonstrated more nativelike sensitivities to Mandarin language, especially in her use of prenominal noun modification with relative clauses, since formation of this structure differs substantively in English. Lexical complexity measures, in contrast, discovered greater word variety and sophistication in Connor's (Grade 2) and Dana's (Grade 5) language production relative to the kindergartener, Rose. However, Connor consistently outperformed Dana on measures of lexical complexity.
Grammatical Complexity: Global Measures
Mean Length of AS-Unit
Table 6 shows an increased number of words produced per AS-unit at higher grade levels. Whereas kindergartener Rose averaged 2.3 words per unit, second grader Connor produced 6.2 on average, nearly three times as many as Rose. Fifth grader Dana generated only a few more AS-units than Rose; however, her mean word count (10.8) represented use of almost five times as many words per AS-unit in comparison to Rose (2.3) and nearly twice as many as Connor (6.2).
Grammatical Complexity (Overall): Mean Word per AS-Unit
Note. Mean is calculated by dividing the total number of words by the total number of AS-units.
Clauses/AS-Unit and Clausal Density
Table 7 summarizes clausal density by displaying mean length of clause per AS-unit alongside a count of clauses per AS-unit by learner. These measures revealed an increase in clausal density that corresponded to the increase in learners’ grade levels. Not surprisingly, half of kindergartener Rose's AS-units were subclausal, such as 这个 // (this one) (Rose, Line 25) and 玩儿 // (play) (Rose, Line 156). Overall, she produced only two AS-units with more than one clause. As a second grader, Connor's speech sample included more than twice as many AS-units (110 vs. 50) and almost seven times as many clauses (164 vs. 27) when compared with Rose's. He also made more extensive use of two- and three-clause AS-units than either Rose or Dana. To exemplify, 它们^到草地上^后 (2) = :: 看见了小猪 // (after they arrived on the grass (2) = :: they saw the little pig //) (Connor, Line 77). Dana's speech sample evidenced the highest proportion of clauses per AS-unit, with a mean of 2.6 relative to Connor (1.5) and Rose (.5). Moreover, two thirds of Dana's AS-units consisted of two or more clauses and nearly one fourth included four to seven clauses. For example, {用-}要把一个水瓶, ^从 (1.5) 六英尺(1.5) {da-um} 放到^一个桌子 :: 不要用手 :: 碰到 :: 拿起来 :: 放在那里 {um}=// ({use-} we want to put a water bottle, from six inches {da-um} onto one table :: you don't want to use your hand :: to touch it :: pick it up :: put it over there {um}=//) (Dana, Line 33–34).
Grammatical Complexity (Sentential): Mean Clause per AS-Unit and Number of Clauses per AS-Unit
Note. Mean is calculated by dividing the total number of clauses by total number of AS-units.
Grammatical Complexity: Specific Measures
Types and Proportional Use of Prenominal Modifiers
For a specific measure of grammatical complexity, we calculated the proportion of modified nouns used by each learner (Table 8). A marked change in students’ use of modified noun phrases is apparent by grade level, which seems to indicate an overall higher level of phrasal complexity across the three learners (Rose had 20% of nouns modified; Connor, 38%; Dana, 58%).
Phrasal Complexity: Relative Percentage of Noun Phrases With Modifiers by Learner
Next, we identified the specific types of modifiers used and their relative frequency of use for each learner. Table 9 displays these findings. Rose produced two types of modified noun phrases: classifier/measure and associative phrases. Her use of modifiers worked similarly to modifier formation in her L1, in that numerals and possessive pronouns preceded the noun. For example, she used possessive phrases such as “我的妈妈 (my mom)” (Line 171). In each case, her use of classifier phrases included a number word followed by a most commonly used classifier 个, e.g., “三个熊” (three bears) (Line 223). More than 80% of Connor's use of modifying elements also fell into two categories: the classifier/measure (“每一个小朋友” (every child) (Line 177) and associative phrase (“种子的生长” (a seed's growth)) (Line 203). In contrast to Rose, he also produced several examples of attributives, for example, “比较长的尾巴” (a relatively long tail) (Line 43) and one example of modifying a noun with a prenominal relative clause, “我最喜欢的课” (the class that I like the most) (Line 188). Dana displayed an even greater facility with modifying noun phrases. She not only used more classifier/measure phrases than Connor but also made use of relative clauses seven times in her speech sample, indicating a more complex and nativelike use of Mandarin. As an example, in response to the interviewer's question about her favorite schoolwork, Dana replied, “我最喜欢的 :: 是我们现在在做的” // (what I like the most :: is what we are doing //) (Line 27).
Phrasal Complexity: Types and Proportional Use of Modifiers in Noun Phrases
Note. N = All instances of noun modifiers produced by the learner; n = the number for each type of modifier used. Because students sometimes used multiple modifiers for a given head noun, the N in this table does not equal the total modified noun count in Table 8.
Lexical Complexity Measures
Guiraud Index
The Guiraud index of all words (Table 10) and content words (Table 11) showed that Connor displayed the greatest lexical variety in all words produced and in nouns and verbs. While Dana's overall diversity of words was less than Connor's, she evidenced slightly greater variety in her production of adjectives. Overall, nouns were produced most frequently by all three learners, followed by verbs, and to a much lesser degree adjectives. However, as a word class, the increase in verb variety was most notable, with Connor and Dana each displaying approximately twice as much variety as Rose.
Guiraud Index-All: All Types to All Tokens
Note. G-All: Guiraud index for all. G-All calculation = all types/square root of all tokens.
Guiraud Index-Content: Content Types to Content Tokens
Note. G-n/N = Guiraud index for nouns; G-v/V = Guiraud index for verbs; G-a/A = Guiraud index for adjectives.
Guiraud index-content calculations = all content types/square root of content tokens.
Proportional Use of Unique Words by Tier
Table 12 displays the proportion of unique words used within each of the three tiers by learner. More than three quarters of all words produced by each of these learners were basic Tier 1 words. Kindergartener Rose produced the largest proportion of Tier 1 words (92%), whereas the largest proportion of more sophisticated Tier 2 and Tier 3 words were produced by second grader Connor. A notable 17% of Connor's unique words were Tier 2 in comparison to 11% for Dana and 7% for Rose. A few Tier 2 words produced by Connor included 上网 (to surf the Internet), 打破 (to break), 敲敲门 (to knock on a door). Unique words categorized as Tier 3 discipline-specific words were much fewer for all three learners, and yet Connor still produced slightly more than Dana, 5% to 4%, respectively. 科学 (science), 过程 (process), 生长 (growth) are examples of Connor's use of Tier 3 words.
Percentage of Unique Words by Tier Across Learners
Note. Unique words are the number of different words produced in one speech sample.
To summarize, results from statistical analyses of oral proficiency assessment data showed significant differences between kindergarten and Grade 2 scores in oral fluency, grammar, vocabulary, and listening comprehension but no further change in proficiency levels between Grades 2 and 5. Subsequent linguistic complexity analysis of speech samples of three students who mirrored median scores for their respective grade levels revealed notable differences in syntactic complexity across all three grade levels. However, while differences in lexis were evident between kindergarten and Grades 2 and 5, there was no evidence of further lexical growth between speech samples in Grades 2 and 5.
DISCUSSION AND IMPLICATIONS
SOPA and COPE assessment results from this study shed additional light on an emerging data set documenting Mandarin language outcomes of early total MI students in the elementary grades (Fortune & Song, Reference Fortune and Song2016). So far, they resemble trends for K–5 Spanish immersion students from early total programs using the same tools at the same grades (Fortune & Tedick, Reference Fortune and Tedick2015). In the Spanish study, significant whole-level differences were found in median scores of kindergarten and Grade 2 for all immersion students; however, median score changes beyond second grade were significant but limited to one sublevel for Spanish. In the current study, Mandarin median scores were unchanged between Grades 2 and 5.
Additionally, MI proficiency ratings were consistently at least one sublevel lower than Spanish immersion in oral fluency, grammar, vocabulary, and listening comprehension at all grades. Possible explanations for overall proficiency differences between immersion languages are many, we would like to call attention to three. First, Spanish immersion fifth graders’ instructional day was still 70% in Spanish in contrast to the 50%–60% of instructional time in Mandarin. Bilingual lexical development is impacted by reduced input (Montrul & Potowski, Reference Montrul and Potowski2007) and may have contributed to these findings. Second, differences in SOPA/COPE task design, administration and rating procedures, and CAL rater interpretations of proficiency domains for Spanish versus Chinese were evident, signaling a study limitation. Third, MI programs are almost exclusively taught by Mandarin L1 teachers from Chinese-speaking societies, primarily mainland China and Taiwan. Pedagogical trends in traditional Chinese-speaking contexts tend toward more teacher-directed, whole-class approaches that are achievement oriented with more time allocated to lecturing with questions (Li, Rao, & Tse, Reference Li, Rao and Tse2012). Thus, disparities in oral proficiency outcomes might also stem from differences in instructional practices due to teacher background as well as the potentially greater time demands of character literacy teaching.
Grade 5 COPE assessment results presented here also compare favorably to an earlier report of speaking and listening proficiency outcomes for fifth-grade MI students (Fortune & Song, Reference Fortune and Song2016). Using a different proficiency assessment tool, the STAndards-based Measurement of Proficiency 4 Skills elementary (STAMP 4Se) (Avant Assessment, 2012), Fortune and Song tested MI students (80 students for speaking; 81 for listening) from three early total programs. In their study, 39 students also participated in the COPE assessment reported here. Results from both assessments find Grade 5 students’ listening skills higher than speaking and generally in the intermediate mid to high range. Results for speaking, on the other hand, indicate overall higher ratings using COPE with three quarters of students scoring between 5–JIM and 7–JAL when compared with the STAMP 4Se finding, 86% at intermediate-low or below. This score discrepancy held even when we compared STAMP 4Se and COPE scores for the exact same 39 students. Such score inconsistency seems to expose the challenge of measuring oral proficiency using global rating scales, and in particular, adequately detailing the morphosyntactic features of Mandarin oral language in the intermediate to early advanced range.
Further evidence of this challenge emerged in the lack of difference found in Grades 2 and 5 SOPA/COPE outcomes for MI students and complexity differences found in MI speech samples. In general, a wide range of complexity metrics confirmed a noticeable difference in the degree of grammatical and lexical complexity between oral language produced by Rose (kindergarten) and the other two students, Connor (Grade 2) and Dana (Grade 5). These findings were not surprising in light of significant differences found in our statistical comparison of median scores for students in these same grades. However, we did not anticipate finding additional, and at times substantive, complexity differences between Connor (Grade 2) and Dana (Grade 5), given the lack of median score changes in any of the three speaking criteria at these grade levels. Instead, Dana's language production displayed greater grammatical complexity as measured by mean words, use of multiple embedded clauses, and prenominal noun modification. SLA scholars have discussed modification of noun phrases as a sign of greater syntactic maturity and complexity (Garman, Reference Garman1990). Similarly, positive changes were not found in Dana's lexical variety and sophistication. In fact, Connor consistently outperformed Dana on these measures.
Dana's (Grade 5) relatively lower lexical complexity measures are interesting and somewhat surprising. Of course, it could simply be due to L2 acquisition strengths and weaknesses of the individual learners among myriad other possibilities. Nonetheless, an earlier follow-up probe of vocabulary results from Spanish immersion students’ performance on SOPA and COPE assessments drew attention to overall lower proficiency scores in vocabulary relative to the other speaking domains of oral fluency and grammar (Fortune & Tedick, Reference Fortune and Tedick2015). In this probe, the percentage of students who scored in the Jr. Advanced Levels 7–9 were compared by grade level and proficiency domain. We replicated this analysis with these data. However, because MI students’ median scores were repeatedly one sublevel lower than those found for Spanish immersion students participating in the same early total immersion model, we looked at the proportion of MI students scoring one sublevel lower, between 6–JIH and 8–JAM. As before, relatively fewer advanced scores were achieved in vocabulary in Grades 2 and 5 when compared with oral fluency and grammar (Figure 5).
Percentage of MI Students at Pre-Advanced Levels (6-8) by Grade and Skill Area
As a result of these proficiency assessment and complexity findings, two curricular and instructional implications and one assessment implication come to mind. First is the critical importance of paying ongoing, systematic attention to vocabulary development with immersion learners (Fortune & Tedick, Reference Fortune and Tedick2015; Met, Reference Met, Fortune and Tedick2008). Given typological differences between how words are identified and formed in English and Mandarin, it is even more imperative that MI teachers work to expand their students’ vocabulary by explicitly building character knowledge and word-forming skills (Anderson & Li, Reference Anderson, Li, McKeough, Phillips, Timmons and Lupart2006). As an example, during the role-play, Dana struggled in her search for a way to say 不公平 (unfair) while responding to a school administrator's new rule about having school on Saturdays. She seemed to recall the sound /ping/ and basic meaning (“something flat”) of the second character in the word she needed, 公平 (fair), and used that partial knowledge to try out two possible words, 平衡 (balanced) and 平均 (even). Both of these words begin with the /ping/ sound and carry the basic meaning of the character 平, but neither were correct. Perhaps with the additional knowledge that individual characters such as平/ping/ can maintain their basic meaning but occur either as the first or second syllable of a word she may have been more successful in finding her word 公平 (fair). In Mandarin, it is important for students to see characters as individual units of meaning that join with other characters in various positions to make words. They need many meaningful and contextualized learning experiences to develop this understanding and practice word formation using characters.
A second pedagogical implication involves noun modification using a prenominal relative clause (RC), a commonly produced syntactic structure in Mandarin oral language. It differs from English in that the Mandarin RC is positioned before the noun it modifies. Dana's speech sample showed a proportionally greater use of this structure than Connor's; however, both Dana's and Connor's use of the RC was mainly restricted to the same basic communication, 我最喜欢的 + “noun” (the “noun” I like the most). It is likely that at school students are given numerous opportunities to talk about their favorite things, so this may be a familiar memorized chunk of language. Because the RC-modifying structure constitutes language that differs in nonobvious and unexpected ways and expands learners’ ability to produce a wide range of similarly structured communications (Harley, Reference Harley1993), it lends itself particularly well to form-focused instruction within an integrated subject-based immersion lesson or unit. Targeting linguistic features for instruction in a way that takes the feature's developmental sequence into account is especially helpful (Pienemann, Reference Pienemann1998). Wang and Feng (Reference Wang, Feng and Nan2014) suggested that in Mandarin direct object RC is the easiest one to acquire, for example, 我正在做的数学题 (the math problem I am solving). The subject RC, 正在上升的滑轮 (the pulley that is going up), and other RC types develop later. Thus, MI teachers would benefit by beginning with integrating a focus on the direct object RC before introducing students to the subject RC.
SOPA/COPE assessments, as global measures of four oral proficiency domains, did not seem able to distinguish the more nuanced developmental changes in MI learner language in the intermediate range and beyond. Findings from this study reinforce those from earlier studies that suggest a need for more progress-sensitive assessment tools especially for the early total immersion context (Fortune & Tedick, Reference Fortune and Tedick2015). If 7- and 8-year-old students are already reaching Jr. intermediate mid to high levels of L2 oral proficiency, it is clear that students, parents, teachers and program leaders need more finely grained measures that can identify linguistic progress as students continue to advance through upper elementary and beyond. Analyzing and reporting on aspects of the linguistic CAF of early total immersion students at various intermediate proficiency levels may further inform the development of new L2 proficiency assessments and improve existing rating scales and tools.
CONCLUSION
This cross-sectional study contributes findings on the speaking and listening skills of early total MI students. Results indicate a significant and rapid change in proficiency by Grade 2 but no further change was evidenced by Grade 5, after three additional years of program participation. While information from assessment instruments that use global rating scales to determine oral proficiency is generally informative, findings from this study underscore global assessments’ inability to capture proficiency changes for early total immersion students beyond Grade 2. We argue that for global proficiency ratings to be useful for instruction and more meaningfully interpreted by early total immersion stakeholders, additional linguistic analysis is important. While the level of analysis used in this study may not be feasible due to time and cost constraints, school-based SOPA/COPE assessment rating teams could take advantage of their note-taking and rating experiences immediately following test administration. Notations of specific linguistic features used during and/or absent from students’ language production at various stages of the intermediate to early advanced proficiency range could inform instructional needs and, if shared with test developers, refine rating scales over time. Written records of questions that emerge as the raters work to distinguish and rate the various language samples could also be helpful in distinguishing gradations within these global levels.
Our multicase study of complexity analysis in three students provided an additional lens to assist our exploration of what learner language is possible for MI students in kindergarten, Grade 2, and Grade 5. It also allowed us to further test our initial global proficiency assessment results and, most importantly, suggest implications for curricular and instructional improvements in MI. Finally, this complexity analysis demonstrates a potentially useful approach to detecting developmental nuance within the intermediate proficiency range. Such knowledge can then inform improvements to existing assessments of oral proficiency in Mandarin.
Acknowledgments
The authors gratefully acknowledge partial funding for this research project from the U.S. Department of Education Foreign Language Assistance Program and the Title VI National Language Resource Centers Program. We are indebted to the teams who collected data for this study and to school administrators for their support. We are especially grateful to Ping Peng and Molly Wieland for their assistance. Finally, we extend our respectful gratitude to colleagues, Elaine Tarone, Kate Paesani, and Roy Lyster and to editorial content reviewers for their insightful feedback.
