INTRODUCTION
The use of openly available rigorous instrumentation has received heightened attention by second language acquisition (SLA) researchers since the creation of the IRIS repository almost 10 years ago (Marsden et al., Reference Marsden, Mackey, Plonsky, Mackey and Marsden2016), and more recently with the endorsement of Open Science policies by several SLA journals. Instrument sharing enables better meta-analyses (Oswald & Plonsky, Reference Oswald and Plonsky2010) and replications (Porte & McManus, Reference Porte and McManus2018). It more broadly supports knowledge generalization about key constructs in SLA research. One of those constructs is second language (L2) proficiency: Whatever the L2 phenomenon of primary interest may be in a given study, researchers will want to characterize the language abilities of their samples, and they will often need to consider proficiency as a control or moderating variable (Norris & Ortega, Reference Norris, Ortega, Gass and Mackey2012).
Yet, much like Thomas (Reference Thomas, Norris and Ortega2006), Tremblay (Reference Tremblay2011), and Hulstijn (Reference Hulstijn2012) reported of past decades, a present perusal of SLA research domains reveals that L2 proficiency continues to be a neglected variable in SLA measurement. For example, many SLA researchers continue to rely on imprecise proxies of language ability such as academic course level or years of instruction (e.g., 73% of 62 replication studies in Marsden et al., Reference Marsden, Morgan-Short, Thompson and Abugaber2018) and all too many still provide no information about the proficiency of their samples (e.g., 17% of 302 primary studies in Plonsky et al., Reference Plonsky, Marsden, Crowther, Gass and Spinner2019). Practicality is likely part of the problem, as collecting scores from standardized tests is time consuming and high cost. The complexity of the construct may also contribute to the problem (e.g., Purpura, Reference Purpura2016; also Leclercq et al., Reference Leclercq, Edmonds and Hilton2014). The current situation would likely improve if researchers knew they could turn to tools for the measurement of L2 proficiency that could meet their research needs in rigorous yet practical ways.
In the present study, therefore, we seek to contribute to instrument sharing and the improvement of L2 proficiency measurement. We do so by focusing on elicited imitation (EI) (also called sentence repetition), a well-established technique (Yan et al., Reference Yan, Maeda, Lv and Ginther2016), whereby participants are asked to listen to a sentence and repeat it back as closely as possible. A suite of EI tasks (EITs) originally developed by Ortega and her colleagues Iwashita, Norris, and Rabie (Ortega et al., Reference Ortega, Iwashita, Norris and Rabie2002) is unique in that it exists in parallel for seven L2s, all available on IRIS. Despite its strengths, only one form exists for each target language. The unavailability of alternate forms limits the usefulness of the instrument, for example whenever a study design calls for the repeated administration of the EIT to the same participants over time, in longitudinal designs, or across languages, in crosslinguistic research. We thus report on a new parallel form of Ortega et al.’s original EIT for L2 English. We created the new form and administered it together with the original to a sample of 82 L2 English participants. We then compared the evidence regarding the two forms’ suitability for measuring L2 English proficiency. In addition, we collected the participants’ perspectives on their EI performances and evaluated those perceptions against their scores.
EITS IN L2 RESEARCH
EI began to be utilized in the 1960s in the field of child language development (e.g., Fraser et al., Reference Fraser, Bellugi and Brown1963). It was quickly taken up in L2 research (e.g., Naiman, Reference Naiman1974) and particularly since the mid-2000s it has witnessed a surge of interest among SLA researchers. Yan et al. (Reference Yan, Maeda, Lv and Ginther2016), who meta-analyzed 21 studies published between 1982 and 2014, identified about 15 different instruments. Since Yan et al., additional EI instruments have been developed, for example, for French (Gaillard & Tremblay, Reference Gaillard and Tremblay2016), Russian (Drackert, Reference Drackert2015), Dutch (Deygers, Reference Deygers2020), and English (Yan, Reference Yan2020). Two main uses for EITs are to empirically characterize the global proficiency of participants and to uncover meaningful proficiency differences within a sample (e.g., Bowden, Reference Bowden2016; Gaillard & Tremblay, Reference Gaillard and Tremblay2016). Other research purposes include determining whether grammatical knowledge is implicit (Ellis, Reference Ellis2005); finding out what linguistic complexity measures can best predict overall L2 proficiency (Iwashita, Reference Iwashita2006; Yu, Reference Yu2021); gauging the extent to which L2 proficiency mediates the relationship between first- and second-language fluency (Duran-Karaoz & Tavakoli, Reference Duran-Karaoz and Tavakoli2020; Huensch & Tracy-Ventura, Reference Huensch and Tracy-Ventura2017); showing grammatical development along preestablished stages (Baten & Cornillie, Reference Baten and Cornillie2019); or screening participants into functionally monolingual and bilingual groups (Park, Reference Park2020). A particular attraction of EITs is their practicality: They can be administered in about 10 minutes per participant, whereas standardized proficiency tests can take an hour or longer. In fact, EITs are even shorter than other short-cut measures of L2 proficiency, such as cloze tests (Tremblay, Reference Tremblay2011) or C-tests (Drackert & Timukova, Reference Drackert and Timukova2020).
Ortega et al. (Reference Ortega, Iwashita, Norris and Rabie2002) originally developed parallel EITs for English as well as German, Japanese, and Spanish. Three other target language versions followed later: French, Korean, and Mandarin. The instrument has been validated and expanded in several studies, notably the Mandarin EIT by Wu and Ortega (Reference Wu and Ortega2013) and in replication by McManus and Liu (Reference McManus and Liu2020); the French EIT by Tracy-Ventura et al. (Reference Tracy-Ventura, McManus, Norris, Ortega, Leclercq, Hilton and Edmonds2014); the Spanish EIT by Bowden (Reference Bowden2016), Park et al. (Reference Park, Solon, Henderson and Dehghan-Chaleshtori2020), and Solon et al. (Reference Solon, Park, Henderson and Dehghan-Chaleshtori2019); and the Korean EIT by Kim et al. (Reference Kim, Tracy-Ventura and Jung2016).
Surprisingly, the EIT for L2 English is the least investigated in the seven-language suite. In the original crosslinguistic EIT development study, which is available only as unpublished data, Ortega et al. (Reference Ortega, Iwashita, Norris and Rabie2002) inspected the EI performances of a sample of 39 learners of English in Japan at low levels of proficiency and reported a high internal reliability of Cronbach’s alpha 0.93 and moderate external validity with TOEFL scores of r = .49. Since then, to the best of our knowledge, the English EIT has been used by Duran-Karaoz and Tavakoli (Reference Duran-Karaoz and Tavakoli2020) and Park (Reference Park2020), but in both studies it was only employed as a proficiency measure rather than being the focus of investigation. Thus, in investigating the newly developed parallel English form, our study also provides validation evidence for the original English EIT form.
WHAT DO EITS MEASURE?
The utility of EITs as a shortcut of L2 proficiency rests on the psycholinguistic rationale that, to be able to repeat a sentence accurately, a speaker needs to parse it into chunks or grammatical units and generate a mental representation using his or her internal productive grammar system—provided of course that the stimuli are long enough to avoid pure memory-based repetition (Bley-Vroman & Chaudron, Reference Bley-Vroman, Chaudron, Tarone, Gass and Cohen1994). It is in this sense that researchers speak of EI as reconstructive.
In their meta-analysis, Yan et al. (Reference Yan, Maeda, Lv and Ginther2016) concluded that EI works better when used to measure global constructs (e.g., general language proficiency or implicit grammatical knowledge) than when used to measure discrete linguistic knowledge or skills (e.g., syntactic or phonological competence). Some insight as to why this may be comes from Klem et al. (Reference Klem, Melby-Lervåg, Hagtvet, Lyster, Gustafsson and Hulme2015), a longitudinal EI study of English as a first language (L1) conducted with 216 children, starting when they were 4 years old and ending when they were 6. Using longitudinal structural equation modeling on data collected over three points in time, these researchers found that EI performances loaded onto a single unidimensional language latent factor together with grammatical and vocabulary scores. Klem et al. took this to mean that EI is “best seen as a complex linguistic task that reflects the integrity of language processing systems at many different levels,” such as speech perception and speech production, vocabulary knowledge, and grammatical processing (p. 152). Thus, in addition to reconstructive, we can think of EI as integrative.
The reconstructive and integrative nature of EI is consistent with Hulstijn’s (Reference Hulstijn2015) Basic Language Cognition (BLC) in his binary proficiency model. To do well on an EIT, a language user must have in place automatic language processing reflective of BLC, which serves mundane communicative abilities and should not vary fundamentally within or between native and competent nonnative speakers. In this sense, by tapping into a speaker’s implicit knowledge under time-limited conditions of oral production, EITs offer some advantages over other proficiency shortcuts that are strictly noncommunicative, such as cloze tests, C-tests, grammaticality judgment tasks, or vocabulary size tests. Still, at face value EI is a psycholinguistic technique that elicits the automatized verbatim repetition of isolated sentences. It is a challenge, therefore, to show conceptual linkages between EI and the kinds of mundane, everyday communicative abilities that are served by core BLC abilities. Nevertheless, in the context of measuring global language ability in adults for a maximally diverse set of SLA research purposes, establishing those linkages remains important.
EITS AND OTHER INDICATORS OF PROFICIENCY
One way to provide evidence for the suitability of a test is to investigate its external validity by comparing the scores it produces with other “measures of … abilities gleaned from outside the test” (Alderson et al., Reference Alderson, Clapham and Wall1995, p. 171). Evidence for the external validity of EITs has been encouraging.
A common choice has been to inspect the relation of EI performances to standardized proficiency test scores. The results show good alignment, as evinced in Pearson correlations, with a range of oral and written tests: (1) the Center for Applied Linguistics’ Simulated Oral Proficiency Interview, which correlated with Spanish EIT scores at r = .88 in Ortega et al. (Reference Ortega, Iwashita, Norris and Rabie2002) and r = .91 in Bowden (Reference Bowden2016), and with Japanese EIT scores at r = .61 in Ortega et al.; (2) the Spanish DELE exam, at r = .92 in Solon et al. (Reference Solon, Park, Henderson and Dehghan-Chaleshtori2019); (3) the TOEFL, at r = .49 in Ortega et al.; (4) a homemade speaking test for L2 Korean modeled after the TOEFL speaking section, at r = .77 in Kim et al. (Reference Kim, Tracy-Ventura and Jung2016); (5) the grammar section of the Oxford Placement Test at r = .51 in Duran-Karaoz and Tavakoli (Reference Duran-Karaoz and Tavakoli2020); and (6) the listening section of the Korean TOPIK, at r = .62 in Kim et al.
It has been also common practice in past L2 EIT research to elicit discourse-level performance, for example using a written essay or a narrative prompted by pictures, to inspect the relationship between EI scores and the dimensions of complexity, accuracy, and fluency (CAF) in the language produced on these communicative tasks. CAF measurement is well established in SLA (e.g., Bulté & Roothooft, Reference Bulté and Roothooft2020), and so is a distinction made by Skehan (Reference Skehan2009) and widely accepted by CAF scholars between general metrics that focus on global CAF qualities of language production, and specific metrics that focus on the produced CAF of certain structures that are assumed to reflect the conceptual demands of a task, such as the verbalization of motion events to describe narratives containing many kinds of movement (e.g., Robinson et al., Reference Robinson, Cadierno and Shirai2009). Table A (see supplementary material) summarizes findings about the relationship between EIT scores and general and specific CAF measures reported across nine EIT studies. A few conclusions can be drawn from that table: (1) the relation between EI performances and syntactic complexity typically yields medium to large positive magnitudes, (2) the more scarce available evidence on lexical complexity and accuracy suggests these two dimensions of CAF might predict EI scores reasonably well, (3) speed fluency (operationalized as speech rate, often measured as syllables per minute) has emerged as a contributor to EI performances, and (4) EI performances align well with both general and specific CAF metrics.
Although less frequently, self-assessments of proficiency have also been used to establish external validity for EITs. Using single statements on a 5-Likert scale response ranging from 1 = very poor to 5 = nativelike, Bowden (Reference Bowden2016) found that holistically self-assessed speaking (r = .75) and listening (r = .73) yielded the highest correlations with the EIT, whereas correlations with ratings for reading and writing were lower but still substantial (r = .62 and .59, respectively). In the original EIT study, Ortega et al. (Reference Ortega, Iwashita, Norris and Rabie2002) employed a self-assessment questionnaire in the form of Can-Do scales developed by Clark (Reference Clark and Barrows1981) and used by the Center for Applied Linguistics in the 1990s. They reported medium correlations for EFL (r = .47), Japanese (r = .53), and Spanish (r = .57), and high for German (r = .76). Can-Do statements (e.g., “I can understand a voice message from the airlines about changes to a flight schedule” or “I can ask for directions to a place”) have gained increasing popularity among language educators as a promising approach to formative self-assessment, and they show good alignment with test scores reflecting both ACTFL (e.g., Ma & Winke, Reference Ma and Winke2019) and CEFR (e.g., Denies & Janssen, Reference Denies and Janssen2016) proficiency descriptors.
In sum, the evidence for the external validity of the EIT is strong. It is particularly abundant vis-à-vis standardized proficiency test scores, rather robust vis-à-vis CAF measures, and relatively positive, albeit limited, when it comes to self-assessments. Based on these insights, in the present study we chose CAF measurement and self-assessment using Can-Do statements as our two main strategies to establish the external validity of the two English EIT forms. In addition, we decided to elicit participants’ perception of the EIT and explore empirically whether these perceptions aligned well with their actual EI performances. This is something that, to the best of our knowledge, has not been attempted before. We reasoned that an investigation into factors that are perceived by L2 users to influence their EI performance can add important insights into how and why EITs work as a short-cut measure of global L2 proficiency.
THE BENEFITS OF PARALLEL FORMS
The Ortega et al.’s (Reference Ortega, Iwashita, Norris and Rabie2002) EIT suite presents many strengths, as shown across the validation studies cited earlier. Moreover, it holds great promise for use in crosslinguistic research because the same parallel design composed of 30 progressively longer and harder sentences is consistently used across all seven languages. For example, Huensch and Tracy-Ventura (Reference Huensch and Tracy-Ventura2017) were able to investigate fluency for French and Spanish within the same study after establishing the respective proficiencies of their two samples through parallel versions of the same EIT in the two target languages. Interest in investigating the same participants’ multiple languages has been on the rise in SLA, in research programs that focus on third-language (L3) learning and multilingual (Ln) learning (e.g., Westergaard’s, Reference Westergaard2021, keynote article and responses). Thus far, participants’ proficiencies across their languages have been rarely measured in these types of studies. Yet, the variable L2, L3, Ln knowledge of multilingual participants should not go unmeasured, as each language can interact with one another, and thus relative proficiency will almost surely affect the interpretation of results. In the future, this crosslinguistic EIT suite may facilitate precision in the measurement of multilinguals’ knowledge by having participants complete the EIT in two or more of their learned languages. However, as is true of all other existing EITs—for example, all those included in Yan et al.’s (Reference Yan, Maeda, Lv and Ginther2016) meta-analysis—only one form exists for each target language. The unavailability of parallel forms reduces the usefulness of any research instrument, including any EIT.
For longitudinal research that calls for repeated instrument administration, the challenge is clear. For example, in the absence of parallel forms, Huensch and Tracy-Ventura (Reference Huensch and Tracy-Ventura2017) for French and Spanish, Serafini and Sanz (Reference Serafini and Sanz2016) for Spanish, and Wu (Reference Wu2017) for Chinese had no recourse but to use the same exact EIT twice, within a short time interval ranging from 3.5 to 5 months. In all three cases, the purpose was to gauge proficiency differences before and after studying abroad. But the repeated administration of the same EIT stimuli to the same participants in these pre- and posttest designs makes it impossible to disentangle study abroad effects from practice effects. Namely, after the first administration, comprehension and thus repetition of the same exact EI stimuli might lead to some (unknown) improvement in subsequent administrations (e.g., as would be predicted by well-known priming effects; Bock, Reference Bock1986; Mahowald et al., Reference Mahowald, James, Futrell and Gibson2016). Practice or carryover effects may be a concern not only in longitudinal designs but also for L3/Ln research because multilinguals would be asked to listen and repeat stimuli that are semantically identical, albeit in different languages. Having two L2 English parallel forms of the same EIT will be particularly useful for L3 and Ln studies featuring samples of participants for whom English is neither the L1 nor the target language because knowledge of L2 English will almost inevitably be part of their multilingual repertoire. For example, Schepens et al. (Reference Schepens, Van der Slik and Van Hout2016) found that 68% of their very large sample of 39,300 multilinguals reported English as their strongest or best L2. Not only are such learners a likely majority in the world, but as Long and Geeslin (Reference Long and Geeslin2017) have argued, evidence from them can yield distinct benefits for knowledge generalization in SLA.
In sum, by developing a parallel form for one of the seven target language EITs made available by Ortega et al. (Reference Ortega, Iwashita, Norris and Rabie2002), L2 English, we hoped to support longitudinal research into L2 English as well as sharper research into L3 and Ln learning by multilingual speakers whose repertoire includes variable knowledge of L2 English.
RESEARCH QUESTIONS
The present study investigated the reliability, form equivalence, external validity, and suitability of a newly developed parallel form of the L2 English EIT (henceforth EIT B), by comparison to the original form (henceforth EIT A) that was developed and investigated by Ortega et al. (Reference Ortega, Iwashita, Norris and Rabie2002). Our research questions were:
RQ1. What is the internal consistency of the EIT B and its parallel forms reliability, as compared with the original EIT A?
RQ2. How well do participants’ EIT A and EIT B performances align with other indicators of L2 proficiency?
RQ3. How well can the EIT A and EIT B distinguish between four proficiency groups in the sample?
RQ4. How well do the items on each EIT form discriminate participants’ performances?
RQ5. What factors did participants say played a large role in their ability to repeat sentences, and how accurate are their perceptions?
METHOD
PARTICIPANTS
A total of 82 L2 English users were recruited. This sample size exceeds the projected sample size of 72 needed for the study, based on an a priori power analysis on the data from Wu and Ortega (Reference Wu and Ortega2013) (N = 80; medium effect size partial η2 = .20 according to Cohen, Reference Cohen1988), which tested the difference among four group means with an alpha of 0.05 and a power of 0.95 using G*Power with F tests (Faul et al., Reference Faul, Erdfelder, Lang and Buchner2007).
The participants had Mandarin Chinese as their L1 and represented a wide range of abilities and language experiences typically seen of adults in areas of the world where English as a foreign language (EFL) is part of the compulsory school curriculum growing up. Three groups of participants were recruited in China or Taiwan and included full-time employees who claimed some knowledge of English (N = 26), non-English-major university students (N = 21), and English-major university students (N = 20). A fourth group (N = 15) was recruited at a public university in the United States and comprised overseas Chinese originally also from China or Taiwan. While they had grown up under the same EFL learning conditions as participants in the other three groups, their learning environment had expanded in recent years, and at the time of the study they had lived in the United States for at least two years and were active users of academic English. The ages of individual participants were not collected, but they ranged between 18 for the youngest participants, who were university freshman students, and 43 for the oldest participants, who were full-time employees.
PROCEDURE AND INSTRUMENTATION
The study was cleared by the Institutional Review Board of the first two authors’ university. All participants signed informed consent and were compensated with an amount equivalent to between US$7 and US$15, depending on the country. They completed four tasks, in this order: EIT A or EIT B, a self-diagnostic survey, an oral narrative task, and EIT B or EIT A. That is, the two EIT forms were counterbalanced, and the self-diagnostic survey and the oral narrative task were done in between the two. The session comprising all four tasks took between 35 and 40 minutes in total. In the following, we describe each instrument in turn.
EIT A AND EIT B
The EIT A was directly taken from Ortega et al. (Reference Ortega, Iwashita, Norris and Rabie2002) on the IRIS repository, and the EIT B was developed by the present authors; the instructions and practice sentences were rerecorded in Chinese and inserted in both EIT forms.
The development of the parallel EIT B required that we closely match the 30 original EIT A items and create 30 equivalent new sentences (see Appendix A in the supplementary material for all stimuli). Findings reported in the literature suggest syntactic structures and sentence length are the two key moderators for sensitivity of EI (e.g., Wu & Ortega, Reference Wu and Ortega2013; Yan et al., Reference Yan, Maeda, Lv and Ginther2016). Therefore, following the same procedure adopted in the development of the original seven-language suite, we first considered retention of key syntactic structures for each item from the EIT A to the EIT B (e.g., a relative clause, a subordinate clause, or a comparative structure in an original item) while maintaining the exact same number of syllables in the fixed order from shortest (7 syllables) to longest (19 syllables). To enhance the comparability of vocabulary difficulty across the two forms while keeping the semantic distance between the parallel items as apart as possible, we selected among word options based on the five-band classification of word frequency by Collins Online English Dictionary (2020) to generate sentences that matched the EIT A as closely as possible. Using the text analysis function of the Corpus of Contemporary American English (COCA), we found that the resulting two EIT forms do align well in terms of vocabulary frequency based on the distribution of frequency bands categorized by COCA as low (1 to 500 rank), mid (501 to 3,000 rank), and high (rank higher than 3,000). Specifically, EIT A contains 4.19% of low-frequency words, 13.55% mid-frequency words, and 82.27% of high-frequency words; EIT B contains 4.82% of low-frequency words, 13.53% mid-frequency words, and 81.65% of high-frequency words. The 30 new sentences were examined by a native speaker of English to ensure they sounded natural and accessible.
After development of the 30 items for the EIT B, the new items were recorded. The EIT A features a female native English-speaking voice from the original version; the newly developed EIT B features a male native English-speaking voice. The stimulus sentences of both EIT A and EIT B are read at a natural reading speed, and the sound recording for the 30 stimuli aligns well with each other, with a comparable length of around 7 minutes and 7 seconds. In all other aspects of the EIT B, we followed Ortega et al.’s (Reference Ortega, Iwashita, Norris and Rabie2002) original design. Both EIT forms contain 30 English sentences sequenced by length in a fixed order that is presented to participants aurally, one sentence at a time, starting with seven syllables in length and becoming progressively longer up to 19 syllables. There is an interval of 2.5 seconds between the end of each sentence and a ring tone prompting participants to start their repetition, to preempt rote repetition. The time allotted for each repetition is standardized, with two extra seconds added to the time it takes the native speaker to enunciate the sentence plus an additional 0.5 second added for every syllable beyond the length of seven syllables. Participants are instructed in their L1 Mandarin to repeat as much as they can and are given a few practice sentences in their L1 before they start repeating the target English sentences in the given order.
SELF-DIAGNOSTIC SURVEY
The self-diagnostic survey (shown in Appendix B in the supplementary material) was given to participants in Mandarin Chinese, immediately after completing one of the two EIT forms. The participants responded to three sections in sequence, tapping: (1) perceptions of performance during the first set of EIT; (2) self-reported proficiency in English; and (3) history of learning English.
Regarding (1), through five questions we asked participants to report the level of difficulty of the EIT, their self-evaluation of how well they did, and how factors such as vocabulary, grammar, accent, or pace of speech of the stimuli affected, in their opinion, their comprehension and repetition of the sentences. Regarding (2), all participants were asked to rate their abilities for English listening and speaking in turn, by (a) self-rating their holistic ability on a 5-point Likert scale (5 = Excellent, 1 = Needs Improvement) and (b) providing a self-evaluation (5 = Not At All Difficult, 1 = Extremely Difficult) of 11 Can-Do statements. Taken from the NCSSFL-ACTFL Can-Do Statements (ACTFL, 2015), they represent linguistic tasks for listening and speaking ranging from a Novice Low to a Distinguished level. Finally, with respect to (3), the participants’ history of English learning was captured in six questions about their major in college, age of acquisition of English, immersive experience in an English-speaking country, and so forth.
Most items in the survey elicited 5-point Likert scale responses. For the scaled questions combined (k = 8, Questions 1 through 4 in Section 1, and 6 through 9 in Section 2), we obtained a Cronbach’s alpha value at 0.71, suggesting satisfactory scale reliability (DeVellis, Reference DeVellis2003). A calculation of reliability on each set of Can-Do statement scales on its own (Questions 7 and 9, respectively) also showed high internal consistency: Cronbach’s alpha was 0.90 for listening and 0.94 for speaking.
L2 ORAL NARRATIVE TASK
The oral narrative task was the same used originally by Wu and Ortega (Reference Wu and Ortega2013) (see Appendix C in the supplementary material). It required participants to tell a story in English based on 12 pictures, depicting how a boy lost his dog when taking a walk, searched everywhere for him, and eventually reunited with him. The story pictures depicted a total of 12 motion events, and participants were instructed to describe the pictures in as much detail as possible. The oral narrative performance was then used to extract the four CAF measures described in the next section.
CODING
Results of the two EITs were first transcribed by a trained English native speaker using Audacity (Audacity Team, 2012). Each sentence was then scored by the first two authors using the five-point scoring rubric developed in Ortega et al. (Reference Ortega, Iwashita, Norris and Rabie2002), shown in Table B (supplementary material). The two raters’ simple percent agreement on initial ratings was 94.5%, and the differences were reconciled through discussion.
All of the 5-point Likert scale ratings of the self-diagnostic survey were tallied. Participants’ ratings on how well they could perform on each of the 11 progressively more challenging tasks described in the Can-Do statements were summed up for listening and speaking, yielding a possible total of between 11 and 55 points on each set.
As for the performances on the oral narrative task, each story was transcribed by an English native speaker. The transcripts were then segmented into clauses, first automatically using the Web-based L2 Syntactic Complexity Analyzer (Lu, Reference Lu2010) and then verifying the results manually. A clause was a unit that contained a unified predicate for a single situation (Berman & Slobin, Reference Berman and Slobin1994, p. 657); clauses that contained more than one unintelligible word were excluded. All clauses that described an entity moving from one location to another were identified and coded as motion clauses.
Four CAF measures were calculated. Number of clauses produced was taken as an indication of total productivity in terms of communicative adequacy in retelling the story. Number of motion clauses produced was considered as an indicator of communicative effectiveness in terms of how well a participant was able to complete the task of narrating a story seeded with 12 motion events. Number of motion verb types was adopted as an indicator of lexical diversity. Finally, speech rate, operationalized as number of syllables per minute, was an indicator of fluency and automaticity in speech. The first three measures (one being general and two with a specific focus on motion events) had been used by Wu and Ortega (Reference Wu and Ortega2013) and McManus and Liu (Reference McManus and Liu2020) on the same narrative prompt, although for Mandarin L2 rather than English L2 discourse. We added number of syllables per minute as a fourth CAF measure to inspect speech fluency, as in several previous EIT studies (see Table A, supplementary material). The number of syllables was calculated using the Syllable Counter tool online (https://www.howmanysyllables.com/syllable_counter/).
RESULTS
INTERNAL CONSISTENCY AND PARALLEL FORMS RELIABILITY
The descriptive statistics for the two EIT forms obtained for the full sample of N = 82 are presented in Table 1. After checking that the main assumptions of normality, linearity, and homoscedasticity were met, we proceeded to address the first research question. Both EIT forms displayed high Cronbach’s alpha values of 0.97 for the EIT B and 0.96 for the original EIT A. The Pearson’s correlation coefficient between the scores of EIT B and EIT A for the total sample was very high (r = .96, p < .01), explaining 91% of the shared variance. In response to Research Question 1, the newly created EIT B form had an excellent internal consistency comparable to that of the EIT A form and the parallel forms reliability was strong (Webb et al., Reference Webb, Shavelson and Haertel2006).
Descriptive statistics on EIT A and EIT B (N = 82)
ALIGNMENT WITH OTHER INDICATORS OF L2 PROFICIENCY
The descriptive statistics for each indicator of L2 proficiency are presented in Table C in the supplementary material. We addressed the second research question first through correlations because they have been reported across most EIT studies as an index of the relationships between EI performances and the external measures. We then ran multiple regressions as an additional index of the predictive power of each external measure.
Table 2 summarizes the strength of correlations between each external indicator of L2 proficiency and the scores obtained on each form of the EIT. As shown in Table 2, both sets of EIT scores yielded strong and statistically significant positive correlations with self-ratings on Can-Do statements for listening and speaking (shared variance between 44% to 64%) and with the CAF measure of fluency (speech rate, shared variance with EITs at 56% for EIT A and 50% for EIT B). The shared variance explained by the other three CAF measures was consistently smaller. Across the board, the magnitudes observed were comparable for both EIT forms.
Pearson’s correlations between EITs and six proficiency indicators
Notes: All correlations were significant at the 0.01 level (2-tailed). Ninety-five percent CIs indicate an approximation of the upper and lower bounds of the true correlation coefficients with a 95% probability.
We followed up the correlation analyses with two separate multiple regressions. For the multiple regression that examined the Can-Do statements, we entered speaking into the equation first, followed by listening. We did this because these are two different constructs, evaluated separately in the self-diagnostic survey, and because speaking is arguably even more directly central to EI performance than listening. All variables were checked for normality, linearity, and homoscedasticity assumptions as well as for multicollinearity. Can-Do statements for speaking was a significant predictor of EIT A and EIT B scores at p < .01 level with a Beta value of 1.44 for EIT A and 1.54 for EIT B. That is, increasing the factor of self-rating speaking by one unit increased EIT A scores by 1.44 points and EIT B scores by 1.54 points. After entry of Can-Do statements for listening, the total variance explained by the model as a whole was 59.2% (R 2 = .592, F (2, 79) = 57.22, p < .01) for EIT A and 63.4% (R 2 = .634, F (2, 79) = 68.51.70, p < .01) for EIT B. However, when listening was added to the model, speaking no longer remained significant, and this was true for both EIT forms. Controlling the variance from Can-Do statements for speaking (44.1% for EIT A; 51.1% for EIT B), listening remained a significant predictor of EIT A and EIT B scores at p < .01 level with a Beta value of 2.08 for EIT A and 1.87 for EIT B, indicating that every unit of increase in the rating for listening increased EIT A scores by 2.08 points and EIT B by 1.87 points.
Next, for the multiple regression analysis that examined the four CAF measures, we entered all four measures simultaneously. We did this because they represent facets of L2 proficiency obtained from the same oral narrative task. Once again, all assumptions were checked and met. The four variables explained 65.6% of the variance in the EIT A scores, (R 2 = .656, F (4, 77) = 36.66, p < .01) and 63.5% of variance in the EIT B scores, (R 2 = .635, F (4, 77) = 33.53, p < .01). In the case of EIT A, number of motion verb types and speech rate made a significant contribution at p < .01 level. When the number of motion verb types increased by one unit, EIT A scores increased by 2.3 points. Likewise, every unit of speech rate increased the EIT A scores by 0.37 points. Additionally, the number of clauses approached significance level at p = .06 with a Beta value at 0.38. In the case of EIT B, number of motion verb types, speech rate, and number of clauses were significant predictors at p < .01 level. Every unit of increase in number of motion verb types increased the EIT scores by 2.55 points. Every unit of speech rate increased the EIT scores by 0.35 points, and every unit increase of number of clauses increased the EIT scores by 0.57 points.
In sum, in response to Research Question 2, participants’ performances on both EIT B and EIT A aligned well with the indicators of L2 proficiency, and the alignment was similar for both EIT forms. For both, the strongest simple correlations with the EITs were yielded by self-reported listening ability (Can-Do statements for listening) followed closely by self-reported speaking ability (Can-Do statements for speaking) and fluency (i.e., speech rate). When examined through the lens of regressions, the contribution of both Can-Do self-assessments to EIT scores was substantial (59.2% for EIT A; 63.4% for EIT B), but self-reported listening was the more important predictor for the two EITs. The contribution of the four CAF measures together was also substantial (65.6% for EIT A; 63.5% for EIT B). Fluency (i.e., speech rate) and number of motion verb types were significant predictors for both EITs.
EIT A AND EIT B PERFORMANCE ACROSS FOUR PROFICIENCY GROUPS
The third research question asked how well the EIT B can distinguish among different English proficiency levels within the sample, compared to the EIT A. We employed a Hierarchical Cluster Analysis (HCA) to identify natural proficiency groupings bottom-up from the full sample of 82 participants (see Staples & Biber, Reference Staples, Biber and Plonsky2015 for a methodological review of the use of HCA in L2 research). Four variables extracted from the responses to the self-diagnostic survey presented considerable variation among the 82 participants: length of English study, duration abroad, duration of inactive use of English, and holistic self-rating of English-speaking skills. They were entered for HCA procedures using the Ward method. There was no issue of multicollinearity, as all coefficients were equal or less than 0.58.
A four-cluster solution was determined to be optimal in providing maximized between-group differences and minimized within-group variation, based on the dendrogram plot of the hierarchical clusters in the data (see Figure A, supplementary material) and the analysis of the distance between fusion coefficients (see Figure B, supplementary material) (cf. Aldenderfer & Blashfield, Reference Aldenderfer and Blashfield1984). Adding a new cluster no longer contributed new information to the previous clusters. Two of the four variables entered in the HCA, duration of inactive use of English and holistic self-rating of speaking skills, best discriminated learners and grouped them into these four clusters.
The clusters in the four-cluster solution were then inspected closely for the descriptive statistics from all variables on the self-diagnostic survey, and four proficiency groups of Low, Mid, High, and Advanced L2 English participants were formed. Table 3 summarizes their background variables and self-assessed proficiency in speaking and listening. Table D (supplementary material) shows how participants in each recruitment group were regrouped into the new proficiency clusters derived from the HCA. A comparison of Table D and Table 3 shows that recruitment strategy does not neatly predict the four cluster groupings. By comparison, the HCA strategy resulted in internally consistent proficiency profiles, as all indices in Table 3 progressed steadily from the Low group to the Advanced group. Table 4 shows the descriptive statistics for the performances on the EIT A and EIT B by proficiency group.
Descriptive statistics of the four proficiency groups identified in the HCA
Notes: All values are means, with SD in parentheses. All data for English background are measured in years. Self-ratings: For holistic listening and speaking, the scale ranged from 1 being the lowest to 5 being the highest; for Can-Do statements, the maximum possible score was 55.
Descriptive statistics for EIT A and EIT B by proficiency groups
Notes: Means are presented with SD in parentheses. The maximum score was 120 for each EIT.
A mixed ANOVA was conducted with the two EIT forms as the within-subjects variable and the groups (i.e., Low, Mid, High, and Advanced) as the between-subjects variable. There was no significant interaction effect between EIT form and group, F (3, 78) = 0.413, p = .744, η 2 = .02, 95% CI for η2 [0, 0.07]. This means that no proficiency group was particularly advantaged or disadvantaged when performing one versus the other EIT form and that the main effect of the between-subjects factor could be inspected using the combined EIT performances (since by default in mixed ANOVA the main effect of a between-subjects factor is generated after combining the within-subjects factor). There was a main effect for group, F (3, 78) = 26.89, p < .01, η 2 = .51, 95% CI for η2 [0.33, 0.61]. This means that differences in proficiency were associated reliably with differences in EIT performance. Contrast analysis revealed that the differences in EIT scores (Estimated Marginal Means) were statistically significant among all four levels. The Advanced group scored highest (EIT A and EIT B combined M = 84.94, SE = 3.26), followed by the High (M = 68.26, SE = 2.57), Mid (M = 55.41, SE = 2.95), and finally the Low group (M = 43.54, SE = 3.84). There was also a significant main effect for versions of EITs, F (1, 78) = 235.86, p < .01, η2 = .75, 95% CI for η2 [0.65, 0.81]. Contrast analysis revealed that participants scored higher on EIT A (M = 68.24, SE = 1.68), compared to EIT B (M = 57.83, SE = 1.58), by about 10 points out of a total 120 possible points or 8% higher. This difference is depicted in Figure 1.
EIT A and EIT B by proficiency group.
We then conducted post hoc one-way ANOVAs separately for EIT A and EIT B to explore whether each set of scores on its own could distinguish across the four proficiency groups. The results showed that the main effect for group was significant for both EIT A (F (3, 78) = 23.048, p < .01, η2 = .47, 95% CI for η2 [0.29, 0.58]) and EIT B (F (3, 78) = 28.78, p < .01, η2 = .53, 95% CI for η2 [0.35, 0.62]). The multiple pairwise comparison analysis demonstrated that the differences among all four groups were statistically significant at the p < .01 level for both EIT A and EIT B. In other words, despite the finding that EIT B was consistently more difficult than EIT A, both EIT forms discriminated equally well across all four proficiency groups.
In answer to Research Question 3, the two forms of the English EIT worked well in distinguishing among all four proficiency groups identified in the sample through HCA. No proficiency group was particularly advantaged or disadvantaged by taking one or the other form. However, the level of difficulty of the EIT B was consistently higher than that of the original EIT A, by a magnitude of 10 points or 8% of the 120-point scoring band.
DISCRIMINATORY POWER OF ITEMS
Discrimination indices (DIs) for the 30 items in EIT A and EIT B were calculated following the steps outlined in Brown (Reference Brown2003). Computation of DIs informs the extent to which an item discriminates between participants who earn the highest 27% of the scores and participants who earn the lowest 27% of the scores (McCowan & McCowan, Reference McCowan and McCowan1999). Thus, information on DI helps gauge whether items were functioning well to a similar extent in EIT B and EIT A. The results are summarized in Table 5.
Discrimination Indices (DIs) and item mean scores for EIT A and EIT B (N = 82)
Note: Means are presented with SD in parentheses.
As seen in Table 5, the newly developed items in EIT B demonstrated a strong ability in differentiating between the top and bottom 27% of performances, and one fully comparable to EIT A: mean DI of = 0.67 for EIT A and 0.68 for EIT B. According to Ebel and Frisbie (Reference Ebel and Frisbie1986), items with a DI of 0.40 or higher can be considered to have good discrimination power, meaning that the item is properly designed to discriminate among test takers on the high and low ends.
Only Items 1 and 4 (on both EIT forms) and Items 2 and 5 (on the EIT A) showed poor discriminatory power below 0.40 (see Table 5). It is important to appreciate that this may reflect the design logic of the EIT as a test to be used with the widest possible range of language abilities: Having particularly short and easy (but less discriminating) items in the beginning of the sentence repetition task enables very low proficiency learners to want to do as well as they can, and it minimizes the risk of participants who freeze from the outset and give up on repeating at all. Starting with Item 6 in both EIT A and EIT B, containing sentences longer than 10 syllables, all DIs consistently show values higher than 0.59 for EIT A and 0.63 for EIT B.
When the range of DIs is compared, 16 items in EIT B surpass their counterparts in EIT A in their DI values and 13 favor DIs in the EIT A. All differences are small, however, ranging from the lowest at 0.00 (Item 27) to the highest at 0.15 (Items 6 & 7) and −0.16 (Items 9 & 11). Such small differences in discriminatory power can be expected even for the same item in a test-retest condition (Price et al., Reference Price, Jhangiani and Chiang2015), and they are certainly reasonable in two parallel forms. Nevertheless, given the discovery that EIT B was consistently more difficult than the original EIT A, we undertook a qualitative inspection of items in the new form that may have been less discriminating and more difficult than their counterpart in the original form. We selected Items 6, 13, and 26 because, although they had DI values above 0.60, they exhibited slightly higher DIs in favor of the EIT A coupled with lower mean scores in the EIT B. They are shown in Table 6.
Three items with lower DI score and mean score on EIT B than EIT A
Note: Word comparisons between the two parallel stimuli analyzed in this section are bolded.
For Item 6, a comparison of percentages of correct repetition for pairs of parallel words across EIT A and EIT B (see bolded in Table 6) revealed that a much higher percentage of participants failed to repeat what than how: 60.98% for how in EIT A versus 26.83% for what in EIT B. Given that frequency is a key factor in acquisition (Ellis, Reference Ellis2017), we conducted a search on the strings “how to drive” and “what to say” in the iWeb corpus (a larger corpus than COCA). Confirming our intuitions, “what to say” yielded a much higher frequency than “how to drive”: 24,681 versus 8,931 raw occurrences. The same finding was confirmed when we search the collocations “knows what” (110,277) and “knows how” (97,522). Thus, frequency was discounted as an explanation in this case. A comparison of the two sound recordings for Item 6 revealed that the male speaker in EIT B pronounced what indistinctly and quietly, as compared to the clear way the female speaker pronounced how in EIT A. It seems possible that the low-salience pronunciation of what in the high-frequency chunk “what to say” made it difficult to hear for many participants. Sample responses that omitted what in Item 6 of EIT B are: “I doubt that she knows it that fast”; “I doubt that she know to stay that fast”; and “I doubt that she would say that fast.” Failing to repeat what resulted in a change in meaning, attracting a score of 2 or lower (cf. rubric in Table B in the supplementary material).
For Item 13, the inclusion of an Adj1-Adj2-Noun (a nice, big house vs. a large, quiet yard) is supposed to tax participants’ proficiency, in addition to contributing to the item length, as these constructions are known to grow in productive use as L2 English users’ proficiency develops within the range from upper-intermediate to very advanced proficiency (e.g., Díez-Bedmar & Pérez-Paredes, Reference Díez-Bedmar and Pérez-Paredes2020). It was found that the second-Adjective retention rate was much lower for quiet in EIT B (only 36.59% of the 82 participants repeated it, whereas the majority simply omitted it) than for big in EIT A (91.46% accurately repeated it). Once again, a plausible culprit could be frequency, and for this item we found confirmation: When we searched these two collocations in the iWeb corpus, “nice and big” showed 853 results, whereas “large and quiet” showed only 29 results. As for the pair of my versus he, a much higher percentage of participants succeeded in repeating my on EIT A (82.93%) than he on EIT B (36.59%). There is a closer semantic relationship between the subject of the relative clause “my animals” and the subject of the main clause “I” in EIT A (i.e., I want a nice, big house in which my animals can live”), compared to the relationship between “he” and “I” in EIT B (i.e., I want a large, quiet yard in which he can grow flowers). We speculate that this made it easier for participants to repeat the EIT A sentence more successfully. Sample responses that showed errors with “quiet” and “he” include: “I want a large yard which can grow flowers”; “I want a quite large yard where I can plant flowers”; and “I want a large quiet yard in which it can grow flowers.” These two sources of added difficulty—big versus quiet plus my versus he—within the same long and syntactically complex sentence strings in Item 13, likely contributed heavily to the lower item mean score in EIT B (M = 2.52) than in EIT A (M = 1.79) by a 0.73 point mean difference, which is rather large on a total possible mean score of 4.
Finally, for Item 26, two prominent contrasts were found between the percentages of correct repetition for the word pairs of book versus glass (76.83% vs. 37.8%) and table versus cupboard (93.90% vs. 17.07%). Our explanation weaves several concerns. In the EIT A, it is possible that the appearance of book and is on the table in the earlier Item 2 (“The red book is on the table ”) may have primed participants to perform better with this Item 26 (“Would you be so kind as to hand me the book which is on the table ?”). Moreover, a search of the iWeb corpus confirmed large frequency differences: The token occurrences of book (3,974,639) and (table 2,004,574) were much higher than glass (1,128,883) and cupboard (42,877). Thus, “glass” and certainly “cupboard” may have been less familiar words for the participants. Many substitutions, such as “glasses” for “glass,” and “cabinet” or “carbon” for “cupboard,” were found in participants’ responses.
To summarize our answer to Research Question 4, the newly developed items differentiated between high scorers and low scorers as well as the items in the original EIT A because the vast majority of the 30 items in both versions exhibited strong discriminatory power well above 0.40, and both DI means were high and practically identical (0.67 for EIT A, and 0.68 for EIT B). When qualitatively examining three items in the EIT B that showed poorer discriminating strength and a lower mean score than their EIT A counterparts, it was found that subtle changes in speaker’s pronunciation, word frequency, and semantic content likely both lowered the discriminatory power of these items and made them more difficulty than their original counterparts.
PARTICIPANTS’ PERCEPTION OF FACTORS AFFECTING THEIR EI PERFORMANCE
Our last research question turns to the evidence collected in section (1) of the self-diagnostic survey, comprising five items that probed into the participants’ experience taking the English EIT (cf. Appendix B, supplementary material). Given that participants completed the self-diagnostic survey immediately after completing either EIT A or EIT B, we first inspected whether responses differed in some way for the two halves of participants.
Based on the evidence from perceived difficulty (Question 1) and self-assessed EIT performance (Question 2), we can say that the greater difficulty of the EIT B was detected in the survey. Namely, participants who answered based on EIT B rated their EI experience as more difficult (M = 3.13, SD = 1.21, a scale of 3 = somewhat difficult) than those responding on the EIT A (M = 2.55, SD = .85, a scale of 2 = slightly difficult). An independent-samples t-test confirmed the difference was statistically significant, t(80) = −2.56, p < .05, and of medium size, Cohen’s d = .56, 95% CI for d [−1.01, −0.13]. Conversely, participants who answered based on EIT A rated their performance higher (M = 2.45, SD = 1.07, a scale of 2 = fair) than those responding on the EIT B (M = 1.89, SD = 1.18, a scale of 1 = needs improvement). Once again, the difference was statistically significant, t(80) = 2.26, p < .05, and of a medium size Cohen’s d = .50, 95% CI for d [0.06, 0.94]. Moreover, participants’ survey responses on these two questions correlated substantially with their actual EIT scores, both for the perceived level of difficulty (Q1) (EIT A: r = −.66, p < 0.01; EIT B: r = −.70, p < .01) and for the self-assessed EIT performance (Q2) (EIT A: r = .73, p < 0.01; EIT B: r = .77, p < .01). In other words, the higher participants rated their perceived difficulty, the more poorly they performed on the EIT, and the higher they rated their EIT performance, the better they scored on the EIT. Thus, not only was the greater difficulty of the EIT B scores reflected in the survey responses, but we can also say that at the individual level participants were realistic in their task perceptions of the EIT experience.
No other differences were found in the remaining of perception responses because (a) independent-samples t-tests comparing participants’ ratings on factors affecting their sentence comprehension (Question 3) and on factors affecting their sentence repetition (Question 4) showed no significant differences on either question; and (b) a Fisher’s exact test found no statistically significant differences in the two halves of participants’ frequencies of selection of whether understanding or repeating the sentence more challenging (Question 5). Therefore, in our analyses of Q3, Q4, and Q5 in the following text, we combined the two halves of responses to EIT A and EIT B.
Table 7 summarizes participant responses when asked to rank the specific factors that affected their comprehension (Question 3) and production (Question 4) of the sentences. As presented in Table 8, when participants were forced to choose between comprehension or production (Question 5) as the most difficult aspect of EI, most in the Low group and many in the Mid group reported that they struggled more with understanding the sentences, whereas the overwhelming majority of High and Advanced groups considered repeating the sentences harder. In response to why, most of the participants that selected comprehension as more challenging wrote that they believed if they could have fully understood the sentence they heard, they would have been better able to repeat it.
Summary of self-ratings of factors affecting EI comprehension and production
Note: Means are presented with SD in parentheses.
Selection on the more challenging aspect: comprehension versus production
Notes: EIT score indicates the mean scores of EIT A and EIT B combined with SD in parentheses. The column marked raw denotes raw frequency counts of participants who selected that answer. Two participants in Mid and one in Advanced did not make a selection.
Using standardized multiple regression, we asked how much variance in the EIT scores could be explained by the top three factors reported in Table 7 to have affected sentence comprehension (i.e., vocabulary, length, and comprehension time) together with the three top factors reported to have affected production (i.e., partial comprehension, length, retention of details). Preliminary analyses showed all assumptions of normality, linearity, homoscedasticity, and multicollinearity were met. For the dependent variable we chose the sum scores of EIT A and EIT B, as the alternative option of using the averaged means of EIT A plus EIT B would have resulted in more loss of variation. When entering all six factors simultaneously, the six variables contributed to 53.1% of the variance in the combined EIT scores, (R 2 = .531, F (6, 75) = 14.15, p < .01).
As can be seen in Table 9, only the factor of partial comprehension made a significant contribution to the EIT scores at a p < .05 level with a very large Beta value of −11.66. This suggests that every unit of increase in the rating of how much participants thought partial comprehension affected their EI performance reduces their EIT scores by 11.66 points, when the other five factors are held constant. Note that the wide confidence intervals around the Beta value for partial comprehension suggested that the sample size may be too small to produce statistically reliable estimates. This preliminary observation awaits further confirmation through a larger sample size.
Regression model of predictors of EIT performance
Note: Confidence intervals are reported in parentheses.
a Factors affecting sentence comprehension.
b Factors affecting sentence production.
Additionally, a large but not statistically significant (p = .06) Beta value of −8.18 was found for the factor of length. Although we cannot be statistically confident, this suggests a possible role of length in influencing EIT scores. In sum, our fifth and final research question can be answered as follows. The half participants who completed the EIT B first did experience more difficulty than the half who first completed the EIT A, and overall individual participants were realistic in their perceptions of how difficult the EIT was and how well they did. In terms of sources of difficulty, and regardless of EIT form, perceived inability to fully understand the aural stimulus sentence was the best predictor, and this was felt most pronouncedly at lower levels of proficiency. Length was also perceived as a top source of difficulty in both comprehending and repeating the stimuli but its effect on actual EIT scores was statistically marginal.
DISCUSSION
The present study investigated the reliability, form equivalence, external validity, and suitability of a newly developed EIT form created in parallel to the original EIT form for English L2 developed by Ortega et al. (Reference Ortega, Iwashita, Norris and Rabie2002). Both the original and the new forms were administered to a sample of 82 Mandarin Chinese users of L2 English recruited to include a broad range of L2 experiences and proficiencies. The newly developed form exhibited excellent internal consistency (α = 0.97, as compared to α = 0.93 in Ortega et al., Reference Ortega, Iwashita, Norris and Rabie2002), strong parallel forms reliability with the original form (r = .96), and strong item discriminatory power (mean DI = 0.68). It also showed good external validity in its satisfactory alignment with other proficiency measures, including self-ratings on Can-Do statements for listening and speaking abilities and CAF-related measures, particularly lexical diversity (i.e., number of motion verb types) and fluency (i.e., speech rate). The EIT B also effectively differentiated the 82 participants across four proficiency groups (Low, Mid, High, and Advanced). However, we also found that participants across all four proficiency groups performed consistently worse on EIT B than on EIT A, by about 10 points of the 120 total possible score, or 8%, and that participants who took the EIT B first also perceived it to be more difficult. A qualitative inspection of some items revealed at least three potential sources for this differential difficulty: speaker’s pronunciation, word frequency, and semantic content. The top factor identified by participants as making EI performance difficulty in general was comprehension, or partial understanding of the sentences.
In this section, we first offer some suggestions for improved use and interpretation of the parallel forms in future research, and we then discuss new insights about the nature of EI drawn from the study findings.
RECOMMENDATIONS FOR USE AND INTERPRETATION OF THE PARALLEL FORMS IN FUTURE RESEARCH
At the heart of validating the EIT B was the question of whether the two EIT forms can be considered alternate forms that can be used interchangeably or equivalently within the same study design and for meaningful comparison across studies. The shared variance between EIT B and EIT A was 91%. This is much higher than the thresholds often proposed by language assessment scholars to proclaim parallel forms reliability (e.g., about 70% shared variance in Weir & Wu, Reference Weir and Wu2006), provided there are other sources of evidence converging to the conclusion that two tests are equivalent. In our study, the additional converging evidence is largely positive too and comes from the fact that the scores obtained from the same participants on the two EITs behaved very similarly across analyses—from degree of alignment with the six external indicators of L2 proficiency, to strong ability to distinguish among four proficiency groups, to item discrimination values largely above 0.40 and mean DIs above 0.65 for both versions. However, a statistically reliable 8% score difference between the two forms—a difference symmetrically observed across all four proficiency levels (see Figure 1)—begs for careful discernment when using and interpreting the two instruments, and it opens space for future improvement.
Researchers interested in using the two English EIT forms as they stand should carefully consider their research purpose for using the instruments. If the purpose is to reliably distinguish among existing proficiency differences within a sample, the 10-point difference would not likely matter, and either form can be used with equal confidence, based on the fact that both EIT A and EIT B distinguished equally well among Low, Mid, High, and Advanced levels in our study. However, if the purpose is replication, we advise researchers to carefully consider whether the EIT A or EIT B was used in the original study, and to choose the same form in their replication study. For meta-analysts wishing to compare and synthesize findings across studies that may have used one of the two present forms, a sensitivity analysis (Iyengar & Greenhouse, Reference Iyengar, Greenhouse, Harris, Hedges and Valentine2009) is recommended to ascertain if differences in form choice affects the aggregation of results, and if so adjustments can be made based on the assumed effect size of an 8% score difference. Our original goal was to make two parallel EIT forms for English L2 that can increase the rigor of proficiency measurement in longitudinal and crosslinguistic research. For these purposes, a statistically reliable 8% score difference between EIT A and EIT B is less than ideal. Again, knowing ahead the magnitude of the predicted differential difficulty can help address the issue in study design (e.g., counterbalancing the forms across multiple administrations) and statistical analysis (adjusting the results by 8% in the predicted direction). Nevertheless, test improvements to further boost parallel forms reliability would be ideal.
An obvious improvement would be to rerecord the stimulus sentences for both sets using the same English-speaking voice. One can also address the small gaps we pointed out in frequency of certain lexical pairs or collocations and in pronoun choice. For example, the pronoun “he” in Item 13 of the EIT B can be replaced with “I,” as in “I want a large, quiet yard in which I can grow flowers,” so as to reduce the larger burden in retaining the semantic content caused by a change of pronoun in the relative clause. Additionally, one could spread out the difficulty difference by creating two new parallel forms out of the 60 existing sentence items, if the odd number items of EIT A were mixed with the even number items of EIT B to form one version and the converse combination was adopted for the other version. This is a simple strategy that preserves the fixed order by ascending syllable length in the suite’s original design (see Appendix D in the supplementary material for the blended and revised forms). We caution, however, that a new study would be necessary to establish whether these suggested improvements succeed in equalizing the difficulty of these two blended EIT forms.
THE ROLE OF COMPREHENSION AND LISTENING IN EI PERFORMANCE
We found that self-reported comprehension difficulties, or the challenge to understand the sentences participants were supposed to repeat, explained EIT scores better than any other sources of difficulty reported in the survey, seemingly overweighing other difficulties that were also perceived to matter, such as vocabulary and sentence length (cf. Table 7). Comprehension loomed particularly large in the mind of Low and Mid proficiency participants (cf. Table 8). For the full sample, too, it made the only significant and large-sized contribution to the EIT scores in the standardized multiple regression (cf. Table 9).
To the best of our knowledge, the role of comprehension in EI has been explicitly investigated only by Deygers (Reference Deygers2020). Reasoning that unfamiliar vocabulary hinders comprehension, he manipulated comprehension levels by insertion of pseudowords in every third sentence in his L2 Dutch EIT. By comparing same-participant performances on regular sentences and sentences with pseudowords, Deygers was able to show that pseudowords made EI performances deteriorate. Furthermore, the debilitating effect grew bigger as sentences become longer and more difficult to comprehend, with more pseudowords packed in them.
Related to comprehension, listening abilities were also of great importance in explaining our participants’ EI performances. Although Can-Do speaking scores predicted 44.1% and 51.1% of the variance in the scores of EIT A and EIT B, respectively, once Can-Do listening scores were entered into the regression models, the overall prediction improved considerably (to 59.2% for EIT A and 63.4% for EIT B) and, moreover, the contribution of speaking ceased to be significant (for both forms), while listening remained the only significant predictor. Two previous EIT studies have included listening ability, and both found it correlated well with EI scores. Of them, Kim et al. (Reference Kim, Tracy-Ventura and Jung2016), in a Korean as a second language context, reported that the listening section of the Korean TOPIK correlated with EIT scores at r = .62, explaining a sizeable 38% of shared variance, but considerably lower than the 59% shared variance for the speaking section (r = .77). Bowden (Reference Bowden2016), in a Spanish as a foreign language context, reported that a holistic rating of self-assessed listening ability (1 = very poor to 5 = nativelike reported) correlated with EIT scores at r = .73, a large shared variance of 53% that was only slightly lower than the 56% observed for holistically self-assessed speaking (r = .75).
Based on the present findings for listening and those reported by Kim et al. (Reference Kim, Tracy-Ventura and Jung2016) and Bowden (Reference Bowden2016), we speculate that listening abilities might be more important in EI performance for L2 users with more foreign-language-like learning experience, and speaking abilities may be more predictive of L2 users with more second-language-like, immersive learning experience. In many foreign language contexts, it is common for learners to exhibit better receptive aural skills than productive oral skills. Listening is typically easier to bring into a foreign language classroom for training than speaking is. Moreover, for a language user’s speaking skills to match their listening competence, more immersive kind of experience with the language may be required. A perusal of our participants’ holistic rating scores for speaking and listening (see Table 3) by proficiency group supports this claim: The three Low, Mid, and High groups’ self-ratings of listening were all higher than their ratings for speaking, although with a gap of less than 1 point on the 5-point Likert scale. However, the gap tended to be reduced and even reserved for the Advanced group, whose mean rating for speaking was slightly higher than for listening (4.39 vs. 4.33). Even though their initial English experiences were typical of their Chinese EFL upbringing, our Advanced participants had a mean length of 1.68 years living abroad and thus more opportunities to functionally use their speaking skills. Thus, future research is warranted that compares the relative contribution of listening versus speaking abilities to EI performance in second- versus foreign-language contexts.
We caution, however, that the potential effect of learning context we submit here for future research will likely be relatively small. This is because there is ample evidence to suggest that the ability of the EIT to gauge global proficiency is modality independent. Namely, as we noted in our review of the literature earlier, the relationships between EI scores and performances on standardized tests are strong when modality is shared (e.g., the Simulated Oral Proficiency Interviews are oral) and when it is not (e.g., the Spanish DELE or the grammar section of the Oxford Placement Exam are written). This modality independence makes the EIT particularly useful and practical: It is suitable for use across diverse learner profiles, including speakers who might have limited familiarity with literacy in the target language (e.g., heritage speakers as in Wu & Ortega, Reference Wu and Ortega2013; low-literacy adults as in Deygers, Reference Deygers2020), as well as speakers who are the product of more reading- and grammar-heavy pedagogies common in many foreign language contexts, such as the EFL contexts for the sample in our study.
LENGTH STILL MATTERS IN EI, AND SO SHOULD VOCABULARY KNOWLEDGE
Our study findings put a spotlight on comprehension and listening as contributors to EI performance. However, they should not be interpreted as canceling out the importance of length and vocabulary in understanding EI performances.
The length of sentences was rated by our participants as one of the top three factors affecting both sentence comprehension and production (cf. Table 7), but did not reach statistical significance as a predictor of actual EIT scores when pitted against comprehension (cf. Table 9). However, sentence length has been found to account for a large proportion of variance in EI scores, as intended by design, in previous studies (e.g., Deygers, Reference Deygers2020; Gaillard & Tremblay, Reference Gaillard and Tremblay2016; Graham et al., Reference Graham, McGhee, Millard, Prior, Watanabe and Lee2010; Kim et al., Reference Kim, Tracy-Ventura and Jung2016; Wu & Ortega, Reference Wu and Ortega2013). We therefore ran a follow-up analysis correlating stimulus length and mean score of the item and found a large negative correlation for both EIT A (r = −.81, p < .01) and EIT B (r = −.83, p < .01). In terms of coshared variance, both results (66% and 69%, respectively) are in the upper end of the previously attested range of between 45% and slightly more than 70%. Thus, it seems as if length of sentences was not as salient in our participants’ experience as comprehension but nevertheless was important in explaining EI performances. The key is, perhaps, to recognize that in EITs, by design, length is intertwined with comprehension, in general, and listening ability, in particular. As Deygers (Reference Deygers2020, p. 21) noted, sentences beyond 9 or 10 syllables in length become difficult to repeat successfully unless they are comprehended. This also helps explain why better rote memory is not found to be a predictor of EI scores in most studies (Graham et al., Reference Graham, McGhee, Millard, Prior, Watanabe and Lee2010; Kim et al., Reference Kim, Tracy-Ventura and Jung2016; Okura & Lonsdale, Reference Okura, Lonsdale, Miyake, Peebles and Cooper2012), with one study (Park et al., Reference Park, Solon, Henderson and Dehghan-Chaleshtori2020, p. 145) uncovering inconsistent and small facilitative effects only for only low-proficiency participants. When stimuli are long enough, pure memory-based echoic repetition is not possible (Bley-Vroman & Chaudron, Reference Bley-Vroman, Chaudron, Tarone, Gass and Cohen1994) and instead comprehension and grammatical processes must apply to reconstruct the sentence.
Compared to length, attention to lexical knowledge has been low in the context of EI (cf. Table A, supplementary material). Yet, of the four CAF measures employed in this study, it was the specific measure of lexical diversity (motion verb types) that contributed most robustly to predicting EIT scores, and our participants appeared to be highly sensitive to certain word frequency differences in some items (e.g., table vs. cupboard). Moreover, they identified vocabulary as one of three top perceived sources of difficulty (cf. Table 7), and the first-rated difficulty, comprehension, is aggravated by unknown vocabulary (Deygers, Reference Deygers2020). More generally, the relationship between vocabulary and proficiency is well established in the L2 field at large (Qian & Lin, Reference Qian, Lin and Webb2020). Thus, we urge EI researchers to consider lexical knowledge in the future, using general metrics (as Tracy-Ventura et al., Reference Tracy-Ventura, McManus, Norris, Ortega, Leclercq, Hilton and Edmonds2014 did) but without overlooking specific ones, given the positive results reported by Wu and Ortega (Reference Wu and Ortega2013), McManus and Liu (Reference McManus and Liu2020), and in the present study.
LIMITATIONS AND CONCLUSION
Several limitations must be acknowledged. First, in retrospect we wish we had rerecorded both EIT forms using the same speaker voice, as we would have avoided the variability introduced by different speakers. Speaker variability poses a processing challenge that ultimately drives rich language learning (Barcroft, Reference Barcroft2015) but in the context of estimating abilities it can destabilize performance and, in our case, it seems to have introduced some added difficulty into the EIT B. Second, the present findings come from a sample of participants from a single L1 background (Mandarin), and validation of the English EIT ought to be expanded to other samples from other L1s and from mixed L1s. Third, we forewent the inclusion of a standardized measure of proficiency for practicality reasons, as any such test would have added one to two more hours to participants’ time commitment; however, a direct measure of proficiency (e.g., TOEFL iBT, IELTS) rather than only self-assessments would have strengthened the evidence for external validity of the EIT A and EIT B.
Despite these limitations, the study was able to offer new and interesting insights. In the context of SLA research, EI instruments are modest tools that offer themselves as short-cut measures of L2 proficiency. The theoretical motivation for using them comes from the claim that EI performance is reconstructive (Bley-Vroman & Chaudron, Reference Bley-Vroman, Chaudron, Tarone, Gass and Cohen1994), integrative (Klem et al., Reference Klem, Melby-Lervåg, Hagtvet, Lyster, Gustafsson and Hulme2015), and modality independent, as we have noted. We can also say that EITs tap Hulstijn’s (Reference Hulstijn2015) BLC dimension of language ability by drawing from “learners’ knowledge and automated ability for the use of core vocabulary and grammar delivered with reasonable pronunciation and fluency” (Wu & Ortega, Reference Wu and Ortega2013, p. 684). Still, at face value, EI is just a psycholinguistic technique that elicits the automatized verbatim repetition of isolated sentences. It is a challenge, therefore, to show conceptual linkages between EI and the kind of abilities served by BLC to communicate in everyday situations and to carry out mundane communicative tasks. Our desire to address this linkage motivated the inclusion of CAF measurement and Can-Do self-assessments, which helped us connect the (reconstructive, integrative, modality independent) psycholinguistic EI task performance to more communicative dimensions of language use. In addition, by eliciting perceptions of EI performances and exploring empirically whether these perceptions did in fact align well with actual EI scores, we were able to show that participants can be rather realistic in their task perceptions of EI, and that their perspectives can add important insights to EI research.
Overall, the present study serves as the first systematic evaluation of the original EIT for L2 English developed by Ortega et al. (Reference Ortega, Iwashita, Norris and Rabie2002), suggesting that this instrument is useful as a short-cut measure of L2 English proficiency for research purposes and providing evidence of strong form equivalence between the original and the present form. By offering a parallel form of the instrument, we hope to support more longitudinal research and sharper research into L3 and Ln learning by multilingual speakers, particularly those whose L1 is not English. We also hope other researchers will feel motivated to further refine the equivalence of the EIT A and EIT B for L2 English in future studies as well as to develop parallel forms for the remaining six L2s available in the Ortega et al. (Reference Ortega, Iwashita, Norris and Rabie2002) EIT suite and make them available to the research community. Continued attention to measurement consistency, the sharing of research instruments, and rigorous measurement of L2 proficiency can only enhance knowledge generalization and strengthen SLA research programs.
Supplementary Materials
To view supplementary material for this article, please visit http://doi.org/10.1017/S0272263121000103.
DATA AVAILABILITY STATEMENT
The following supplementary materials are available on the IRIS database (http://www.iris-database.org):
Table A: The relationship between EI performances and CAF measures across nine EIT studies
Table B: EIT Scoring Rubric (based on Ortega et al., Reference Ortega, Iwashita, Norris and Rabie2002)
Table C: Descriptive statistics on Can-Do Statements and Complexity-Accuracy-Fluency (CAF) measures (N = 82)
Table D: Recruitment grouping by proficiency cluster grouping
Figure A: Dendrogram of cluster analysis.
Figure B: Distance between fusion coefficients against number of clusters.
Appendix A: Sentence stimuli for Forms A and B (with syllable length in parentheses)
Appendix B: Self-diagnostic survey
Appendix C: Oral narrative task
Appendix D: Sentence stimuli (revised and blended)
