Introduction
SLA has a long-standing interest in defining and assessing second-language (L2) proficiency. However, proficiency is a complex, multilevel construct and, as such, is particularly difficult to measure (e.g., Leclercq et al., Reference Leclercq, Edmonds and Hilton2014; Thomas, Reference Thomas, Norris and Ortega2006). In an attempt to define this construct, Tracy-Ventura et al. (Reference Tracy-Ventura, McManus, Norris, Ortega, Leclercq, Edmonds and Hilton2014), following Hulstijn’s (Reference Hulstijn2011, Reference Hulstijn2012) definitions of L1 and L2 proficiency, suggest that general L2 proficiency refers to basic language cognition (BLC) that includes the processing of oral language that contains high-frequency lexico-grammatical and phonological elements. Tracy-Ventura et al. further propose that general proficiency tests should focus on assessing oral language in a literacy-independent way concentrating on “operations involving high-frequency elements during the integrative use of oral language, while recruiting in real time all interrelated basic linguistic levels of lexicon, grammar, phonotactics and prosody” (p. 144).
One instrument that has emerged from researchers’ attempts to assess general L2 proficiency accurately, reliably, and efficiently is the elicited imitation task (EIT). This type of test involves having learners listen to a series of utterances each of which they are prompted to repeat as accurately as possible. Underlying this task is the assumption that to accurately repeat a sentence, learners must possess sufficient ability to parse and understand it (Bley-Vroman & Chaudron, Reference Bley-Vroman, Chaudron, Tarone, Gass and Cohen1994; Vinther, Reference Vinther2002). Put more simply, greater ability to repeat serves as a proxy for proficiency.
Since their introduction in the 1970s, EITs have been used as a measure of L2 proficiency in various languages. For example, Ortega et al. (Reference Ortega, Iwashita, Norris and Rabie2002) created EITs in four different languages (English, German, Japanese, and Spanish). Building on Ortega et al.’s study, Wu and Ortega (Reference Wu and Ortega2013) developed an EIT to measure L2 Mandarin Chinese proficiency. Likewise, Tracy-Ventura et al. (Reference Tracy-Ventura, McManus, Norris, Ortega, Leclercq, Edmonds and Hilton2014) and Gaillard and Tremblay (Reference Gaillard and Tremblay2016) both presented evidence for the validity of a French EIT.
Despite multiple studies reporting sizeable correlations between EIT and other more traditional measures, the validity of EIT scores representing L2 proficiency has been questioned. Vinther (Reference Vinther2002), for example, suggested that it can be hard to distinguish between the subjects’ actual knowledge of a model and sound imitation in EIT. Moreover, learners’ individual differences impacting EIT performance have been called into question. Kim et al. (Reference Kim, Tracy-Ventura and Jung2016), for instance, investigated the potential role of phonological short-term memory in task performance, finding a significant but weak correlation between EIT and memory measures. Similarly, Park et al. (Reference Park, Solon, Henderson and Dehghan-Chaleshtori2020) examined the relative contributions of language ability and phonological short-term memory (PSTM) to EIT performance and found that PSTM capacity may act as a moderator in EIT performance of beginner L2 learners. The strongest predictors of L2 learners’ EIT performance overall, however, were language-related measures.
In response to such concerns, and to contribute to the validity argument that undergirds the use of EIT as a measure of L2 proficiency, several scholars have sought to synthesize findings on the relationship between EIT performance and other measures of proficiency (Yan et al., Reference Yan, Maeda, Lv and Ginther2016; Zhou, Reference Zhou2012). As described in the following text, these meta-analyses are both somewhat dated, given the steady flow of primary studies in this domain. Moreover, Yan et al. (Reference Yan, Maeda, Lv and Ginther2016) presents valuable validity evidence of EIT from one particular angle, prioritizing its ability to discriminate high- versus low-proficiency learners. The present study seeks to provide an updated and complementary review of correlational research to add to the findings of Zhou (Reference Zhou2012) and Yan et al. (Reference Yan, Maeda, Lv and Ginther2016) by estimating the criterion-related and concurrent validity of EITs.
Defining and Measuring L2 Proficiency
L2 proficiency is a construct that has long been debated in applied linguistics. In their attempts to define proficiency, researchers such as Bachman (Reference Bachman1990) and Leclercq et al. (Reference Leclercq, Edmonds and Hilton2014) referred to L2 proficiency as one’s overall knowledge of or ability to use language, while other definitions and frameworks have focused on specific skills needed to successfully understand and produce a target language (e.g., Gaillard & Tremblay, Reference Gaillard and Tremblay2016, p. 420; for a recent overview of different frameworks and their corresponding scales, see Harsch & Malone, Reference Harsch, Malone, Winke and Brunfaut2021). In L2 learning contexts, proficiency has been broadly defined using its components through which language ability is assessed. Initially, Lado (Reference Lado1961) and later Carroll (Reference Carroll and Davies1968) proposed a comprehensive model that represented proficiency as consisting of four skills that could be assessed discretely: reading, writing, listening, and speaking. However, as Son (Reference Son2018) points out, proficiency has since developed to incorporate language abilities beyond mere linguistic forms and reflects the application of those abilities in different contexts. Regarding the operationalization of L2 proficiency specifically in SLA research, a fairly recent trend has been to use the variables of complexity, accuracy, and fluency in learner production (Housen et al., Reference Housen, Kuiken and Vedder2012; Pallotti, Reference Pallotti, Winke and Brunfaut2021; Skehan, Reference Skehan1989). Taking into account L1 proficiency, Hulstijn (Reference Hulstijn2011, Reference Hulstijn2012) incorporated automaticity of processing and core linguistic knowledge into the L2 proficiency model.
Perhaps due to the various approaches to defining L2 proficiency, there seems to be little agreement on how to measure it, especially for the purposes of SLA research. While some of the existing approaches to L2 proficiency assessment have included speed of lexical access (Bairstow et al., Reference Bairstow, Lavaur, Laxén, Aparicio, Leclercq and Edmonds2014), self-assessment (Li & Zhang, Reference Li and Qian2021; Tigchelaar et al., Reference Tigchelaar, Bowles, Winke and Gass2017), course grades (see Brown et al., Reference Brown, Plonsky and Teimouri2018), C-tests (e.g., Norris, Reference Norris2018), and corpus-based analysis of learner written and oral production (e.g., Gablasova et al., Reference Gablasova, Brezina and McEnery2017), researchers are calling for more rigorous and targeted methods of assessing learner proficiency (Derrick, Reference Derrick2016; Norris & Ortega, Reference Norris, Ortega, Gass and Mackey2012). Moreover, due to the multifaceted, “fuzzy” nature of proficiency (Carlsen, Reference Carlsen2012), problems associated with reliability and validity of nonstandardized L2 proficiency measures have been noted (Alderson, Reference Alderson2007; Hulstijn, Reference Hulstijn2007; Pienemann et al., Reference Pienemann, Johnston and Brindley1988). Bowden (Reference Bowden2016) characterized proficiency as difficult to define due to the lack of reliable standardized proficiency measures that can be used in research settings (p. 648). Furthermore, Leclercq et al. (Reference Leclercq, Edmonds and Hilton2014) noted that the validity of a proficiency scale and its representativeness of stages of L2 development should not be taken for granted as emphasizing the need for clear definitions of scale bands that include language descriptors and their functions to create a basis for reliable level assignment. This view is further supported by Long, Gor, and Jackson’s (Reference Long, Gor and Jackson2012) position that existing proficiency rating guidelines (e.g., CEFR levels, ACTFL scale) largely disregard the linguistic side of L2 proficiency focusing instead on functional criteria (see Norris, Reference Norris2005, for further discussion and evidence on linguistic/morphosyntactic correlates of proficiency). However, the practicality and face validity of standardized proficiency measures have been questioned by researchers. In particular, standardized tests are lengthy and expensive, especially when it comes to proficiency measurement for research purposes, leading some to look for alternative measures (Kokhan, Reference Kokhan2013).
The importance of designing and implementing principled L2 assessments for SLA research has been repeatedly demonstrated through secondary research. Tremblay (Reference Tremblay2011), for example, surveyed 144 studies published from 2000 to 2008 that examined the linguistic knowledge and behavior of L2 learners. Out of the 57 studies that measured proficiency using an independent test, 40% reported using sections of standardized proficiency or placement tests, 30% used cloze tests, 26% oral interviews or accent ratings, and 23% used other measures. In the remaining studies, proficiency was usually estimated through classroom level or years of instruction (60%), length of residence in the L2 environment (24%), existing proficiency scores (19%), or self-ratings (8%). Interestingly, 14% of the studies did not include estimates of proficiency at all. In a similar review, Hulstijn (Reference Hulstijn2012) detected an increase of “objective” L2 proficiency assessments in studies from 19% to 54% over the period of 1998 and 2011 with measures varying from standardized tests (e.g., IELTS, TOEFL) to cloze tests, translation tasks, and panel judgments. Test format features have also been found to affect the ability of different instruments to distinguish between learners of different proficiencies, as shown in recent meta-analytic research on L2 proficiency measures other than EIT. A methodological synthesis and meta-analysis of grammaticality judgment tasks (GJTs) showed that scores on untimed GJTs were substantially higher than on timed ones (d = 1.35; Plonsky et al., Reference Plonsky, Marsden, Crowther, Gass and Spinner2020). In a similar fashion, a meta-analysis conducted by In’nami and Koizumi (Reference In’nami and Koizumi2009) demonstrated that scores on a listening test that consisted of multiple-choice items were substantially higher compared to open-ended items (Hedges’s g = 1.11). Thus, it seems that the need for agreed upon tests of proficiency for research purposes remains (e.g., Carlsen, Reference Carlsen2012; Ginther & Yan, Reference Ginther and Yan2018).
EIT as a Measure of L2 Proficiency
In response to some of the concerns over proficiency assessment just described, EITs have been tested and employed extensively as a measure of L2 proficiency. Generally, an EIT includes sentence-level stimuli that include a range of grammatical structures and increase in the number of syllables as the test progresses. This assessment technique consists of learners hearing a stimulus and repeating it as accurately as possible with the assumption that learners can only do so if they have parsed the sentences and understood it (Bley-Vroman & Chaudron, Reference Bley-Vroman, Chaudron, Tarone, Gass and Cohen1994; Vinther, Reference Vinther2002). The designs of individual EITs vary, however. For example, some contain ungrammatical stimuli (e.g., Spada et al., Reference Spada, Shiu and Tomita2015) or a delay after the sentence is presented to the test-taker (e.g., Tracy-Ventura et al., Reference Tracy-Ventura, McManus, Norris, Ortega, Leclercq, Edmonds and Hilton2014). EITs can also focus on a specific target-language feature (or set of features) (e.g., Sarandi, Reference Sarandi2015) or measure global proficiency (e.g., Ortega, Reference Ortega2000).
One critique commonly leveled toward EITs is that the task may not be reconstructive in nature and that subjects may merely imitate the sounds they hear relying on rote memorization of stimuli—a process known as “parroting” (Fraser et al., Reference Fraser, Bellugi and Brown1963; McDade et al., Reference McDade, Simpson and Lamb1982; Vinther, Reference Vinther2002). Despite such concerns (e.g., Kaplan, Reference Kaplan1994), EITs have been used to assess different L2 constructs including communicative competence (Savignon, Reference Savignon1982), implicit knowledge (Ellis, Reference Ellis2005; Han & Ellis, Reference Han and Ellis1998), as well as oral proficiency (Bowden, Reference Bowden2016; Hameyer, Reference Hameyer1980; Solon et al., Reference Solon, Park, Henderson and Dehghan-Chaleshtori2019) and overall proficiency (e.g., Ortega et al., Reference Ortega, Iwashita, Norris and Rabie2002; Spada et al., Reference Spada, Shiu and Tomita2015; Suzuki & DeKeyser, Reference Suzuki and DeKeyser2015; Tracy-Ventura et al., Reference Tracy-Ventura, McManus, Norris, Ortega, Leclercq, Edmonds and Hilton2014; Wu & Ortega, Reference Wu and Ortega2013).
To bolster the validity argument for EIT as a trustworthy measure of L2 proficiency, L2 research has largely focused on examining the relationship between learners’ performance on EI tasks and more established measures of proficiency. Several EIT validation studies found significant positive correlations between learners’ EIT scores and scores on standardized proficiency measures with correlations ranging from 0.62 for IELTS (Erlam, Reference Erlam2006) to 0.76 for Test of Proficiency in Korean (TOPIK) (Kim et al., Reference Kim, Tracy-Ventura and Jung2016; see also Isbell & Son, Reference Isbell and Son2021). Similarly, strong and positive correlations from r = 0.87 to 0.91 have been found between EIT and Simulated Oral Proficiency Interviews (SOPI), particularly in L2 Spanish (Bowden, Reference Bowden2016; Ortega, Reference Ortega2000; Ortega et al., Reference Ortega, Iwashita, Norris and Rabie2002). Still other studies have examined the relationship between EIT and self-assessed proficiency scores for speaking (.75 > r > .93), listening (.73 > r > .93), reading (.62 > r > .87), and writing (.59 > r > .89) (Bowden, Reference Bowden2016; Chaudron et al., Reference Chaudron, Ngyuen and Prior2005; Gaillard & Tremblay, Reference Gaillard and Tremblay2016; Mozgalina, Reference Mozgalina2015) as well as the overall self-rated proficiency, r = .62 (Baten & Cornillie, Reference Baten and Cornillie2019). The all-encompassing nature of elicited imitation as a measure of proficiency is also reinforced given the strong relationship between EITs and cloze tests, r = .89 (Gaillard & Tremblay, Reference Gaillard and Tremblay2016) and end-of-year grades, r = .78 (Tracy-Ventura et al., Reference Tracy-Ventura, McManus, Norris, Ortega, Leclercq, Edmonds and Hilton2014). Finally, further strengthening the validity argument for the EIT, estimates of internal consistency for EITs tend to be fairly high.
Given the large body of research on EIT as a measure of L2 proficiency, it is not surprising that this relationship has attracted the attention of secondary researchers. In a synthesis of 24 studies, Zhou (Reference Zhou2012) found EI to be a dependable measure of proficiency with internal consistency ranging from .78 to .96. The study also provided support for construct-related validity of EITs as the correlation between EIT scores and other L2 proficiency measures was higher than .5 in most of the studies in the sample (p. 90). A more recent and more comprehensive two-phase narrative review and meta-analysis by Yan et al. (Reference Yan, Maeda, Lv and Ginther2016) surveyed 76 studies to provide a comprehensive account of the use of elicited imitation in research and assessment contexts. The narrative review demonstrated the variability of EITs with regard to their task features. Nevertheless, EITs were found to be useful in distinguishing participants at higher and lower proficiency levels according to the meta-analysis of 21 studies (Hedges’s g = 1.34). Moderator analyses revealed that the overall effect size differed significantly across the target constructs (e.g., general language proficiency, implicit grammatical knowledge), sentence length, and scoring method. Critically, the authors converted the effect sizes obtained from correlational studies into a standardized mean difference index (Hedges’s g). Although this decision allowed the authors to include a greater selection of studies, it also led to a loss of data in that proficiency, measured in many cases as a continuous variable, was reduced to a dichotomized one (i.e., low vs. high; see discussion on the perils of doing so in Plonsky & Oswald, Reference Plonsky, Loewen and Sato2017). The overall meta-analytic effect, therefore, cannot be interpreted as the relationship between proficiency tests and EIT scores but, rather, the difference between lower and higher proficiency groups, determined variously across studies, on EIT scores.
The approach taken by Yan et al. (Reference Yan, Maeda, Lv and Ginther2016) provides an important type of criterion-related validity evidence. However, addressing EIT validity from a different angle—that is, by including and focusing on correlational data as in the present study—can preserve the variance in both EITs and proficiency tests to estimate the overall relationship between them as well as the extent to which that relationship might be moderated by sample- and task-level features. In doing so, the present study adds important criterion-related concurrent validity evidence, complementary to that of Yan et al., by estimating the strength of the relationship between learners’ scores on EITs with those of other proficiency measures. In what follows, we briefly describe relevant sample- and task-level features of EIT.
Features of EITs That May Affect Its Construct-Related Validity
There are several features that have been examined in EIT-based studies with regard to their effect on the relationship between EIT performance and test scores. Yan et al. (Reference Yan, Maeda, Lv and Ginther2016), building on previous reviews of elicited imitation (e.g., Vinther, Reference Vinther2002), considered four such features: (a) length of sentence stimuli, (b) delayed repetition, (c) grammatical features of the stimuli, and (d) scoring methods. Adding to that list, Erlam (Reference Erlam2006) investigated the role of delay in the validity of EIT. Overall, the evidence from these studies seems to suggest that control of these variables may increase the ability of EIT to discriminate between learners’ proficiency levels while reducing variance caused by extraneous constructs (e.g., short-term memory).
Previous studies have also attempted to address the “parroting” issue of EITs by including stimuli of different lengths. Nearly five decades ago, Naiman (Reference Naiman1974) suggested that EITs can measure comprehension if sentences are long enough. Additionally, Perkins, Brutten, and Angelis (Reference Perkins, Brutten and Angelis1986) named stimuli length the most robust discriminator of item difficulty. Graham and colleagues (Reference Graham, McGhee, Millard, Prior, Watanabe and Lee2010) reported that stimuli length was the best predictor of L2 learners’ performance (R2 = .728); by comparison, lexical frequency explained only 8% of the variance in participants’ scores. Sentence length was also found to be a strong predictor of participants’ EIT performance as related to other proficiency tests in Hendrickson et al. (Reference Hendrickson, Aitken, McGhee, Johnson, Prior, Watanabe and Lee2010), Ortega et al. (Reference Ortega, Iwashita, Norris and Rabie2002), and Yan et al. (Reference Yan, Maeda, Lv and Ginther2016). Gaillard and Tremblay (Reference Gaillard and Tremblay2016) attributed this effect to the sentence complexity that increases with length and the added cognitive load imposed on the parser. They also noted that length is best represented by the number of syllables in a sentence. However, to date, no standard has been established for determining an optimal length or range of lengths. Graham et al. (Reference Graham, McGhee, Millard, Prior, Watanabe and Lee2010) reported that participants’ performance dropped significantly on sentences containing more than 16 syllables. Other studies found that learners’ accuracy declined when sentences contained 5 to 10 words and was extremely low with stimuli of 17 words or more (Chaudron et al., Reference Chaudron, Ngyuen and Prior2005). In comparison, Erlam (Reference Erlam2006) found only a moderate and negative correlation between sentence length and success at imitation (r = –.28).
Another EIT feature that varies across studies is whether learners are prompted to repeat the target sentence immediately or following a delay. A delay in the imitation can be included either to give the participants an opportunity to rehearse a stimulus before repeating it or to limit parroting; however, it can also present an additional load on the working memory and cause the cue to fade away. In her validation study, Erlam (Reference Erlam2006) added a delay to the EIT by asking the test-takers to focus on the meaning of the sentence and respond to a comprehension question. According to the author, this design feature helped the participants access explicit linguistic knowledge instead of focusing on the form of the utterance. In contrast to Erlam, Ortega et al. (Reference Ortega, Iwashita, Rabie and Norris1999) did not include meaning-based judgments to avoid adding another cognitive dimension to the task. Instead, they added a 3-second pause after the stimuli to prevent overestimation of the L2 learners’ linguistic abilities.
The difficulty of items on EIT has also been shown to be affected by syntactic, lexical, and phonological complexity of the stimuli (Graham et al., Reference Graham, McGhee, Millard, Prior, Watanabe and Lee2010; Ortega, Reference Ortega2000). Another practice in the development of EIT has been the inclusion of ungrammatical stimuli with the purpose of studying the learners’ reaction to such anomalies. The assumption underlying this practice is that only learners who can identify the grammatical error are truly processing and understanding the sentence (Kaplan, Reference Kaplan1994). As explained by Yan et al. (Reference Yan, Maeda, Lv and Ginther2016), when an ungrammatical sentence is processed, the speakers will reconstruct its meaning using their explicit grammatical knowledge and correct the error in the output. Although some researchers maintain that spontaneous correction of ungrammatical sentences can be a powerful indicator of the reconstructive nature of EIT and its validity as a proficiency assessment tool (e.g., Erlam, Reference Erlam2006; Markman et al., Reference Markman, Spilka and Tucker1975), others are skeptical about the usefulness of including ungrammatical stimuli. Yan et al. (Reference Yan, Maeda, Lv and Ginther2016) and Bowden (Reference Bowden2016) pointed out that the repetition of ungrammatical sentences verbatim can simply be a result of test administration procedure because participants are often not given explicit instructions to correct errors.
In their meta-analysis, Yan et al. (Reference Yan, Maeda, Lv and Ginther2016) hypothesized that the way EIT is scored may influence its reliability and, therefore, its ability to distinguish between speakers of different proficiency levels. Moreover, as observed by Vinther (Reference Vinther2002), even those scoring procedures that attempt to reach maximum objectivity have an element of arbitrariness in them. So far, three main scoring methods have been common: the binary approach, the ordinal rating scale approach, and the interval scale approach (Yan et al., Reference Yan, Maeda, Lv and Ginther2016). In the binary scale, repetitions are scored as either correct (1 point) or incorrect (0 points) (e.g., Gray, Reference Gray2017; Spada et al., Reference Spada, Shiu and Tomita2015). The ordinal rating scale approach involves the use of a rubric (e.g., 1–5) to measure the accuracy of each repetition (e.g., Gaillard & Tremblay, Reference Gaillard and Tremblay2016; Tracy-Ventura et al., Reference Tracy-Ventura, McManus, Norris, Ortega, Leclercq, Edmonds and Hilton2014). Studies that use the interval scale approach measure the number or percentage of errors in participants’ production (Lonsdale & Christensen, Reference Lonsdale and Christensen2011).
The opinions of researchers in the domain regarding the necessity of imposing a time limit on participants’ production vary substantially. Some maintain that an EIT must be performed under time pressure for it to measure implicit language knowledge (Bowles, Reference Bowles2011; Ellis, Reference Ellis2005). Others insist that such pressure is disadvantageous because it adds unnecessary stress for the participants leading to underestimations of their ability (Gaillard & Tremblay, Reference Gaillard and Tremblay2016).
One final consideration is the number of items in the EIT. Generally speaking, scales with a greater number of items yield higher reliability estimates (McKay & Plonsky, Reference McKay, Plonsky, Winke and Brunfaut2021). It is also important for a test to have enough items to be able to achieve adequate coverage of the construct of interest. However, including too many items increases risk of fatigue, thus introducing a threat to the validity of the instrument (Phakiti, Reference Phakiti2014). The role of the number of sentences in an EIT has not been discussed in this domain per se. However, previous meta-analyses of L2 research have examined this variable. For example, Marsden et al. (Reference Marsden, Morgan-Short, Thompson and Abugaber2018) synthesized self-paced reading studies in SLA and found that the number of target items in an experiment affected learners’ outcomes. The possibility that the length of an EIT has an effect on its ability to accurately represent the proficiency of the test-takers merits investigation.
The preceding review clearly demonstrates that certain task features of an EIT may play a crucial role in its reliability as a proficiency measure. The authors call for a more in-depth investigation of these features in relation to the concurrent validity of EIT.
The Present Study
The current study is a conceptual replication and expansion of Yan et al.’s (Reference Yan, Maeda, Lv and Ginther2016) meta-analysis in that it also investigates the criterion-related validity of EIT as a measure of L2 proficiency but takes a different approach by examining correlational studies. In other words, instead of operationalizing proficiency categorically and focusing only on differences in EIT scores for groups at different proficiency levels, this meta-analysis will view proficiency as a continuous variable and will focus on the correlation between EIT scores and other measures of L2 proficiency. This study will also examine the extent to which that correlation might vary as a function of several moderators. The following research questions will be addressed:
RQ1: To what extent do EI scores correlate with other measures of L2 proficiency?
RQ2: To what extent does the relationship between EI scores and other measures of proficiency vary across different features of the research design and task/instrument?
RQ3: How reliable are measures in studies of the criterion-related validity of EITs?
Method
Retrieval of Studies and Inclusion Criteria
An exhaustive and principled search was carried out to locate and retrieve a representative if not exhaustive sample of studies to be included in the meta-analysis. Both published and unpublished studies (e.g., doctoral dissertations, master theses, conference presentations) were eligible to reduce the risk of inflated average effect sizes due to publication bias (see Borenstein et al., Reference Borenstein, Hedges, Higgins and Rothstein2010 and Ferguson & Brannick, Reference Ferguson and Brannick2012 for discussion). Before the search was conducted, we established that we would include only those studies that:
-
1. Examined the relationship between EIT scores and other measures of L2 proficiency;
-
2. Reported correlations between EIT and other proficiency measures (e.g., Pearson’s r); and
-
3. Were written in English because this was the language shared by both authors.
Following Plonsky and Brown (Reference Plonsky and Brown2015), several techniques were applied to locate relevant studies. First, nine online databases were searched: ERIC, LLBA (Linguistics and Language Behavior Abstracts), PsycINFO, DOAJ (Directory of Open Access Journals), ProQuest Dissertation and Theses, ScienceDirect Journals, JSTOR Arts and Sciences, SAGE Journals Premier, and Wiley Online Library. Additionally, Google and Google Scholar were searched. The keywords used in this step of the search were a combination of phrases related to (a) the task including “elicited imitation (task)” or “elicited imitation test” or “EI” or “EIT” or “sentence repetition” or “imitation” or “repetition”; and (b) phrases related to language learning (“second language” or “foreign language” or “L2”). Furthermore, manual backward citation searches were performed on the identified studies as well as previous syntheses (Yan et al., Reference Yan, Maeda, Lv and Ginther2016; Zhou, Reference Zhou2012). When a report was not readily available online or through the university library, the authors were contacted. Specifically, we contacted two first authors of studies who sent us the requested reports or directed us toward a copy online.
The final sample consisted of 46 studies published between 1975 and 2021 (full citations appear immediately before the References section): thirty peer-reviewed journal articles, one book chapter, seven dissertations, two theses, and six conference proceedings. The list of peer-reviewed journals that contributed to the sample is given in Table 1.
Journal sources included in the sample
Coding Scheme
The coding scheme in this study was an adaptation of the instruments used in two conceptually related meta-analyses (Yan et al., Reference Yan, Maeda, Lv and Ginther2016 and Plonsky et al., Reference Plonsky, Marsden, Crowther, Gass and Spinner2020). The studies were coded on five categories of items: (a) study identification, (b) study design and participant characteristics, (c) instrument design features, (d) reporting practices, and (e) reported results. The first draft of the coding scheme was piloted on a subsample of three studies. After the necessary adjustments were made to the coding scheme based on the pilot coding and recommendations from colleagues, the rest of the studies were coded. Additionally, a second coder with meta-analytic experience recoded 20% of the studies to estimate interrater reliability. The average agreement between the two coders was 82%. In 9 out of 79 categories, agreement was below 70%. The first coder then revisited those items with the second coder and requested that they double check their coding. Agreement was then recalculated for those categories and reached 83% or higher in all cases. The overall agreement also improved to 94%. The complete coding scheme is available on IRIS (Marsden et al., Reference Marsden, Mackey, Plonsky, Mackey and Marsden2016).
Analysis
Attending to the first research question, concerned with the overall relationship between EIT performance and scores on other measures of L2 proficiency, correlation coefficients obtained from the studies in the sample were weighted and averaged using the meta-analytic functions in JASP (version 0.12.2). Because several studies included various r coefficients in the reporting of their results, the principles of handling multiple effect sizes outlined in Yan et al. (Reference Yan, Maeda, Lv and Ginther2016) were followed. Effect sizes from four EITs that were administered separately in different target languages reported in one study were not combined and were, rather, included in the calculations separately (Ortega, Reference Ortega, Iwashita, Norris and Rabie2002). However, if a study reported correlations between an EIT and multiple proficiency measures from a single sample, all effect sizes were averaged for the calculation of the aggregated meta-analytic correlation (e.g., Bowden, Reference Bowden2016; Graham et al., Reference Graham, Lonsdale, Kennington, Johnson, McGhee, Calzolari, Choukri, Maegaard, Mariani, Odijk and Piperidis2008). Finally, if both overall and subsection proficiency test scores were reported, the correlation between EIT and total proficiency scores was included for analysis (e.g., Erlam, Reference Erlam2006). In total, 94 dependent and 45 independent correlation coefficients were retrieved from the 46 studies. As part of the analysis, a funnel plot of the obtained correlations was also created to assess the potential effect of publication bias on the overall effect size.
To examine the extent to which the relationship in question might vary across different study- and EIT-related features, a number of moderator analyses were performed. To do so, effect sizes were grouped, weighted, and aggregated across each of the following potential moderators: (a) type of proficiency measure, (b) immediate versus delayed repetition, (c) grammatical features of the stimuli, (d) construct measured, (e) scoring method, (f) timed versus untimed repetition, and (g) whether or not an estimate of EIT reliability was reported. Importantly, if a study reported correlations between EIT and multiple other proficiency measures, these correlations were kept separate in analyzing the moderating effect of the type of proficiency measure on the relationship in question. The role of two continuously measured moderators (length of stimuli, number of target items) was also considered by calculating the correlation between those measures and their corresponding observed correlations.
Finally, to address RQ3, which was concerned with reliability in this domain, we conducted a reliability generalization meta-analysis (see, e.g., Saito & Plonsky, Reference Saito and Plonsky2019). This step of the analysis involved calculating (unweighted) descriptive statistics (Ms, SDs, CIs) of reliability estimates from EITs as well as the other proficiency measures correlated with EITs. In parallel to the different EIT scoring methods described in the preceding text, two types of reliability were meta-analyzed from EITs: internal consistency (i.e., instrument reliability such as Cronbach’s alpha) and interrater reliability, which was separated out by the different reliability indices reported in the sample (i.e., % agreement, correlation between raters, and Cohen’s κ).
Results
To better understand the body of research in the domain of EIT as a measure of proficiency, we calculated frequencies and percentages of different participant demographics and study contexts. Table 2 presents the results for the demographic features of age and proficiency.Footnote 1 Among those studies that reported participants’ age, the majority included adult L2 learners (k = 23) with one study (Kang & Lust, Reference Kang and Lust2019) focusing on children. Although advanced learners were the most common participant group in the sample, it is important to note that proficiency was measured continuously in the studies with the participants being spread out across proficiency levels. Most often, participant proficiency was assumed from institutional level (k = 24). In some studies, proficiency was justified by the results of a standardized test taken by the participants beforehand (k = 8), self-reports (k = 5), or nonstandardized tests (k = 4). In four of the studies, no proficiency justification was given.
Summary of participant demographics
*Most were likely adults, given the university-based context of the majority of the sample.
Regarding the native and target languages of the participants, several clear patterns emerged. Most of the studies included speakers of multiple languages (k = 20, 43%) followed by L1 English speakers (k = 10, 22%) and bilinguals (k = 2, 4%). In addition, there was one study each that included speakers of Japanese, Spanish, Dutch, Chinese, Arabic, and Turkish. English was the most popular L2 in the sample (k = 17, 37%). Spanish and French were next (k = 7 and 5, respectively, or 15% and 11%). Other studies included learners of Korean, German, Korean, Russian, Japanese, Chinese, Dutch, and Portuguese. There was one study that employed an artificial language in its design (Ettlinger et al., Reference Ettlinger, Morgan-Short, Faretta-Stutenberg and Wong2016). Table 3 summarizes the learning contexts of the reports in the sample. Most of the studies took place at colleges or universities (k = 37, 74%) with the target language being foreign for the participants (k = 25, 48%). Some of the studies took place in language institutes (k = 4, 9%) and one study (Risdiani, Reference Risdiani2016) involved hospitality workers.
Summary of study contexts
Overall Effect Size
The results regarding the first research question, concerning the criterion-related validity of elicited imitation as a measure of L2 proficiency, are shown in Figure 1. This figure presents a forest plot of all the obtained effect sizes and their corresponding 95% confidence intervals. In the aggregate, these effect sizes produced a positive and fairly strong correlation between EIT and other proficiency measures: r = .66, 95% CI [.61, 71]. To understand the extent of this relationship in the absence of measurement error, we applied a correction for attenuation. Because not all studies reported reliability estimates, we applied this correction at the meta-analytic (as opposed to the individual study) level. Adjusted for the internal consistency reliability of both EIT measures and the other measures of proficiency they were correlated with (0.92 and 0.86, respectively, see the following text), the overall correlation increases to .74. This weighted effect size can be considered generally strong according to Plonsky and Oswald’s (Reference Plonsky and Oswald2014) guidelines for interpreting effect sizes in L2 research. At the same time, a correlation of .66 (or .74) may not seem especially strong in the context of criterion-related validity evidence for proficiency tests. Thus, the result adds to the validity argument behind the EIT as a reasonably trustworthy measure of L2 proficiency. The tight confidence interval around this effect size, indicating the precision of the overall estimate, offers further support to this conclusion.
Forest plot of effect sizes (r) across the sample.
To assess the presence and extent of publication bias in the sample, a funnel plot was created using meta-essentials (Suurmond et al., Reference Suurmond, van Rhee and Hak2017; Figure 2). Before creating the plot, all correlations were transformed into z-scores to stabilize the variance in the sample. Borenstein et al. (Reference Borenstein, Hedges, Higgins and Rothstein2010) explain that if the meta-analytic sample is biased, the funnel plot will be asymmetrical with studies missing at the lower left-hand area (p. 283), which is only slightly the case. The larger spread of the scores to the right of average (indicated by the dot with the confidence intervals to the right of the median) further supports the slight presence of publication bias.
Funnel plot of observed and imputed effect sizes.
The “trim and fill”’ technique was then applied to the sample to impute the values of potentially missing studies in the sample and estimate the unbiased effect size accordingly. The five open circles in Figure 2 represent the imputed values, and the dot with confidence intervals around it is the z-score of the adjusted average effect size. When converted to r, the effect size was only slightly smaller than the observed one, r = .63, 95% CI [.60, .66]. Consequently, despite some evidence of publication bias, EIT scores can still be considered to be relatively strongly associated with other measures of proficiency.
Moderator Analysis
To answer the second research question, the variability of correlation coefficients across the sample was examined across a set of potential moderators. First was the type of proficiency measure correlated with EIT. Additionally, the effect size variability was examined with regard to design features of the EIT: (a) immediate versus delayed repetition, (b) grammatical features of the stimuli, (c) measured construct, (d) scoring method, (e) timed versus untimed repetition, (f) length of stimuli, and (g) number of target items. The detailed descriptions of the moderator categories are given in the coding scheme in Appendix A.
Type of Proficiency Measure
The first moderator analysis in the study considered the type of other proficiency measures correlated with EIT scores. When categorizing proficiency measures, we used a modified version of the framework presented by Gass et al. (Reference Gass, Behney and Plonsky2020) who proposed three common ways of classifying learners by proficiency: standardized tests (e.g., TOEFL), nonstandardized researcher-made tests, also called locally developed or research-specific tests in Thomas (Reference Thomas, Norris and Ortega2006), and institutional data (e.g., results of an end-of-year achievement test). We decided to add another category in this classification, namely, self-reports. It is noteworthy that some studies in our sample used multiple proficiency measures; for example, Bowden (Reference Bowden2016) included both a nonstandardized test as well as self-reports and correlated the results of both of these measures with EIT scores. To avoid inappropriate data dependencies, the different r coefficients from such studies were included in the calculations separately. The results, summarized in Table 4, revealed that EIT correlated most strongly with self-reports (r = .81) followed by standardized tests (r = .74). In contrast, EIT scores did not relate as strongly with the results of nonstandardized tests and institutional reports (r = .60 and r = .49, respectively). We note, though, that some of the subgroups in these analyses included only a few studies, as shown in the k columns and should be interpreted with caution.
Summary of moderator analyses across types of proficiency measures
Note: Ettlinger (Reference Ettlinger, Morgan-Short, Faretta-Stutenberg and Wong2016) was not included in this moderator analysis as the study focused on an artificial language.
EIT Design Features
The next set of moderator analyses was concerned with the features of EITs. The comparison of effect sizes based on immediate versus delayed repetition showed that the EITs with a delay revealed slightly stronger associations with other measures of proficiency (Table 5). Within the group of studies with delayed repetition, two types of delay were found: 2.5- to 3-second pause (k = 22) and comprehension questions (k = 9). Interestingly, the average effect size of studies with a pause after the stimuli was substantially larger than in studies where participants had to answer a question after hearing the sentence (r = .76 vs. .52, respectively).
Summary of moderator analyses across EIT design features
Furthermore, the results showed that EITs were more consistent in studies that did not include ungrammatical stimuli in their tests, as seen in Table 5. Although the use of ungrammatical stimuli did not lead to higher consistency of EITs, a similar trend was noted by Yan et al. (Reference Yan, Maeda, Lv and Ginther2016) who found tasks that used only grammatical sentences to be more discriminating across proficiency levels (g = 1.34 vs. g = 1.16).
There was a slight difference in effect sizes from studies that used EIT to measure global proficiency as opposed to specific constructs (Table 5). Most frequently, the tasks were controlled for lexical complexity of stimuli (k = 14) followed by syntactic complexity (k = 13) and morphological/morphosyntactic complexity (k = 11). Some studies also included control of phonological features of the sentences like presence or absence of liaison and schwa deletion (e.g., Gaillard & Tremblay, Reference Gaillard and Tremblay2016). It is important to note that EITs oftentimes focused on more than one construct.
The analysis of the type of scoring used to measure the results of the EITs revealed a stronger correlation among studies that used ordinal scales compared to binary scales. The strongest correlation was observed for EITs that used interval and mixed scales, although there were only two such studies (e.g., Serafini, Reference Serafini2013). Most of the studies used holistic assessment (k = 41) whereas some researchers chose to use analytic rubrics to assess participants’ performance. For example, Burger and Chrétien (Reference Burger and Chrétien2001) graded participants on accuracy of pronunciation, syntactic, and discourse features on top of overall exact repetition while Gaillard (Reference Gaillard2014) used a rubric that consisted of six categories representing constructs within L2 proficiency (e.g., vocabulary, morphology, syntax). Finally, as shown in Table 5, there was virtually no difference in effect sizes between timed and untimed EITs suggesting that this design feature did not play a major role in observing the relationship between EIT and other measures of proficiency. We also note that the majority of the studies in the sample (k = 31) failed to report this information; therefore, this result should be taken as tentative.
In contrast with previous meta-analyses of elicited imitation, we decided to maintain the continuous nature of variables as much as possible when examining the moderators. Thus, correlational analyses were conducted between the length of stimuli as well as the number of items in the EIT and the observed effect sizes (Tables 6 and 7). In examining the length of the test items, we used two indicators: the maximum number of syllables and the range in syllables. We chose these indicators because they allowed us to shed light on the relationship between the difficulty of EIT and its concurrent validity.
Summary of moderator analyses across continuously measured EIT features
Moderator analysis of EIT reliability reporting
The analysis revealed a moderate and positive correlation between the maximum length of stimuli and correlations seeking to estimate criterion-related validity of EITs. Moreover, the range of lengths included in the task was also found to be positively linked with the observed effect sizes. Thus, it seems that the wider the range of sentence lengths and the longer the sentences are, the better an EIT is at measuring proficiency.
A closer examination of the sample revealed that the most common lengths of sentences included in EITs were 7–17 syllables (k = 8) and 7–19 syllables (k = 10). It seems that this length became especially popular after Ortega’s (Reference Ortega2000) and Ortega et al.’s (Reference Ortega, Iwashita, Norris and Rabie2002) seminal EIT studies. The three effect sizes with r >.9 were found in the studies that had target sentences up to 27, 32, and 34 syllables long (Gaillard, Reference Gaillard2014; Lonsdale & Lever, Reference Lonsdale and Lever2015; Solon et al., Reference Solon, Park, Henderson and Dehghan-Chaleshtori2019). Five studies did not consider sentence length as important in developing EITs (e.g., Baten & Cornillie, Reference Baten and Cornillie2019) or stated that it varied but did not report the exact values (e.g., Gray, Reference Gray2017).
With regard to the number of target items included in the test, the studies in our sample displayed a large variation with the shortest task including eight items (Kang & Lust, Reference Kang and Lust2019) and the longest 112 (Meir et al., Reference Meir, Walters and Armon-Lotem2016). After calculating the average number of items in the studies and its standard deviation, Meir et al. (Reference Meir, Walters and Armon-Lotem2016) was found to be an outlier as the number of items in their test was higher than two SDs above the mean with a moderate corresponding correlation (r = .51); therefore, this study was excluded from this part of the analysis. Training items were not included in these calculations either. The significant moderate correlation between the number of items and EIT consistency presented in Table 6 suggests that the ability of the task to measure participants’ proficiency increases with the length of the test.
Reliability in EIT Research
Critical to the validity argument for EITs, like any quantitative measure, is the presence of measurement error. Most, but not all, studies in the sample reported one or more estimates of the internal consistency of the test expressed by either Cronbach’s alpha or Kuder–Richardson Formula 20 coefficients. As indicated in Table 7, there was a small difference in r coefficients, with the studies that reported reliability demonstrating a slightly stronger correlation between EITs and other proficiency measures. However, an analysis of reliability coefficients and r coefficients of the relationship between EITs and other proficiency measures revealed a moderate positive correlation between the two values: r = .59, 95% CI [.32, .77]. In other words, higher internal consistency of EIT was associated with higher criterion-related validity evidence.
The final research question involved conducting a reliability generalization meta-analysis of the reliability estimates found in the sample. To carry out this phase of the study, we averaged (without weighting) the available EIT reliability coefficients, EIT interrater reliability estimates, and the reliability of other proficiency tests correlated with EITs. As seen in Table 8, we found that the overall internal consistency reliability of EIT was quite high (.92) with coefficients ranging from .7 to .99. Given the range of different proficiency tests, mean reliability of other measures of proficiency was also quite high (.86). In addition to internal consistency, three indices of interrater reliability were recorded and aggregated as well. Here, too, overall meta-analytic estimates were quite high whether calculated as agreement among raters (.88 for % agreement, .91 for kappa) or correlation between raters (.95). As with other results, these mean coefficients should be interpreted with caution because less than half of the studies in the sample reported interrater reliability of EIT and reliability of other proficiency measures.
Average reliability coefficients reported in the sample
To summarize, in addition to an overall moderate-to-strong relationship between EI and other measures of proficiency (r = .66 or r corrected = .74, the results of the moderator analyses revealed fairly consistent variability in this relationship based on the type of proficiency measure EIT was correlated with (e.g., r = .74 for standardized tests vs. .49 for course grades) as well as a number of test design features such as pauses, target features, and scoring methods (e.g., r = .85 for interval and mixed scoring vs. .53 for binary scoring). Overall, reliability estimates in this domain were found to be fairly high, both for EITs and other measures of proficiency.
Discussion
The main objective of this study was twofold: We sought to (a) estimate the overall correlation between EITs and other measures of L2 proficiency, as an indicator of the former’s concurrent validity (RQ1) and (b) to assess variability in that relationship as a function of study design and EIT features. We also aimed to investigate measurement error in this domain using reliability generalization meta-analysis. In doing so, we tried to update and expand on previous synthetic efforts in this domain (Yan et al., Reference Yan, Maeda, Lv and Ginther2016; Zhou, Reference Zhou2012) by including a number of fairly recent studies and by preserving the correlational nature of the data as observed in many primary studies.
The overall aggregated effect size of r = .66 (r corrected = 0.74) confirmed that EITs can be viewed as a fairly dependable measure of L2 proficiency, at least from the point of view of its criterion-related and concurrent validity. Undoubtedly, a moderate to strong correlation between EIT and other proficiency measures does not automatically signify that the test is an inherently valid measure of proficiency (see Révész & Brunfaut, Reference Révész, Brunfaut, Winke and Brunfaut2021, for a discussion on validity evidence in SLA). This is especially important because the proficiency tests EITs have been correlated with differ in design and administration. Nevertheless, the overall result of this study seems to confirm in complementary fashion the results of previous meta-analyses that found EIT to be well-aligned with other measures of L2 proficiency. Yan et al. (Reference Yan, Maeda, Lv and Ginther2016) found that EIT can clearly differentiate between lower- and higher-level learners as demonstrated by a large effect size (Hedges’s g = 1.34). Although Zhou did not present an average effect size, the author did mention that the majority of studies in her sample reached a correlation larger than .50 (p. 91). This finding adds further support to the large body of research seeking to present validity evidence for the use of elicited imitation for L2 assessment purposes (e.g., Bowden, Reference Bowden2016; Chaudron et al., Reference Chaudron, Ngyuen and Prior2005; Gaillard & Tremblay, Reference Gaillard and Tremblay2016; Kim et al., Reference Kim, Tracy-Ventura and Jung2016; Ortega, Reference Ortega2000; Solon et al., Reference Solon, Park, Henderson and Dehghan-Chaleshtori2019). Another way of understanding the overall value of r = .66 (r corrected = 0.74) is to compare it to correlations between proficiency tests and other alternative measures. Li and Zhang (Reference Li and Zhang2021), for example, recently meta-analyzed the relationship between self-assessed proficiency and measures of language performance, observing an overall correlation of r = .47. McKay et al. (Reference McKay, Teimouri, Sağdiç, Salen, Reagan and Malone2021), however, observed much stronger evidence of the criterion-related validity in his meta-analysis of the relationship between C-tests and other measures of proficiency (r corrected = .87).
The utility of EIT as a measure of proficiency is promising for both L2 educators and researchers. In L2 research, EITs can and perhaps should be used more widely to bring consistency into the way proficiency is measured and, thus, contribute to comparability across studies. The results of this study also suggest that teachers and program administrators might even employ EIT as a means to assess learner proficiency because this test is more efficient and economical than proficiency measures such as OPIs (Oral Proficiency Interviews) or other standardized tests (e.g., Wu & Ortega, Reference Wu and Ortega2013).
Despite the promising overall result of the study, there is still little consistency in the way EITs are designed and administered. Oftentimes, researchers justify their choices in designing the tasks based on a previous study involving the same target language. Similar inconsistencies in instrument design have been noted in previous methodological syntheses by Plonsky et al. (Reference Plonsky, Marsden, Crowther, Gass and Spinner2020) on GJTs and by Marsden et al. (Reference Marsden, Thompson and Plonsky2018) on self-paced reading (SPR) tasks. Marsden et al., for example, found evidence for a lack of systematicity and standardization in SPR stimuli development. Likewise, Plonsky et al. argued for standardization across GJT features including time pressure, scoring, number of items, and others to avoid potential threats to validity in the form of nonconstruct relevant variance. Also related to the issue of test development, Plonsky et al. stressed the value of clear justification of the test developers’ choices based on specific learner populations, the type of knowledge being assessed (i.e., the target construct), and so forth.
Critically, variability in designs should not be seen as a simple matter of researcher preference. The moderator analyses of the present study found meta-analytic correlations to vary across several design and task features. One critical finding was that dependability of EIT as a measure of proficiency increases with the length and number of the stimuli included in the task. Other design features examined in this study were not found to be substantial predictors of the EIT × proficiency relationship. Nevertheless, it is important to consider these features when choosing or developing an EIT because they can still lead to higher or lower validity evidence. For instance, although the presence of an overall delay did not seem to affect the utility of EIT, the tasks that included silent pauses after stimuli presentation demonstrated substantially stronger correlations with other proficiency measures than those that did not include a delay. It is possible that some of these findings are related to the role of the parser in elicited imitation (Gaillard & Tremblay, Reference Gaillard and Tremblay2016; Serafini, Reference Serafini2013).
Overall, it seems that longer sentences and a pause before repetition may present additional cognitive load for the participants, which help limit their reliance on working memory and thereby serve to tap learner proficiency more clearly. Furthermore, longer sentences have been associated with greater difficulty of EITs (e.g., Perkins et al., Reference Perkins, Brutten and Angelis1986). Gradually increasing the length of stimuli can help to detect the point when the task becomes too challenging for a learner reflecting their level of ability in the L2. In contrast, including a comprehension question after the stimuli presentation may result in the EIT becoming a less useful signal of proficiency, potentially due to the parser overload since the test-taker would have to process the content of the sentences, answer the question, and then repeat the original sentence (Papadopoulou & Clahsen, Reference Papadopoulou and Clahsen2003). Thus, based on the results in this study, the optimal design of an EIT consists of including a pause after stimuli presentation and sentences with a wide range of syllables.
Similar to Yan et al. (Reference Yan, Maeda, Lv and Ginther2016), the findings of this study also suggest that the type of scoring method employed can be critical to the ability of EIT to distinguish learner proficiency. Simply counting repetitions as correct or incorrect was found to be associated with weaker correlations than using more nuanced scales. That is, although a binary scale may be faster and more efficient, an ordinal scale, and even more so, an interval or mixed scale, can provide a more accurate and fine-grained representation of learner ability. In contrast to Yan et al., however, the nature of the measured construct (global vs. specific) was not found to affect the dependability of EIT thus allowing the test creator more freedom in choosing whether to focus on a certain construct or not.
Contrary to previous research that recommended the inclusion of ungrammatical stimuli (e.g., Erlam, Reference Erlam2006; Hamayan et al., Reference Hamayan, Saegert and Laraudee1977; Kaplan, Reference Kaplan1994), this design feature was not found to improve evidence of concurrent validity of EITs. A similar finding was observed in the meta-analysis by Yan et al. (Reference Yan, Maeda, Lv and Ginther2016). It is clear, therefore, that ungrammatical stimuli do not aid in the EIT’s ability to assess learner proficiency. On the contrary, if participants are not properly instructed, such sentences may confuse them and muddle the results. On a related note, our results show that making elicited imitation a timed activity may add nothing to the task’s value. However, it is important to note that this finding must be interpreted with a grain of salt due to a large part of the sample not reporting whether EITs were timed or not.
Turning to the study design features, our analysis revealed several noteworthy trends. Importantly, we found that EIT had the highest correlations with standardized tests and self-reports. This observation lends support to our earlier point that this instrument can successfully substitute standardized testing in many instances such as when being used for research purposes.
Our results also showed that most studies in the sample (88%) included internal reliability estimates of EITs, which is much higher than the 5.25% reported by Zhou (Reference Zhou2012) for the domain of EI. This degree of reliability reporting is also high compared to many other L2 research domains (see meta-analysis of reliability coefficients in L2 research by Plonsky & Derrick, Reference Plonsky and Derrick2016). It is possible that this difference is due to the inclusion of more recent studies that have taken more care in reporting reliability estimates. In addition being frequently reported, reliability estimates in this domain were quite high, indicating little error stemming from a lack of consistency across items or raters.
Related to the relatively high overall estimates of reliability is the moderate to strong positive association between the reported EIT reliability coefficients and correlations of EIT scores with other proficiency measures. This is not entirely surprising because high reliability implies that the items in a test probe the same construct and the test is less affected by measurement error resulting from a lack of internal consistency. Nevertheless, this finding serves as a useful reminder to pilot and refine EITs to ensure the instrument is maximally consistent.
Conclusion and Implications
The results of the present study lend strong support to elicited imitation as a useful measure of L2 proficiency and lead to several implications for research in the domain. It is obvious that in the current landscape, there is a prevalence of studies that have centered around adult, college-level learners of English as a second language. To provide a more trustworthy and robust picture of the usefulness of EIT to assess L2 proficiency, future studies need to expand their research into other contexts and participant demographics (see Andringa & Godfroid, Reference Andringa and Godfroid2020; Plonsky, Reference Plonsky, Loewen and Sato2017; Isbell & Son, Reference Isbell and Son2021). Additionally, studies that focus on target languages other than English will not only help us understand the appropriateness of elicited imitation for learners of these languages but also contribute to the research on EIT as a functional assessment instrument independent of the L2. Regarding EIT design, there are several features of which we need to improve our understanding. For example, quite a few studies in the sample employed a type of delay in their EITs. Future research could examine the effect of different types of delays on task performance. Finally, in terms of EIT reliability, further studies at the primary and/or secondary level focused specifically on the relationship between reliability and task features (i.e., test features as moderators of reliability) would be a valuable addition to the domain as well.
Beyond presenting substantive results, one of the roles of meta-analysis is to provide methodological guidance to future research. One area in need of improvement is reporting practices, a recurring issue in L2 research (e.g., Larson-Hall & Plonsky, Reference Larson-Hall and Plonsky2015; Plonsky, Reference Plonsky2013). Specifically, 4 out of 11 moderators contained missing values because some studies in the sample did not include the necessary information to code them. Moreover, the majority of the reports did not include assumption checks, confidence intervals for the observed correlations, or the participant sampling method (see related study and commentary in Hu & Plonsky, Reference Hu and Plonsky2021). There is also room for improvement in the reporting of reliability, particularly with respect to interrater reliability.
Despite these weaknesses and corresponding suggestions for future research, the available evidence in this domain indicates an overall strong relationship between scores on EITs and other proficiency measures. We are hopeful that these results will both empower and inform future studies involving EITs as well as those that seek to better understand their design and application in L2 research.
SUPPLEMENTARY MATERIALS
To view supplementary material for this article, please visit http://doi.org/10.1017/S0272263121000395.
