Hostname: page-component-74656f9884-8rzl6 Total loading time: 0 Render date: 2026-11-05T22:25:08.298Z Has data issue: false hasContentIssue false
  • English
  • Français

EXPLORING AN ELICITED IMITATION TASK AS A MEASURE OF HERITAGE LANGUAGE PROFICIENCY

Published online by Cambridge University Press:  08 February 2022

Megan Solon Open the ORCID record for Megan Solon [Opens in a new window] ,
Hae In Park Open the ORCID record for Hae In Park [Opens in a new window] ,
Marzieh Dehghan-Chaleshtori Open the ORCID record for Marzieh Dehghan-Chaleshtori [Opens in a new window] ,
Carly Carver  and
Avizia Y. Long Open the ORCID record for Avizia Y. Long [Opens in a new window]
Megan Solon*
Affiliation:
Indiana University, Bloomington, IN, USA
Hae In Park*
Affiliation:
University at Albany, SUNY, Albany, NY, USA
Marzieh Dehghan-Chaleshtori
Affiliation:
University at Albany, SUNY, Albany, NY, USA
Carly Carver
Affiliation:
Augusta University, Augusta, GA, USA
Avizia Y. Long
Affiliation:
San José State University, San José, CA, USA
*
*Corresponding authors. E-mail: msolon@iu.edu; hpark9@albany.edu
*Corresponding authors. E-mail: msolon@iu.edu; hpark9@albany.edu
Rights & Permissions [Opens in a new window]

Abstract

This study examines a 30-item Spanish elicited imitation task (EIT) as a measure of global language proficiency for heritage language (HL) learners of Spanish. Results from Rasch modeling suggest that, while EIT scores demonstrated excellent reliability, the ability of much of the HL sample far exceeded the difficulty of the items. Differential item functioning analysis revealed that several items functioned differently for the present study’s HL sample than for the second language learners from a prior study. Additionally, although EIT scores were strongly correlated with another proficiency measure for the present HL sample (e.g., the Diploma de español como lengua extranjera or DELE), common-person equating suggested that the EIT and the DELE may measure different latent variables. Overall, the results provide evidence of the EIT’s utility as a proficiency assessment tool for research on HL learners, however, with limitations and the need for additional analyses discussed.

Information

Type
Research Article
Information
Studies in Second Language Acquisition , Volume 44 , Issue 4 , September 2022 , pp. 1095 - 1123
Check for updates
Copyright
© The Author(s), 2022. Published by Cambridge University Press

Introduction

Interest in heritage language (HL) acquisition and development has experienced a surge in recent years (Montrul, Reference Montrul2016). Fueled by changing demographics and the growing presence of students with cultural/familial connections to and native background in the “foreign” language(s) taught in schools, researchers and educators have dedicated new energy to exploring the nature of these learners’ linguistic knowledge, the processes by which their language(s) develop and change, and the pedagogical practices best suited for instruction of so-called heritage languages. According to Montrul (Reference Montrul2016), “As used in the United States, heritage speaker refers to individuals from language minority groups who grow up exposed to a minority language in the home and the majority societal language” (p. 16). In addition to constituting its own field of empirical inquiry, research on HL knowledge and development also contributes to broader understandings of the nature of bilingualism and the process of language acquisition (including first, second, and additional; e.g., Ortega, Reference Ortega2020). Nevertheless, a commonly recognized challenge of HL research and HL pedagogy is the vast heterogeneity of HL speakers (e.g., Polinsky & Kagan, Reference Polinsky and Kagan2007). This heterogeneity encompasses (but is not limited to) speakers’ age of acquisition of the heritage and majority languages, the quantity and quality of input in both languages as well as the levels of use of each language, and speakers’ formal education and literacy in the HL. Likewise, proficiency in the HL varies widely among HL speakers and learners and can also differ greatly between different skills (e.g., speaking and listening vs. reading and writing). This heterogeneity highlights the great challenge of grouping HL speakers for teaching and research purposes and has spurred calls for careful and deliberate proficiency assessment for these purposes (e.g., Son, Reference Son2017, Reference Son2021).

Similarly, within second language (L2) acquisition research, repeated calls for reliable and validated proficiency assessment techniques (e.g., Hulstijn, Reference Hulstijn, Unsworth and Blom2010, Reference Hulstijn2012; Thomas, Reference Thomas1994, Reference Thomas, Norris and Ortega2006; Tremblay, Reference Tremblay2011) have drawn attention to the challenges and the importance of proficiency assessment in research on language acquisition and development. Such calls have spurred empirical action related to the development and validation of various proficiency assessment tools for particular purposes and populations. One technique—elicited imitation—has garnered substantial recent attention. Following Ortega et al.’s (Reference Ortega, Iwashita, Norris and Rabie2002) design and exploration of parallel elicited imitation tasks (EITs) in four languages (i.e., English, German, Japanese, and Spanish), numerous studies have explored the validity and reliability of the use of various EITs for proficiency assessment of L2 learners for research purposes (e.g., Chinese: McManus & Liu, Reference McManus and Liu2020; Wu & Ortega, Reference Wu and Ortega2013; Zhou & Wu, Reference Zhou and Wu2009; French: Gaillard & Tremblay, Reference Gaillard and Tremblay2016; Tracy-Ventura et al., Reference Tracy-Ventura, McManus, Norris, Ortega, Leclercq, Edmonds and Hilton2014; Korean: Kim et al., Reference Kim, Tracy-Ventura and Jung2016; Spanish: Bowden, Reference Bowden2016; Solon et al., Reference Solon, Park, Henderson and Dehghan-Chaleshtori2019). Related studies have also explored the role of various EIT design features (e.g., scoring method: Gaillard & Tremblay, Reference Gaillard and Tremblay2016; inclusion of ungrammatical sentences: Erlam, Reference Erlam2006) and have examined the characteristics of EIT items that contribute to their relative difficulty and discriminability (e.g., Graham et al., Reference Graham, McGhee, Millard, Prior, Watanabe and Lee2010; Hendrickson et al., Reference Hendrickson, Aitken, McGhee, Johnson, Prior, Watanabe and Lee2010). Through these studies, EITs, and especially those following Ortega et al.’s 30-item, delayed repetition design, have been repeatedly shown to provide valid, reliable, and practical indices of L2 proficiency, with item difficulty most significantly affected by sentence length (see Kostromitina & Plonsky, Reference Kostromitina and Plonsky2021, for a recent meta-analysis). The present study, then, asks whether an EIT, as a common, recognized, and validated L2 proficiency assessment tool, can similarly be used to reliably assess HL proficiency for research purposes. Specifically, the present study explores the use of the Spanish EIT for HL proficiency assessment and compares the performance of the EIT for HL proficiency assessment to that for L2 proficiency assessment.

Background

Heritage language assessment

Existing research on HL assessment, especially as related to Spanish in the United States, has largely focused on assessment for course placement purposes. For example, articles dedicated to this very topic in the 2012 special issue of the Heritage Language Journal emphasize assessment as a “necessary first step” for distinguishing L2 and HL populations (Beaudrie, Reference Beaudrie2012) and a step that provides important information to educators and curriculum designers about the knowledge that HL learners bring to the classroom (Fairclough, Reference Fairclough2012). MacGregor-Mendoza (Reference MacGregor-Mendoza2012) argues that, to access HL speakers’ linguistic knowledge, HL assessments should reflect activities with which HL speakers would be familiar. Additionally, she emphasizes that assessments used for placement purposes should be developed in-house and must be site-specific so as to ensure that they align both with the needs of the local HL student population and with the goals and realities of the specific language program for which linguistic knowledge and language abilities are being assessed.

In the context of research, however, other priorities and goals—including practicality and replicability across studies and samples—take precedence. Likewise, because research on HL learners often incorporates comparisons to L2 learners or other bilinguals as well as monolinguals, the importance of the ability to reliably establish proficiency across participant groups and backgrounds is heightened. Son (Reference Son2018), for example, explored the validity of interpreting scores of a Korean C-test as a measure of Korean HL and L2/foreign language proficiency for research purposes. Her study, which included assessment using the Korean EIT (to which we will return in a later section), showed that both speaking and writing proficiency contributed to explaining C-test score variance (R2 = .80). Nevertheless, writing proficiency was a stronger predictor of C-test scores than speaking proficiency for HL learners, whereas, for foreign language learners, the two skills equally predicted C-test outcomes. Ultimately, Son concluded that the Korean C-test offers a valid, reliable, and practical option for Korean HL (and foreign language) proficiency assessment (with some caveats) and constitutes an important tool for researchers exploring Korean HL acquisition.

Researchers investigating Spanish as a HL in the United States have largely adopted a modified version of the Diploma de español como lengua extranjera “Diploma of Spanish as a Foreign Language” (DELE) exam (Montrul, Reference Montrul2012) for such purposes (e.g., Bowles, Reference Bowles2011; Montrul, Reference Montrul2002, Reference Montrul2004; Montrul & Foote, Reference Montrul and Foote2014; Montrul et al., Reference Montrul, Foote and Perpiñán2008; Montrul & Ionin, Reference Montrul and Ionin2012, Pascual y Cabo & Gómez, Reference Pascual y Cabo and Gómez2015; Torres & Sanz Reference Torres and Sanz2015). The full DELE is “the official accreditation degree of fluency in the Spanish language, issued and recognized by the Ministry of Education, Culture and Sports of Spain” (Montrul, Reference Montrul2016, p. 187). According to Garvida (Reference Garvida2009), the test follows criteria established by the Common European Framework of Reference for Languages, and its results are supported by royal decree. Nevertheless, little empirical information is available regarding validation of the use or interpretation of the DELE or reliability in its scoring. The modified version commonly used for research on both L2 and HL Spanish consists of 50 test items that measure vocabulary (30 items, multiple choice) and grammar (20 items, cloze) knowledge in the written modality and includes a scoring breakdown for grouping learners into proficiency levels: advanced = score of 40–50; intermediate = score of 30–39; low = score of 0–29 (Montrul, Reference Montrul2012).

The validity of using this adapted DELE test as a measure of HL proficiency has been called into question, especially as related to its focus on formal aspects of the language only in the written modality (which may not fully capture HL speakers’ often more developed oral-aural abilities or informal language systems). Carreira and Potowski (Reference Carreira and Potowski2011), for example, urge researchers to include an oral component to HL proficiency assessment for research purposes, especially when researchers plan to compare L2 learners and HL speakers.

Elicited imitation task for proficiency assessment

As previously mentioned, one shortcut method of assessing (oral/aural) language proficiency that has received substantial recent attention is that of elicited imitation. Elicited imitation tasks involve a technique whereby learners listen to and then repeat a (typically sentence-level) stimulus as accurately as they can (Bowden, Reference Bowden2016). The assumption underlying the use of elicited imitation is that if participants can easily repeat the stimuli, they have comprehended, parsed, and acquired the grammatical features displayed in the stimuli; if participants have not yet acquired the features of the stimuli, sentence repetition should be more difficult (Yan et al., Reference Yan, Maeda, Lv and Ginther2016). Elicited imitation has been implemented in first language and L2 research to assess various features of language performance such as listening comprehension (e.g., Jenson & Vinther, Reference Jenson and Vinther2003), implicit knowledge (e.g., Erlam, Reference Erlam2006), syntax (e.g., Li, Reference Li2010), morphology (West, Reference West2012), phonology (e.g., Trofimovich et al., Reference Trofimovich, Lightbown, Halter and Song2009), and lexis (e.g., Verhagen, Reference Verhagen2011). More recently, researchers have revisited the EIT to assess whether it is an appropriate tool for measuring global (oral/aural) proficiency (Yan et al., Reference Yan, Maeda, Lv and Ginther2016). Scholars such as Bley-Vroman and Chaudron (Reference Bley-Vroman, Chaudron, Tarone, Gass and Cohen1994) have argued that participants will use what they have comprehended from a stimulus to reconstruct a conceptual meaning and then establish a linguistic form. This processing—necessary for sentence repetition and, thus, successful completion of the EIT—is expected to become more automatized and, consequently, easier for L2 learners as their proficiency increases. Two recent meta-analyses of the use of EITs to assess L2 proficiency provide additional evidence to support the use of EITs for this purpose. Based on their survey of studies employing EITs to assess global or specific proficiency, Yan et al. (Reference Yan, Maeda, Lv and Ginther2016) reported that, overall, EITs are effective for discriminating between speakers across proficiency levels. Kostromitina and Plonsky’s (Reference Kostromitina and Plonsky2021) conceptual replication and expansion of Yan et al. (Reference Yan, Maeda, Lv and Ginther2016) included 60 independent effect sizes and likewise found EITs to generally correlate highly with other measures of L2 proficiency (aggregated r = .66; r corrected = 0.74).

Concerns have been expressed that learners may simply parrot what they hear in an EIT stimulus (Bley-Vroman & Chaudron, Reference Bley-Vroman, Chaudron, Tarone, Gass and Cohen1994; Vinther, Reference Vinther2002) by relying on their working memory, rather than constructing meaning and form upon hearing the stimulus. A counter to this argument, however, is that EIT stimuli tend to be of a length that exceeds working memory capacity (i.e., five to seven words, or six to eight syllables; Vinther, Reference Vinther2002). Furthermore, several studies have found that EITs tend to correlate most highly with, and thus are best regarded as a measure of, language abilities (such as production performance features like complexity, accuracy, and fluency) as opposed to memory capacity (e.g., Dosi et al., Reference Dosi, Papadopoulou, Tsimpli, Scott and Waughtal2016; Kim et al., Reference Kim, Tracy-Ventura and Jung2016; Park et al., Reference Park, Solon, Henderson and Dehghan-Chaleshtori2020).

With regard to the Spanish EIT, researchers have consistently found that Ortega et al.’s (Reference Ortega, Iwashita, Norris and Rabie2002) version, with some modifications, provides a valid and reliable index of L2 proficiency and that the task can be used to distinguish between learners of varying L2 proficiency levels. Bowden’s (Reference Bowden2016) study of the EIT performance of 37 L2 learners of low (second to fourth semester with no immersion experience), advanced (6.7 semesters of university-level Spanish study on average and 1–2 semesters of immersion experience), and very advanced (a minimum of 19 months immersion experience) L2 experience revealed the Spanish EIT to correlate highly with learners’ Simulated Oral Proficiency Interview (SOPI) scores (Pearson r = .91), to have high internal consistency of items (Cronbach’s α =.98), to have high discriminatory power among learners (discrimination indices above .20 for every item except one), and to have high discriminatory power for learners of low (M discrimination index = .35) and advanced (M discrimination index = .26) L2 experience, though not as high for the very advanced group (M discrimination index = .15). Bowden also found that three proficiency levels or clusters, determined using a k-means cluster analysis of EIT scores, resulted in groups with less overlap in SOPI scores than the a priori experience levels (determined through course level and immersion experience).

Solon et al. (Reference Solon, Park, Henderson and Dehghan-Chaleshtori2019) made a few modifications to the Spanish EIT used by Bowden to make it more dialectally universal and, building upon Bowden (Reference Bowden2016), further examined the Spanish EIT with respect to its relationship with existing proficiency measures, internal reliability, discriminatory power, item difficulty, and its ability to discriminate across and cluster a larger sample of L2 learners (n = 88) with a broader range of L2 experience levels. Additionally, to address the EIT’s potential weakness in distinguishing among the most advanced learners as found by Bowden, Solon et al. added six additional and more difficult items to the end of Bowden’s 30-item test. Similar to Bowden (Reference Bowden2016), results from Solon et al. (Reference Solon, Park, Henderson and Dehghan-Chaleshtori2019) with regard to the 30-item test revealed strong correlations with participants’ DELE scores (r = .92), high internal reliability (α = .99), and overall high item discriminatory power with a mean item discrimination index of .74.Footnote 1 Additionally, not only did the longer, 36-item test exhibit similarly strong correlations with existing proficiency measures and similar internal reliability as the 30-item test, it also exhibited slightly higher discriminatory power among learners at higher experience levels and a greater ability to distinguish among high level learners for clustering purposes. Thus, Solon et al. corroborated previous findings regarding the validity and reliability of using the Spanish EIT for L2 proficiency assessment purposes and suggested that the 36-item version may be a useful tool for researchers who require greater discriminability at higher proficiency levels.

In sum, previous research on elicited imitation suggests that it can provide valid and reliable indices of L2 oral/aural proficiency for a variety of languages and can serve as a practical (inexpensive and quick to administer and score) proficiency assessment tool for research purposes. The Spanish EIT, in particular, has been repeatedly shown to correlate highly with established measures of L2 Spanish proficiency (e.g., SOPI and DELE scores), to be scored with high internal reliability, and to be useful for clustering learners into proficiency levels (e.g., Bowden, Reference Bowden2016; Ortega et al., Reference Ortega, Iwashita, Norris and Rabie2002; Solon et al., Reference Solon, Park, Henderson and Dehghan-Chaleshtori2019). Additionally, recent versions of the Spanish EIT have been modified both to ensure more dialectal universality and to target proficiency assessment within particular groups or levels of learners (i.e., more advanced learners; Solon et al., Reference Solon, Park, Henderson and Dehghan-Chaleshtori2019). Notably, however, existing research on the Spanish EIT has purposefully excluded HL learners from its learner samples; thus, the question remains, is the use of the (Spanish) EIT as a proficiency assessment tool for HL learners also valid?

Elicited imitation for assessing heritage language proficiency

To avoid the valid test fallacy, which refers to a researcher’s tendency to falsely assume that validity is a property of a test instrument and, therefore, a test validated in one measurement context may be suitable for use in another context (Norris & Ortega, Reference Norris, Ortega, Doughty and Long2003), it is necessary to validate the use of the EIT for HL proficiency assessment for research purposes. To our knowledge, only two studies exist that have explored EIT performance of HL learners. As previously mentioned, Son (Reference Son2018) carried out a validation study of a C-test designed to assess Korean HL and L2 proficiency. Unlike the EIT, a C-test is a written measure of language proficiency that requires the test taker to complete an authentic text (or set of texts) with words that best complete the text. Forty-one HL learners and 52 L2 learners of Korean completed the C-test in Son’s study. Learners’ performance on the C-test was compared to four other measures of Korean language proficiency: the ACTFL Oral Proficiency Interview-computer (OPIc), the ACTFL Writing Proficiency Test (WPT), a self-assessment questionnaire, and, notably, Kim et al.’s (Reference Kim, Tracy-Ventura and Jung2016) Korean EIT. Son reported strong, positive correlations between learners’ C-test scores and scores on each of the four measures of Korean proficiency examined. Correlation coefficients for the relationship between C-test and EIT performance were similar for both learner groups (⍴ = .84 for L2 group, ⍴ = .86 for HL group).

Wu and Ortega (Reference Wu and Ortega2013) explored curricular level (high and low) as well as learner background (L2 and HL) on performance on a Chinese EIT.Footnote 2 Eighty learners were sampled from a university Chinese language program—40 from lower-division courses and 40 from upper-division courses. Within each curricular level, half of the participants were L2 Chinese learners and half were HL Chinese learners. The high-level learner group significantly outperformed the low-level group, regardless of learner background, F(1, 76) = 19.53, p < .001. A significant main effect for learner background (L2 vs. HL) on Chinese EIT performance was also found, F(1, 76) = 6.84, p = .011. Specifically, mean EIT scores for HL learners were higher than mean scores for L2 learners, regardless of curricular level (lower vs. upper division). Although specific differences between HL and L2 learner samples were not a focus of their study, Wu and Ortega’s findings, at least preliminarily, suggest that the EIT is able to capture proficiency differences by learner background and that the EIT may be a useful tool for assessing HL proficiency.

Present Study

The present study explores the use of the Spanish EIT for HL proficiency assessment. Additionally, given that the use of the EIT and of this specific Spanish EIT for proficiency assessment of L2 learners has been repeatedly explored, we examine the functioning of EIT items for measuring the same ability across HL and L2 subgroups, using Solon et al.’s (Reference Solon, Park, Henderson and Dehghan-Chaleshtori2019) L2 sample for comparison purposes. Following and building upon Bowden (Reference Bowden2016) and Solon et al. (Reference Solon, Park, Henderson and Dehghan-Chaleshtori2019), the present study explores the following research questions:

  1. 1. To what extent does the Spanish EIT assess a wide range of HL proficiency?

  2. 2. What is the internal reliability of the EIT using a HL sample?

  3. 3. To what extent do the EIT items assess the same ability for HL learners as for L2 learners?

  4. 4. How do Spanish EIT scores relate to a more commonly used measure of HL proficiency (i.e., the DELE) in SLA research?

Methodology

Participants

The HL participants for this study (n = 63) were recruited from university Spanish language classes ranging from beginning Spanish classes for HL speakers to graduate-level courses at three different US universities. For the present study, participants were included in the HL sample if they reported Spanish to be their native language or the native language of at least one of their parents, they reported acquiring Spanish naturalistically in early childhood, and they had spent a substantial part of their childhood in the United States.Footnote 3 For the present sample, all HL participants were exposed to Spanish from birth. The majority of participants (n = 51) were born in the continental United States; of the 12 participants born outside of the continental United States, the mean age of arrival to the continental United States was 5.63 years (range = 1 month–12 years). Participants were of Mexican, Puerto Rican, Dominican, Peruvian, Panamanian, Ecuadorian, Venezuelan, and Guatemalan descent, as well as a mixture therein. Table 1 provides additional information regarding the background characteristics of these participants, taken from information self-reported on the background questionnaire. Mean scores on the DELE (max: 50) and EIT (max: 120) are also included in Table 1; information on how these assessments are scored is provided in following sections.

TABLE 1.

Background characteristics of HL participants organized by participant course level

Tasks and procedures

Participants met individually with a researcher to complete a series of tasks; these included (1) a background questionnaire, (2) a phonological working memory task (Grey et al., Reference Grey, Cox, Serafini and Sanz2015), (3) the Spanish EIT, (4) a monologic oral production task, and (5) the modified DELE (Montrul, Reference Montrul2012). Only information from the background questionnaire, the EIT, and the modified DELE are considered in the present study.

The background questionnaire elicited information regarding participants’ family and linguistic background and their exposure to and experience with English and Spanish as well as other languages. The information summarized in Table 1 was collected using the background questionnaire.

The EIT instrument was adapted from the 30-item EIT developed by Ortega et al. (Reference Ortega, Iwashita, Norris and Rabie2002) and modified by Bowden (Reference Bowden2016). This version, described in detail in Solon et al. (Reference Solon, Park, Henderson and Dehghan-Chaleshtori2019), consists of 30 Spanish sentences that increase in length from 7 to 18 syllables (see Appendix A); these Spanish sentences are preceded by six English practice items. Each sentence is presented aurally, one by one, followed by a 2-second pause and then a 0.5-second tone, which prompts participants to repeat the sentence just heard. Solon et al. (Reference Solon, Park, Henderson and Dehghan-Chaleshtori2019) made two minor lexical/grammatical changes to Bowden’s (Reference Bowden2016) version to make this EIT more dialectally universal. This new version was recorded by a female, native speaker of Mexican Spanish following the procedures outlined in Bowden (Reference Bowden2016) and is freely available on IRIS (www.iris-database.org).Footnote 4

As previously described, the modified DELE exam, developed by Montrul (Reference Montrul2012), is a 50-item test that includes a 30-item multiple-choice vocabulary test followed by a 20-item cloze grammar test. This standardized test has been widely used in linguistics research to determine L2 and, notably, HL proficiency levels (see discussion in Montrul, Reference Montrul2016, and examples of HL research that employ this measure in Montrul & Foote, Reference Montrul and Foote2014; Pascual y Cabo & Gómez, Reference Pascual y Cabo and Gómez2015; Torres & Sanz, Reference Torres and Sanz2015, among others). As evidence of its prominence in research related to Spanish as a HL, the modified test is included as a freely available example of proficiency assessments on the National Heritage Language Resource Center’s website (https://nhlrc.ucla.edu/nhlrc/data/example).

Coding

Each participant received a score on the EIT and on the DELE. The EIT was scored following the rubric developed by Ortega et al. (Reference Ortega, Iwashita, Norris and Rabie2002) and recreated in Table 2.

TABLE 2.

Spanish EIT rating criteria (abbreviated)

Following this rubric, each item received a score of 0–4 based on the participant’s repetition of the sentence. As can be observed in Table 2, scoring is primarily based on the preservation of meaning and, secondarily, considers grammatical content. Given the possibility to score a maximum of 4 on each item, the total possible EIT score a participant could achieve is 120. For the present study, all EITs were coded by the first author, and 30% of EITs were also coded by the fourth author. The interrater reliability coefficient indicated a substantial level of agreement between coders (Cohen’s kappa = .64); percent absolute agreement on item-level coding was 73.39% (see Table B1 in Appendix B). Each item on the DELE is scored as correct or incorrect according to the answers provided by Montrul (Reference Montrul2012). The total possible score on the DELE is 50.

Analysis

Rasch analysis (Rasch, Reference Rasch1960) was performed using Winsteps 4.4.1. There are many benefits to this type of statistical modeling as compared to classical test theory (CTT; Green, Reference Green2013; Knoch & McNamara, Reference Knoch and McNamara2015). Most notably, unlike CTT, Rasch modeling is sample- and test-independent allowing the results of its analysis to be generalizable beyond the sample and test. Because both persons and items are placed on the same logit scale in a person-item (or Wright) map, one can determine how well the test items are distributed across the ability levels of test takers. Comparing a learner’s test performance directly to the difficulty level of an item provides a more accurate picture of the learner’s ability than simply identifying where the learner ranks in a normative sample. Furthermore, Rasch analysis enables assessment of item bias through investigation of differential item functioning (DIF), allowing one to determine whether the item is endorsed similarly across different subgroups of the sample. Rasch is also appropriate to investigate scales with nonnormal distributions as it compares scales across groups regardless of the distribution of the latent trait (i.e., the attribute being measured) among the groups (Drouin et al., Reference Drouin, Horner and Sondergeld2012; Fischer & Molenaar, Reference Fischer and Molenaar1995; Fox & Jones, Reference Fox and Jones1998).

Before proceeding with the main analyses, a set of preliminary analyses were conducted. The basic requirements for Rasch analysis include unidimensionality (i.e., that the test measures a unitary latent construct) and local independence (i.e., that the probability of endorsing one item is not associated with any other in the test). Following the recommendations of Aryadoust et al. (Reference Aryadoust, Ng and Sayama2020), unidimensionality was evaluated using a principle component analysis of residuals with three criteria: (a) the first principle component of residuals (or first contrast) must yield an eigenvalue less than 2.0, (b) disattenuated correlations between person measures must be > .7, and (c) a random pattern should be observed in a standardized residual contrast plot (Linacre, Reference Linacren.d.-a, Reference Linacre2018). The results indicated that the Rasch model explained 88.09% of the variance in the measures (60.23% explained by persons and 27.86% explained by items), while the unexplained variance yielded an eigenvalue of 3.11 for the first contrast and 2.28 for the second contrast. However, the disattenuated correlation for person measures was between .93 and 1.00. These results indicated that there were no meaningful components beyond the primary dimension of measurement, confirming unidimensionality of the EIT. To evaluate local independence, a standardized residual correlation between each pair of item residuals was examined. According to Linacre (Reference Linacren.d.-b), items with a substantially high correlation (> .7) are indicative of local dependency, and the highest correlation observed in the data was .47 (between items 1 and 5). Thus, the principle of local independence of items also held for the EIT.

To investigate our research questions, a Rating Scale Model was applied to the data to examine the psychometric properties of the Spanish EIT. First, the ability of the EIT to measure a wide range of HL proficiency (RQ 1) was assessed by examining item difficulty and person ability through a Wright map and fit statistics. Fit statistics provide information about misfitting items that display patterns of responses that do not follow predictions (e.g., some learners with high ability might have answered an easy item incorrectly although they were predicted to be able to answer it correctly) and thus do not contribute much to the reliability of the test. Item goodness-of-fit was evaluated using infit and outfit mean-square (MNSQ) and standardized Z values: Infit and outfit MNSQ indices are sensitive to inliers and outliers that exhibit unexpected response patterns, respectively. Both the infit and outfit MNSQ indices have an expected value of 1, and values between 0.5 to 1.5 are typically considered acceptable (Wright & Linacre, Reference Wright and Linacre1994). Standardized Z values reflect the probability that the misfit is statistically significant, and the range from –2 to 2 suggests a reasonable fit of the data to the model. If the infit MNSQ is acceptable, the standardized Z can be ignored (Green, Reference Green2013). Because the goal of this validation study was not to modify the test based on the judgments about the suitability of items (this would defeat the purpose of developing parallel EITs in a variety of languages to facilitate comparability across studies and languages), items identified as misfitting were not removed.

To determine the internal reliability of the EIT (RQ 2), two measures of reliability—one for the items and one for the persons—are reported. As in CTT, these reliability indices range from zero to 1.0; the higher the person and item reliability level, the more confidence we can have in the accuracy of the measure. Separation, which refers to the number of statistically distinct levels of item difficulty or person ability in the data, provides another index for reliability. Item separation reliability measures the extent to which the items are separated along the scale, whereas person separation reliability measures how reliably persons can be differentiated on the measured variable. These reliability indices are the Rasch equivalent of the traditional KR-20 or Cronbach alpha indices typically reported in true score theory. There is no ceiling effect for the separation indices, and high separation (>2) indicates that the test is able to separate between items/persons of different levels. Lastly, the overall model fit of the data is assessed using chi-square statistics. The chi-square value reflects invariance across the latent trait and is expected to be nonsignificant (p > .05) to indicate a good fit of the data to the Rasch model.

To examine whether the EIT measures the same ability across subgroups of test takers (RQ 3), DIF analyses were conducted with the HL learners from the current study and L2 learners (N = 88) from Solon et al. (Reference Solon, Park, Henderson and Dehghan-Chaleshtori2019). DIF is a technique to discover unexpected item behavior, which can be indicative of whether responses to items are biased toward one subgroup of test takers. If an item in the test functions differently for a certain subgroup, it means that there may be some other factor besides the test construct that affects one subgroup but not the other. According to Linacre (Reference Linacren.d.-c), DIF contrast values greater than .64 coupled with the rejection of the null hypothesis (p < .05) provide evidence in support of DIF. Because DIF analysis does not explain why an item benefits one group over the other, a qualitative analysis was performed to further examine the items that were found to exhibit DIF.

Finally, to compare the EIT with another commonly used proficiency measure for HL learners (RQ 4), two sets of analyses were conducted: First, we performed a Spearman correlation between the EIT scores and the DELE as correlation calculation is a widely adopted operation in validation research for exploring concurrent validity. High correlation results would indicate that there is a significant overlap between a new test (i.e., the EIT) and an established and recognized test (i.e., DELE). Under true score theory, many EIT validation studies (e.g., Bowden, Reference Bowden2016; Kim et al., Reference Kim, Tracy-Ventura and Jung2016; Solon et al., Reference Solon, Park, Henderson and Dehghan-Chaleshtori2019) as well as meta-analyses on the EIT (Kostromitina & Plonsky, Reference Kostromitina and Plonsky2021; Yan et al., Reference Yan, Maeda, Lv and Ginther2016) have heavily relied on correlational evidence to demonstrate that two instruments are measuring the same underlying construct (in this case, L2 proficiency).

However, Bond and Fox (Reference Bond and Fox2015) point out that a true score correlation calculation does not take into account standard error (SE), and the ensuing r values are attenuated as a result of ignoring this error. Therefore, we additionally conducted common-person equating to assess the comparability of the EIT and the DELE. Common-person equating through Rasch analysis is considered a more rigorous way of assessing concurrent validity (than correlation calculation) as it takes into account person estimates and their error estimates. This was done by calculating person ability estimates for the EIT and DELE scores and plotting them against each other, with a 95% confidence interval (CI) band derived from the person estimate standard errors. The 95% CI band provided a means to evaluate the extent to which the two tests are assessing the same construct within a reasonable degree of measurement error.

Results

Descriptive statistics revealed that, in the present sample, HL learners’ EIT scores ranged from 16 to 120, suggesting that the Spanish EIT was effective in eliciting a wide range of test-taker abilities. Within this range, slight negative skewness (–.82) and kurtosis (–.43) values and the mean score of 84.65 (SD = 28.65) out of 120 indicate that more HL learners scored at the higher end of the range. The Kolmogorov–Smirnov test statistic indicated a significant deviation from normality (D[63] = .15, p = .002); thus, the data were analyzed using nonparametric tests when such tests were needed.

Ability of EIT to assess range of HL proficiency (RQ 1)

Results from Rasch modeling demonstrated generally good fit to the Rasch model, with 83% of the items (k = 25/30) falling within the 0.5 to 1.5 infit and outfit MNSQ range of fit indices. As shown in Table 3, there were five misfitting items that fell outside the acceptable threshold: four items (Items 1, 4, 18, and 30) displayed MNSQ statistics higher than 1.5, indicating that they underfitted the model, or were unpredictable, possibly as a result of carelessness or guessing (Wright & Linacre, Reference Wright and Linacre1994); one item (Item 10) displayed MNSQ statistics below 0.5, suggesting that it overfitted the model, or was too predictable/redundant.

TABLE 3.

Rasch fit statistics for the EIT items

Note. Misfit is indicated with bolding.

Figure 1 plots person ability and item difficulty on a single scale, thereby providing information about how items are distributed in relation to the ability of HL learners and whether there are sufficient items measuring all levels of the construct. Ability estimates of the test takers are on the left side, and item difficulty is on the right side. Items that are at the same logit value as a person have a 50% probability of being answered correctly by that person. The HL learners’ ability covered a range of 10.82 logits, which is far wider than the range of 5.73 logits for item difficulty. This suggests that the range of item difficulty for the EIT did not map adequately with the person ability of the sample. Approximately half of the participants (n = 37) lay above the ability estimates that the items can measure, and the person ability mean (2.05; marked “M” next to the Xs to the left of the vertical center line in Figure 1) was above item mean difficulty (0.00; marked “M” next to the items to the right of the vertical center line), indicating that the test was too easy for many of the HL learners.

Figure 1.

Wright map of person ability and item difficulty (n = 63); M represents the mean, S is 1 standard deviation from the mean, and T is 2 standard deviations from the mean.

Reliability of the EIT (RQ 2)

Results from Rasch modeling revealed that the EIT demonstrated excellent reliability indices. Both the item and person reliability coefficients were 0.97, meaning that only 3% of the variability in item and person measures was attributable to error. This suggests that the results would be fairly consistent over any given number of administrations. Item and person separation indices were 5.54 and 5.37, respectively, demonstrating that the EIT consistently discriminated approximately six statistically distinct strata of performance in persons and five difficulty levels in items. The Rasch analysis further demonstrated a good fit of the data to the model, with the chi-square (X 2) value of 3018.55 at a nonsignificant level (p = .69).

Comparison of item functioning between HL learner and L2 learner samples (RQ 3)

To examine whether the EIT measures the same ability across subgroups of test takers, DIF analyses were conducted with the HL learners from the current study and L2 learners (n = 88) from Solon et al. (Reference Solon, Park, Henderson and Dehghan-Chaleshtori2019). The L2 learners from Solon et al. (Reference Solon, Park, Henderson and Dehghan-Chaleshtori2019) were recruited from two of the three universities that served as recruitment sites for the present study’s HL sample and were divided, in that study, into experience levels based on course matriculation, months of Spanish immersion, and years of formal Spanish study. Table 4 presents an overview of Solon et al.’s L2 learner sample including the mean EIT scores by level. More detailed information about the L2 sample as well as the procedure for determining experience levels and coding EITs is available in Solon et al. (Reference Solon, Park, Henderson and Dehghan-Chaleshtori2019).Footnote 5

TABLE 4.

Overview of L2 sample from Solon et al. (Reference Solon, Park, Henderson and Dehghan-Chaleshtori2019)

a For course level, 1 = 100 level; 2 = 200 level; 3 = 300 level; 4 = 400 level; 5 = graduate level in coursework; 6 = graduate level beyond coursework.

Figure 2 plots the relationship of item difficulty estimates between the present study’s HL learners and Solon et al.’s (Reference Solon, Park, Henderson and Dehghan-Chaleshtori2019) L2 learners. The correlation of item difficulty estimates between the two groups was .79 (p < .001), and all the items except for Item 1 fell within the 95% confidence interval. This suggests that item difficulty differences between HL learners and L2 learners were not substantial enough to significantly change the hierarchies of item difficulty. Additionally, rating scale diagnostics for both learner samples (see Appendix C) suggest that the ratings applied to the two learner groups patterned similarly (with a score of 4—perfect repetition—being the most frequent category chosen by raters for each sample) with threshold and infit statistics supporting the functioning interpretation of EIT scores across both groups.

Figure 2.

Scatterplot of item difficulty estimates for HL learners (y axis) and L2 learners (x axis).

Next, item-level DIF analyses were conducted to evaluate item-level invariance across subgroups. As presented in Table 5, 9 out of 30 items displayed DIF contrast values greater than 0.64 logits: items 3, 12, and 27 favored HL learners and Items 2, 4, 9, 10, 18, and 24 favored L2 learners. To better understand why certain items differed in how they measured person abilities for HL learners as compared to L2 learners, we examined these nine items and learners’ repetitions of them more closely. A qualitative analysis of the L2-favoring items revealed patterns in HL responses consistent with literature on heritage Spanish morphosyntactic patterns that likely resulted in lower scores on these items for HL learners than for L2 learners at the same person ability. For example, item 10 reads Me gustan las películas que acaban bien “I like movies that end well.” Despite this item being among the least difficult for both HL and L2 learners, HL learners even of high abilities exhibited consistent patterns of difficulty, especially as related to the conjugation of the verb gustar “to be pleasing to,” a dative experiencer verb for which verbal agreement is controlled by a postverbal theme (in this case las películas “movies”) rather than a preverbal experiencer (in this case, first person as indicated by the first-person dative clitic me). Heritage Spanish conjugation of gustar invariably in the third-person singular form gusta regardless of subject-verb agreement has been well documented (de Prada Pérez & Pascual y Cabo, Reference de Prada Pérez, Pascual, Cabo, Herschensohn and Tanner2011; Pascual y Cabo, Reference Pascual y Cabo, Amaro, Judy, Pascual and Cabo2013), but here likely resulted in lower scores on this item for HL learners than L2 learners at corresponding person abilities. Items 18 and 24 also favored L2 learners; HL learners’ performance on these items suggested challenges related to the imitation of the present perfect verb forms (i.e., ha terminado “has finished”; ha disminuido “has decreased”). Although scant research exists on use of this specific verb tense in heritage Spanish, results from Cuza et al. (Reference Cuza, Pérez-Tattam, Barajas, Miller, Sadowski and Schwieter2013) suggest that it rarely occurs in the oral production of child and adult heritage Spanish speakers.

TABLE 5.

DIF analysis of EIT items for HL learners and L2 learners (n = 151)

Note. DIF contrast greater than .64 is indicated with bolding.

In contrast, HL learners scored higher than ability-matched L2 learners on Item 3 El carro lo tiene Pedro “Pedro has the car,” where L2 learners showed more consistent and prevalent difficulties than HL learners with object-verb-subject word order and the reduplicative direct object clitic pronoun lo referring to el carro “the car.” Likewise, Item 12 (i.e., Después de cenar me fui a dormir tranquilo “After eating dinner, I went to sleep peacefully”) appeared to present lexical difficulties (e.g., with the words cenar “to eat dinner” and tranquilo “peacefully”) for L2 learners that were less prevalent for the HL learners.

Comparison of EIT to DELE (RQ 4)

Finally, to explore the relationship between HL learners’ EIT performance and performance on another common HL proficiency measure, we conducted correlation analysis and common-person equating with the EIT and DELE scores. The results of Spearman correlation (ρ = .88, p < .001) demonstrated that there is a strong positive relationship between the scores of the two tests. Nevertheless, Figure 3 also shows that several of the highest scorers on the EIT received DELE scores between 30–39, classifying them as intermediate according to Montrul’s (Reference Montrul2012) divisions. Figure 3 includes a scatterplot of each HL learner’s EIT score against their DELE score.

Figure 3.

Scatterplot of EIT and DELE scores by individual HL participant.

Next, person ability estimates (in logits) of the EIT and DELE scores were calculated for common-person equating. The mean person ability estimates of the EIT and DELE were 2.05 logits (SD = 2.27) and .79 logits (SD = 1.26), respectively, demonstrating that the HL learners’ ability estimates of the EIT were higher than those of the DELE (see Appendix D for raw scores of the EIT and DELE along with their ability estimates with errors). The person ability estimates of the two measures were then plotted against each other as shown in Figure 4. DELE scores were adjusted for the mean difference between EIT and DELE scores (1.26) because the DELE was more difficult for this sample than the EIT. When one underlying construct is expected to be the subject of two measures, at least 95% of the person ability estimates should remain invariant (identical allowing for error) across the testing situation, and the regression line should have a slope of 1 and an intercept of zero (Bond & Fox, Reference Bond and Fox2015). However, Figure 4 demonstrates that approximately half of the cross-plotted estimates (n = 35/63, 55%) fell outside the 95% CI and that the slope and intercept of the regression line were 1.30 and .15, respectively, suggesting that the two proficiency measures may not be sufficiently comparable with respect to measurement error. Overall, the common-person equating plot does not support the high concurrent reliability correlation coefficient found between the EIT and DELE scores. The Spearman correlation between the EIT and DELE ability estimates further demonstrated a weak relationship ρ = .27, p = .003, lending support for the result of common-person equating.

Figure 4.

EIT and DELE common-person equating.

Discussion

The present study investigated the utility of the Spanish EIT for HL proficiency assessment. Specifically, it explored the reliability of the 30-item Spanish EIT first developed by Ortega et al. (Reference Ortega, Iwashita, Norris and Rabie2002) and revised by Bowden (Reference Bowden2016) and, most recently, by Solon et al. (Reference Solon, Park, Henderson and Dehghan-Chaleshtori2019) and the validity of its use as a measure of HL proficiency for research purposes. It also compared item functioning for the present HL learner sample with that of the L2 learner sample from Solon et al. (Reference Solon, Park, Henderson and Dehghan-Chaleshtori2019). Results from Rasch modeling indicated excellent reliability (item and person reliability coefficients of 0.97); 83% of the EIT items demonstrated good fit to the model. Nevertheless, a Wright map comparison of person and item ability also revealed that the Spanish EIT was generally too easy for this HL sample, with person ability far exceeding item difficulty. A DIF analysis comparing HL and L2 samples revealed nine differentially functioning items, with more items favoring L2 learners than HL learners (i.e., six as compared to three).

It is important to reiterate that the purpose of the present study was not to modify the existing EIT nor to design or create a new proficiency measure specific to HL learners, but rather to explore the validity of employing an existing and widely used tool for such purposes so as to facilitate valid and reliable proficiency testing across learner samples. The present results indicate that while the existing tool may assess HL proficiency, it also may not be able to do so adequately so as to capture or differentiate between all levels of HL ability. An anonymous reviewer questioned whether differentiating among highly proficient learners is necessary given that the intended and most common use of instruments like the EIT for research purposes is to be able to assess a wide range of proficiency and group learners into proficiency groups (e.g., low, intermediate, and high). Nevertheless, we argue that the present findings add to those of Bowden (Reference Bowden2016) and Solon et al. (Reference Solon, Park, Henderson and Dehghan-Chaleshtori2019) in suggesting that the ceiling effect observed with this EIT may, in fact, result in the grouping of learners (e.g., into one “high” group) with quite varied language proficiencies, thus potentially limiting our understanding of developmental processes especially at later or higher level stages. For example, Bowden’s (Reference Bowden2016) k-means cluster analysis using EIT scores grouped 18 of her 37 learners together into the highest proficiency level, even though these participants varied widely with regard to Spanish language learning experience and SOPI scores. Likewise, Solon et al.’s (Reference Solon, Park, Henderson and Dehghan-Chaleshtori2019) cluster analyses using scores from the 30-item resulted in three clusters, with the highest cluster containing 37 (of 88 total) learners including learners grouped a priori into intermediate, high, and very high levels. The results of the present study suggest similar limitations with assessing and discriminating among higher proficiency learners that could result in learners with quite different proficiencies and abilities being grouped together. For example, among the 37 learners whose ability exceeded that of item ability were learners from all course levels (100 through graduate level) and learners who would be categorized by their DELE scores as belonging to low, intermediate, and advanced proficiency levels (Montrul, Reference Montrul2012). Although course matriculation is likely even less informative regarding HL proficiency than it is for L2 proficiency, we believe these patterns provide additional evidence of the limitations of the EIT with regard to capturing and discriminating among higher level learners. Additionally, while this limitation does not appear to be specific to the HL sample, we would also argue that our understanding of HL acquisition, in particular, may be enhanced with assessment tools that do have the capability to distinguish among learners at higher levels than the present EIT does.Footnote 6

While we believe the present results offer important insight into the utility of the EIT for HL proficiency assessment for research purposes, we must acknowledge limitations of the present analysis. First, as pointed out by a reviewer, the small HL sample size limits the reliability of the DIF analyses presented here. We also acknowledge that the HL sample, like the L2 sample in Solon et al. (Reference Solon, Park, Henderson and Dehghan-Chaleshtori2019), was recruited from a university student population, which likely resulted in a rather homogenous sample with regard to characteristics such as educational attainment as well as perhaps such traits as working memory. We believe that this choice was warranted to facilitate comparison between similar L2 and HL groups. Additionally, the present samples reflect research samples often recruited and compared in research in SLA and for whom researchers in the field generally lack (or at least do not employ; see Park et al., Reference Park, Solon, Dehghan-Chaleshtori and Ghanbar2021) tools to assess proficiency consistently and reliably across groups. Thus, this sampling decision was motivated by a desire to choose a starting point that may be well-recognized by and serve a wide range of researchers working with HL and L2 populations. Nevertheless, this decision limits the generalizability of the findings to other learner, bilingual, and heritage populations. Future research that explores the utility of the EIT for other (sub)populations, such as that conducted by Deygers (Reference Deygers2020) on learners with diverging educational backgrounds, as well as research that explores the role of various potential intervening variables (e.g., working memory), is certainly warranted and will further our understanding of the validity of the interpretation of EIT scores as proficiency measures for a wider range of bilingual populations.

Additionally, the proficiency measure against which we compared EIT performance was the DELE, an instrument comprised of discrete, written items testing vocabulary and grammar knowledge. We readily acknowledge the limitations of this comparison: The DELE provides a proficiency measure tied to literacy; the EIT focus on oracy. However, despite these clear differences, we believed that comparing EIT scores to the DELE was a meaningful place to begin this analysis given the current reliance on the DELE as the primary tool used for Spanish HL proficiency assessment for research purposes. As noted in our “Results” section, although our correlation analysis suggested a strong, positive relationship between DELE and EIT scores, a common-person equating analysis found that 55% of cross-plotted person ability estimates fell outside the 95% confidence intervals, suggesting that the two tests, in fact, measure different underlying constructs. This finding emphasizes the limitation of our comparison between the EIT and the DELE, but it also constitutes an important finding in its own right. It is notable (as can be observed in Table D1) that for 49 of our 63 HL participants, the EIT ability estimates were higher than DELE estimates, lending credence to the idea that proficiency measures that focus on writing and literacy may have a tendency to underestimate HL proficiency or at least miss other components or types of proficiency (e.g., Carreira & Potowski, Reference Carreira and Potowski2011). Future analyses that compare EIT performance with other oral measures of proficiency, including for example, complexity, accuracy, and fluency (i.e., CAF) analyses of the monologic oral production task performed by the present study’s participants, will provide additional evidence regarding the relationship between EIT performance and other measures of HL (oral) proficiency. Likewise, further analyses regarding the reliability and validity of use of existing proficiency assessment tools like the DELE that are commonly used to assess HL proficiency are needed.

Conclusion

This study represents the first substantive analysis of the validity and reliability of the EIT for HL proficiency assessment purposes, although HL proficiency has previously been assessed via EIT (e.g., Son, Reference Son2018; Wu & Ortega, Reference Wu and Ortega2013). The present results provide evidence for the validity of using EIT as a shortcut measure of HL proficiency. Nevertheless, the present results also suggest that the 30-item Spanish EIT may not adequately capture the abilities of HL samples, especially those including learners with higher ability levels, and that certain (and more) items may differentially favor ability-matched L2 learners given the specific grammatical and lexical components of these particular sentences. Thus, the present study suggests both that the EIT can be a useful tool for proficiency assessment across learner samples and that work to develop additional tools that further cover and better discriminate among the wide range of HL abilities may be warranted.

SUPPLEMENTARY MATERIALS

To view supplementary material for this article, please visit http://doi.org/10.1017/S0272263121000905.

Appendix A

Sentences included in EIT with their English translations

  1. 1. Quiero cortarme el pelo. (7 syllables)

“I want to get my hair cut.”

  1. 2. El libro está en la mesa. (7 syllables)

“The book is on the table.”

  1. 3. El carro lo tiene Pedro. (8 syllables)

“Pedro has the car.”

  1. 4. Él se ducha cada mañana. (9 syllables)

“He showers every morning.”

  1. 5. ¿Qué dice usted que va a hacer hoy? (9 syllables)

“What do you say that you are going to do today?”

  1. 6. Dudo que sepa manejar muy bien. (10 syllables)

“I doubt that he knows how to drive very well.”

  1. 7. Las calles de esta ciudad son muy anchas. (11 syllables)

“The streets of this city are very wide.”

  1. 8. Puede que llueva mañana todo el día. (12 syllables)

“It is possible that it will rain tomorrow all day.”

  1. 9. Las casas son muy bonitas pero caras. (12 syllables)

“The houses are very pretty but expensive.”

  1. 10. Me gustan las películas que acaban bien. (12 syllables)

“I like movies that end well.”

  1. 11. El chico con el que yo salgo es español. (13 syllables)

“The boy that I am dating is Spanish.”

  1. 12. Después de cenar me fui a dormir tranquilo. (13 syllables)

“After eating dinner, I went to sleep peacefully.”

  1. 13. Quiero una casa en la que vivan mis animales. (14 syllables)

“I want a house in which my animals can live.”

  1. 14. A ustedes les fascinan las fiestas grandiosas. (14 syllables)

“You [plural] are fascinated by grandiose parties.”

  1. 15. Ella sólo bebe cerveza y no come nada. (15 syllables)

“She only drinks beer and does not eat anything.”

  1. 16. Me gustaría que el precio de las casas bajara. (15 syllables)

“I would like for the price of the houses to come down.”

  1. 17. Cruza a la derecha y después sigue todo derecho. (15 syllables)

“Cross to the right and then continue straight.”

  1. 18. Ella ha terminado de pintar su apartamento. (14 syllables)

“She has finished painting her apartment.”

  1. 19. Me gustaría que empezara a hacer más calor pronto. (15 syllables)

“I would like for it to start being warmer soon.”

  1. 20. El niño al que se le murió el gato está triste. (15 syllables)

“The boy whose cat died is sad.”

  1. 21. Una amiga mía cuida a los niños de mi vecino. (16 syllables)

“A friend of mine takes care of my neighbor’s children.”

  1. 22. El gato que era negro fue perseguido por el perro. (16 syllables)

“The cat that was black was followed by the dog.”

  1. 23. Antes de poder salir él tiene que limpiar su cuarto. (16 syllables)

“Before being able to leave he has to clean his room.”

  1. 24. La cantidad de personas que fuman ha disminuido. (16 syllables)

“The number of people who smoke has decreased.”

  1. 25. Después de llegar a casa del trabajo tomé la cena. (17 syllables)

“After arriving home from work, I ate dinner.”

  1. 26. El ladrón al que atrapó la policía era famoso. (17 syllables)

“The thief that the police caught was famous.”

  1. 27. Le pedí a un amigo que me ayudara con la tarea. (17 syllables)

“I asked a friend to help me with the homework.”

  1. 28. El examen no fue tan difícil como me habían dicho. (17 syllables)

“The exam was not as hard as they had told me.”

  1. 29. ¿Serías tan amable de darme el libro que está en la mesa? (18 syllables)

“Would you be so kind as to give me to book that is on the table?”

  1. 30. Hay mucha gente que no toma nada para el desayuno. (18 syllables)

“There are many people who do not eat anything for breakfast.”

  1. 31. Son ellas las que acaban de decorar la sala de espera. (19 syllables)

“They [fem.] are the ones who just finished decorating the waiting room.”

  1. 32. ¿Sabe usted si el tren de las once y media ya ha salido de la estación? (20 syllables)

“Do you [form.] know if the 11:30 train has already left the station?”

  1. 33. Nunca me divertí tanto como cuando fui a la pista de hielo. (20 syllables)

“I have never had as much fun as when I went to the ice rink.”

  1. 34. Cuanta más prisa tenía en su trabajo, menos calidad producía. (21 syllables)

“The more rushed he was in his work, the lower quality he produced.”

  1. 35. Ellos lo organizaron el año pasado en la universidad cercana. (23 syllables)

“They organized it last year at the university nearby.”

  1. 36. Acabamos de volver del supermercado donde las ofertas eran muy interesantes. (27 syllables)

“We just got back from the supermarket where the offers were very interesting.”

Appendix B

Table B1.

Crosstabulation of interrater agreement

Appendix C

Table C1.

Rating scale diagnostics for present HL sample and Solon et al.’s (Reference Solon, Park, Henderson and Dehghan-Chaleshtori2019) L2 sample

Appendix D

Table D1.

Raw scores and ability estimates with errors on the DELE and EIT

Footnotes

1 Note that Solon et al. (Reference Solon, Park, Henderson and Dehghan-Chaleshtori2019) followed Ebel’s (Reference Ebel1979) guidelines for interpreting item discrimination indices (indices below .19 indicate poor discriminability; indices between .20 and .29 demonstrate marginal discriminability; indices between .30 and .39 show reasonably good discriminability, and levels above .40 are considered very good).

2 Wu and Ortega (Reference Wu and Ortega2013) also examined the relationship between EIT performance and learners’ performance on an oral narrative task evaluated in terms of three complexity, accuracy, and fluency (CAF)-related measures. These results are not discussed in the present review.

3 Numerous definitions of HL learners exist, ranging from those that focus on familial connections (e.g., Fishman, Reference Fishman2001) to those that require some proficiency in the HL (e.g., Valdés, Reference Valdés2001). We do not wish to imply that defining HL learners in the context of the present study or distinguishing them from L2 learners was straightforward. Nevertheless, we attempt to clearly lay out the criteria we used for inclusion/exclusion of participants in the HL group so that readers have an understanding of the characteristics used to define and recruit this sample.

4 Solon et al. (Reference Solon, Park, Henderson and Dehghan-Chaleshtori2019) compared the utility of six additional EIT items for assessing and discriminating advanced language capacities. For this reason, the EIT version available on IRIS includes 36 items. Nevertheless, the first 30 items correspond to the Ortega et al. (Reference Ortega, Iwashita, Norris and Rabie2002) and Bowden (Reference Bowden2016) versions with the lexical changes described.

5 The coding of the L2 EIT data in Solon et al. (Reference Solon, Park, Henderson and Dehghan-Chaleshtori2019) and the present HL EIT data was performed by the same two raters.

6 Given that Solon et al. (Reference Solon, Park, Henderson and Dehghan-Chaleshtori2019) proposed a 36-item EIT with more challenging items as more appropriate for L2 learners with higher abilities, we conducted a post hoc analysis of this 36-item EIT for the present study’s HL sample to better understand the Spanish EITs suitability for assessing HL proficiency. Thus, the present study’s HL learners’ performance on Solon et al.’s (Reference Solon, Park, Henderson and Dehghan-Chaleshtori2019) 36-item Spanish EIT were examined using the same procedure described above for the 30-item EIT. As in Solon et al. (Reference Solon, Park, Henderson and Dehghan-Chaleshtori2019), the participants of the present study completed just one EIT: the 36-item version. However, because this version is the revised 30-item Ortega et al. (Reference Ortega, Iwashita, Norris and Rabie2002) EIT plus an additional six items, it permits the analysis of 30-item and 36-item EIT results for the same sample. The results of these post hoc analyses are presented and discussed in supplementary materials Appendix S1.

References

Aryadoust, V., Ng, L. Y., & Sayama, H. (2020). A comprehensive review of Rasch measurement in language assessment: Recommendations and guidelines for research. Language Testing. Advance online publication. https://doi.org/10.1177/0265532220927487.CrossRefGoogle Scholar
Beaudrie, S. (2012). Introduction: Developments in Spanish heritage language assessment. Heritage Language Journal, 9, i–xi.CrossRefGoogle Scholar
Bley-Vroman, R., & Chaudron, C. (1994). Elicited imitation as a measure of second-language competence. In Tarone, E. E., Gass, S., & Cohen, A. D. (Eds.), Research methodology in second-language acquisition (pp. 245–261). Lawrence Erlbaum.Google Scholar
Bond, T. G., & Fox, C. M. (2015). Applying the Rasch model: Fundamental measurement in the human sciences. Routledge.Google Scholar
Bowden, H. W. (2016). Assessing second-language oral proficiency for research: The Spanish elicited imitation task. Studies in Second Language Acquisition, 38, 647–675.CrossRefGoogle Scholar
Bowles, M. A. (2011). Exploring the role of modality: L2-heritage learner interactions in the Spanish language classroom. Heritage Language Journal, 8, 30–65.CrossRefGoogle Scholar
Carreira, M., & Potowski, K. (2011). Commentary: Pedagogical implications of experimental SNS research. Heritage Language Journal, 8, 134–151.CrossRefGoogle Scholar
Cuza, A., Pérez-Tattam, R., Barajas, E., Miller, L., & Sadowski, C. (2013). The development of tense and aspect morphology in child and adult heritage Spanish: Implications for heritage language pedagogy. In Schwieter, J. (Ed.), Innovative research and practices in second language acquisition and bilingualism (pp. 193–220). Benjamins.CrossRefGoogle Scholar
de Prada Pérez, A., & Pascual, y Cabo, D. (2011). Invariable gusta in the Spanish of heritage speakers in the US. In Herschensohn, J. & Tanner, D. (Eds.), Proceedings of the 11th Generative Approaches to Second Language Acquisition conference (pp. 110–120). Cascadilla Proceedings Project.Google Scholar
Deygers, B. (2020). Elicited imitation: A test for all learners? Examining the EI performance of learners with diverging educational backgrounds. Studies in Second Language Acquisition, 42, 933–957.CrossRefGoogle Scholar
Dosi, I., Papadopoulou, D., & Tsimpli, I. M. (2016). Linguistic and cognitive factors in Elicited Imitation Tasks: A study with mono- and biliterate Greek-Albanian bilingual children. In Scott, J. & Waughtal, D. (Eds.), Proceedings of the 40th annual Boston University Conference on Language Development (pp. 101–115). Cascadilla Press.Google Scholar
Drouin, M., Horner, S. L., & Sondergeld, T. A. (2012). Alphabet knowledge in preschool: A Rasch model analysis. Early Childhood Research Quarterly, 27, 543–554.CrossRefGoogle Scholar
Ebel, R. L. (1979). Essentials of educational measurement (3rd ed.). Prentice-Hall.Google Scholar
Erlam, R. (2006). Elicited imitation as a measure of L2 implicit knowledge: An empirical validation study. Applied Linguistics, 27, 464–491.CrossRefGoogle Scholar
Fairclough, M. (2012). A working model for assessing Spanish heritage language learners’ language proficiency through a placement exam. Heritage Language Journal, 9, 121–138.CrossRefGoogle Scholar
Fischer, G. H., & Molenaar, I. W. (1995). Rasch models: Foundations, recent developments, and applications. Springer Verlag.CrossRefGoogle Scholar
Fishman, J. (2001). 300-plus years of heritage language education in the United States. In J. Kreeft Peyton, D. A. Ranard, & S. McGinnis (Eds.), Heritage languages in America: Preserving a national resource (pp. 81–97). Center for Applied Linguistics and Delta Systems.Google Scholar
Fox, C. M., & Jones, J. A. (1998). Uses of Rasch modeling in counseling psychology research. Journal of Counseling Psychology, 45, 30.CrossRefGoogle Scholar
Gaillard, S., & Tremblay, A. (2016). Linguistic proficiency assessment in second language acquisition research: The elicited imitation task. Language Learning, 66, 419–447.CrossRefGoogle Scholar
Garvida, M. M. (2009). A review of Diplomas de Español como Lengua Extranjera (DELE). International Journal of Pedagogies and Learning, 5, 4–10.CrossRefGoogle Scholar
Graham, C. R., McGhee, J., & Millard, B. (2010). The role of lexical choice in elicited imitation item difficulty. In Prior, M. T., Watanabe, Y., & Lee, S. (Eds.), Selected proceedings of the 2008 Second Language Research Forum (pp. 57–72). Cascadilla Proceedings Project.Google Scholar
Green, R. (2013). Statistical analyses for language testers. Palgrave Macmillan.CrossRefGoogle Scholar
Grey, S., Cox, J. G., Serafini, E. J., & Sanz, C. (2015). The role of individual differences in the study abroad context: Cognitive capacity and language development during short‐term intensive language exposure. The Modern Language Journal, 99, 137–157.CrossRefGoogle Scholar
Hendrickson, R., Aitken, M., McGhee, J., & Johnson, A. (2010). What makes an item difficult? A syntactic, lexical, and morphological study of elicited imitation test items. In Prior, M. T., Watanabe, Y., & Lee, S. (Eds.), Selected Proceedings of the 2008 Second Language Research Forum (pp. 48–56). Cascadilla Proceedings Project.Google Scholar
Hulstijn, J. H. (2010). Measuring second language proficiency. In Unsworth, S. & Blom, E. (Eds.), Experimental methods in language acquisition research (pp. 185–199). Benjamins.CrossRefGoogle Scholar
Hulstijn, J. H. (2012). The construct of language proficiency in the study of bilingualism from a cognitive perspective. Bilingualism: Language and Cognition, 15, 422–433.CrossRefGoogle Scholar
Jenson, E. D., & Vinther, T. (2003). Exact repetition as input enhancement in second language acquisition. Language Learning, 53, 373–428.CrossRefGoogle Scholar
Kim, Y., Tracy-Ventura, N., & Jung, Y. (2016). A measure of proficiency or short-term memory? Validation of an elicited imitation test for SLA research. Modern Language Journal, 100, 655–673.CrossRefGoogle Scholar
Knoch, U., & McNamara, T. (2015). Rasch analysis. In Advancing quantitative methods in second language research (pp. 275–304). Routledge.CrossRefGoogle Scholar
Kostromitina, M., & Plonsky, L. (2021). Elicited imitation tasks as a measure of L2 proficiency: A meta-analysis. Studies in Second Language Acquisition. Advance online publication. https://doi.org/10.1017/S0272263121000395 CrossRefGoogle Scholar
Li, S. (2010). Corrective feedback in perspective: The interface between feedback type, proficiency, the choice of target structure, and learners’ individual differences in working memory and language analytic ability (Unpublished doctoral dissertation). Michigan State University, East Lansing, MI.Google Scholar
Linacre, J. M. (n.d.-a). Dimensionality: Contrasts and variances. https://www.winsteps.com/winman/principalcomponents.htm Google Scholar
Linacre, J. M. (n.d.-b). Table 23.99 Largest residual correlations for items. https://www.winsteps.com/winman/table23_99.htm Google Scholar
Linacre, J. M. (n.d.-c). Table 30.1 Differential item functioning DIF pairwise. https://winsteps.com/winman/table30_1.htm Google Scholar
Linacre, J. M. (2018). Detecting multidimensionality in Rasch data using Winsteps Table 23. https://youtu.be/sna19QemE50 Google Scholar
MacGregor-Mendoza, P. (2012). Spanish as a heritage language assessment: Successes, failures, and lessons learned. Heritage Language Journal, 9, 1–26.CrossRefGoogle Scholar
McManus, K., & Liu, Y. (2020). Using elicited imitation to measure global oral proficiency in SLA research: A close replication study. Language Teaching. Advance online publication. https://doi.org/10.1017/S026144482000021X CrossRefGoogle Scholar
Montrul, S. (2002). Incomplete acquisition and attrition if Spanish tense/aspect distinctions in adult bilinguals. Bilingualism: Language and Cognition, 9, 50–69.Google Scholar
Montrul, S. (2004). Subject and object expression in Spanish heritage speakers: A case of morpho-syntactic convergence. Bilingualism: Language and Cognition, 7, 125–142.CrossRefGoogle Scholar
Montrul, S. (2012). DELE Proficiency Test. Unpublished instrument. http://international.ucla.edu/nhlrc/data/example Google Scholar
Montrul, S. (2016). The acquisition of heritage languages. Cambridge University Press.Google Scholar
Montrul, S. & Foote, R. (2014). Age of acquisition interactions in bilingual lexical access: A study of the weaker language of L2 learners and heritage speakers. International Journal of bilingualism, 18, 274–303.CrossRefGoogle Scholar
Montrul, S., Foote, R., & Perpiñán, S. (2008). Gender agreement in adult second language learners and Spanish heritage speakers: The effect of age and context of acquisition. Language Learning, 58, 503–553.CrossRefGoogle Scholar
Montrul, S., & Ionin, T. (2012). Dominant language transfer in Spanish heritage speakers and L2 learners in the interpretation of definite articles. The Modern Language Journal, 9, 70–94.CrossRefGoogle Scholar
Norris, J. M., & Ortega, L. (2003). Defining and measuring SLA. In Doughty, C. & Long, M. H. (Eds.), The handbook of second language acquisition (pp. 716–761). Blackwell Publishing Ltd.CrossRefGoogle Scholar
Ortega, L. (2020). The study of heritage language development from a bilingualism and social justice perspective. Language Learning, 70, 15–53.CrossRefGoogle Scholar
Ortega, L., Iwashita, N., Norris, J. M., & Rabie, S. (2002, October). An investigation of elicited imitation tasks in crosslinguistic SLA research. Paper presented at the Second Language Research Forum, Toronto, Canada.Google Scholar
Park, H. I., Solon, M., Henderson, C., & Dehghan-Chaleshtori, M. (2020). The roles of working memory and oral language abilities in elicited imitation performance. Modern Language Journal, 104, 133–151.CrossRefGoogle Scholar
Park, H. I., Solon, M., Dehghan-Chaleshtori, M., & Ghanbar, H. (2021). Proficiency reporting practices in research on second language acquisition: Have we made any progress? Language Learning. Advance online publication. https://doi.org/10.1111/lang.12475 CrossRefGoogle Scholar
Pascual y Cabo, D. (2013). Knowledge of gustar-like verbs in Spanish heritage speakers. In Amaro, J. Cabrelli, Judy, T., & Pascual, D. y Cabo, (Eds.), Proceedings of the 12th Generative Approaches to Second Language Acquisition conference (pp. 162–169). Cascadilla Proceedings Project.Google Scholar
Pascual y Cabo, D., & Gómez, I. (2015). Preposition stranding in Spanish as a heritage language. Heritage Language Journal, 12, 186–209.CrossRefGoogle Scholar
Polinsky, M., & Kagan, O. (2007). Heritage languages: In the “wild” and in the classroom. Language and Linguistics Compass, 1, 368–395.CrossRefGoogle Scholar
Rasch, G. (1960). Probabilistic models for some intelligence and attainment tests. The Danish Institute for Educational Research.Google Scholar
Solon, M., Park, H. I., Henderson, C., & Dehghan-Chaleshtori, M. (2019). Revisiting the Spanish elicited imitation task: A tool for assessing advanced language learners? Studies in Second Language Acquisition, 41, 1027–1053.CrossRefGoogle Scholar
Son, Y.-A. (2017). Toward useful assessment and evaluation of heritage language learning. Foreign Language Annals, 50, 367–386.CrossRefGoogle Scholar
Son, Y.-A. (2018). Measuring heritage language learners’ proficiency for research purposes: An argument-based validity study of the Korean C-Test (Unpublished doctoral dissertation). Georgetown University, Washington, DC.Google Scholar
Son, Y.-A. (2021). Testing heritage language learners. In P. Winke & T. Brunfaut (Eds.), The Routledge handbook of second language acquisition and language testing (pp. 422–431). Routledge.Google Scholar
Thomas, M. (1994). Assessment of L2 proficiency in second language acquisition research. Language Learning, 44, 307–336.CrossRefGoogle Scholar
Thomas, M. (2006). Research synthesis and historiography: The case of assessment of second language proficiency. In Norris, J. M. & Ortega, L. (Eds.), Synthesizing research on language learning and teaching (pp. 279–298). Benjamins.CrossRefGoogle Scholar
Torres, J., & Sanz, C. (2015). Is there a cognitive advantage for Spanish heritage bilinguals? A first look. Heritage Language Journal, 12, 292–309.CrossRefGoogle Scholar
Tracy-Ventura, N., McManus, K., Norris, J., & Ortega, L. (2014). “Repeat as much as you can”: Elicited imitation as a measure of oral proficiency in L2 French. In Leclercq, P., Edmonds, A., & Hilton, H. (Eds.), Measuring L2 proficiency: Perspectives from SLA (pp. 143–166). Multilingual Matters.CrossRefGoogle Scholar
Tremblay, A. (2011). Proficiency assessment standards in second language acquisition research: “Clozing” the gap. Studies in Second Language Acquisition, 33, 339–372.CrossRefGoogle Scholar
Trofimovich, P., Lightbown, P. M., Halter, R. H., & Song, H. (2009). Comprehension-based practice: The development of L2 pronunciation in a listening and reading program. Studies in Second Language Acquisition, 31, 609–639.CrossRefGoogle Scholar
Valdés, G. (2001). Heritage language students: Profiles and possibilities. In J. Kreeft Peyton, D. A. Ranard, & S. McGinnis (Eds.), Heritage languages in America: Preserving a national resource (pp. 37–77). Center for Applied Linguistics and Delta Systems.Google Scholar
Verhagen, J. (2011). Verb placement in second language acquisition: Experimental evidence for the different behavior of auxiliary and lexical verbs. Applied Psycholinguistics, 32, 821–858.CrossRefGoogle Scholar
Vinther, T. (2002). Elicited imitation: A brief overview. International Journal of Applied Linguistics, 12, 54–73.CrossRefGoogle Scholar
West, D. E. (2012). Elicited imitation as a measure of morphemic accuracy: Evidence from L2 Spanish. Language and Cognition, 4, 203–222.CrossRefGoogle Scholar
Wright, B. D., & Linacre, J. M. (1994). Reasonable mean-square fit values. Rasch Measurement Transactions, 8, 370.Google Scholar
Wu, S.-L., & Ortega, L. (2013). Measuring global oral proficiency in SLA research: A new elicited imitation test of L2 Chinese. Foreign Language Annals, 46, 680–704.CrossRefGoogle Scholar
Yan, X. Maeda, Y., Lv, J., & Ginther, A. (2016). Elicited imitation as a measure of second language proficiency: A narrative review and meta-analysis. Language Testing, 33, 497–528.CrossRefGoogle Scholar
Zhou, Y., & Wu, S.-L. (2009). Development and pilot of a Mandarin L2 elicited imitation task. Unpublished manuscript, University of Hawai‘i at Mānoa, Honolulu, HI.Google Scholar
Figure 0

TABLE 1. Background characteristics of HL participants organized by participant course level

Figure 1

TABLE 2. Spanish EIT rating criteria (abbreviated)

Figure 2

TABLE 3. Rasch fit statistics for the EIT items

Figure 3

Figure 1. Wright map of person ability and item difficulty (n = 63); M represents the mean, S is 1 standard deviation from the mean, and T is 2 standard deviations from the mean.

Figure 4

TABLE 4. Overview of L2 sample from Solon et al. (2019)

Figure 5

Figure 2. Scatterplot of item difficulty estimates for HL learners (y axis) and L2 learners (x axis).

Figure 6

TABLE 5. DIF analysis of EIT items for HL learners and L2 learners (n = 151)

Figure 7

Figure 3. Scatterplot of EIT and DELE scores by individual HL participant.

Figure 8

Figure 4. EIT and DELE common-person equating.

Figure 9

Table B1. Crosstabulation of interrater agreement

Figure 10

Table C1. Rating scale diagnostics for present HL sample and Solon et al.’s (2019) L2 sample

Figure 11

Table D1. Raw scores and ability estimates with errors on the DELE and EIT

Supplementary material: File

Solon et al. supplementary material

Solon et al. supplementary material

Download Solon et al. supplementary material(File)
File 58.3 KB