1. Introduction
Lexical proficiency is an important component of L2 learning (Alderson Reference Alderson2005; David Reference David2008), reading ability (Laufer Reference Laufer1992; Albrechtsen, Haastrup & Henriksen Reference Albrechtsen, Haastrup and Henriksen2008), academic achievement (Daller, Van Hout & Treffers-Daller Reference Daller, Van Hout and Treffers‐Daller2003), and communicative ability (Wilkins Reference Wilkins1972; Gould, Nation & Read Reference Gould, Nation and Read1990; Manchon, Murphy & de Larios Reference Manchon, Murphy and de Larios2009). Research on lexical development in L2 learners can provide important information about L2 acquisition in general and, more specifically, about the development of L2 lexical networks. Such research can also provide information about lexical proficiency, which remains a difficult construct to define (Henriksen Reference Henriksen1999; Zareva, Schwanenflugel & Nikolova Reference Zareva, Schwanenflugel and Nikolova2005; Boers et al. Reference Boers, Eyckmans, Kappel, Stengers and Demecheleer2006; David Reference David2008).
Previous studies examining L2 lexical proficiency have taken multiple theoretical and empirical approaches. Historically, the two most common lines of inquiry have explored either the distinction between receptive and productive knowledge (Melka Reference Melka, Schmitt and McCarthy1997; Henriksen Reference Henriksen1999; Read Reference Read2000; Schmitt Reference Schmitt2010) or the overall size of a learners’ vocabulary (Nation Reference Nation1990; Meara Reference Meara1992; Laufer & Nation Reference Laufer and Nation1999; Nation & Beglar Reference Nation and Beglar2007). Other approaches to understanding L2 lexical proficiency include assessing lexical performance (T. McNamara Reference McNamara1996; Polio Reference Polio, Silva and Matsuda2001; Bachman Reference Bachman2002), examining the inherent difficulty of lexical items (Laufer Reference Laufer, Huckin and Coady1997; Nation Reference Nation1990), investigating the development of L2 associative and lexical network knowledge (Meara Reference Meara1983; Haastrup & Henriksen Reference Haastrup and Henriksen2000; Fitzpatrick Reference Fitzpatrick2006), measuring the growth of lexical automaticity (Segalowitz, Segalowitz & Wood Reference Segalowitz, Segalowitz and Wood1998; Siyanova & Schmitt Reference Siyanova and Schmitt2008; Hulstijn, Van Gelderen & Schoonen Reference Hulstijn, Van Gelderen and Schoonen2009) and analyzing measures of lexical output in production (Morris & Cobb Reference Morris and Cobb2004; Crossley, Salsbury & McNamara Reference Crossley, Salsbury and McNamara2009, Reference Crossley, Salsbury and McNamara2010a, Reference Crossley, Salsbury and McNamara2010b; Crossley & Salsbury Reference Crossley and Salsbury2011). Research focusing on the vocabulary items produced by L2 learners is guided by the premise that vocabulary use directly taps into learners’ lexical knowledge and is closely aligned with language proficiency and levels of vocabulary acquisition (Daller & Xue Reference Daller, Xue, Daller, Milton and Treffers‐Daller2007; Crossley & Salsbury Reference Crossley and Salsbury2011; Crossley et al. Reference Crossley, Salsbury, McNamara and Jarvis2011; Lu Reference Lu2012; Jarvis Reference Jarvis2013; Saito et al. Reference Saito, Webb, Trofimovich and Isaacs2016).
While a number of studies have investigated lexical proficiency in L2 learners, few of these studies have been replicated. One reason for this may be the tendency for researchers in applied linguistics not to perform replication studies, whether due to the perceived difficulty of replicating quasi-experimental studies, the belief that only true experimental studies can be replicated, a distrust of traditional quantitative research methodology (Porte Reference Porte and Porte2012, Reference Porte2013, Reference Porte, Brown and Coombe2015), or the belief that replication is not as valued as original studies (Porte & Richards Reference Porte and Richards2012). Compounded upon those potential reasons, the investigation of lexical proficiency is still a relatively new research topic which lends itself to multiple perspectives and approaches. Because the possibility to explore new areas of lexical proficiency are ripe, there may exist a perception that there is little need to replicate lexical proficiency studies yet (and also because few studies exist). However, these reasons do not mitigate the need for replication, which is a key element of the scientific process (Porte & Richards Reference Porte and Richards2012; Porte Reference Porte2013, Reference Porte, Brown and Coombe2015). In this study, we provide an approximate or partial replicationFootnote 1 of a study by Salsbury, Crossley & McNamara (Reference Salsbury, Crossley and McNamara2011) that examined the longitudinal development of a number of lexical features in a spoken corpus of English L2 learners. More precisely, Salsbury et al. investigated whether the lexical sophistication of English L2 learners’ productive vocabulary changed over time with regard to word imageability, concreteness, familiarity, and meaningfulness. They found significant changes as a function of time for all lexical properties except for familiarity. The results suggested that, in general, learners begin to produce more sophisticated vocabulary as they increase their English proficiency. Salsbury et al. (Reference Salsbury, Crossley and McNamara2011) were the first to formally identify this relation between English proficiency and lexical sophistication, and a large number of studies investigating lexical proficiency have cited this article as a foundational study (at the time of writing, the study has been cited 52 times).
There were, however, a number of limitations to the Salsbury et al. (Reference Salsbury, Crossley and McNamara2011) study that make it appropriate for replication. First, the sample size was relatively small (six learners sampled six times over the course of a year) in a single context (first language [L1] speakers interviewing L2 speakers). In addition, the study did not control for a number of factors important in L2 acquisition studies (e.g., age, proficiency level, gender) and used a statistical technique that averaged group means and did not properly account for individual participant variation. Thus, as suggested by Mackey (Reference Mackey and Porte2012), the original study is a candidate for replication in that it failed to control for a number of potentially significant variables and focused on a single context.
In the current study, we provide an approximate replication of the Salsbury et al. study using a larger data sample containing a greater number of individual and contextual differences. We do not change the major variables of the original study (i.e., we focus on longitudinal lexical growth in L2 learners), but do introduce a number of appropriate, new variables to the study related to population and setting (Porte Reference Porte and Porte2012). The individual differences include gender, age, language distance, and proficiency. The contextual differences include whether or not the L2 speaker was conversing with another L2 speaker or with an L1 speaker. The goal of the study is to replicate Salsbury et al. (2012) ‘with changes’ to assess whether the results of the original study are generalizable to a new population in a new setting (Porte Reference Porte and Porte2012).
1.1 General background
Traditionally, lexical sophistication has been defined as the proportion of relatively advanced words in a learner's output (Read Reference Read2000). Advanced words have generally been conceptualized as low frequency words (Laufer & Nation Reference Laufer and Nation1995) and there is a rich history of exploring links between the knowledge and use of less frequent words and general language proficiency (Meara Reference Meara, Brown, Malmkjaer and Williams1996; Daller & Xue Reference Daller, Xue, Daller, Milton and Treffers‐Daller2007; Milton Reference Milton2009) as well as acquisition order (Read Reference Read and Kunnan1998; Schmitt, Schmitt & Clapham Reference Schmitt, Schmitt and Clapham2001). In addition, more recent research has begun to examine relationships between lexical sophistication and human judgments of L2 writing, speaking, and lexical proficiency (Crossley et al. Reference Crossley, Salsbury and McNamara2010b, Reference Crossley, Salsbury, McNamara and Jarvis2011; Kyle & Crossley Reference Kyle and Crossley2015, Reference Kyle and Crossley2016). A robust finding from this research has been that L2 learners that use less frequent words or have greater knowledge of less frequent words are generally judged to have greater lexical or language proficiency.
Another approach to defining lexical sophistication can be found in global traits models, which primarily consist of two dimensions: breadth of lexical knowledge (i.e., the size of a vocabulary) and depth of lexical knowledge (i.e., the manner and degree to which known words are organized; Meara Reference Meara, Brown, Malmkjaer and Williams1996, Reference Meara, Butler, Gómez-González and Suárez2005; Read Reference Read and Kunnan1998). These two dimensions are believed to account for the majority of lexical knowledge (Henriksen Reference Henriksen1999; Zareva et al. Reference Zareva, Schwanenflugel and Nikolova2005). A problem with a two-trait system is that not all lexical features can easily be slotted into the breadth and depth categories. For instance, lexical features related to orthography, phonologic principles, and grammatical complexity do not easily fit within the two categorizations (Read Reference Read2000; Nation Reference Nation2001), nor do lexical features related to core lexical items such as word concreteness, imageability, meaningfulness, and familiarity. Features related to core lexical items have a strong history of research in first language (L1) psycholinguistic studies, which show that these features allow for quicker lexical processing and retrieval. However, they have been largely ignored in L2 studies to date with Salsbury et al. (Reference Salsbury, Crossley and McNamara2011) being the most prominent exception.
1.2 The original study
The aim of the Salsbury et al. (Reference Salsbury, Crossley and McNamara2011) study was to examine lexical growth in L2 learners beyond traditional breadth and depth vocabulary measures. As such, Salsbury et al. examined changes in lexical behavior as a function of time in reference to vocabulary features related to core lexical items. The core lexical items they were interested in were concreteness, imageability, familiarity, and meaningfulness. These are discussed briefly below.
1.2.1 Concreteness
Concreteness indexes here-and-now concepts, ideas, and things or, more specifically, how perceptible and tangible the mental representation of a word is (Paivio, Yuille & Madigan Reference Paivio, Yuille and Madigan1968; Toglia & Battig Reference Toglia and Battig1978; Gilhooly & Logie Reference Gilhooly and Logie1980). Concreteness ratings come from human judgments of how abstract a word is based on how simple it is to describe the meaning of that word. If one can describe a word by simply pointing to it, such as the word ‘apple’, a word can be said to be concrete, while if a word can only be explained using other words, it can be considered more abstract (Brysbaert, Warriner & Kuperman Reference Brysbaert, Warriner and Kuperman2014). Concrete words such as apple can be perceived through visual and haptic experiences, while abstract words (e.g., problem) cannot be conceptually understood with direct, sensual experience (Paivio Reference Paivio1990; Brysbaert et al. Reference Brysbaert, Warriner and Kuperman2014).
A number of studies have demonstrated processing advantages for more concrete words. Such processing advantages are likely because concrete words are represented in both verbal and imagery codes in memory (i.e., dual coding; Paivio Reference Paivio1990) and are strongly linked to a narrow range of supporting perceptual contexts available in memory that allow words to be more easily retrieved (i.e., context availability; Schwanenflugel, Harnishfeger & Stowe Reference Schwanenflugel, Harnishfeger and Stowe1988). For instance, studies have shown that word concreteness ratings correlate strongly with free recall, word recognition, short-term memory, associative learning, and associative speed in learning for L1 speakers (Paivio Reference Paivio1990; Gee, Nelson & Krawczyk Reference Gee, Nelson and Krawczyk1999), providing evidence that concrete words are recalled and recognized more quickly as well as learned earlier. For L2 speakers, concrete words are translated and learned more quickly than abstract words (Ellis & Beaton Reference Ellis and Beaton1993; De Groot & Poot Reference De Groot and Poot1997; Kaushanskaya & Rechtzigel Reference Kaushanskaya and Rechtzigel2012). There is likely an additional advantage for concrete word processing in L2 speakers based on the notion that most concrete words will share semantic features across languages and activate richer semantic representations derived from both L1s and L2s as compared to abstract words (Kaushanskaya & Rechtzigel Reference Kaushanskaya and Rechtzigel2012).
1.2.2 Imageability
Word imageability indexes how easy it is to construct a mental image of a word with the understanding that words differ in their capacity to arouse mental images of things or events. Imageability scores are important because a word or concept that triggers a mental image quickly and easily is more likely to be recalled (e.g., Paivio et al. Reference Paivio, Yuille and Madigan1968). For example, highly imageable words have more context availability because they tend to be experienced and analyzed visually, which facilitates lexical processing for L1 speakers (Balota et al. Reference Balota, Cortese, Sergent-Marshall, Spieler and Yap2004) and word learning for L2 learners (Ellis & Beaton Reference Ellis and Beaton1993).
Like concreteness ratings, imageability ratings come from human judgments of word properties. These judgments show that high-imagery words such as cow evokes images easily, while a word such as perdition does not. There are strong overlaps between the constructs of imageability and concreteness with many words scoring highly for both properties. However, there are differences between the two properties in that highly imageable words are not always concrete. For instance, the words accident and marriage are not very concrete, yet they evoke strong sensory experiences or mental pictures for participants such that these words are scored higher on the imagery dimension than on the concreteness dimension.
1.2.3 Familiarity
Familiarity scores are based on human judgments of how familiar words are to adults. Generally, the ratings reflect if a person knows the word and if they are likely to use it. For instance, words such as breakfast, radio, and book all have high familiarity scores, while words such as encephalon and egress have low familiarity scores. In terms of core lexical items, more familiar words are recognized more quickly (Toglia & Battig Reference Toglia and Battig1978).
Researchers have interpreted word familiarity scores as measures of word exposure because the scores reflect word distribution patterns in spoken data (demonstrating a bias toward natural exposure; Stadthagen-Gonzalez & Davis Reference Stadthagen-Gonzalez and Davis2006). Thus, word familiarity is often associated with indices of word frequency (e.g., Schmitt & Meara Reference Schmitt and Meara1997). While word familiarity and word frequency are often presumed to measure the same construct, differences do exist. Specifically, word frequency scores are based on word counts from corpora and are only an approximation of familiarity. This can be seen in words such as mumble, giggle, and drowsy, which are rated as familiar even though they have low frequency counts and occur with similar frequency as unfamiliar words such as cohere, rend, and char (Gernsbacher Reference Gernsbacher1984).
1.2.4 Meaningfulness
Word meaningfulness refers to how likely words are to activate other words. Meaningfulness scores are based on human judgments of how related a word is to other words. Words with high meaningfulness scores (e.g., cup) evoke many word associations (such as coffee, plate, and saucer), while words with low meaningfulness scores (e.g., astuteness) evoke fewer word associations. Word meaningfulness has been shown to be an important predictor of lexical decision times such that words with more associations are processed more quickly (Paivio et al. Reference Paivio, Yuille and Madigan1968; Whaley Reference Whaley1978).
Word meaningfulness scores are similar to free association norms in that both measure the number of similar words associated with another word. Unlike meaningfulness scores, free association norms count the number of distinct first words produced in free association tasks (i.e., tasks in which participants are provided with a stimulus word and asked to produce the first word that comes to mind; Kiss et al. Reference Kiss, Armstrong, Milroy, Piper, Aitken, Bailey and Hamilton-Smith1973; Nelson, McEvoy & Schreiber Reference Nelson, McEvoy and Schreiber1998). L1 lexical research has found that words with many associates (e.g., book) are contextually more diverse than words with few associates (e.g., lone; Nelson et al. Reference Nelson, McEvoy and Schreiber1998), that free association norms play an important role in cued recall (Nelson, Schreiber & McEvoy Reference Nelson, Schreiber and McEvoy1992), and that words with more associates are processed faster than words with fewer associates (Buchanan, Westbury & Burgess Reference Buchanan, Westbury and Burgess2001). In L2 lexical studies, research has reported that L2 learners have more diverse responses to word association tasks than L1 speakers (Meara Reference Meara1978), and that as L2 learners become more proficient, their associative behaviors develop toward native speaker patterns (Zareva & Wolter Reference Zareva and Wolter2012).
1.3 Summary of original study
Salsbury et al. (Reference Salsbury, Crossley and McNamara2011) was the first (and potentially only) longitudinal study to examine lexical growth in terms of features related to core lexical items. To do so, they analyzed the spontaneous speech data of six adult, L2 English learners in a one-year-long study. The L2 speakers were interviewed about every two weeks over the one-year period. The L2 speakers were all enrolled in an intensive English program, and all L2 speakers began the study during the first level of a six-level intensive English program and continued their participation in the study as they progressed through the program. Participants’ language proficiency was tested every two months using an institutional Test of English as a Foreign Language (TOEFL). The spoken data was collected in naturalistic settings in which native speaking interviewers and the L2 learners engaged in naturally occurring discourse. The total corpus comprised 99 transcripts across the six learners. These transcripts were cleaned and then lexical features were derived from the MRC Psycholinguistic Database for each of the lexical features of interest using the computational tool Coh-Metrix (Graesser et al. Reference Graesser, McNamara, Louwerse and Cai2004), which automatically calculated the lexical properties of the words in the corpus. Salsbury et al. then conducted a series of repeated measure Analyses of Variance (ANOVAs) using the TOEFL scores and MRC database results from Coh-Metrix as the dependent variable and time as the independent variable. The ANOVAs were used to examine linear changes in language proficiency and lexical output over time. The results indicated that the L2 learners’ showed linear growth in English proficiency over the course of the year. In addition, the ANOVAs demonstrated that meaningfulness, concreteness, and imageability scores decreased significantly over the course of a year indicating the use of more sophisticated lexical items. No significant linear trends were noted for familiarity scores. The authors concluded that the findings suggested that L2 learners’ spoken vocabulary became less dependent on physical, visual, and semantic contexts over time and that these findings had important implications for word learnability and L2 pedagogical practices.
1.4 Need for replication
The Salsbury et al. (Reference Salsbury, Crossley and McNamara2011) study indicated that lexical properties related to core lexical items may serve as important predictors of L2 development. However, this study did not control for a number of fixed factors (e.g., English proficiency), and examined changes in lexical development using statistical analyses based on averaged group means which do not properly account for individual participant variation. In addition, the sample size was relatively small (six learners). In the current study, we provide an approximate or partial replication of the Salsbury et al. study using a larger corpus with a greater number of individual and contextual differences. The individual differences include gender, age, language distance, and proficiency. The contextual differences include whether or not the L2 speaker was conversing with another L2 speaker or with an L1 speaker. The goal of the study is to replicate Salsbury et al.’s (Reference Salsbury, Crossley and McNamara2011) study ‘with changes’ to assess whether the results of the original study are generalizable to a new population in a new setting (Porte Reference Porte and Porte2012). Thus, the research questions that guide this study are:
1. Do L2 learners produce more sophisticated lexical items in terms of familiarity, concreteness, imageability, and meaningfulness as a function of time spent studying in an ESL environment?
a. This question does not differ from the original study.
2. How do factors such as proficiency level, gender, age, and interlocutor language status affect the development and use of lexical items?
a. This question was not addressed in the original study.
2. Methods
2.1 Data
Salsbury et al.’s (Reference Salsbury, Crossley and McNamara2011) original study examined the naturalistic lexical production of six L2 learners over the course of a year. The data in this replication study consists of transcripts of naturalistic conversations between matriculated and non-matriculated students at a university in the southeastern United States. Matriculated participants were L1 and L2 English speakers enrolled in undergraduate and graduate-level TESOL courses. Non-matriculated participants were L2 English learners attending an intensive English program (IEP) at the same university. In total, 50 matriculated students, 25 of whom were L2 English speakers, and 50 non-matriculated students, all of whom were L2 English speakers, participated. In all cases, matriculated students interacted with non-matriculated students. Conversations between the participants took place every month over a period of four months. Participants conversed with the same partner each time. Conversations lasted an average of 30 minutes. However, only a portion of each conversation was transcribed because of cost. Each transcript was split into two files containing the oral production of the matriculated student and the IEP participant separately.
In this study, we only focus on the 50 non-matriculated students, which is almost a ten-fold increase on that used by Salsbury et al. (Reference Salsbury, Crossley and McNamara2011). Descriptive data for the corpus derived from the non-matriculated students is presented in Table 1. The table includes the number of transcripts per month, number of participants without transcripts per month, and average word counts per month. Thirty-five of the participants were male and 15 were female. Their ages ranged between 18 and 55 years with an average of 26.1 years (SD = 8.26). Their L1 backgrounds included Korean, Mandarin Chinese, Japanese, Mexican Spanish, Arabic, and Uzbek.
Descriptive statistics for corpus
2.2 Individual difference measures
In the original replication, Salsbury et al. (Reference Salsbury, Crossley and McNamara2011) examined linear changes in the six participants’ TOEFL scores over the course of the year. In this replication study, the proficiency for each IEP student was determined based on an ACT ESL Compass test which was administered at the beginning of the semester. The test includes sections related to listening, reading, and grammar. We relied on the test maker's suggested proficiency levels and descriptors in classifying the students. As such, students who scored 126 or below on the ACT Compass ESL reading/grammar tests were classified as high beginning level. Students that scored 127 to 142 were classified as low intermediate level and those that scored 143 to 162 were classified as high intermediate. Students who scored 163 or above were classified as advanced.
Participants also provided their age, sex, and L1. We included age as a factor because multiple studies have demonstrated that younger learners develop proficiency in a L2 faster than older learners (Johnson & Newport Reference Johnson and Newport1989; DeKeyser Reference DeKeyser2000; McDonald Reference McDonald2000; Birdsong & Molis Reference Birdsong and Molis2001). We included gender because many studies have shown that females outperform males on language tasks (Sunderland Reference Sunderland2000; Pavlenko & Piller Reference Pavlenko, Piller and Hornberger2008; Roquet, Llopis & Pérez-Vidal Reference Roquet, Llopis and Pérez-Vidal2016). We also computed language distance scores for each L2 learner based on their L1, following research that demonstrated lower L2 acquisition correlates with larger distances between an L1 and L2 (Van der Slik Reference Van der Slik2010; Schepens, Van der Slik & Van Hout Reference Schepens, Van der Slik and Van Hout2013a, Reference Schepens, Van der Slik, Van Hout, Borin and Saxena2013b). The language distance scores were based on linguistic distance between the learner's L1 and English. The score ranges were from 1 to 3, with a score of one being further from English than a score of three (Miller & Chiswick Reference Miller and Chiswick2005). The score is meant to capture how difficult it is for participants with specific L1s to learn English. These additional demographic and language factors were not analyzed in the original Salsbury et al. (Reference Salsbury, Crossley and McNamara2011) study.
2.3 Lexical features
The original Salsbury et al. (Reference Salsbury, Crossley and McNamara2011) study relied on word concreteness, imageability, familiarity, and meaningfulness scores reported by Coh-Metrix. In the current replication study, we used the Tool for the Automatic Analysis of Lexical Sophistication (TAALES; Kyle & Crossley Reference Kyle and Crossley2015) to compute lexical features related to word familiarity, concreteness, imageability, and meaningfulness derived from the MRC Psycholinguistic Database (Coltheart Reference Coltheart1981). Like Coh-Metrix, TAALES is freely available. However, unlike Coh-Metrix, TAALES is housed on a user's computer and allows for batch processing of texts, enabling easier replication. From TAALES, we used counts for all words in the texts in a manner identical to Salsbury et al. TAALES includes familiarity scores for 4,943 lemmas, concreteness scores for 4,315 lemmas, imageability scores for 4,848 lemmas, and meaningfulness scores for 2,644 lemmas. Salsbury et al. demonstrated that the words reported in the MRC database provide coverage of about 85% of the words found in L2 spoken data.
2.4 Statistical analysis
Salsbury et al. (Reference Salsbury, Crossley and McNamara2011) depended on simple ANOVA analyses which calculate linear trends based on averaged group means, but do not properly account for individual participant variation. ANOVAs also do not allow for the concurrent co-varying of multiple fixed factors that may be associated with language learning such as language proficiency, age, gender, and language distance. To address this, we constructed linear mixed effects (LME) models in R (R Core Team 2016) using the lme4 package (Bates et al. Reference Bates, Mächler, Bolker and Walker2015) to examine linear growth trends. We also used the lmerTest (Kuznetsova, Brockhoff & Christensen Reference Kuznetsova, Brockhoff and Christensen2016) package to derive p values from the models. For each lexical feature of interest (i.e., familiarity, concreteness, imageability, and meaningfulness), we constructed a separate model with that feature as the dependent variable. We then entered participant gender (male or female), age, language proficiency (high beginner, low intermediate, high intermediate, and advanced), interlocutor match (native speaker (NS) with non-native speaker (NNS) or non-native speaker with non-native speaker), session order (range of 1–4), and language distance as fixed effects (i.e., predictor variables). In addition to these fixed effects, we entered participants as random effects with session order entered as a random slope for participants. Random effects and random slopes account for the non-independence of data (i.e., repeated measures) and better capture the individual variation of participants in general, as well as the individual effects session order may have on each participant in specific.
For each model, we tested interactions between session order and proficiency and session order and language distance, only retaining interactions in the models if they were significant. We used the multcomp package (Hothorn, Bretz & Westfall Reference Hothorn, Bretz and Westfall2008) to obtain full comparisons between the four proficiency levels. Additionally, we derived effect sizes from the models using the MuMIn package (Nakagawa & Schielzeth Reference Nakagawa and Schielzeth2013), which provides two measures of variance explained: a marginal R2 measuring the variance explained by the fixed effects only, and a conditional R2 measuring the variance explained by the fixed and random effects combined. We visually inspected residuals of all models to ensure the absence of heteroscedasticity using the plot() and qqPlot() functions. Plots and figures for significant effects were created using the effects package (Fox Reference Fox2003).
3. Results
3.1 MRC concreteness (all words)
An LME model predicting concreteness in all words reported a significant main effect for session order, in that an increase in session order resulted in lower levels of word concreteness (t = −2.918, p = .004). The model also reported a significant main effect for proficiency. Specifically, high beginner proficiency learners had significantly higher levels of concreteness compared to advanced (t = 3.194, p = .001) and high intermediate (t = 2.775, p = .005) learners. No other fixed effects or interactions were significant. This model reported a marginal R2 of .251 and a conditional R2 of .610. Table 2 displays the results for all fixed effects entered into the model, and Figures 1 and 2 display proficiency and time differences.Footnote 2
LME model predicting word concreteness (all words)
Note. Sex baseline = female; Match baseline = NNS=NNS
3.2 MRC imageability (all words)
An LME model predicting imageability in all words reported a significant main effect for session order, in that as session order increased, imageability decreased (t = −3.492, p < .001). The model also reported a significant main effect for proficiency. Specifically, high beginner learners had significantly higher levels of imageability when compared to advanced (t = 3.336, p < .001) and high intermediate (t = 2.979, p = .002) learners. No other fixed effects or interactions were significant. This model reported a marginal R2 of .256 and a conditional R2 of .644. Table 3 displays the results for all fixed effects entered into the model and Figures 3 and 4 visually display proficiency and time differences.
LME model predicting word imageability (all words)
Note. Sex baseline = female; Match baseline = NNS=NNS
3.3 MRC familiarity (all words)
An LME model predicting familiarity for all words reported no significant main effects or interactions for any of the fixed effects. This model reported a marginal R2 of .025 and a conditional R2 of .282. Table 4 displays the coefficients, standard errors, t, and p values for the model and Figures 5 and 6 visually display proficiency and time differences.
LME model predicting word familiarity (all words)
Note. Sex baseline = female; Match baseline = NNS=NNS
3.4 MRC meaningfulness (all words)
An LME model predicting meaningfulness of all words reported a significant main effect for session order, in that as session order increased, word meaningfulness decreased (t = −2.171, p = .031). The model also reported a significant main effect for proficiency. Specifically, advanced learners had significantly lower amounts of word meaningfulness when compared to high beginner (t = −3.863, p < .001) and low intermediate (t = −2.783, p = .005) learners, and high intermediate learners had significantly lower amounts of word meaningfulness when compared to high beginner (t = −2.775, p = .005) learners. This model reported a marginal R2 of .317 and a conditional R2 of .647. Table 5 displays the results for all fixed effects entered into the model, and Figures 7 and 8 visually display proficiency differences.
LME model predicting word meaningfulness (all words)
Note. Sex baseline = female; Match baseline = NNS=NNS
4. Discussion
4.1 Lexical development
Research on L2 lexical development can provide important information about L2 acquisition including the growth of L2 lexical networks. There is a long history of exploring lexical growth in terms of word frequency and lexical diversity (both related to breadth of lexical knowledge) which has provided strong evidence that higher proficiency L2 learners produce less frequent words and a greater variety of words (Meara Reference Meara, Brown, Malmkjaer and Williams1996; Daller & Xue Reference Daller, Xue, Daller, Milton and Treffers‐Daller2007; Milton Reference Milton2009). Research into depth of lexical knowledge has shown that more proficient L2 learners produce lexical features that demonstrate a greater degree and manner of word knowledge (Schmitt Reference Schmitt1998; Crossley et al. Reference Crossley, Salsbury and McNamara2009, Reference Crossley, Salsbury and McNamara2010a). However, not all lexical features fit into a binary two-trait system involving breadth and depth of lexical knowledge. Lexical features related to core lexical items are a good example and ones that have not received deep attention in the literature on L2 lexical development.
The purpose of this study was to provide an approximate or partial replication of Salsbury et al. (Reference Salsbury, Crossley and McNamara2011), who examined the longitudinal growth of lexical features related to core lexical items (i.e., word concreteness, imageability, familiarity, and meaningfulness). In contrast to the Salsbury et al. study, the current study increased the sample size and examined a number of factors important in L2 acquisition studies (e.g., age, proficiency level, gender, language distance). Thus, we introduced a number of variables in order to increase the confirmatory power of the replication (Porte Reference Porte and Porte2012). The additional variables were meant to examine if a replication ‘with changes’ can generalize the original study to a new population and setting. The results of the replication provide support for the findings of Salsbury et al. Like them we found no developmental growth patterns for word familiarity, indicating that L2 speakers used words with similar familiarity ratings at the beginning and at the end of the study. In addition, like Salsbury et al. we found significant growth patterns for word concreteness, imageability, and meaningfulness as a function of time such that L2 learners began to produce words that were less concrete, less imageable, and less meaningful (i.e., they started to produce words that were contained fewer features found in core lexical items).
Importantly, this study also included a number of individual differences variables that have important links with L2 acquisition. The reported models indicated that proficiency was a significant predictor of concreteness, imageability, and meaningfulness, such that lower level learners produced more concrete, imageable, and meaningful words than higher level learners. The results indicated no differences in lexical production as a function of age, gender, or language distance, which is surprising considering the number of studies that have demonstrated language advantages for younger learners (Johnson & Newport Reference Johnson and Newport1989; DeKeyser Reference DeKeyser2000; McDonald Reference McDonald2000; Birdsong & Molis Reference Birdsong and Molis2001), for females (Sunderland Reference Sunderland2000; Pavlenko & Piller Reference Pavlenko, Piller and Hornberger2008; Roquet et al. Reference Roquet, Llopis and Pérez-Vidal2016), and for speakers whose first language is closer to the language being studied (Van der Slik Reference Van der Slik2010; Schepens et al. Reference Schepens, Van der Slik and Van Hout2013a, Reference Schepens, Van der Slik, Van Hout, Borin and Saxena2013b).
These replication findings provide strong evidence that as L2 learners progress, they generally begin to produce more sophisticated words that contain fewer features related to core lexical items. This appears to the case even within a short time period of learning (i.e., the four months for which data was collected in this study). We discuss each of the lexical features assessed below
4.1.1 Concreteness
As in the original Salsbury et al. (Reference Salsbury, Crossley and McNamara2011) study, we found that L2 learners begin to produce less concrete words as a function of time indicating the production of words with fewer features related to core lexical items over time. Thus, L2 learners begin to produce more sophisticated words that are less tangible and perceptible within a four-month period. These results indicate that L2 learners’ lexicons become more complex and allow for the production of words that rely less on verbal and imagery codes (i.e., dual coding; Paivio Reference Paivio1990) as well as depend on less perceptual contexts (i.e., context availability; Schwanenflugel et al. Reference Schwanenflugel, Harnishfeger and Stowe1988). The findings also hint at the growth of lexical networks in that the production of less concrete words indicates longer free recall and word recognition times as well as increased association speeds (Paivio Reference Paivio1990; Gee et al. Reference Gee, Nelson and Krawczyk1999). An important finding from this study is that word concreteness scores also reported a main effect with proficiency such that less advanced learners produced more concrete words when compared to more advanced learners. This finding provides evidence that the lower proficiency learners in this study consistently produced more concrete words than high intermediate and advanced learners.
4.1.2 Imageability
Like the original Salsbury et al. (Reference Salsbury, Crossley and McNamara2011) study, we also found that word imageability decreased as a function of time for the L2 learners in our sample. This finding suggests that learners begin to use words that require more effort to construct mental images (i.e., words that are less likely have features related to core lexical items) because the words have less context availability. Like the concreteness finding, this finding also signals that learners’ lexical networks are developing such that they begin to produce words that are more difficult to recall. In addition, like the concreteness finding, imageability also reported main effects for proficiency such that less advanced learners produced more imageable words when compared to more advanced learners. This finding helps to support the notion that learners use less imageable words as a function of time.
4.1.3 Familiarity
Like the original study, we find no evidence that learners begin to use less familiar words. It may be the case that familiar words are produced at a similar rate regardless of proficiency because of their importance in shaping meaning. However, lexical proficiency studies would seem to contradict this presumption (Crossley et al. Reference Crossley, Salsbury and McNamara2012, Reference Crossley, Salsbury, McNamara and Jarvis2011) in that word familiarity measures show strong correlations with human ratings of lexical proficiency such that speakers with more proficient lexicons use less familiar words (both in speech and in writing). It may also be the case that familiarity is highly conflated with word frequency, which shows contradictory patterns in L2 studies. While some studies have indicated that frequency decreases as a function of proficiency (Laufer & Nation Reference Laufer and Nation1995; Bell Reference Bell2003; Crossley et al. Reference Crossley, Salsbury, McNamara and Jarvis2011, Reference Crossley, Salsbury and McNamara2012), others have indicated that L2 learners use more frequent words as a function of time (Crossley et al. Reference Crossley, Salsbury and McNamara2010a), although this is most likely a result of repetition of infrequent words at the early stages of learning (Crossley et al. Reference Crossley, Salsbury, Titak and McNamara2014). Perhaps, as mentioned by Salsbury et al. greater intervals of time spent learning a language are needed to note changes in word familiarity in longitudinal studies. Alternatively, the production of familiar words may not change as a function of time.
4.1.4 Meaningfulness
In a similar fashion as the original study, this replication found that learners produced less meaningful words as a function of time. Since less meaningful words have fewer associations, the results support the notion that learners begin to use more sophisticated words over time and words that are less likely to contain features related to core lexical items. Like the concreteness and imageability findings, the results support the development of lexical networks in that learners begin to produce words that have longer lexical decision times. Also, like concreteness and imageability, a main effect was reported for proficiency level such that less proficient L2 learners produced more meaningful words when compared to more advanced learners. These findings provide additional support that developing L2 learners begin to produce less meaningful words.
In total, the findings from this study provide a replication of research examining L2 lexical sophistication and L2 lexical development. It is not accurate to characterize advanced lexical proficiency based solely on the number of words a learner knows or how well they know the word. While these two features are important in describing lexical proficiency and have an extensive research history supporting their basic notions, lexical proficiency goes beyond breadth and depth of knowledge. Specifically, any definition of lexical proficiency should also consider the notion of core lexical items. These core lexical items are not defined based on frequency, but rather on word properties that relate to their core lexical properties such as concreteness, imageability, familiarity, and meaningfulness. Such properties explore the visual, haptic, and associative representations contained in words as well as the likelihood of use based on ratings of familiarity. All these properties have important cognitive effects on word learning and are easy to automatically calculate using freely available natural language processing (NLP) tools like TAALES (Kyle & Crossley Reference Kyle and Crossley2015).
The study also provides support for the manner in which L2 lexicons develop over time. While the study is a replication of Salsbury et al. (Reference Salsbury, Crossley and McNamara2011) and the findings mirror the original study, the differences between the two studies provide unique contributions to our understanding of L2 lexical development, and elements of the study can be viewed as a follow-up study in addition to a replication (Porte Reference Porte, Brown and Coombe2015). A key difference between the two studies is that Salsbury et al. tracked the development of six learners over the course of a year-long period while this study examined 50 learners over a four-month period. That the results of the two studies were similar indicates that significant trends in lexical development seem to occur within the first four months of studying English in an English as an L2 academic environment. In addition, this study examined a number of individual difference features along with word properties. In many cases the findings provided greater support for the notion that lexical sophistication increases with language experience in that the more proficient speakers in the sample produced more sophisticated words when compared to less proficient speakers. Thus, the study provided both longitudinal and cross-sectional support highlighting a trend wherein developing learners move away from producing words with features related to core lexical items and toward more sophisticated words. In addition, the analysis factored out gender, age, and language distance and demonstrated that they were not significant predictors of lexical acquisition. This finding counters other research in L2 acquisition that has shown strong effects for age, gender, and language distance in terms of grammar, syntactic, and phonological acquisition, which indicates that these effects may not hold for lexical acquisition. Lastly, the study examined the potential for context differences to influence learning. The results demonstrated that regardless of whether L2 speakers spoke with other L2 speakers or L1 speakers, lexical development still occurred. Of course, much of this development likely happened outside of the interview situation, where learners at this level are exposed to a wide variety of English input sources (e.g., Bell, Skalicky & Salsbury Reference Bell, Skalicky and Salsbury2014), but it is still an important factor to consider.
The findings also have important implications for language teaching. For instance, many language text books rely on frequency lists to develop word lists for pedagogical purposes (Hill & Thomas Reference Hill and Thomas1988a, Reference Hill and Thomas1988b, Reference Hill and Thomas1989; Tickoo Reference Tickoo1989). However, frequency may not be the best predictor of lexical acquisition, and it is surely not the only indicator. A more principled approach to developing word lists would include assessing lexical properties of words that go beyond frequency, such as those related to core lexical items (e.g., concreteness). These properties are strong predictors of lexical development in L2 learners, whereas frequency is often confounded by word repetition (Crossley et al. Reference Crossley, Salsbury, Titak and McNamara2014). Moreover, lexical properties related to frequency, such as familiarity, do not demonstrate strong learning trends, suggesting that frequency may be an imperfect metric of language acquisition. Using more principled selection criteria when developing word lists would likely assist language teachers in better meeting the needs of their developing students, especially at the beginning levels. This study, along with other studies that examine lexical properties beyond frequency, can provide such criteria.
Beyond lexical properties, the findings from this study also provide some evidence that lexical acquisition occurs within a short time period of intensive study, regardless of age, gender, or L1. Furthermore, this growth occurred alongside free-form conversations between learners and L1 speakers, suggesting that acquisition of these lexical properties occurred relatively unconsciously in tandem with intensive study. Thus, language teachers may take solace in the knowledge that repeated, casual conversation sessions in the target language benefit L2 learners in expanded lexical knowledge without any direct pedagogical intervention.
Conclusion
In conclusion, this paper presents an approximate replication of Salsbury et al. (Reference Salsbury, Crossley and McNamara2011) with a number of important changes. First, the sample size present in this paper was larger (almost ten times the original size). Second, a number of individual differences were examined including age, gender, language distance, and proficiency. Third, linear mixed effect models that examine both group and individual variation were used to analyze growth over time. With these different parameters, the results of the replication provide support for the basic notion that developing L2 lexicons move from the production of words with stronger links to core lexical items to words with weaker links to core lexical items over time as reported in Salsbury et al. Thus, the findings provide some measure of external validity to the original study in that these can be extrapolated to a different setting, subjects, and times. To our knowledge, this is one of the first replications to take place in the domain of L2 lexical research and the results are heartening in that the basic premises of lexical development were upheld and the knowledge gained from the original study can be separated from its circumstances (Porte Reference Porte, Brown and Coombe2015).
While the replication was principled and the findings robust, the results still call for additional work to be conducted in lexical development studies. Chiefly, more replication studies are needed on a variety of different studies beyond production studies. Survey-based studies and psycholinguistic studies examining depth of knowledge are likely in need of replication. In addition, the findings reported here should be subjected to behavioral studies that assess links between word naming and lexical decision time for words varied by concreteness, imageability, and meaningfulness. Using L2 learners from different proficiency levels would afford the chance to examine how levels of knowledge interact with response times. Lastly, the findings from both this study and Salsbury et al. (Reference Salsbury, Crossley and McNamara2011) obtained no differences over time in the use of familiar words. Additional studies should examine links between word familiarity and use in a variety of other tasks (such as writing tasks) as well as examine word familiarity and its similarity with word frequency. In combination, these studies would provide a much deeper understanding of L2 lexical proficiency and L2 lexical development.
Supplementary Material: To view supplementary material for this article, please visit https://doi.org/10.1017/S0261444817000362
Dr Scott Crossley is an Associate Professor of Applied Linguistics at Georgia State University. Professor Crossley's primary research focus is on natural language processing and the application of computational tools and machine learning algorithms in language learning, writing, and text comprehensibility. His main interest area is the development and use of natural language processing tools in assessing writing quality and text difficulty. He is also interested in the development of second language learner lexicons and the potential to examine lexical growth and lexical proficiency using computational algorithms.
Stephen Skalicky is a Ph.D. Candidate in the Department of Applied Linguistics at Georgia State University. His research focuses on variables that affect the comprehension and production of figurative language using psycholinguistic, corpus, and natural language processing methods, with a particular emphasis on examining these phenomena among multilingual language users.
