The raw frequency of linguistic items in language input is a powerful predictor of language acquisition (Ellis, Reference Ellis2002). Frequent exposure to linguistic items can help explain syntactic, morphological, phonological, and lexical development in both a first language (L1) and a second language (L2). Approaches concerning the predictive power of frequency effects mainly rely on Zipfian distributions (Zipf, Reference Zipf1935) and implicit learning mechanisms (Ellis, Reference Ellis2002). More recently, these theories have expanded to include construction learning, which includes not only Zipfian distributions but also the importance of form (i.e., item saliency) and function (i.e., prototypicality of meaning and redundancy in cues; Boyd & Goldberg, Reference Boyd and Goldberg2009; Ellis & Collins, Reference Ellis and Collins2009; Ellis & Ferreira-Junior, Reference Ellis and Ferreira-Junior2009; Wulff, Ellis, Römer, Bardovi-Harlig, & LeBlanc, Reference Wulff, Ellis, Römer, Bardovi-Harlig and LeBlanc2009). However, even in construction learning research, frequency effects are generally most predictive of language acquisition.
Recently, the primacy of frequency effects has been called into question in many L1 studies that examine word-naming and lexical decision tasks. Historically, research into such tasks has demonstrated that word frequency is a strong predictor of how quickly a word is (a) read (Balota, Cortese, Sergent-Marshall, Spieler, & Yap, Reference Balota, Cortese, Sergent-Marshall, Spieler and Yap2004; Forster & Chambers, Reference Forster and Chambers1973; Frederiksen & Kroll, Reference Frederiksen and Kroll1976), (b) judged to be a word versus a nonword (Forster, Reference Forster, Wales and Walker1976), and (c) spelled accurately (Barry & Seymour, Reference Barry and Seymour1988). However, recent research indicates that other properties of words such as contextual diversity (i.e., the number of different contexts in which a word appears)Footnote 1 and contextual distinctiveness (i.e., the contextual constraints surrounding a word) are better predictors of lexical decision latencies (Adelman, Brown, & Quesada, Reference Adelman, Brown and Quesada2006; Johns & Jones, Reference Johns, Jones, Sloutsky, Love and McRae2008; McDonald & Shillcock, Reference McDonald and Shillcock2001) and word naming (Adelman et al., Reference Adelman, Brown and Quesada2006; Johns & Jones, Reference Johns, Jones, Sloutsky, Love and McRae2008) than word frequency. This line of research suggests that there are no facilitative effects for word frequency independent of both contextual diversity (Adelman et al., Reference Adelman, Brown and Quesada2006) and contextual distinctiveness (McDonald & Shillcock, Reference McDonald and Shillcock2001) for L1 users.
The purpose of this study is to examine word frequency, contextual diversity, and contextual distinctiveness effects in explaining L2 lexical production. To investigate these lexical features, we conduct a corpus analysis of L1 and L2 spoken discourse to examine lexical differences between frequent words (both nouns and verbs) produced by beginning-level L2 learners and frequent words produced by L1 speakers but not produced by beginning-level L2 learners. We derive frequency and contextual diversity values from the SUBTLEXus (Brysbaert & New, Reference Brysbaert and New2009) and British National Corpora (BNC; 2001). We derive contextual distinctiveness values from the University of South Florida word association norms (USF; Nelson, McEvoy, & Schreiber, Reference Nelson, McEvoy and Schreiber1998) and from latent semantic analysis (LSA; Landauer, Foltz, & Laham, Reference Landauer, Foltz and Laham1998). We use these values to classify whether a word is produced or not produced by beginning-level L2 learners. Such an approach was used by Crossley and Salsbury (Reference Crossley and Salsbury2010) and affords an examination of word properties that are important in early lexical production and, by extension, early lexical acquisition. In selecting word properties related to lexical frequency, contextual diversity, and contextual distinctiveness, we are able to address the following research questions:
1. Is word frequency—independent of contextual diversity and distinctiveness—the strongest predictor of early L2 word production?
2. Do differences exist in the lexical properties of nouns and verbs that may explain early word production?
BACKGROUND
Word Frequency Effects
Every level of language (i.e., phonology, morphology, lexicon, syntax, and discourse) is characterized by disproportionate frequencies of individual items within its inventory of features. These frequencies generally follow a Zipfian distribution (Zipf, Reference Zipf1935) in which the most frequently occurring types account for a majority of tokens. Theoretical accounts of these Zipfian distributions hold that they aid in the process of language learning due to the increased probability of form-function mapping for more frequent tokens (Ellis, Reference Ellis2006; MacWhinney, Reference MacWhinney, de Groot and Kroll1997). These more frequent tokens may then serve as exemplars for the acquisition of other less frequent tokens (Boyd & Goldberg, Reference Boyd and Goldberg2009; Ellis & Collins, Reference Ellis and Collins2009). For example, Ellis and Ferreira-Junior (Reference Ellis and Ferreira-Junior2009) examined L2 learner production of verb-argument constructions. They found that learners first produced the most frequent, prototypical exemplar types from the inventories of verb-argument constructions (e.g., transitive or ditransitive constructions) in English from the input to which they were exposed (e.g., they first produced give for ditransitive constructions and put in for verb-locative constructions). In the same vein, Wulff and colleagues (2009) explored the effects of frequency on the acquisition of tense and aspect in English. They found that when the verbs that occurred most frequently in a specific tense-aspect occurred consistently in that tense-aspect—and not another—then there was a facilitative learning effect based on frequency because these items could serve as exemplars. The authors point to progressive aspect as an example because, unlike other tenses or aspects (e.g., past tense), the most frequently occurring verbs in progressive aspect were consistently found in the progressive.
Frequency effects are also thought to be important in L2 lexical acquisition (Ellis, Reference Ellis2002). Laufer and Nation (Reference Laufer and Nation1995) found that the production of frequent words in L2 writing decreased as a function of time spent learning English, although studies examining spoken language have not supported this (Crossley, Salsbury, & McNamara, Reference Crossley, Salsbury and McNamara2010). Corresponding evidence for the strength of word frequency indices to explain L2 lexical acquisition come from a series of studies conducted by Crossley and colleagues, which have demonstrated that word frequency is an important predictor of L2 lexical proficiency (Crossley, Salsbury, McNamara, & Jarvis, Reference Crossley, Salsbury, McNamara and Jarvis2011a, Reference Crossley, Salsbury, McNamara and Jarvis2011b) and that word frequency is a significant classifier of proficiency level for L2 writers (Crossley, Salsbury, & McNamara, Reference Crossley, Salsbury and McNamara2012). In all cases, more advanced L2 learners produced less frequent words than beginning-level L2 learners. Other studies focusing on L1 processing have demonstrated the strength of word frequency in predicting linguistic performance. For example, studies have supported the notion that high-frequency words are recognized (Kirsner, Reference Kirsner and Ellis1994) and named (Balota et al., Reference Balota, Cortese, Sergent-Marshall, Spieler and Yap2004; Forster & Chambers, Reference Forster and Chambers1973; Frederiksen & Kroll, Reference Frederiksen and Kroll1976) more rapidly.
In theory, frequency effects can help explain language acquisition because the faster processing of high-frequency items testifies to associative learning from usage. With language experience, a learner’s perceptual system becomes tuned to expect linguistic items according to their probability of occurrence in the input (Ellis, Reference Ellis2006). Thus, language learners acquire language patterns on the basis of association through exposure. This exposure follows Zipf’s law in that the frequency of any word in the corpus will be inversely proportional to its rank in a frequency table. This law indicates that, although learners may be exposed to thousands of different words, the words they are regularly exposed to will be a much smaller set of highly frequent words, which signifies a decrease in input variability. Because learners hear and read the same sets of words consistently, these words are learned and produced more quickly (Mintz, Newport, & Bever, Reference Mintz, Newport and Bever2002).
Contextual Diversity
Some L1 researchers have begun to question the superiority of frequency effects over other conceptually and statistically related measures in explaining lexical processing (Adelman et al., Reference Adelman, Brown and Quesada2006; Brysbaert & New, Reference Brysbaert and New2009; Johns & Jones, Reference Johns, Jones, Sloutsky, Love and McRae2008). These researchers focus instead on contextual diversity indices (i.e., the number of unique contexts in which a word occurs). Whereas contextual diversity and word frequency are highly correlated, memory research has demonstrated that it is not the number of repeated exposures to an item that affects retrieval but rather the separation of these exposures in time and context (Glenberg, Reference Glenberg1976, Reference Glenberg1979). Specifically, if the time and context of exposures do not change substantially, there may be no benefit to repetition on subsequent retrieval (Verkoeijen, Rikers, & Schmidt, Reference Verkoeijen, Rikers and Schmidt2004) because the more contexts in which an item occurs, the more likely it is that the item will be needed in a new context; that is, the principle of likely need will increase (Anderson & Milson, Reference Anderson and Milson1989; Anderson & Schooler, Reference Anderson and Schooler1991). Therefore, word frequency may be a poor indicator of need because words may cluster within a particular context and may not be shared among contexts. Adelman et al. (Reference Adelman, Brown and Quesada2006) argue that if item retrieval and lexical retrieval operate similarly, then the effect of word frequency will be reduced if repeated exposures to a word happen in a similar environment. Thus, word retrieval—and possibly word acquisition—may be less related to the number of exposures to a word that a learner receives and more related to the number of different contexts in which a word is experienced (i.e., a word’s contextual diversity).
To test this theory, Adelman et al. (Reference Adelman, Brown and Quesada2006) examined response times in six different word-naming and lexical decision tasks using indices of word frequency and contextual diversity. To obtain their contextual diversity index, Adelman and colleagues counted the number of documents in a corpus that contained each word in their analysis under the assumption that a greater number of unique contexts would directly relate to a word’s contextual diversity. Contextual diversity was found to predict word processing times independently of word frequency. Additionally, there was no evidence of a facilitative effect for word frequency independent of contextual diversity. That is, when both contextual diversity and word frequency indices were used to measure the same data simultaneously, contextual diversity was a significant predictor of the variance in the data (for both word-naming and lexical decision tasks), but word frequency indices were not. These results suggest that contextual diversity indices are better predictors of lexical decision and word-naming tasks and that word frequency indices do not contribute to explaining human responses to these tasks when investigated concurrently with contextual diversity indices. Similar results have been found in other studies comparing the strengths of word frequency and contextual diversity indices in explaining human word response times (Johns & Jones, Reference Johns, Jones, Sloutsky, Love and McRae2008; Steyvers & Malmberg, Reference Steyvers and Malmberg2003).
Contextual Distinctiveness
Another approach to examining how context affects the principle of likely need is to examine a word’s contextual distinctiveness. The underlying assumption of contextual distinctiveness is that the immediate context of a word has priority in explaining language processing effects (McDonald & Shillcock, Reference McDonald and Shillcock2001). Like indices of contextual diversity, indices of contextual distinctiveness can be derived from language corpora. However, they are operationalized as co-occurrence counts of words that are located in the immediate environment. According to this view, it is thus not the number of different texts that a word occurs in (i.e., distant context) but rather the number of words that strongly associate with a word (i.e., local context) that best predicts language processing. Context, in this sense, involves semantic representations of a word that are influenced by the immediate environment in which the word is encountered. Co-occurring words are argued to be the best indicators of this environment (McDonald & Shillcock, Reference McDonald and Shillcock2001). For instance, the word amok is extremely constrained by its immediate environment (usually only occurring in the bigram run amok). However, run has many fewer constraints than amok and occurs with a variety of different words (McDonald & Shillcock, Reference McDonald and Shillcock2001). Consequently, amok has high contextual distinctiveness, whereas run has low contextual distinctiveness.
Although it has not received the same amount of attention as contextual diversity, the effects of contextual distinctiveness have also been compared to word frequency effects in explaining lexical decision latencies. McDonald and Shillcock (Reference McDonald and Shillcock2001) implemented indices of contextual distinctiveness by developing co-occurrence vectors between words in a lemmatized version of the BNC (2001) spoken subcorpus. In two different experiments that assessed lexical decision response time latencies, the index of contextual distinctiveness was a stronger predictor of human responses than the index of word frequency. These findings indicate that words that appear in relatively constrained linguistic contexts (i.e., have high distinctiveness) have longer lexical decision latencies, whereas words in unconstrained contexts (i.e., words that have low distinctiveness) have shorter response times. McDonald and Shillcock concluded that, although word frequency and contextual distinctiveness indices are highly correlated, word frequency likely approximates contextual distinctiveness. As such, contextual distinctiveness is not only a more powerful predictor of lexical response latencies but also a more theoretically compelling variable than word frequency because it is based on a word’s lexical environment.
METHOD
The purpose of this study is to assess the predictive ability of computational indices related to word frequency, contextual diversity, and contextual distinctiveness to categorize nouns and verbs as either being produced or not produced by beginning-level L2 learners. Such an approach enables us to examine the effects of these lexical properties in explaining lexical production, which is unlike most previous studies that have examined word frequency, contextual diversity, and contextual distinctiveness in relation to word-naming and lexical decision tasks. Additionally, our focus is on beginning-level L2 learners and not fluent L1 speakers. Thus, we are able to examine the effects of these lexical properties on lexical production as compared to lexical processing.
We associate lexical production with lexical acquisition under the hypothesis that productive vocabulary taps into word knowledge at the processing, storage, and, most importantly, word retrieval level. For our study, the word frequency and contextual diversity indices are collected from the SUBTLEXus (Brysbaert & New, Reference Brysbaert and New2009) and the BNC (2001). Our contextual distinctiveness indices are taken from the USF word association norms (Nelson et al., Reference Nelson, McEvoy and Schreiber1998) and from LSA (Landauer et al., Reference Landauer, Foltz and Laham1998) near-neighbor counts. Our goal is to better understand the role of word frequency, contextual diversity, and contextual distinctiveness in explaining lexical production by beginning L2 learners.
Word Lists
We used the lists of produced and unproduced words reported in Crossley and Salsbury (Reference Crossley and Salsbury2010). The lists comprise (a) frequent words produced by beginning L2 learners and (b) words that are not produced by L2 learners, but that are frequently produced by native speakers of English. The basic assumption behind these word lists was that words produced by beginning-level L2 learners are easier to acquire and that words not produced by L2 learners but commonly produced by L1 speakers are more difficult to acquire.
The L2-produced word list was developed using a language learner corpus. The corpus was collected during a yearlong longitudinal study of L2 learners as reported by Salsbury (Reference Salsbury2000). The corpus has been used in a variety of lexical development studies (Crossley, Salsbury, & McNamara, Reference Crossley, Salsbury and McNamara2009, Reference Crossley and Salsbury2010; Salsbury, Crossley, & McNamara, Reference Salsbury, Crossley and McNamara2011) and is characterized by naturalistic speech (i.e., spoken, unprepared speech). The corpus consists of oral data collected in bimonthly (approximately every two weeks) elicitation sessions with L2 English learners enrolled in an intensive English program at a large American university. All learners were tested on arrival to the program and were placed into the lowest proficiency level of the program. The learners were also assessed through the institutional TOEFL examination. The portion of the corpus used by Crossley and Salsbury (Reference Crossley and Salsbury2010) focused on the six learners who completed the yearlong study, focusing specifically on the first 3 months of the study (i.e., when standardized tests indicated the learners were at the beginning level). The learners came from various L1 backgrounds: Spanish (n = 1), Japanese (n = 1), Korean (n = 1), and Arabic (n = 3).
The average number of word tokens per elicitation session ranged from 1,100 to 2,100, which varied by learner and by session. In the first 3 months of the study (from which data for the present article were collected), learners were provided with a total of 50 unique topics about which to talk. The topics were prepared in advance and included emotion cards (e.g., happy, sad, lonely), topic cards (e.g., wishes, travel, movies), photograph descriptions (from Life: Faces), questions related to attitudes toward learning English, and questions about imaginary scenarios (e.g., Imagine you have a wish for any talent or skill. What is that talent or skill?). Learners selected the topics they wanted to discuss but typically chose all of the sessions’ prepared topics and added in their own topics. Elicitation sessions lasted 30‒45 min, with an average number of learner conversational turns ranging from 50 to 70. The interviewers (all native English-speaking university students) were instructed to ask follow-up questions and comment on the learners’ content. Thus, the elicitation sessions were dominated by learner talk. Additionally, the elicitation sessions were informal, and participants were on a first-name basis.
To select the words for the L2 unproduced word list, Crossley and Salsbury (Reference Crossley and Salsbury2010) used the SUBTLEXus corpus (Brysbaert & New, Reference Brysbaert and New2009). The corpus consists of 51 million words from subtitles of American television shows and films. The SUBTLEXus corpus was selected because it focuses on everyday interactions and contains oral language similar in register to the interactions recorded in the L2 corpus.
The word lists did not contain all produced and unproduced words found in the corpora but, instead, included only words that met three criteria. The first criterion was word frequency: Only those words that were within the first 3,000 most common words in the English language (as reported by SUBTLEXus) were included. This criterion was selected to ensure that the participants in the study would have most likely been exposed to the words in either instructional or naturalistic settings. The second criterion was that the words in the L2-produced list had to be accurately used by at least half of the L2 participants. The third criterion was that the words had to fit clearly into a noun or verb categorization. If questions arose, Crossley and Salsbury (Reference Crossley and Salsbury2010) analyzed word use in context to ensure its part-of-speech category. The 45 most frequent verbs and nouns from each group that fit these criteria were then selected, which produced a word list of 90 nouns (i.e., 45 produced and 45 unproduced) and 90 verbs (i.e., 45 produced and 45 unproduced). Close examination revealed a repetition in the use of the word steal in the unproduced verbs. This extra instance of the word steal was removed, which left 89 verbs (45 produced and 44 unproduced).
Selected Indices
We report on a variety of indices related to word frequency, contextual diversity, and contextual distinctiveness, which will be discussed in more detail in the following section. The selected nouns and verbs along with their respective word frequency, contextual diversity, and contextual distinctiveness values are presented in Appendices A–D.
Contextual Diversity Indices
SUBTLEXus contextual diversity
We selected a contextual diversity index from the SUBTLEXus corpus (Brysbaert & New, Reference Brysbaert and New2009). The foundations of the SUBTLEXus corpus are subtitles taken from 2,046 U.S. films from the years 1900–1990, 3,218 U.S. films from the years 1990–2007, and 4,575 U.S. television series. The total size of the corpus is 51 million words: 16.1 million words from television shows, 14.3 million words from films before 1990, and 20.6 million words from films after 1990. The SUBTLEXus database derives a contextual diversity score for each word in the corpus by calculating the number of different television shows and films in which the words appear. We predict that words that have greater SUBTLEXus contextual diversity values will be learned and produced earlier.
British National Corpus contextual diversity
The BNC consists of 100 million words of British English collected from the latter part of the 20th century. The BNC contains roughly 90% written texts and 10% spoken texts. We divided the BNC into its spoken and written components and then lemmatized the samples by replacing each inflected token with its canonical form to ensure that all root forms of the words in the word lists were captured. We then conducted a text count (i.e., the number of text samples in which a word occurs) using the corpus linguistics tool WordSmith (Scott, Reference Scott2004). We used the percentage of text samples in which a word occurs in the BNC as our BNC contextual diversity measure. From this measure, we computed three indices of contextual diversity: (a) for the entire BNC, (b) for the written subsection, and (c) for the spoken subsection. We investigated both written and spoken texts to assess potential differences in L2 input. We predict that the words that have greater BNC contextual diversity values will be acquired and produced earlier than words that do not.
Contextual Distinctiveness Indices
University of South Florida word association norms
Our first index of contextual distinctiveness was derived from the word association scores reported in the USF word association norms (Nelson et al., Reference Nelson, McEvoy and Schreiber1998). The USF database contains nearly 750,000 association responses for more than 5,000 words generated by 6,000 participants. In the database, word associations were collected by giving native English-speaking participants a prime word—that is, a word meant to stimulate a response—and by asking them to write the first word that came to their mind that was meaningfully related to that prime word. For our index of contextual distinctiveness, we calculated the average number of associations for each word in our word lists. We used these associations to identify the distributional information contained in the prime word under the hypothesis that the associations reflect the environment in which the word is encountered. We predict that words with many associations will be less semantically distinct (i.e., the words will occur in a greater number of local contexts) and thus lead to quicker acquisition and production.
Latent semantic analysis
Latent semantic analysis measures the semantic associations between words through a mathematical technique known as singular value decomposition (SVD), which reduces thousands of dimensions and relationships between words in large corpora to a smaller number. This procedure promotes quick calculations of associations between words and the context in which they occur (Landauer et al., Reference Landauer, Foltz and Laham1998). These associations represent how often a word or words occur within documents. To compare words, LSA computes the cosine between the two sets of vectors reported by the SVD for each word. These vectors report values between −1 (no similarity) and 1 (high similarity). Thus, two words will acquire semantic similarity to each other—but not to the other words—if (a) the words appear in the same context and (b) other words in that context appear in many other contexts without them (Landauer & Dumais, Reference Landauer and Dumais1997). As an example, the LSA cosine between dog and cat is .36. The LSA cosine between dog and fur is .24. The LSA cosine between dog and table is .04.
For our LSA index of contextual distinctiveness, we computed the number of associations reported for each word in our word lists that occurred at least 20 times in the Touchstone Applied Science Association (TASA) corpusFootnote 2 and had a LSA value greater than .26 using logarithmic entropy values.Footnote 3 We predict that words with more word associations will be learned and produced earlier by L2 learners.
Word Frequency Indices
SUBTLEXus word frequency
Frequency values were taken from the SUBTLEXus corpus (Brysbaert & New, Reference Brysbaert and New2009). As previously discussed, the SUBTLEXus corpus provides not only a contextual diversity count but also a frequency count for the number of times a word appears in the 51-million-word corpus (i.e., frequency per million words on the basis of word form). The word frequency counts taken from SUBTLEXus should better reflect the frequency of words in spoken language than more commonly available corpora of written language. We predict that L2 learners will acquire and produce words occurring more frequently in SUBTLEXus first.
British National Corpus word frequency
We derived word frequency values for each of the words in the word lists using frequency counts (i.e., frequency per thousand words on the basis of word forms) calculated from the spoken and written BNC subcorpora and from the entire BNC corpus. We used WordSmith (Scott, Reference Scott2004) to calculate the frequency values using a lemmatized version of the BNC. Like the SUBTLEXus word frequency values, we predict that L2 learners will acquire and produce words that occur more frequently in the BNC corpus and subcorpora first.
Word familiarity
Word familiarity refers to how commonly a word is experienced in natural discourse (Wilson, Reference Wilson1988). We collected word familiarity values from the Medical Research Council (MRC) database, which reports familiarity scores for 9,392 words taken from Toglia and Battig (Reference Toglia and Battig1978) and Gilhooly and Logie (Reference Gilhooly and Logie1980). Participants in these studies were instructed to score words on an interval scale from 1 to 7 on the basis of how familiar they were with the given word; higher scores indicated greater familiarity. For example, the word adze received a low mean familiarity score of only 2.12, whereas the word eat had a mean score of 6.71. Researchers have interpreted word familiarity scores as measures of word exposure because the scores reflect word distribution patterns in spoken data (demonstrating a bias toward natural exposure; Stadthagen-Gonzalez & Davis, Reference Stadthagen-Gonzalez and Davis2006). Thus, word familiarity is often associated with indices of word frequency (e.g., Schmitt & Meara, Reference Schmitt and Meara1997), and some research indicates that frequency effects are actually the result of spurious correlations between frequency and familiarity (Gernsbacher, Reference Gernsbacher1984). We predict that more familiar words should be learned and produced first by L2 learners.
Statistical Analysis
For each word list (nouns or verbs), we first conducted a MANOVA to examine if the contextual diversity, contextual distinctiveness, and word frequency indices demonstrated significant differences between the produced and unproduced words. We next conducted a stepwise discriminant function analysis (DFA) using selected indices for each word list that demonstrated significant differences between the produced and unproduced words but did not exhibit multicollinearity with other indices in the set.Footnote 4 The DFA generates a discriminant function, which provides an algorithm to predict group membership (i.e., whether or not the words were produced).
We first used the DFA on the entire word list and then used the DFA model reported for the entire set to predict group membership of the words in the word list using leave-one-out cross-validation (LOOCV). For LOOCV, we selected a fixed number of folds that equaled the number of observations (i.e., words). In the cross-validation, one observation is left out, and the remaining instances are used as the training set (in this case either the 89 remaining words in the noun word list or the 88 remaining words in the verb word list). We then tested the accuracy of the model by examining the model’s ability to predict the proficiency classification of the omitted instance. Thus, we were able to test the accuracy of the model on an independent data set (i.e., on data that is not used to train the model). If the results of the discriminant analysis in both the entire set and the n fold cross-validation set are similar, then we have increased confidence that the findings support the extension of the analysis to external data sets.
We also report the results of the LOOCV using recall, precision, and F1 scores. Recall scores are computed by tallying the number of hits (i.e., correct predictions) over the number of hits + false negatives (i.e., the number of produced words that were misclassified as unproduced words). Precision is the number of hits divided by the number of hits + false positives (i.e., the number of unproduced words that were classified as produced words). The precision and recall scores allow us to better understand the accuracy of the mode, whereas the F1 scores give us a weighted average of the precision and recall results.
RESULTS
Nouns
MANOVA
A MANOVA was conducted using the contextual diversity, contextual distinctiveness, and word frequency indices as the dependent variables and the produced and unproduced nouns as the independent variables. All variables that reported significant differences after a Bonferroni correction (in which p < .005) were then assessed using Pearson correlations for multicollinearity (with a threshold of r > .70). All the variables except the LSA index demonstrated significant difference between the produced and unproduced nouns. The word frequency index taken from SUBTLEXus demonstrated the highest effect size and demonstrated multicollinearity with the contextual diversity indices from SUBTLEXus and the BNC spoken corpus. For this reason, the SUBTLEXus word frequency index was kept in the analysis, and the other two indices were removed. The word frequency index taken from the entire BNC corpus demonstrated multicollinearity with the word frequency indices taken from the BNC spoken and written corpora and the contextual diversity index from the entire BNC corpus and the BNC written corpus. Because the word frequency for the entire BNC index demonstrated a greater effect size with the independent variable, it was retained in the analysis, and the other four indices were removed (see Table 1 for the descriptive statistics and the MANOVA results for the variables selected for the DFA analysis).
Mean (standard deviations), f value, p value, and effect size for nouns
Note.
BNC = British National Corpus; USF = University of South Florida.
Discriminant Function Analysis
The stepwise DFA selected variables from column 1 in Table 1 on the basis of a statistical criterion that retains the variables that best classify the grouping variable (produced or unproduced nouns) and helps control for potential multicollinearity. For our analysis, the significance level for a variable to be entered or to be removed from the model was set at the norm generally adopted in applied linguistics: p ≤ .05. The stepwise DFA retained two variables as significant predictors of whether a noun was produced or not produced by beginning-level L2 learners (word frequency SUBTLEXus and word familiarity) and removed the remaining two variables as nonsignificant predictors.
The results demonstrate that the DFA using the two indices correctly allocated 77 of the 90 nouns in the total set, χ2 (df = 1, n = 90) = 45.534, p < .001, for an accuracy of 86.0% (the chance level for this analysis and all analyses is 50%). For the LOOCV, the discriminant analysis correctly allocated 75 of the 90 nouns for an accuracy of 83.3% (see the confusion matrix reported in Table 2 for results).Footnote 5 The measure of agreement between the actual text type and that assigned by the model produced a weighted Cohen’s kappa of .711, which demonstrated a substantial agreement.Footnote 6
Predicted word type (produced or not produced) from total set and LOOVC set for nouns
Note.
LOOCV = leave-one-out cross-validation.
The precision scores (i.e., the number of correct results divided by the number of all retrieved results) and recall scores (i.e., the number of correct results divided by the number of all relevant results) from the model for predicting the nouns as either produced or not produced are presented in Table 3. The model performed equally well regardless of whether the noun was produced or unproduced. The overall accuracy of the model for the total set was .855 (the average F1 score) and .822 for the cross-validated set. The results demonstrate that the combination of two indices related to word frequency can substantially discriminate between produced and unproduced nouns.
Precision and recall finding (total set and LOOCV set) for nouns
Note.
LOOCV = leave-one-out cross-validation.
Verbs
MANOVA
A MANOVA was conducted using the contextual diversity, contextual distinctiveness, and word frequency indices as the dependent variables and the produced versus unproduced conditions as the independent variables. All variables that reported significant differences after a Bonferroni correction were then assessed using Pearson correlations for multicollinearity (with a threshold of r > .70). Like the noun analysis, all the variables except the LSA index demonstrated significant difference between the produced and unproduced verbs. The contextual diversity index from SUBTLEXus reported the highest effect size and also demonstrated multicollinearity with the contextual diversity index from the BNC spoken corpus, which was removed. The word frequency index from SUBTLEXus had a larger effect size and demonstrated multicollinearity with all three of the word frequency indices calculated from the BNC corpus, which were also removed. Lastly, the contextual diversity index calculated from the total BNC corpus had a larger effect size and demonstrated multicollinearity with the contextual diversity index from the BNC written corpus, which was removed. Descriptive statistics and MANOVA results for the final indices used in the verb DFA are presented in Table 4.
Mean (standard deviations), f value, p value, and effect size for verbs
Discriminant Function Analysis
The stepwise DFA retained the contextual diversity SUBTLEXus index and the word familiarity index. The results demonstrate that the DFA using the two indices correctly allocated 70 of the 89 verbs in the total set, χ2 (df = 1, n = 89) = 29.223, p < .001, for an accuracy of 78.6%. For the cross-validated set, the discriminant analysis correctly allocated 69 of the 89 verbs for an accuracy of 77.5% (see Table 5 for results). The measure of agreement between the verb type (produced and unproduced) and that assigned by the model produced a weighted Cohen’s kappa of .573, which demonstrated a moderate agreement.
Predicted word type (produced or not produced) from total set and LOOCV set for verbs
Note.
LOOCV = leave-one-out cross-validation.
Table 6 provides the index precision and recall scores for predicting whether the verbs were produced or unproduced. The model performed similarly for predicting both produced and unproduced verbs. The overall accuracy of the model for the total set was .786 (the average F1 score). The accuracy for the cross-validated set was .775. The results demonstrate that the combination of two indices related to contextual diversity and word familiarity can moderately discriminate between produced and unproduced nouns.
Precision and recall finding (total set and LOOCV set) for verbs
Note.
LOOCV = leave-one-out cross-validation.
Post Hoc Analysis
Noting that the leading predictor of whether a noun was produced or not produced was SUBTLEXus word frequency and that the leading predictor of whether a verb was produced or not produced was SUBTLEXus contextual diversity, we ran a post hoc analysis to investigate differences in the lexical properties of nouns and verbs under the premise that lexical differences between nouns and verbs may explain why frequency is predictive of nouns and contextual diversity is predictive of verbs. We predicted that verbs would be more ambiguous and more abstract than nouns, and, thus, the learning of verbs would benefit from greater textual diversity because context would assist with developing stronger lexical associations. Because nouns already contain richer lexical properties (i.e., they are less ambiguous and more concrete), we predicted that their production would not strongly rely on contextual diversity. For this analysis we focused on lexical properties related to depth of knowledge such as word hypernymy (i.e., word specificity), word polysemy (i.e., word ambiguity), and word meaningfulness (i.e., number of word associations) and core lexical items such as word concreteness and word imageability. These indices will be discussed briefly.
Depth of Knowledge Indices
Depth of knowledge indices tap into lexical associations among words and are important indicators of word knowledge (Crossley et al., Reference Crossley, Salsbury and McNamara2009, Reference Crossley, Salsbury and McNamara2010). We calculated depth of lexical knowledge using WordNet (Fellbaum, Reference Fellbaum1998; Miller, Beckwith, Fellbaum, Gross, & Miller, Reference Miller, Beckwith, Fellbaum, Gross and Miller1993) and the MRC Psycholinguistic Database (Wilson, Reference Wilson1988). From WordNet, we calculated an approximate polysemy value (i.e., the number of senses a word has) and a word hypernymy value (i.e., the number of levels a word has in a conceptual, taxonomic hierarchy).Footnote 7 These indices relate to word ambiguity and specificity, respectively. Using the MRC Psycholinguistic Database, we calculated word meaningfulness (i.e., the number of associations a word has according to human raters; Gilhooly & Logie, Reference Gilhooly and Logie1980; Paivio, Yuille, & Madigan, Reference Paivio, Yuille and Madigan1968; Toglia & Battig, Reference Toglia and Battig1978) for each word.
Core Lexical Items
Core lexical items are more likely to be basic category words, which are words that are generally learned first and that are characteristic of emerging lexicons (R. Brown, Reference Brown1958; Murphy, Reference Murphy2004). One method to measure core lexical items is through reporting word concreteness and imageability scores as found in the MRC Psycholinguistic Database. These scores measure lexical constructs such as word abstractness (i.e., concreteness) and the evocation of mental and sensory images (i.e., imageability).
MANOVA
A MANOVA was conducted using the depth of knowledge and core lexical indices as the dependent variables and the noun and verb conditions as the independent variables. The analysis demonstrated significant differences between nouns and verbs for all indices. Overall, nouns contained much richer lexical properties such that nouns were significantly more specific, concrete, imageable, meaningful, and unambiguous. Verbs, however, were more abstract and ambiguous and, thus, less concrete, imageable, and meaningful. Descriptive statistics and MANOVA results for this analysis are located in Table 7.
Mean (standard deviations), f value, p value, and effect sizes between nouns and verbs
DISCUSSION
This study has demonstrated that indices related to both word frequency and contextual diversity are strong classifiers of whether a word is produced or not produced by beginning-level L2 learners. Indices related to contextual distinctiveness were not strong predictors of whether words were produced or not produced. However, word frequency and contextual diversity indices appear to act independently of each other. Word frequency indices are strong classifiers of whether or not a noun is produced, whereas contextual diversity indices are strong classifiers of whether or not a verb is produced. A post hoc analysis showed that nouns and verbs significantly differed in a variety of lexical properties such that nouns were more specific, concrete, imageable, meaningful, and unambiguous; this finding provides potential evidence for the differences in classification variables between nouns and verbs.
Our noun analysis demonstrated that noun production was best explained using two indices related to word frequency and word familiarity. These two indices categorized 83% of the nouns correctly as being either produced or not produced by the L2 learners in our study. The MANOVA demonstrated that both of these indices had large effect sizes (Cohen, Reference Cohen1992). Additionally, the standardized coefficients reported by the DFA, which indicate the strength of the contribution that each lexical index makes in predicting the dependent variable, demonstrated that word frequency was a stronger classifier of word production than familiarity was (coefficients of .803 and .431, respectively). Contextual diversity indices were also strong predictors of noun production, but they were highly collinear with word frequency indices and did not report stronger effect sizes than our word frequency indices. This finding indicates that word frequency is a stronger predictor of noun production than contextual diversity. The findings also indicate that word familiarity contributes additional classification power independent of word frequency and provides evidence that word frequency and word familiarity may measure different lexical properties.
Our verb analysis—unlike our noun analysis—did not retain a word frequency index but, instead, retained a contextual diversity index along with a word familiarity index. These two indices correctly classified 78% of the verbs as being produced by beginning-level L2 learners. The two indices demonstrated large effect sizes, which indicate a strong relationship between the lexical properties and the dependent variable. The standardized coefficients suggested that the contextual diversity index was a stronger classifier than the word familiarity index (reporting coefficients of .668 and .502, respectively). Word frequency indices, although strongly predictive of verb production, did not report stronger effect sizes than contextual diversity indices, even though they were highly collinear. These findings indicate that contextual diversity is the strongest predictor of verb production for the sampled population, with word familiarity providing additive power to the analysis.
A comparison of the two analyses demonstrates important differences in the effects of word frequency and contextual diversity on lexical production. It appears that one lexical property is not superior to the other in explaining lexical production but that both are important qualities of word production that differ depending on the syntactic category of the word. Nouns—which are more concrete, more imageable, more specific, more meaningful, and less ambiguous—are best predicted by word frequency indices. This is likely due to the fact that the lexical properties of nouns allow them to be learned on the basis of repetition alone. Thus, the distributional properties of nouns allow lexical acquisition that is based on association through exposure. This is in contrast to verbs, which are more abstract and ambiguous. The abstractness and ambiguity found in verbs likely make them more difficult to acquire through repetition alone, and, thus, acquisition is likely aided by exposure to the words in a variety of contexts, following a principle of likely need. It is these contexts that may provide the lexical properties that afford acquisition. Therefore, context is an important element of abstract and ambiguous word learning, but words that have richer innate properties (i.e., concreteness, imageability, specificity) can be acquired through frequency alone.
In addition to word frequency and contextual diversity, word familiarity is also an important indicator of lexical acquisition and provides additive classification accuracy to both noun and verb production. Unlike word frequency and contextual diversity, the exact lexical properties that underlie human judgments of word familiarity are difficult to pinpoint. Word familiarity has often been interpreted as a measure of word exposure because words that are rated as more familiar are recognized more quickly (Connine, Mullennix, Shernoff, & Yelen, Reference Connine, Mullennix, Shernoff and Yelen1990; Gernsbacher, Reference Gernsbacher1984; Stadthagen-Gonzalez & Davis, Reference Stadthagen-Gonzalez and Davis2006). A variety of studies have demonstrated that word familiarity ratings—unlike written, corpus-based frequency indices—more accurately reflect spoken word frequency and therefore demonstrate a bias toward natural exposure (Stadthagen-Gonzalez & Davis, Reference Stadthagen-Gonzalez and Davis2006). Thus, word familiarity ratings may correlate more strongly to the age of acquisition of lexical items (G. Brown & Watson, Reference Brown and Watson1987). The familiarity indices used in this study were not strongly correlated with word frequency nor with contextual diversity and were the second-strongest predictors in both our verb and noun analysis. Thus, beyond word frequency and contextual diversity, human judgments of familiarity are important indicators of whether a word will be produced or not produced.
We were surprised that our indices of contextual distinctiveness were not selected as classifiers in our DFAs. Our LSA indices yielded no significant differences between produced and unproduced nouns and verbs, whereas our USF indices demonstrated significant differences between produced and unproduced nouns but not verbs. For the noun analysis, the USF indices showed medium effect sizes with the dependent variables, but they were not selected as independent variables in the DFA, which indicates that they provided no facilitative effects independent of word frequency. It is possible, then, that contextual distinctiveness, although important for explaining L1 lexical processing, is not an important property of L2 lexical production. One potential explanation for this is that beginning-level L2 learners may not attend to local contexts as much as global contexts when acquiring lexical items.
From a pedagogical perspective, this study has implications for L2 instruction and materials design. The findings indicate that, when learning nouns, L2 instructors can likely rely on the natural distributions of words as found in representative corpora. That is to say, nouns may not need to be taught in widely dispersed contexts, although such dispersion would likely not prove detrimental. However, verb learning is likely facilitated by introducing L2 learners to specific verbs in a variety of contexts to provide the learners with the extralinguistic information necessary for acquisition of these more abstract and less specific words. Thus, L2 practitioners would do well to complement verb learning by introducing students to verbs in a variety of different contexts. Similar suggestions hold true for material designers. Because L2 learners appear to benefit from a wide range of contexts when learning verbs, textbooks should also present new verbs in a variety of contexts to introduce learners to the range of possible uses of verbs.
CONCLUSION
Our primary research goals in this study were to investigate if word frequency was the strongest predictor of L2 word production and if lexical differences existed between nouns and verbs that would explain word production. We have demonstrated that, although word frequency is the strongest predictor of noun production, it is not the strongest predictor of verb production. We thus call into question the primacy of frequency effects in lexical production and acquisition. We have also demonstrated that differences in the lexical properties (i.e., concreteness, imageability, meaningfulness, specificity, and ambiguousness) can help explain why nouns are best classified using a word frequency index and why verbs are best explained using a contextual diversity index.
Even though the statistical results from this study are quite robust—reporting classification accuracies that average about 80%—we see this study as an initial investigation into comparing the power of word frequency, contextual diversity, and contextual distinctiveness effects in explaining L2 lexical acquisition. Future studies would benefit from examining lexical acquisition as a receptive skill by analyzing word-naming or lexical decision tasks. The indices used in this study could also be operationalized differently. Specifically, as larger corpora become available, additional word frequency and contextual diversity indices could be calculated. This is especially true for large corpora that contain a greater variety of American English words, which may differ slightly from the English found in the BNC. Future studies should also consider examining contextual diversity not only as calculated in the current study but also as a result of unique semantic contexts (as compared to context defined as an individual text; Johns & Jones, Reference Johns, Jones, Sloutsky, Love and McRae2008) or as a measure of collocational productivity (i.e., the number of collocates a word demonstrates in a selected corpus that is over a specific threshold; Wolter, Reference Wolter, Fitzpatrick and Barfield2009). Such additions may combine to provide us with a better understanding of L2 word learning in relation to the effects of word frequency, contextual diversity, and contextual distinctiveness.
APPENDIX A LEXICAL VALUES OF L2-PRODUCED NOUNS
APPENDIX B LEXICAL VALUES OF L2-UNPRODUCED NOUNS
APPENDIX C LEXICAL VALUES OF L2-PRODUCED VERBS
APPENDIX D LEXICAL VALUES OF L2-UNPRODUCED VERBS
