INTRODUCTION
The ability to recognize a word’s visual form and associate it with meaning constitutes a basis for higher-level processing at both the sentence- or discourse-level (Cortese & Balota, Reference Cortese, Balota, Spivey, McRae and Joanisse2012; Stanovich, Reference Stanovich, Barr, Kamil, Mosenthal and Pearson1991). Accordingly, a number of researchers have examined the features that impact the speed with which words are accurately recognized. With exceptions (e.g., Dijkstra & Van Heuven, Reference Dijkstra and Van Heuven2002; Fender, Reference Fender2003; Portin, Lehtonen, & Laine, Reference Portin, Lehtonen and Laine2007; Segalowitz & Segalowitz, Reference Segalowitz and Segalowitz1993), the majority of such work has relied exclusively on native speakers while significantly less work has examined how word features explain the speed and accuracy with which nonnative speakers (NNS) recognize visual words. Insight into NNS word recognition is important not only for theoretical and empirical investigations of NNS reading but also for broader inquiry into how words are acquired, accessed, and processed by NNS. A better understanding of NNS word recognition can also help identify which features contribute to the difficulty (or ease) with which NNS learn words.
The purpose of the current study is to examine NNS English word recognition in a large-scale dataset of lexical decision performance measures. We examine the degree to which lexical predictors attested in L1 studies explain NNS word recognition to better understand NNS lexical processing and determine which features are predictive of word recognition. Our study is motivated by the following question: To what degree do lexical variables predict word recognition latencies and accuracy for NNS subjects?
The goal of this project is to examine which factors impact NNS visual word recognition to better understand the intrinsic difficulty of English words for NNS learners. By intrinsic difficulty, we refer to the properties of words that are fundamental to their form or meaning that may make them more difficult to learn (cf. Ellis, Reference Ellis, Schmitt and McCarthy1997; Hemchua & Schmitt, 2006).
NNS LEXICAL ACCESS AND WORD RECOGNITION
Psycholinguistic approaches to NNS lexical knowledge investigate the manner in which words are organized and accessed from NNS’s mental lexicons, as well as what factors may influence NNS lexical processingFootnote 1 (Dijkstra, Reference Dijkstra, Kroll and De Groots2005; Sunderman & Campbell, Reference Sunderman, Campbell and Chapelle2013). One approach to better understanding NNS lexical organization involves the examination of lexical retrieval, or access (Sunderman & Campbell, Reference Sunderman, Campbell and Chapelle2013), during word recognition. Visual word recognition is one of the fundamental lower-level processes of successful lexical decoding (Stanovich, Reference Stanovich, Barr, Kamil, Mosenthal and Pearson1991) and includes subprocesses involving recognition and identification of letters, formation of grapheme-phoneme correspondence, and association of words to their conceptual/semantic representations (Segalowitz, Poulsen, & Komoda, Reference Segalowitz, Poulsen and Komoda1991). In skilled decoders, these subprocesses are automatized, thus freeing up cognitive resources for more higher-level processing, such as syntactic and discoursal-level processing.
A common technique for exploring word recognition is the lexical decision task (Meyer & Schvaneveldt, Reference Meyer and Schvaneveldt1971), during which subjects are presented with a string of letters and asked to judge as quickly and accurately as possible whether the letters constitute a real word in the target language. Typically, subjects press a designated key or button to indicate their decision. Both the speed and accuracy of subjects’ decisions are analyzed, under the presumption that the relative speed of accurate word recognition indexes ease of processing. However, with exceptions (e.g., Cook & Gor, Reference Cook and Gor2015; Cook, Pandža, Lancaster, & Gor, Reference Cook, Pandža, Lancaster and Gor2016), word recognition in NNS research has received relatively little attention when compared to L1 studies (Koda, Reference Koda2005). The majority of studies that have investigated NNS word recognition have examined the influence of the L1 on NNS word recognition (e.g., Elston-Güttler, Paulmann, & Kotz, Reference Elston-Güttler, Paulmann and Kotz2005; Fender, Reference Fender2003) or the bilingual lexical organization (e.g., De Groot, Borgwaldt, Bos, & van den Eijnden, Reference De Groot, Borgwaldt, Bos and van den Eijnden2002; Dijkstra & Van Heuven, Reference Dijkstra and Van Heuven2002). Of those studies that have investigated within-language (vs. cross-linguistic) NNS word recognition, most have focused on the development of automaticity (e.g., Segalowitz & Segalowitz, Reference Segalowitz and Segalowitz1993; Segalowitz, Segalowitz, & Wood, Reference Segalowitz, Segalowitz and Wood1998), although at least one focused on a large number of within- and between-linguistic variables to predict NNS performance (Lemhöfer et al., Reference Lemhöfer, Dijkstra, Schriefers, Baayen, Grainger and Zwitserlood2008).
VARIABLES KNOWN TO IMPACT VISUAL WORD RECOGNITION BEHAVIOR
While a great deal of research has investigated the type of lexical features that impact L1 visual word recognition in lexical decision tasks, little NNS literature exists on the matter. In general, L1 research demonstrates that features such as word frequency, semantic features, and neighborhood effects explain word recognition behaviors. These effects are discussed in more detail in the following text.
Frequency
Word frequency is the most robust predictor of speed of response and accuracy in L1 word recognition tasks (Balota & Chumbley, Reference Balota and Chumbley1990; Balota, Cortese, Sergent-Marshall, Spieler, & Yap, Reference Balota, Cortese, Sergent-Marshall, Spieler and Yap2004; Forster & Chambers, Reference Forster and Chambers1973; Whaley, Reference Whaley1978). While word frequency effects are found in essentially any task that involves visual recognition, they are especially strong in lexical decision tasks (Rastle, Reference Rastle and Gaskell2009). An assumption for this effect is that language speakers’ experience with a known word is encoded in the activation level of its orthographic representation (McClelland & Rumelhart, Reference McClelland and Rumelhart1981). Words that occur more frequently in input are more readily activated than words that occur rarely. Accordingly, it is easier to reach a threshold of activation for frequent words than for less frequently encountered words, which have a higher level of activation (Coltheart, Rastle, Perry, Langdon, & Ziegler, Reference Coltheart, Rastle, Perry, Langdon and Ziegler2001; Grainger & Jacobs, Reference Grainger and Jacobs1996; McClelland & Rumelhart, Reference McClelland and Rumelhart1981).
Word frequency has also been found to predict lexical decision (De Groot et al., Reference De Groot, Borgwaldt, Bos and van den Eijnden2002) and word identification task performance (Lemhöfer et al., Reference Lemhöfer, Dijkstra, Schriefers, Baayen, Grainger and Zwitserlood2008) in second language studies, a finding that aligns with evidence of frequency-based learning and word representation during second language acquisition (Ellis, Reference Ellis2002a, Reference Ellis2002b). While there is evidence to suggest that NNS subjects demonstrate a stronger frequency effect in lexical decision tasks than L1 subjects (Duyck, Vanderelst, Desmet, & Hartsuiker, Reference Duyck, Vanderelst, Desmet and Hartsuiker2008), Diependaele, Lemhöfer, and Brysbaert (Reference Diependaele, Lemhöfer and Brysbaert2013) have demonstrated that stronger NNS frequency effects can be explained by differences in language-specific proficiency rather than qualitative differences in NNS processing due to language competition. This same argument—the Lexical Quality Hypothesis—has been used to explain differential frequency effects in L1 speakers as well (Perfetti, Reference Perfetti2007).
Contextual Diversity
Traditionally, most word frequency counts are corpus-based values built on occurrence frequencies gathered from large reference corpora (e.g., Kučera & Francis, Reference Kučera and Francis1967), though spoken corpus-based frequencies are available as well (e.g., Brysbaert & New, Reference Brysbaert and New2009b). Because this manner of operationalizing frequency effectively treats language as a randomly ordered collection of words, some researchers have begun to argue that the diversity of contexts in which a word is encountered may offer a more psychologically valid explanation of the word frequency effect (Adelman, Brown, & Quesada, Reference Adelman, Brown and Quesada2006; Brysbaert & New, Reference Brysbaert and New2009a; McDonald & Shillcock, Reference McDonald and Shillcock2001). For example, Adelman et al. (Reference Adelman, Brown and Quesada2006) found that contextual diversity measures (operationalized as the number of documents within a corpus that contain a given word) explained more variance in L1 lexical decision latencies than word frequency measures (the total number of times a given word occurs in a corpus).
Age of Acquisition
Another factor found to impact L1 word recognition performance is age of acquisition (AoA) (Brown & Watson, Reference Brown and Watson1987; Ellis & Morrison, Reference Ellis and Morrison1998), which refers to the estimated age at which a native speaker of a language first encounters a word. The theoretical assumption underlying the analysis of AoA is that earlier acquired words play a more central role in the lexicon (Steyvers & Tenenbaum, Reference Steyvers and Tenenbaum2005) while also boasting greater cumulative frequency over time (Cortese & Balota, Reference Cortese, Balota, Spivey, McRae and Joanisse2012). Past studies have demonstrated strong AoA effects in word recognition tasks independent of frequency (Morrison & Ellis, Reference Morrison and Ellis1995), and studies have also demonstrated evidence of AoA effects on visual word recognition in languages other than English (e.g., Brysbaert, Lange, & Wijnendaele, Reference Brysbaert, Lange and Wijnendaele2000). For example, Brysbaert et al. (Reference Brysbaert, Lange and Wijnendaele2000) found a significant effect of AoA on L1 lexical decision performance in L1 Dutch, even when stimuli were controlled for length and imageability.
While the age of learners’ initial exposure to a second language is a strong determinant of nativelike acquisition (DeKeyser, Reference DeKeyser2013; Flege, Yeni-Komshian, & Liu, Reference Flege, Yeni-Komshian and Liu1999; Johnson & Newport, Reference Johnson and Newport1989), particularly for pronunciation (Saito, Reference Saito2015), given individual differences in learning experiences, it is difficult to estimate the average AoA for individual NNS words the way researchers have for L1 words and, to our knowledge, no such metrics exist. However, some researchers (e.g., Crossley & McNamara, Reference Crossley and McNamara2012; Salsbury, Crossley, & McNamara, 2011) have successfully used L1 AoA ratings to investigate NNS lexical acquisition. These studies report that L1 AoA is a strong predictor of NNS acquisition (Salsbury, Crossley, & McNamara, 2011) and that NNS learners produce words with higher AoA than L1 writers (Crossley & McNamara, Reference Crossley and McNamara2012)
Length
The length of a word has also been found to influence performance in word recognition (Coltheart et al., Reference Coltheart, Rastle, Perry, Langdon and Ziegler2001), with some research suggesting that the impact of length is stronger for low-frequency words than high-frequency words (Balota et al., Reference Balota, Cortese, Sergent-Marshall, Spieler and Yap2004) and that word length facilitates visual word recognition, but only for words ranging from three to five letters long (New et al., Reference New, Ferrand, Pallier and Brysbaert2006). In a bilingual study, De Groot et al. (Reference De Groot, Borgwaldt, Bos and van den Eijnden2002) found that NNS English lexical decision time—but not L1 Dutch lexical decision time—was significantly influenced by word length in Dutch-English bilingual subjects. The study also reported an interaction between length and frequency for NNSs of English such that subjects responded more slowly to longer words only if they were also less frequent.
Neighborhood Size
Orthographic Neighborhood
An orthographic neighbor is formed by changing one letter of a real word to form another real word (Coltheart, Davelaar, Jonasson, & Besner, Reference Coltheart, Davelaar, Jonasson, Besner and Dornic1977). For example, the word dumpster has zero orthographic neighbors while the word fire has eight, including fare, file, firm, and wire. While orthographic neighborhood size has been found to facilitate lexical decision performance (Andrews, Reference Andrews1997; Balota et al., Reference Balota, Yap, Hutchison, Cortese, Kessler, Loftis and Treiman2007), its impact may be modulated by other variables (Andrews, Reference Andrews1992), differences between word processing and word production, or the presence of nonsense or pseudowordsFootnote 2 as distractors (Johnson & Pugh, Reference Johnson and Pugh1994). For instance, Balota et al. (Reference Balota, Cortese, Sergent-Marshall, Spieler and Yap2004) found that while orthographic neighborhood size was facilitatory in younger L1 subjects (M = 20.5 years), it had an inhibitory effect for older subjects (M = 73.6 years) and slow-processing young subjects. NNS research analyzing orthographic neighborhood size has found that for Dutch L1 subjects highly proficient in English, increasing orthographic neighborhood density in English had a facilitatory effect on English word recognition but inhibitory effects on Dutch word recognition (Van Heuven et al., Reference Van Heuven, Dijkstra and Grainger1998).
Phonological Neighborhood
While orthographical neighbors differ by one letter, phonological neighbors differ by one phoneme. For example, the word tomb has 16 phonological neighbors, including room, time, and zoom, while the word stone only has three phonological neighbors: stoat, stole, and stove. Previous L1 research suggests that phonological neighborhood size may facilitate lexical decision performance independent of orthographic neighborhood size (Yates, Locker, & Simpson, Reference Yates, Locker and Simpson2004). Investigations of NNS phonological neighborhood size have typically examined cross-language bilingual word recognition and suggest that orthographic and phonological neighborhood are likely to have a similar impact on bilingual processing Marian and Blumenfeld (Reference Marian and Blumenfeld2006).
Semantic Characteristics
Previous studies have also demonstrated that the meaning-based characteristics of a word (e.g., concreteness, meaningfulness, imageability) may impact how quickly and accurately a word is recognized (Balota et al., Reference Balota, Cortese, Sergent-Marshall, Spieler and Yap2004; Balota, Ferraro, & Connor, Reference Balota, Ferraro, Connor and Schwanenflugel1991). In an L1 study on visual word recognition, Yap and Balota (Reference Yap and Balota2008) found semantic effects on word recognition even after controlling for other lexical characteristics (e.g., length, neighborhood size). Cortese and Balota (Reference Cortese, Balota, Spivey, McRae and Joanisse2012) note that the existence of relatively larger semantic effects on lexical decision performance (compared to other word recognition tasks, such as word naming) is not surprising given that the central task in lexical decision is to distinguish meaningful strings of letter from nonmeaningful strings. We discuss a number of these features in greater depth in the text that follows.
Concreteness
Concrete words evoke mental images quickly and can be contrasted with abstract words. For instance, a word like tree is more concrete than a highly abstract word like solidarity. Previous studies have demonstrated that concrete words are responded to faster in lexical decision tasks than abstract words (Paivio, Reference Paivio1991; Schwanenflugel & Shoben, Reference Schwanenflugel and Shoben1983; Schwanenflugel & Stowe, Reference Schwanenflugel, Stowe and Schwanenflugel1991). Concreteness may also have a greater effect on lexical decision latencies for low-frequency than high-frequency words (Carlton, Reference Carlton1975). Concrete words are also translated faster and more accurately by NNS subjects (De Groot, Dannenburg, & Van Hell, Reference De Groot, Dannenburg and Van Hell1994).
Hypernymy
The term hypernym refers to a word whose meaning subsumes more specific word-members within a semantic hierarchy. Words higher in a hypernymic hierarchy—such as plant—are typically considered to be less specific than words found lower on a hypernymic hierarchy—such as succulent (Lakoff, 1987). To our knowledge, no L1 study has used hypernym values to investigate lexical decision performance outside of category judgments or semantic priming. In a priming study that included both L1 and NNS subjects, Crossley (Reference Crossley2013) found evidence of bidirectional priming (hyponym to hypernym, and vice versa) for L1 subjects but only unidirectional effects (hyponym to hypernym only) for NNS subjects.
Contextual Distinctiveness
More recently, researchers have begun to investigate the degree of semantic clustering between a word and other words in the lexicon (Steyvers & Tenenbaum, Reference Steyvers and Tenenbaum2005) and its impact on visual word recognition. For example, Balota et al. (Reference Balota, Cortese, Sergent-Marshall, Spieler and Yap2004) found that semantic connectivity—operationalized as the number of associations produced in response to a given word in a free association task (Nelson, McEvoy, & Schreiber, Reference Nelson, McEvoy and Schreiber1998)—explained significant variance in L1 lexical decision latencies, with faster reaction times for words with greater connectivity. Semantic connectivity is subsumed within a broader construct referred to as contextual distinctiveness. Contextual distinctiveness refers to the constraints made on a word by its frequent lexical, associative, and semantic contexts, and the degree to which such constraints contribute to a word’s meaning representation (Berger, Crossley, & Kyle, Reference Berger, Crossley and Kyle2017; Crossley, Subtirelu, & Salsbury, Reference Crossley, Subtirelu and Salsbury2013; McDonald & Shillcock, Reference McDonald and Shillcock2001). For example, McDonald and Shillcock found that a word’s frequent lexical environment was more predictive of L1 lexical decision performance than occurrence frequency, with words occurring in more constrained (i.e., more distinct) lexical contexts resulting in longer decision latencies.
CURRENT STUDY
Traditionally, researchers who investigated relationships between lexical features and lexical decision tasks manipulated and controlled specific features in small sets of word recognition stimuli. More recently, researchers have begun to forego such small-scale factorial designs to investigate word recognition in favor of larger-scale item-analyses (e.g., Balota et al., Reference Balota, Cortese, Sergent-Marshall, Spieler and Yap2004; Keuleers, Diependaele, & Brysbaert, Reference Keuleers, Diependaele and Brysbaert2010; Keuleers, Lacey, Rastle, & Brysbaert, Reference Keuleers, Lacey, Rastle and Brysbaert2012; Spieler & Balota, Reference Spieler and Balota1997). In the latter, multiple regression techniques are used to determine the unique variance of lexical features of stimuli on word recognition performance.
In this study, we examine NNS lexical decision performance (response latencies and accuracies) in visual word recognition for 3,318 words in English. We select a number of empirically attested predictors associated with L1 word recognition response behavior to examine NNS lexical decision behavior. Our goal is to determine which factors impact NNS visual word recognition to better understand the intrinsic difficulty of English words for NNS learners and better understand NNS lexical access. Analyses include the examination of relationships between measures (i.e., lexical measures) and outcomes (i.e., NNS lexical decision response times [RTs] and accuracies). Attested interactions among variables are investigated as well.
METHODS
NNS LEXICAL DECISION RESPONSE DATA
Participants
NNS lexical decision performance measures were collected from 1,315 self-identified NNS speakers of English residing within the United States using the online crowdsourcing platform Amazon Mechanical Turk (AMT). All participants were asked to complete a brief linguistic and demographic questionnaire, which included questions pertaining to their dominant language (the language that they use most often), other languages spoken besides English, confidence in English ability (scale of 1–4), hours per day spent using English, years spent learning English, years spent studying English, months lived in an English-speaking country, gender, and handedness.Footnote 3 The results of this questionnaire are reported in Appendix A. However, because the current study was item based and not subject based, participants’ demographic and individual difference information could not be included in the data analyses.
AMT workers independently choose from among a list of available tasks on the AMT site to complete in exchange for monetary compensation. Studies of AMT workers have demonstrated that this population is just as reliable as traditional psychology subject-pool populations (Buhrmester, Kwang, & Gosling, 2011), and several attested psycholinguistic phenomena have been replicated with AMT workers using online measurement techniques (Enochson & Culbertson, 2015). However, because AMT workers complete the tasks online and anonymously, researchers must take care to ensure the data collected represents unique individuals who were paying attention during the task. To account for these challenges, we conducted participant management through the website TurkPrime (Litman, Robinson, & Abberbock, 2016), which blocks workers from repeating tasks multiple times and helps ensure their Internet connection originated from within the United States.
Stimuli
For the NNS lexical decision tasks, 3,318 English content words and 3,318 pseudowords were selected from the English Lexicon Project (ELP), a large publicly available behavioral and descriptive dataset collected by Balota et al. (Reference Balota, Yap, Hutchison, Cortese, Kessler, Loftis and Treiman2007).Footnote 4 The ELP dataset includes lexical decision performance behavior from 816 L1 subjects with approximately 3,400 responses per subjects. Latencies (i.e., how long it took for a subject to make a decision regarding whether a word was a real word in English) were calculated in response to 80,962 real word and pseudoword stimuli (40,481 each), including both mono- and multisyllabic words. All subjects in the ELP were native speakers of English recruited from participant pools across six different universities in the United States. The 40,481 real words were selected to represent a range of frequencies (based on Kučera-Francis and CELEX norms) while the pseudowords were generated by changing one to two letters in selected real words. For each word in the ELP, mean item latencies, z-scores, and standard deviations (SDs) were calculated using accurate observations across subjects who encountered a particular item. Mean accuracies were calculated for each word as well.
Word selection for this study followed that of the original ELP study except that function words were excluded from the current study because function words convey relatively little meaning compared to content words (Cann, Reference Cann and Borsely2000) and were thus hypothesized to offer little predictive power in an analysis based on word representation (McDonald & Shillcock, Reference McDonald and Shillcock2001). In addition, proper nouns were excluded to lessen the impact of cultural knowledge on word recognition. The pseudowords used in this study were randomly selected from the ELP project’s pseudowords. Sixty-four sublists were created to ensure that all 6,636 real/pseudowords were evenly distributed across test subjects and that at least 20 observations per word were obtained. Following Balota et al. (Reference Balota, Yap, Hutchison, Cortese, Kessler, Loftis and Treiman2007), all real word stimuli were organized into randomly assigned sublists consisting of adjacent alphabetized words. In this manner, the likelihood that a subject would be presented with words with the same morphemic stem was minimized. Sixty-three of the sublists contained 52 real words and 52 pseudowords each (104 total), while the 64th sublist contained 42 real words and 42 pseudowords (84 total). Analyses of variance (ANOVA) found no significant differences in length between real words and pseudowords in the 64 sublists.
Procedure
Data collection was facilitated using two online services: Qualtrics and Testable. Data from 256 participants (26,564 trials) was first collected using the Qualtrics Reaction Time Engine (QRTE; Barnhoorn, Haasnoot, Bocanegra, & van Steenbergen, 2015), a JavaScript plug-in for the online survey service Qualtrics that allows for collection of reaction time data. Due to unanticipated changes in the Qualtrics platform, the authors of the QRTE engine were forced to retire the engine while we were still collecting data. We then collected the remaining 1,059 participants (109,796 trials) using Testable, a self-contained website that uses JavaScript to collect online RT data.
AMT workers first viewed a description of the experiment on the AMT website and decided whether to participate. The instructions specified that only NNS speakers of English were able to participate. Once agreeing, participants were redirected to the experiment website and were presented with the informed consent. Participants who agreed were then asked again to complete a brief linguistic and demographic questionnaire (see Appendix A). After completing these questions, the participants moved on to the lexical decision task.
Each participant completed 104 trials (52 real word, 52 pseudowords) in the online lexical decision task. Following Balota et al. (Reference Balota, Yap, Hutchison, Cortese, Kessler, Loftis and Treiman2007), a fixation point was presented for 250 ms prior to each target word. Subjects were instructed to press “Q” if the string of letters was a “nonsense word” and “P” if the string of letters was a real word in English and were told that capitalization of the letters did not matter. Participant reaction time (measured in milliseconds) and accuracy was recorded for each trial. Each of the 64 subexperiments was completed by a minimum of 20 participants, and a total of 136,360 observations were collected overall. On average, participants took 8.5 minutes to complete the entirety of the experiment and were paid $0.80USD for their participation.
Calculation of NNS Item Norms
Following Balota et al. (Reference Balota, Yap, Hutchison, Cortese, Kessler, Loftis and Treiman2007), a two-step outlier identification and removal process was followed. In the first step, 4.780 observations with RTs ≤ 200 ms were removed from analysis. Another 1,685 observations with RTs ≥ than 3,000 ms were excluded as well. In the second step, all RTs were standardized within subjects and any observation for which the RT was less than three SDs below the mean for a given subject or greater than three SDs above the mean for that subject were removed. One observation was removed that fell less than three SDs below the subject’s mean, while 2,361 observations were removed that fell three SDs above the mean for given subjects. Following this two-step outlier process, 127,533 observations remained for analysis.
Next, mean item latencies, standardized item RTs (z-scores), and SDs for every real word were calculated based on accurate observations only (90.44% of the observations after outlier removal). Z-scores were calculated by subtracting individual subjects’ mean RT for accurate observations from each raw RT and dividing by their SD. The z-scores are useful because they minimize individual differences in speed or variation during the lexical decision task. The average number of accurate observations per word was also calculated. The resulting real word NNS lexical decision performance measures were used as variables in the current study.
LEXICAL MEASURES
All measures investigated in the current study operationalize variables that have previously demonstrated a consistent influence on L1 and/or NNS lexical decision task performance. In Table 1 and in the following text, we offer further details regarding the specific measuresFootnote 5 selected.
Lexical measures selected as predictors for regression analyses
Word Frequency
A frequency index derived from written texts in the Corpus of Contemporary American English (COCA; Davies, Reference Davies2009) was used as a measure of written word frequency. COCA contains more than 520 million words of American English collected since 1990. Written genres that comprise the corpus include fiction, magazine, newspaper, and academic genres. A COCA log-transformed measure was assessed in the current study. A frequency measure derived from the SUBTLEXus corpus (Brysbaert & New, Reference Brysbaert and New2009b) was also used to determine the spoken frequency of each word. SUBTLEXus frequency scores were derived from a corpus of American film and television subtitles approximating spoken language but excluding highly formal or academic written registers. A word with a relatively higher frequency value is assumed to occur more often in language usage. A measure of log-transformed frequency was used in this study.
Contextual Diversity
A range score derived from the SUBTLEXus corpus (Brysbaert & New, Reference Brysbaert and New2009b) was used to measure contextual diversity. This measure reports the number of individual texts (in this case, film or television shows) in which a word occurs in the corpus. A high contextual diversity value means that a word occurs in a greater variety of usage contexts, while a low value is indicative of a word that is relatively less contextually diverse. A log-transformed contextual diversity was used in this study.
Age of Acquisition
We analyzed average L1 AoA ratings collected by Kuperman, Stadthagen-Gonzalez, and Brysbaert (Reference Kuperman, Stadthagen-Gonzalez and Brysbaert2012). These AoA scores are derived from human judgments compiled using web-based crowdsourcing. A high AoA value is reflective of a word that is typically acquired later in life than a word with a lower AoA value.
Length
The number of letters per word was counted to measure length due to the strength of orthographic length in predicting lexical decision latencies (Coltheart et al., Reference Coltheart, Rastle, Perry, Langdon and Ziegler2001) independent of frequency (New et al., Reference New, Ferrand, Pallier and Brysbaert2006).Footnote 6
Neighborhood Size
We used measures of both orthographic neighborhood and phonological (excluding homophones) neighborhood size obtained from the ELP (Balota et al., Reference Balota, Yap, Hutchison, Cortese, Kessler, Loftis and Treiman2007). A higher neighborhood size value means that a word has more orthographical or phonological neighbors. In addition to a classic measure of neighborhood size (the number of words derived by changing one letter in the target word at a time), we also analyzed a computer science metric based on string edit distance: orthographic Levenshtein distance 20, or OLD20 (Yarkoni & Balota, 2008).
Semantic Characteristics
Concreteness
We used a measure reporting the average human judgments of a word’s concreteness. The original judgments were collected by Brysbaert, Warriner, and Kuperman (Reference Brysbaert, Warriner and Kuperman2014) using web-based crowdsourcing. A word with a higher value on the selected concreteness measure can be considered to have a more concrete referent than a word with a lower value.
Hypernymy
The hypernymy measure assessed in this study uses WordNet (Fellbaum, Reference Fellbaum1998; Miller, Beckwith, Fellbaum, Gross, & Miller, Reference Miller, Beckwith, Fellbaum, Gross and Miller1990) to determine the mean value of noun hypernymy in each text of the learner corpus. Words scoring lower on the hypernymy scale are less specific, while words receiving a higher score on the scale are considered to be more specific.
Contextual Distinctiveness
Four measures were selected to operationalize the contextual distinctiveness of words. One way of characterizing contextual distinctiveness is through the number of associations a word has, that is, its degree of semantic connectivity (Balota et al., Reference Balota, Cortese, Sergent-Marshall, Spieler and Yap2004). Two databases were used to quantify a word’s associations: the Edinburgh Associative Thesaurus (EAT) (Kiss, Armstrong, Milroy, & Piper, Reference Kiss, Armstrong, Milroy, Piper, Aitken, Bailey and Hamilton-Smith1973), which reports the number of word types produced in response to a given stimuli in a free association task; and the University of South Florida (USF) norms (Nelson et al., Reference Nelson, McEvoy and Schreiber1998), which work backward by reporting the number of stimuli words that resulted in production of the target word as an associate in free association tasks. EAT values measures the number of associates a word elicits, with a higher number indicating a greater number of semantic associates, while USF values may be interpreted as the ease with which a word comes to mind in response to a variety of different cues (Nelson et al., Reference Nelson, McEvoy and Schreiber1998).
A third measure was selected to operationalize a word’s contextual distinctiveness based on the variability of contexts in which it occurs. This computer-derived latent semantic analysis (LSA) measure, originally calculated by Hoffman, Ralph, and Rogers (Reference Hoffman, Ralph and Rogers2013), reports the semantic similarities of 1,000-word contexts in which a word occurs in the written British National Corpus (British National Corpus, 2007). Contextual distinctiveness (variety of semantic contexts) quantifies the degree to which the contexts associated with a given word vary in meaning. A word with a high value occurs in a greater variety of semantic contexts than a word with a lower value.
Finally, a relative entropy measure originally calculated by McDonald and Shillcock (Reference McDonald and Shillcock2001) was used to reflect the frequency distribution of the immediate lexical contexts in which a word occurs. This computational measure, derived from the spoken British National Corpus (British National Corpus, 2007) and referred to here as contextual distinctiveness (variety of lexical contexts), reports a higher value for words that co-occur statistically in the company of relatively few words, and a lower value for words that occur in a variety of lexical contexts.
STATISTICAL ANALYSIS
NNS RTs and accuracies for the lexical decision tasks were used as dependent (i.e., outcome) variables in separate multiple regressions. Models 1 and 2 used the lexical measures described previously as independent variables to predict NNS response latencies and accuracies, respectively.Footnote 7 Because frequency effects have been demonstrated to be moderated by length in past analyses of word recognition reaction times (Balota et al., Reference Balota, Cortese, Sergent-Marshall, Spieler and Yap2004), interactions between length and frequency were included in both models. In addition, interactions between length and neighborhood size were investigated, under the presumption that longer words would have fewer orthographic or phonological neighbors.
Prior to each regression, correlations among the independent variables were checked for instances of multicollinearity. If any two measures were highly collinear (r > .80), only the measure with the strongest relationship to the response behavior was retained. Remaining measures were standardized to reduce multicollinearity and associated problems caused by the inclusion of interaction terms. Prior to final regressions, an initial model was obtained to determine significant interactions. All remaining measures and significant interaction terms were then entered into a subsequent model to accurately obtain results. Any predictor demonstrating suppression was excluded from entry in the final regression. The resulting models are reported in the following text.
RESULTS
DESCRIPTIVE STATISTICS
Table 2 lists the mean and SD for the NNS lexical decision performance variables. Visual inspection of histograms indicated that values were relatively normally distributed (with slight evidence of positive skewness), with the exception of accuracy, which was nonnormally distributed. Table 3 reports descriptive statistics for predictor variables and correlations between the selected variables and the RTs and accuracy data.
Descriptive statistics for lexical decision performance measures
Descriptive statistics and correlations for initial lexical measures
*** p < 0.001; ** p < 0.01; * p < 0.05; p < 0.10
NNS STANDARDIZED ITEM RESPONSE TIMES
Ten variables and two interactions were manually entered in a multiple regression as independent variables with NNS standardized RTs as the dependent variable. Three predictors (hypernymy, length, and neighborhood size [orthographic]) demonstrated suppression effects and were excluded from further analysis. The two interactions did not demonstrate significance and were excluded from additional analysis as well. The seven remaining predictors were entered in a subsequent multiple regression. A significant regression equation for Model 1 was found (F (7, 3247) = 223.000, p < .001), with an adjusted R Footnote 2 of .323. Five variables contributed significantly to Model 1 (p < .05) (see Table 4).
Model 1 regression results predicting NNS standardized RTs
Note: β is standardized coefficient, SE is standard error, and no unstandardized coefficients reported because all predictors were standardized; hypernymy, length, and neighborhood size were removed from analysis due to suppression effects; interaction terms did not demonstrate significant in initial analysis and were removed from Model 1.
*** p < 0.001; ** p < 0.01; * p < 0.05; p < 0.10
Model 1 findings indicate that contextual diversity, AoA, and three operationalizations of contextual distinctiveness (two word association measures and a measure reporting the distinctiveness of lexical contexts) explained roughly 32% of the variance in average item RTs in the NNS lexical decision task. The coefficients for these measures (Table 6) indicate the size of each variable’s effect on NNS RT z-scores as well as the direction of the effect. The findings indicate that NNS subjects more quickly recognized English words that occur in a variety of contexts (i.e., contextual diversity), have more associations in native-speaker word association tasks, and are acquired earlier.
MODEL 2: NNS MEAN ITEM ACCURACIES
The same variables and interactions were entered in a multiple regression as independent variables with NNS mean item accuracy as the dependent variable. Three predictors (concreteness, hypernymy, and contextual distinctiveness [word associations, USF]) demonstrated suppression effects and were excluded from further analysis. The seven remaining indices and two interaction terms were then reentered into a final model (Model 2). A significant regression equation was reported for Model 2, (F (9, 3245) = 112.300, p < .001), with an adjusted R Footnote 2 of .235. Five measures contributed significantly to the model (Table 5), along with two interactions between length and contextual diversity (log) and length and neighborhood size (orthographic).
Model 2 regression results predicting NNS mean item accuracy
Note: β is standardized coefficient, SE is standard error, and no unstandardized coefficients reported because all predictors were standardized; concreteness, hypernymy, and contextual distinctiveness (word associations, USF) removed from analysis due to suppression.
*** p < 0.001; ** p < 0.01; * p < 0.05; p < 0.10
Model 2 findings indicate that contextual diversity, length, AoA, and two operationalizations of contextual distinctiveness (one word association measure and a measure reporting the distinctiveness of lexical contexts), as well as interactions between contextual diversity and length (Figure 1) and between neighborhood size and length (Figure 2), explained roughly 24% of the variance in average item accuracy in the NNS lexical decision task.
Demonstrates that the interaction between length and contextual diversity is strongest for shorter words, in that accuracy was significantly higher for shorter words if they had higher contextual diversity values (i.e., upper bound) compared to shorter words with lower contextual diversity values (i.e., lower bound). For longer words, differences in contextual diversity had less of an impact on accuracy. In other words, if a shorter word is encountered in fewer diverse contexts, it is more difficult to accurately recognize. However, this effect attenuates as length increases, suggesting that for a very long word, the diversity of contexts in which the word naturally appears has less of an effect on accurate word recognition.
Represents the interaction between length and orthographic neighborhood size. For shorter words, those with fewer orthographic neighbors were recognized more accurately compared to shorter words with a greater number of orthographic neighbors. In other words, if a shorter word appears orthographically similar to several other shorter words, that word is more difficult to accurately recognize. While this interaction appears to reverse as length increases in Figure 2, confidence intervals suggest a great deal of variance in Model 2 predictions for words with a large number of orthographic neighbors. This is likely due to the fact that longer words generally have fewer (if any) orthographic neighbors, and also due to the relatively few number of words in our dataset at this range in length.
The remaining coefficients suggest that NNS subjects were more likely to accurately recognize English words they are more contextually diverse, are longer, and have more associations in word association tasks. NNS subjects were less likely to accurately recognize English words that are acquired later in life and occur in more distinct lexical contexts.
DISCUSSION
While predictors of L1 word recognition are well attested, relatively little research has examined components of NNS word recognition. A better understanding of NNS word recognition is important not only for investigations of NNS lexical decoding but also for broader inquiry into how words are accessed and processed by NNS speakers. In particular, knowledge of within-language (vs. cross-linguistic) components of NNS word recognition can help identify which features contribute to the difficulty (or ease) with which NNS speakers from a variety of L1 backgrounds learn new words. Significant predictors of NNS latencies and accuracy included frequency, AoA, and contextual distinctiveness, with length moderating the impact of frequency and neighborhood size on accuracy. These predictors collectively explained approximately 32% of variance in NNS lexical decision item RTs and 24% of the variance in item accuracy. For comparison, Balota et al.’s (2004) L1 study demonstrated that lexical variables explained 49% and 52% of variance in ELP lexical decision and accuracy RTs, respectively. Because Balota et al. analyzed monosyllabic RTs, one would anticipate participants’ decisions to multisyllabic words to exhibit greater variability. In addition, factors present in the current study but not analyzed—specifically L1 background and NNS proficiency—likely account for some of this unexplained variance.
Item RTs (Model 1) were predicted by contextual diversity, AoA, and three contextual distinctiveness measures (two word association measures and a measure reporting the distinctiveness of lexical contexts). Meanwhile, item accuracy (Model 2) was impacted by contextual diversity, AoA, and two contextual distinctiveness measures (one word association measure and a measure reporting the distinctiveness of lexical contexts). In addition, length was found to moderate the effect of contextual diversity and neighborhood size on accuracy of NNS word recognition (Figures 1 and 2).
Taken as a whole, these results demonstrate that the word recognition behavior of NNS speakers of English is impacted significantly by contextual diversity, (L1) AoA, and contextual distinctiveness, and that the impact of contextual diversity and neighborhood size on NNS word recognition may be moderated by length. The relationship of each of these variables to NNS word recognition and lexical access is further discussed in the text that follows.
CONTEXTUAL DIVERSITY
The strongest predictor of both RTs and accuracy was contextual diversity. Like L1 subjects, NNS subjects respond more quickly and accurately to words that they are more frequently exposed to in a greater variety of contexts. Because of the high correlation between our contextual diversity measure and a word frequency measure (r = .980), it is reasonable to interpret the two as overlapping operationalizations of frequency. In this regard, our results support previous evidence suggesting a strong frequency effect in NNS word representation and recognition (De Groot et al., Reference De Groot, Borgwaldt, Bos and van den Eijnden2002; Ellis, Reference Ellis2002a).
AGE OF ACQUISITION
Another variable that predicted both RTs and accuracy was AoA. The AoA measure used here is based on human estimates of L1 age of word learning. While one would not assume that the age or sequence of learning for L1 children and NNS learners are the same, the unique variance explained by AoA in NNS lexical performance suggests that this measure may capture cumulative frequency or centrality of representation in the NNS lexicon, as it has been argued to do for the L1 lexicon (Cortese & Balota, Reference Cortese, Balota, Spivey, McRae and Joanisse2012; Steyvers & Tenenbaum, Reference Steyvers and Tenenbaum2005). Our findings also highlight the importance of including AoA in analyses to avoid falsely attributing word recognition effects to frequency or contextual diversity alone (Morrison & Ellis, Reference Morrison and Ellis1995).
CONTEXTUAL DISTINCTIVENESS
Contextual distinctiveness also predicted lexical decision behavior for both RTs and accuracy. The two contextual distinctiveness variables retained in Models 1 and 2 measured number of word associations and the distinctiveness of a word’s lexical context. This means that target words with a greater number of associations in native speakers’ minds were recognized more quickly and accurately by NNS subjects. This finding aligns with previously attested L1 behavior (Balota et al., Reference Balota, Cortese, Sergent-Marshall, Spieler and Yap2004), demonstrating that both L1 and NNS subjects are quicker to recognize words with greater semantic connectivity in the lexicon.
Words that occur in the company of a variety of different words (vs. in more distinct lexical contexts) were also recognized more quickly and with greater accuracy by NNS subjects. Following contextual diversity, this lexical contextual distinctiveness measure (McDonald & Shillcock, Reference McDonald and Shillcock2001) was the second strongest predictor of NNS decision times. Whereas contextual diversity reports the variety of global contexts (e.g., documents, newspapers, television shows) in which a word is encountered, the preceding contextual distinctiveness measure operationalizes context more locally as the immediate lexical environment in which a word frequently occurs (see Berger et al., Reference Berger, Crossley and Kyle2017). The independent variance explained by both contextual diversity and a lexical contextual distinctiveness measure indicates that the constraints of both global and local context impact NNS subjects’ lexical access in a manner unique from raw frequency alone, a finding that has been demonstrated in NNS lexical access as well (McDonald & Shillcock, Reference McDonald and Shillcock2001).
LENGTH AS MODERATING VARIABLE
The observed interaction between length and contextual diversity (Figure 1) for NNS accuracy (i.e., differences in contextual diversity values have a reduced influence on accuracy as word length increases) may reveal a proficiency effect, whereby subjects respond more quickly and accurately to more contextually diverse words—regardless of length—because they know them better (De Groot et al., Reference De Groot, Borgwaldt, Bos and van den Eijnden2002). This impact of length on NNS accuracy runs parallel to a similar trend found in L1 studies, whereby length demonstrates a stronger effect for low-frequency words than high-frequency words (Balota et al., Reference Balota, Cortese, Sergent-Marshall, Spieler and Yap2004; Weekes, Reference Weekes1997). Length was also found to moderate the effect of neighborhood size on accuracy of NNS word recognition (Figure 2) such that shorter words with fewer orthographic neighbors were recognized more accurately when compared to those with a greater number of orthographic neighbors. Taken as a whole, the two interactions demonstrated in Model 2 suggest that length’s impact on accurate NNS word recognition is strongest for less contextually diverse words with a greater number of similarly spelled neighbors; conversely, the impact of length wanes for more contextually diverse words with fewer orthographic neighbors. It is also the case that longer words naturally have fewer orthographic neighbors.
Previously attested predictors of L1 word recognition including frequency (written or spoken), concreteness, hypernymy, and the variety of semantic contexts in which a word typically occurs (Balota et al., Reference Balota, Cortese, Sergent-Marshall, Spieler and Yap2004; Balota, Ferraro, & Connor, Reference Balota, Ferraro, Connor and Schwanenflugel1991; De Groot et al., Reference De Groot, Borgwaldt, Bos and van den Eijnden2002) were not significant predictors in our models. It should be noted, however, that while these features were not predictive in our models, they all correlated significantly with NNS lexical decision performance variables (one exception was concreteness, a purely semantic variable that correlated significantly with NNS accuracy but not RTs). Thus, the findings suggest that NNS word recognition is not entirely distinct from L1 word recognition and that the two are influenced by similar lexical variables. To further test links between NNS and L1 word recognition, post hoc independent-samples t-test and Pearson’s correlations were conducted to compare differences between mean item RTs, mean item standardized RTs (z-scores), mean item RT SD, and accuracies. Significant differences between all L1 and NNS performance variables (p < .001) were demonstrated, such that L1 recognition was faster (by ∼100 ms) and more accurate (see Table 6). In addition, moderate and significant correlations were reported between L1 and NNS mean average RTs, mean RTs z-scores, and mean item accuracy. These post hoc analyses indicate that while L1 subjects recognized real words in English faster and more accurately, some shared variance (as evidenced by correlations) was observed for RTs and accuracies between L1 and NNS subjects. The effect sizes of these correlations suggest that at least a moderate amount of variance in the NNS lexical decision datasets—particularly for RTs—is systematically similar to L1 subjects.
Comparing lexical decision performance between L1 subjects (Balota et al., Reference Balota, Yap, Hutchison, Cortese, Kessler, Loftis and Treiman2007) and L2 subjects (current study)
*** p < 0.001; ** p < 0.01; * p < 0.05; p < 0.10
a Due to nonnormality, tie-corrected Spearman’s rho (ρ) and Mann-Whitney (U) test are reported for mean accuracy.
CONCLUSION
This study was motivated by the degree to which lexical variables could predict NNS subject data. In total, these analyses were conducted to better understand the intrinsic difficulty of English words for NNS speakers and how this difficulty may affect second language word recognition and acquisition. Findings indicated that contextual diversity, AoA, and contextual distinctiveness measures related to number of word associations and lexical context significantly predicted variance in both NNS RTs and accuracy. In addition, significant interactions were observed between length and contextual diversity, and between length and neighborhood size, for NNS accuracy.
This study demonstrates that contextual diversity, AoA, and contextual distinctiveness may be useful in understanding the intrinsic difficulty of English words for NNS users across a variety of L1 backgrounds. Measures derived from this study could also help mitigate concerns about using L1 baseline data to examine NNS phenomena (Ortega, Reference Ortega, Cook and Wei2016) in that NNS baseline norms would now be available for use alongside L1 norms. Ultimately, more studies involving NNS word norms will contribute to the cumulative development of knowledge regarding NNS lexical access specifically, and the processes of NNS word learning and SLA more broadly. Finally, our post hoc analysis found that while L1 and NNS subjects differed significantly in their lexical decision task performance, they demonstrated shared variance (moderate correlations) across most lexical decision measures, suggesting that differences in L1 and NNS lexical access are more a matter of degree than kind.
The current findings should be interpreted carefully in consideration of limitations. One limitation is our lack of control for L1 background and the potential for cross-linguistic influence and/or nonselective lexical access (cf. Bultena & Dijkstra, Reference Bultena, Dijkstra and Chappele2013; Van Heuven, Dijkstra, & Grainger, Reference Van Heuven, Dijkstra and Grainger1998). For example, it has been shown that NNS lexical decision performance can be facilitated by L1 cognates (Dijkstra, Miwa, Brummelhuis, Sappelli, & Baayen, Reference Dijkstra, Miwa, Brummelhuis, Sappelli and Baayen2010) and L1 orthography effects (Wang & Koda, Reference Wang and Koda2005). A subject-based analysis of these data—specifically, one that includes research participants’ L1 background—would be a crucial next step in exploring this limitation. Another constraint of the dataset is the inability to account for proficiency in NNS speakers. While all our participants were adult NNS English speakers in the United States that were advanced enough to navigate the AMT online platform, it is likely that a range of proficiencies were represented. Future studies of a similar nature would do well to control for both L1 background and NNS proficiency. Another limitation is the population. We had a greater percentage of L1 Spanish speakers who may demonstrate patterns that differ from other L1s (although see Lemhöfer et al. [Reference Lemhöfer, Dijkstra, Schriefers, Baayen, Grainger and Zwitserlood2008] as a counterpoint). Additionally, many participants reported English as their dominant language. These participants, although NNS, may perform in a more nativelike way than NNS that do use English as their dominant language. Also, like all behavioral studies, we are not able to thoroughly control when a participant was potentially distractor or was not paying attention. We partially controlled for this by removing outliers, but some variance in responses may have been missed. Finally, differences in the method of data collection between the current study and Balota et al. (Reference Balota, Yap, Hutchison, Cortese, Kessler, Loftis and Treiman2007) should be acknowledged. Whereas Balota et al.’s research participants were seated in a physical lab and presumably monitored while taking the experiment, the AMT workers who participated in the current study were unsupervised. Even though previous research suggests that AMT workers are reliable (Buhrmester, Kwang, & Gosling, 2011; Enochson & Culbertson, 2015), it is possible that the use of AMT workers introduced some additional variability in the data. Furthermore, the fact that Balota et al.’s participants were university students makes it likely that the age of their research subjects was less variable than the age of AMT workers who contributed to the current study’s lexical decision response data.
SUPPLEMENTARY MATERIAL
To view supplementary material for this article, please visit https://doi.org/10.1017/S0272263119000019
