One of the most fundamental issues in the field of SLA concerns the (non)existence of a critical period for language learning: Can late second language (L2) learners ever achieve nativelike levels of mastery in the L2? The critical period hypothesis (CPH) for language learning was proposed in 1959 by Penfield and Roberts, and later by Lenneberg (Reference Lenneberg1967). In its strongest form, the CPH predicts that late L2 learners cannot achieve a nativelike level of proficiency because language learning is fundamentally different after a certain age (e.g., Bley-Vroman, Reference Bley-Vroman, Gass and Schachter1989; Gregg, Reference Gregg, Ritchie and Bhatia1996; Long & Robinson, Reference Long, Robinson, Doughty and Williams1998). Weaker versions of the CPH acknowledge that L2 learning is affected by age effects, making nativelike attainment less likely for later onsets of learning, but nativelikeness is sometimes possible given the right circumstances (Birdsong & Molis, Reference Birdsong and Molis2001; Bongaerts, Van Summeren, Planken, & Schils, Reference Bongaerts, Van Summeren, Planken and Schils1997). Other versions of the CPH hold that nativelike levels of attainment may be possible but state that the learning mechanisms underlying early and late L2 acquisition are not the same. These versions of the CPH contend that nativelike levels can only be attained through more explicit types of learning because the ability to learn implicitly is gradually lost ( DeKeyser, Reference DeKeyser2000). By now, many studies have been published that address aspects of the CPH (for reviews see DeKeyser, Reference DeKeyser, Gass and Mackey2012; Hyltenstam & Abrahamsson, Reference Hyltenstam, Abrahamsson, Doughty and Long2003), and these studies often include a native speaker (NS) sample. The goal of this study is to evaluate how NSs have been used in CPH research and to assess how the choice of NSs may have affected the results of such studies. First, I review if and how NSs have been used to falsify claims of the CPH, who the NSs in CPH research were, and how variable NS performance may be. Then, I present data demonstrating how the selection of the NS sample may affect the outcomes of CPH research.
One approach to falsification of the CPH does not involve NSs: Many have considered the correlation between age of onset (AoO) and L2 proficiency measures (e.g., Abrahamsson & Hyltenstam, Reference Abrahamsson and Hyltenstam2009; Bialystok & Miller, Reference Bialystok and Miller1999; DeKeyser, Reference DeKeyser2000). A strong correlation (negative or positive, depending on whether accuracy or speed measures are used) means that early onset is related to higher levels of ultimate attainment. However, DeKeyser (Reference DeKeyser, Gass and Mackey2012) pointed out that such correlations are not in support of the CPH unless they show the noncontinuity of the relationship between AoO and proficiency: Before the critical period, the relationship between AoO and achievement should be strong; after the critical period, the effect should cease to exist. DeKeyser’s review showed that, in the vast majority of studies that provide relevant information, the effect of AoO does level off somewhere in puberty. Although this finding is in support of the CPH because it suggests a qualitative change in the ability to learn the L2, it merely proves that L2 acquisition is subject to age effects (DeKeyser, Reference DeKeyser, Gass and Mackey2012).
Another approach to falsification of the CPH is to demonstrate that the proficiency of a particular L2 learner is indistinguishable from that of NSs. In fact, Long (Reference Long1990) suggested that finding one such learner would suffice to falsify the strong version of the CPH, which predicts that nativelike levels cannot be achieved. Abrahamsson and Hyltenstam (Reference Abrahamsson and Hyltenstam2009) reviewed most of the extant CPH work and found considerable variation in the reported incidence rates of nativelikeness, from none (e.g., Johnson & Newport, Reference Johnson and Newport1989) to rates toward 75% of a study’s participant sample (e.g., Birdsong, Reference Birdsong1992). Several researchers have pointed to the fact that differences in design may have caused this variation in outcomes (e.g., Abrahamsson & Hyltenstam, Reference Abrahamsson and Hyltenstam2009; DeKeyser, Reference DeKeyser2013; Long, Reference Long2005). The incidence of nativelikeness obtained in any study is the result of (a) which L2 learners are selected, (b) which target structures and test tasks are used, and (c) which NSs are selected. Not only do the choices that are made in this regard determine the yardstick by which L2 learners’ performance is gauged, but they actually reflect how the construct of nativelike mastery has been operationalized. The selection of L2 learners and study design choices has been subject to discussion; the selection of the NS controls has received little attention in the literature.
When it comes to the first aspect—the selection of the L2 sample—the majority of researchers seem to agree that it is best to select highly advanced learners (e.g., Hyltenstam & Abrahamsson, Reference Hyltenstam, Abrahamsson, Doughty and Long2003; White & Genesee, Reference White and Genesee1996). Abrahamsson and Hyltenstam (Reference Abrahamsson and Hyltenstam2009), who presented probably the most sophisticated CPH study so far, tried to select only those L2 learners whose speech could not be perceived by NS judges as being nonnative. In their review of the literature, they illustrated that many studies explicitly stated that they selected advanced learners and provided information about length of residence, amount of exposure, and some other background variables. However, they also noted that many studies were unclear about how advanced their learners were in terms of L2 proficiency and that many samples may well have included L2 learners who were easily identifiable as nonnative. Long (Reference Long2005) pointed to another crucial requirement with regard to the selection of appropriate L2 learners for CPH research: The L2 should have been acquired clearly past the hypothesized critical period. There still is considerable debate about where the end of the critical period lies, and some have claimed that there are multiple critical periods for different domains of language (Long, Reference Long1990). This is supported by the findings of Granena and Long (Reference Granena and Long2013), who reported discontinuities in the correlation between AoO and performance, first for phonology, then for lexis and collocation, and finally for morphosyntax. In this light, it seems that it is best to select learners well past their midteens, especially if they are tested on multiple aspects of proficiency. An additional concern was raised by Muñoz (Reference Muñoz2008), who points out that it may not be fair to apply the concept of nativelike attainment to L2 learners in instructed settings because the nature of the input received in these settings is such that nativelikeness cannot be expected. In other words, the selection of the L2 sample may also have to be motivated by the nature and the amount of the learners’ experience with the L2.
When it comes to the choice of target structures and test tasks (the second aspect that determines a study’s obtained incidence rate), a crucial question was raised by Birdsong (Reference Birdsong2005): “What is the range of linguistic behaviors that should be considered for falsification of the CPH/L2A [the critical period hypothesis for second language acquisition]?” (p. 322). DeKeyser’s (Reference DeKeyser, Gass and Mackey2012) review shows that the existing body of research is largely limited to grammatical competence and pronunciation. Studies focusing on grammatical competence have often used grammaticality judgment tasks pertaining to one or several rules of grammar. Studies about pronunciation mostly involve holistic ratings of nativelikeness by NSs, although acoustic analysis techniques have also been used (e.g., B. Lee, Guion, & Harada, Reference Lee, Guion and Harada2006). Hyltenstam and Abrahamsson (Reference Hyltenstam, Abrahamsson, Doughty and Long2003) stated that it is not valid to draw conclusions about critical periods on the basis of investigations into just one or two aspects of the L2, and they insisted that L2 learners should be subjected to in-depth linguistic scrutiny, which means that a multitask approach should be used so that many different aspects of L2 proficiency are investigated. This diminishes the possibility that the L2 learner happens to be nativelike in the domain of language that was tested but not in other domains. Additionally, Hyltenstam and Abrahamsson argued that, for statistical reasons, the tests used should not exhibit ceiling effects in NS performance. Differences between NSs and nonnative speakers (NNSs) may then be due to insufficient variation in NS performance. Abrahamsson and Hyltenstam (Reference Abrahamsson and Hyltenstam2009) adopted these recommendations and found that none of the late learners fell within the NS range, but this was also true for many of the early L2 learners.
Several researchers have claimed that criteria for testing the CPH should be derived from theories of language (Birdsong, Reference Birdsong2005; Eubank & Gregg, Reference Eubank, Gregg and Birdsong1999; Hyltenstam & Abrahamsson, Reference Hyltenstam, Abrahamsson, Doughty and Long2003). In studies conducted within the Universal Grammar framework, grammatical structures were selected that required parameter (re)setting for learners of a particular first language (L1; e.g., Johnson & Newport, Reference Johnson and Newport1989). However, this limits the choice to only those features whose development is indeed subject to Universal Grammar. More recently, Hulstijn (Reference Hulstijn2011) suggested a view on language proficiency that can potentially provide a theoretically motivated criterion for falsifying the CPH. He proposed the constructs of basic and higher language cognition. Basic language cognition refers to those features of the language that are shared by all NSs of the language and that can be applied automatically. Anything beyond basic language cognition is higher language cognition and falls outside the realm of what it means to be nativelike. Although the suggestion is interesting, as things stand, the theory cannot be used to test the CPH, as it has not yet been verified empirically.
Decisions about what to test seem to be based mostly on empirical rather than theoretical grounds. Hyltenstam and Abrahamsson (Reference Hyltenstam, Abrahamsson, Doughty and Long2003) advocated a more empirical approach when they claimed that inquiry into the CPH should not be limited to core aspects of grammar. They pointed out that L2 learners should be tested on a wide array of language features to make sure that NNSs are nativelike in each and every domain of the L2, not just in one. Tests should include peripheral and language-specific features, such as knowledge of metaphors and idiomatic constructions, as these are the domains in which NSs and NNSs are likely to differ. Birdsong (Reference Birdsong2005) pointed out, however, that such an approach renders the CPH unfalsifiable. Even if L2 learners pass all tests, there is always the possibility that the one test that can identify them as NNSs is not part of the test battery. This leaves CPH researchers in a paradoxical situation: One can and should test L2 learners on as many aspects of language as possible, but it might never be enough to falsify the CPH.
The selection of the NS controls is the third aspect that determines whether NNSs can perform within the NS range. This aspect seems to have received very little consideration in the literature so far. To be more precise, few have raised concerns about the actual makeup of the NS norm groups, but the practice of comparing NSs and NNSs itself has not remained unchallenged within the CPH debate. The most important objection raised is that CPH research generally involves the comparison of monolingual and bilingual speakers (e.g., Birdsong, Reference Birdsong2005; Cook, Reference Cook and Cook2002; Grosjean, Reference Grosjean1998). There is abundant evidence that bilingualism may affect both the L1 and the L2 systems (see Montrul, Reference Montrul2008) as well as more general cognitive processes (see Bialystok, Reference Bialystok, Kroll and De Groot2005). Because the language systems of bilinguals are connected in intricate ways, it may be impossible by definition for L2 learners to have a L2 system that is identical to that of monolingual L1 speakers. As long as the L1 is active, it will exert some influence on the L2, even if the L2 is the dominant language. In a recent review, Hulstijn (Reference Hulstijn2012) made specific recommendations concerning the selection of NSs for comparison with bilinguals. Most important, he recommended that NSs should not be more advanced in terms of the factors that shape linguistic experience, such as literacy and level of education.
Comparisons of NSs and NNSs have often been undertaken despite concerns about the validity of such comparisons. An inspection of the NS groups used in CPH studies shows that they are generally not in line with Hulstijn’s recommendation. Table 1 lists 35 studies that tried to establish whether NNSs were able to score within the range of a group of NS controls. It documents both the NS and NNS samples used in these studies; for the NSs, it also documents whatever was reported about the background of the NS sample used. Several things stand out. First, 2 studies did not include any NSs; they compared early and late learners only. The remaining 33 studies all made use of a NS group, although some referred to the norms provided by other studies (e.g., Birdsong & Molis, Reference Birdsong and Molis2001; DeKeyser, Reference DeKeyser2000) instead of testing NSs themselves. In cases in which pronunciation was tested, the studies also included NS judges, which are usually small groups of expert language users (e.g., teachers). Second, the NS samples were often quite small: 14 studies made use of a sample of 10 speakers or fewer. Third, and perhaps most important, very little background information was provided in terms of NS literacy levels and levels of education: 21 out of 33 studies (64%) reported the group’s mean age or whether they spoke with or without an accent, but further information about the NSs tested was not given. When information was provided, the NS group was invariably highly educated. It is fair to conclude that NSs were merely mentioned in passing in many studies. Finally, in some studies, NSs were selected to behave according to a certain standard, and speakers were removed from the sample because they did not meet this standard. For example, Johnson and Newport (Reference Johnson and Newport1989) excluded a NS with accented speech, and Bongaerts et al. (Reference Bongaerts, Van Summeren, Planken and Schils1997) excluded speakers who made many errors and exhibited hesitations and pauses in their speech. Bongaerts and colleagues’ decision may have been inspired by a previous experience in which the NS controls were found to perform relatively poorly (Bongaerts, Planken, & Schils, Reference Bongaerts, Planken, Schils, Singleton and Lengyel1995), a finding that was attributed to the fact that the NSs of English spoke with a slight regional accent, whereas the Dutch learners were trained to use Received Pronunciation.
List of CPH studies that compared NS and NNS performance
Note.
AoO = age of onset.
Of all studies reviewed and listed in Table 1, Coppieters (Reference Coppieters1987) is the only study that discussed the NSs at length, acknowledging that NSs may exhibit considerable variation in performance. Coppieters noted that “any study pretending to compare and contrast NS’s [sic] and NNS’s [sic] will have to face the problem of NS variability” (p. 548), and he acted accordingly by trying to select a sample of NSs of French that might be representative of the entire L1 community. He selected speakers from different regions of France and Belgium and tried to include people of lower and higher educational backgrounds. In a footnote to his replication of Coppieters’s study, Birdsong (Reference Birdsong1992) concurred that “the putative ‘ultimate attainment’ of one group of native speakers may be different from that of another group of natives” (p. 707), but he also pointed out that in practice, most studies selected NS controls “that are similar along relevant social and educational dimensions” (p. 707). Coppieters’s study also demonstrates the difficulties of constructing a good comparison group. Most important, it requires decisions about who can provide a fair NS norm. For Coppieters, this included Belgian speakers of French but not Canadian or Swiss speakers.
The practice of NS comparison group sampling as it has been employed up until now is valid only if NSs exhibit very little variation. Until now, NS variation has not been a popular topic of investigation (Dąbrowska, Reference Dąbrowska1997). The reason for this may be rooted in Chomsky’s (Reference Chomsky1965) proposal that the competence to be studied in linguistics is that of the ideal speaker-listener “who knows its language perfectly and is unaffected by such grammatically irrelevant conditions as memory limitations, distractions, shifts of attention and interest, and errors (random or characteristic) in applying his knowledge of the language in actual performance” (p. 3). Although Chomsky acknowledges that sources of individual variation may be rooted in knowledge and the ability to apply that knowledge, many linguists have taken this to mean that NSs have uniform mental grammars and that individual differences have nonlinguistic—and therefore, theoretically uninteresting—causes only (Dąbrowska & Street, Reference Dąbrowska and Street2006). As stated by Dąbrowska (Reference Dąbrowska2012), the assumption of uniform success in L1 acquisition actually forms the basis for claims about a critical period and the fundamental difference between L1 and L2 acquisition, as it has been contrasted with the general failure of L2 learners to become nativelike.
However, insofar as evidence has been accumulated, it has been shown that NSs also vary considerably, even in the domain of grammar. Dąbrowska (Reference Dąbrowska2012) recently reviewed a number of studies on NS variation, some of which explicitly challenged the notion of a uniform NS competence, and the results supported the notion of experience-based individual differences (Dąbrowska, Reference Dąbrowska1997; Dąbrowska, Reference Dąbrowska2008; Dąbrowska & Street, Reference Dąbrowska and Street2006; Ferreira, Reference Ferreira2003; Street & Dąbrowska, Reference Street and Dąbrowska2010). The common denominator in these studies was the testing of heterogeneous samples of adult NSs (in terms of educational level) on their knowledge of fairly common, but grammatically complex, structures such as passives (Dąbrowska & Street Reference Dąbrowska and Street2006) or the Polish genitive inflection (Dąbrowska Reference Dąbrowska2008). The studies all attest to considerable differences between NSs in knowledge of the constructions under investigation. Dąbrowska (Reference Dąbrowska2012) suggests that the results are probably caused by differences in quantitative or qualitative experiences with processing linguistic texts, due to the participants’ different educational backgrounds. In other words, NSs exhibit variation due to differences in quality and quantity of the input—that is, for the same reasons that NNSs do.
In light of such variation, the standard practice of NS norm group sampling in CPH research may be a cause for concern. In this study, several tasks with high- and low-frequency items were administered to 113 fairly advanced L2 learners as well as a heterogeneous sample of 120 NSs to compare the results of the NNSs against two different samples of NSs that differed depending on educational background. The data that are presented in this study were not gathered with the intention of falsifying the CPH and are not fit to do so for some of the reasons discussed in this introduction: The L2 learners were probably not all highly advanced L2 users, and the tasks were not designed to test the CPH. The goal of the present study was to illustrate that the use of nonrepresentative NS norm groups affects the observed incidence of NNSs falling within the NS range. Additionally, I investigated how the use of basic versus more peripheral task items (in terms of frequency) affects these incidence rates. The expectation was that incidence rates drop as the educational level of the norm group increases and that incidence rates drop when low-frequency test items are used to provide the NS range.
METHOD
Participants
Native and nonnative speakers of Dutch were compared in this study; they were recruited in and around the city of Amsterdam through advertisements posted in several educational institutes, supermarkets, and community centers and through networks of relatives and friends. Selection was based on level of education (LoE; vocational or higher) and age (approximately between 20 and 35). The NS sample consisted of 124 participants (85 females, 39 males; 63 of low LoE, 61 of high LoE) ranging from 19 to 40 years old (M = 25, SD = 5.2). The low LoE participants were actively enrolled in vocational education or had already completed their studies, and some of the occupations present in the sample were house painter, gardener, mechanic, secretary, and train driver. The high LoE participants were actively enrolled in or had completed education at the bachelor’s level (27 participants) or master’s level (34 participants); in two cases these were language related. The high LoE participants held jobs such as managers, financial analysts, pedagogy counselors, and journalists. Although the NS group was rather heterogeneous in terms of occupations and educational backgrounds, the sample cannot be considered an accurate reflection of Dutch society. The lowest types of education were still underrepresented, and the sample was not randomly drawn.
The NNS sample consisted of 118 participants (80 females, 38 males; 52 of low LoE, 66 of high LoE) ranging from 19 to 40 years old (M = 29, SD = 5.3). Twelve participants were studying (or had studied) in a language program. As described before, the participants were not selected to be nativelike in Dutch; they varied in Dutch proficiency from B1 on the Common European Framework of Reference (Council of Europe, 2001) scales to C2, which means that they ranged from intermediate levels to nativelike levels of proficiency. Participants were asked to judge their own proficiency in reading, writing, listening, speaking, grammar, and vocabulary on a 5-point Likert scale, yielding a summed maximum score of 30. The mean score was 21.5 (SD = 4.7). Seven participants gave themselves 30 out of 30 points and rated themselves as nativelike. The mean age at arrival in the Netherlands ranged from 0 to 36 years old (M = 21; SD = 8.1), and length of stay in the Netherlands varied from 8 months to 27 years (M = 8 years; SD = 6.1 years). Thirty-five different L1s were present in the NNS sample, which included German (9 speakers), Russian (9 speakers), Bahasa Indonesian (9 speakers), and Spanish (8 speakers).
All participants were financially compensated. Participants with hearing problems and people using medication that might impair their ability to perform reaction-time tasks were excluded from the study. All participants signed a consent form.
Tasks and Materials
Participants performed a range of tasks in a fixed order of administration; these included tasks not relevant for the present study, such as measures of working memory and intelligence. The tasks were mostly administered in two sessions of 2 hr on separate days with short breaks between the tasks, although some participants did all tasks in 1 day, with a long break between the two sessions. All tasks were newly developed, and their quality was assessed on the basis of pilots with 27–51 NSs and NNSs (this varied per task). Another goal was to make sure that the tasks were suitable for both groups. None of the tasks were designed with falsification of the CPH in mind, although vocabulary and grammaticality judgment tasks have regularly been used in CPH research. All tasks are also described in Andringa, Olsthoorn, Van Beuningen, Schoonen, and Hulstijn (Reference Andringa, Olsthoorn, Van Beuningen, Schoonen and Hulstijn2012) and Olsthoorn, Andringa, and Hulstijn (Reference Olsthoorn, Andringa and Hulstijn2012).
Segmentation (Word Recognition)
The segmentation task was designed to test to what extent participants were able to recognize common words in normal speech. The test consisted of two parts. In the first part, participants heard short fragments of speech, and they were instructed to indicate as quickly and accurately as possible how many words they had heard by pressing the corresponding number on the keyboard. The test consisted of five practice trials and 45 experimental trials presented in random order and with 2 s intervals. The speech fragments consisted of two, three, and four words, but participants could press the numbers 1 to 5. In the second part of the test (not reported), participants listened to the same trials again and had to identify which words they heard by typing them out. Twenty-two fragments were fully articulated, and the articulation of the other 23 was reduced according to the principles of vowel reduction or consonant reduction (or both) as described in Ernestus (Reference Ernestus2000), Kloots, De Schutter, Gillis, and Swerts (Reference Kloots, De Schutter, Gillis, Swerts, Koole, Nortier and Tahitu2003), and Coussé, Gillis, and Kloots (Reference Coussé, Gillis and Kloots2007). These were typical of normal colloquial Dutch (compare He has not and He hasn’t in English). Fragments always consisted of highly common words only: If the task was difficult, it was due to the reductions. All fragments were spoken by a female NS of Dutch with a neutral accent who was recorded in a soundproof studio; the recording was digitized at 44 kHz (16-bit quantization). The task was set up and run with the E-Prime software package (Schneider, Eschman, & Zuccolotto, Reference Schneider, Eschman and Zuccolotto2002). Both speed (response time from fragment offset) and accuracy (number of trials correctly answered) were logged.
Grammatical Processing
The grammatical processing task assessed participants’ knowledge of word-order properties of the Dutch language. Participants heard speech fragments of three to four words long and had to indicate whether the string was permissible at the beginning of a sentence by pressing yes or no as fast as possible. The task consisted of four practice trials and 34 experimental trials that were presented in random order and with 2 s intervals. Half of the stimuli constituted grammatically correct sentence beginnings, and the other half of the stimuli were ungrammatical and thus cannot occur in sentence-initial position. For instance, stimuli such as Ze hebben overal “They have everywhere” and Die stad lijkt heel “That city seems very” required a yes response, whereas Precies ik weet “Exactly I know” and Er altijd moet iemand “There always must somebody” required a no response. The stimuli were spoken at a normal pace by a female NS of Dutch with a neutral accent who was recorded in a soundproof studio; the recording was digitized at 44 kHz (16-bit quantization). The task was set up and run with the E-Prime software package (Schneider et al., Reference Schneider, Eschman and Zuccolotto2002). Both response accuracy and response speed (delay between stimulus onset and response) were logged.
Semantic Processing
The semantic processing task was aimed at assessing the ability to comprehend the semantic and pragmatic meaning of full sentences. The task contained 60 trials: four practice trials and 56 test trials. Each trial consisted of a spoken proposition and two short response alternatives shown on the computer screen; one was a semantically and pragmatically meaningful response to the spoken proposition, and one was an inappropriate response. The participants’ task was to indicate as quickly as possible which was the appropriate response by pressing the associated key. The response alternatives were visible on screen for 3 s before onset of the spoken proposition and remained visible until a decision was made for either. The interval between a response and the presentation of the next stimulus was 2 s. The propositions were spoken at a normal pace by a female NS of Dutch with a neutral accent who was recorded in a soundproof studio; the recording was digitized at 44 kHz (16-bit quantization). The task was set up and run with the E-Prime software package (Schneider et al., Reference Schneider, Eschman and Zuccolotto2002). Both response accuracy and speed were recorded. Response speed was computed as the interval between proposition onset and response-key press.
The spoken propositions varied along three parameters: length (between 7 and 12 words or between 13 and 23 words), syntactic complexity (with or without a subordinate clause), and content word frequency. In this study, only analyses pertaining to frequency will be presented: The high- and low-frequency stimuli were balanced in length and syntactic complexity. To determine frequency, the three least frequent content words of the sentence were identified in the CELEX corpus (Baayen, Piepenbrock, & Gulikers, Reference Baayen, Piepenbrock and Gulikers1995). The CELEX database is a corpus of written Dutch that consists of 124,000 lemmas of Dutch, with frequency information based on a corpus of written Dutch of 50 million word forms. A sentence was highly frequent if its three least frequent words occurred an average of more than 30 times per million words. For low- frequency sentences, the mean frequency of the content words had to be lower than 30 times per million. The response alternatives that participants chose from always consisted of highly common phrases, maximally four words long, as can be seen in the following examples:
(1) Proposition (short, simple, frequent):
Gaat de bus naar Amsterdam nog wel?
“Is the bus to Amsterdam still in service?”
Reactions:
Nee sorry / Goed idee
“No sorry” / “Good idea”
(2) Proposition (short, simple, infrequent):
Uit die reactie beluister ik onwil om mee te werken.
“In this response I detect reluctance to cooperate.”
Reactions:
Nee sorry / Ik ook
“No sorry” / “I do too”
Word Monitoring
This task tested participants’ ability to use their knowledge of the distributional and combinatorial properties of the Dutch language to predict upcoming information. In each trial, a target word to be monitored appeared on the screen for 1 s, and participants were instructed to remember that word. Subsequently, a carrier sentence was played, and participants were instructed to press the space bar as soon as they heard the target word. The monitor task consisted of 46 experimental trials and 4 practice trials, and they were presented in random order with 2 s intervals between each trial. The sentences varied in length from 7 to 17 words; sometimes they consisted of main clauses only and other times of main clauses with coordinate, subordinate, or relative clauses. The position of the word to be monitored in the carrier sentence varied from 2nd to 14th position. The words to be monitored were nouns (n = 23), verbs (n = 8), adjectives (n = 6), prepositions (n = 5), and adverbs (n = 4). All carrier sentences consisted of simple, high-frequency words. The propositions were spoken at a normal pace by a female NS of Dutch with a neutral accent who was recorded in a soundproof studio; the recording was digitized at 44 kHz (16-bit quantization). The task was set up and run with the E-Prime software package (Schneider et al., Reference Schneider, Eschman and Zuccolotto2002). Response time was logged from the onset of the word to be monitored. Participants could press before the word actually appeared, which means that negative values could and did occur.
Self-Paced Listening
The self-paced listening task was a measure of lower level sentence processing efficiency. The task was set up in accordance with the specifications provided by Ferreira, Henderson, Anes, Weeks, and McFarlane (Reference Ferreira, Henderson, Anes, Weeks and McFarlane1996). Participants listened to sentences one word at a time by pressing the space bar to hear the next word. If the space bar was pressed before the end of the word, it was truncated, and the next word was played. A third of the sentences were followed by a simple yes-no comprehension question to encourage participants to execute the task faithfully. Sentences were presented in random order. Participants were instructed to be as fast and accurate as possible. The task consisted of 56 trials, and participants were given four trials to practice. The sentences were spoken at a normal pace by a female NS of Dutch with a neutral accent who was recorded in a soundproof studio; the recording was digitized at 44 kHz (16-bit quantization). The task was set up and run with the E-Prime software package (Schneider et al., Reference Schneider, Eschman and Zuccolotto2002). Response times were measured from the offset of each fragment within a sentence, and for each sentence, an average response time was calculated. Responses to sentence-initial and sentence-final fragments were removed because these tend to be highly variable.
The stimulus materials consisted of 56 grammatical sentences of Dutch. As in the semantic processing task, sentences varied in length, complexity, and frequency, and these parameters were operationalized according to the same criteria. The utterances were read word by word by a female NS of Dutch with a neutral accent who was recorded in a soundproof studio; the recording was digitized at 44 kHz (16-bit quantization). Although sentences were presented and recorded word by word, care was taken to retain the Dutch intonation contours as much as possible.
Receptive Vocabulary Knowledge
The vocabulary test was a computer-administrated, receptive multiple-choice test that measured passive knowledge of vocabulary. The test was based on a selection of items from Hazenberg and Hulstijn (Reference Hazenberg and Hulstijn1996), a test originally developed for speakers of Dutch as a L2. Target vocabulary words were presented in a carrier sentence from which the meaning of the target word could not be deduced. Participants had five options to choose from, the last one always being “I really don’t know.” New low-frequency items were added to make the test suitable for NSs. They were constructed according to the same principles as those employed by Hazenberg and Hulstijn. First, target words were not jargon and did not represent specialized fields of knowledge. Also, items were created so that it should have been possible to capture the meaning of the target words in simple, high-frequency language and so that it was possible to present the target word in a simple carrier sentence. Care was taken not to introduce any systematicity in the length of the options and the way in which meanings were described.
The test consisted of 60 items. The target words were selected on the basis of frequency information from CELEX (Baayen et al., Reference Baayen, Piepenbrock and Gulikers1995), and they gradually decreased in frequency. For the purpose of this study, frequency was operationalized as a two-level factor by separating the 30 highest frequency items (once or more per million/42 occurrences or more in the total corpus of 50 millions words) from the 30 lowest frequency items (once or less per million/40 or fewer occurrences in the total corpus). Examples of high-frequency items are glanzen “to shine,” traject “trajectory,” and toelage “allowance”; examples of low-frequency items are oploop “gathering,” “stir,” affreus “hideous,” and hagiografie “hagiography.”
Statistical Analyses
Following a commonly used procedure in CPH research, I ascertained the incidence of NNSs falling within the NS range of scores. In previous studies, two procedures have been used to determine the NS range. One is simply the use of the observed statistical range, which means that the NNS score should be between the lowest and the highest NS score. Alternatively, the range has also been based on the mean and the standard deviation, whereby a NNS score should fall within a particular number of NS standard deviations from the NS mean to count as nativelike. Some have preferred the use of the statistical range (based on minimum and maximum scores), as it tends to be more inclusive (Abrahamsson & Hyltenstam, Reference Abrahamsson and Hyltenstam2009), but this measure has two disadvantages. First, one person—the slowest or the lowest scoring NS participant—ultimately sets the standard. Good performance on a task is hardly ever the result of coincidence, but poor performance may well be due to factors that are irrelevant to the measurement, such as attention lapses caused by any number of personal reasons or an off day, among others. Such factors should not determine the threshold. Additionally, the statistical range is sensitive to sample size: It tends to increase as the sample size grows. For these reasons, I adopted a range based on the mean score and the standard deviation. Also, I removed and replaced clearly outlying responses to individual trials. The threshold value beyond which I considered a NNS score within the NS range was the mean plus (for speed tasks) or minus (for accuracy tasks) 2 NS standard deviations. However, if this value exceeded the minimum or maximum observed value, it was replaced by that value, so that the threshold never fell outside the range of observed NS values. The obtained threshold was always rounded upward for accuracy tasks and downward for speed tasks: If the accuracy mean minus 2 standard deviations was 26.3, then scores of 27 and higher were considered within the NS range.
The first step in the analyses, then, was to inspect the scores for outliers. For the segmentation task, one NS was removed because he or she scored more than 40% incorrect on the fully articulated items. This was taken to mean that this person had not understood the task: In a second run of the task, participants were asked to type in the exact words heard, and no mistakes were made by this person on the fully articulated items. Nonnative speakers were also removed occasionally because of clearly outlying responses or because their data were not correctly logged (five NNSs were removed from the segmentation results, four from the grammatical processing results, two from the semantic processing results, and two from the word monitoring results). For the speed tasks (segmentation speed, grammatical processing speed, semantic processing speed, self-paced listening, and word monitoring), outliers on individual trials were identified. Responses were considered outliers if they fell outside the range of 2.5 standard deviations from the group mean for that item (this was done separately for NSs and NNSs), in which case they were coded as missing. This never constituted more than 2.4% of the data points in any task. Additionally, latencies of inaccurate responses were considered invalid and were also set to missing for segmentation speed, grammatical processing speed, and semantic processing speed (for the NSs: 12.8%, 3.6%, and 1.7%, respectively, and for the NNSs: 16.5%, 13.1%, and 6.7%). All missing data points were then newly estimated by imputation in SPSS by means of the full information maximum likelihood estimation procedure, and mean scores per participant were calculated.
To investigate how the NS sample affects the observed rate of NNSs falling within the NS range, representative and nonrepresentative NS norm groups were needed. For the nonrepresentative sample (NRS) norm, all master’s-level graduates in the NS sample were selected, totaling 34 participants. For the representative sample (RS) norm, I attempted to construct a sample that would be a fair reflection of the spread of educational levels in Dutch society. According to data gathered by the Centraal Bureau voor de Statistiek (CBS; 2011), 9% of the Dutch labor force possessed a master’s-level degree in 2011; 18% possessed a bachelor’s-level degree, and 73% possessed a vocational degree or no degree (the CBS distinguishes as many as seven subcategories within the vocational level, but these were not identifiable in the data). Thus, the educational backgrounds in the representative sample were equally proportioned and consisted of all lower level participants (58 participants), supplemented with 13 randomly drawn bachelor’s-level participants (from a total of 27) and 6 randomly drawn master’s-level participants (from a total of 34). Consequently, the representative sample norms were based on 77 participants. The data from master’s-level participants were present in both the RS and NRS norms.
The two NS samples were compared for mean and standard deviation differences by means of one-way ANOVAs and Levene’s test for equality of variances: Means and standard deviations determined the actual span of the RS and NRS ranges against which NNSs were tested. For the NNSs, I simply determined for each task whether their score fell inside or outside the NS RS and NRS ranges, which resulted in two nominal variables (inside or outside the RS and NRS) for each score. I ran exact McNemar tests and a chi-square test for related pairs (there were two decisions for each participant: one based on the RS norm and one based on the NRS norm) to assess whether the NRS norm excluded significantly more NNSs as nativelike than did the RS norm, which was the hypothesis. Figure 1 illustrates how the McNemar test works. It determines whether the probability of b is equal to the probability of c, in which the letters refer to observed cell frequencies. Thus, the test considers only those observations that fall inside the RS range and outside the NRS range and vice versa; it ignores those observations that are included or excluded by both norms.
Example contingency table.
RESULTS
Table 2 displays the descriptive statistics for the RS and the NRS NS norm groups on all measures. One-way ANOVAs were used to test for differences between the two groups. The table shows that significant differences in mean scores were observed for the four accuracy tasks. The NRS group was better on these tasks. For the speed tasks, no mean differences were observed, with the exception of segmentation speed, in which the NRS group was faster. Additionally, differences in standard deviations were observed for grammatical processing accuracy, segmentation speed, and word monitoring as indicated by violations of Levene’s test for equality of variances. In all cases, the NRS group displayed less variation in performance. For semantic processing speed, grammatical processing speed, and self-paced listening, no differences were observed in either mean scores or variances.
Descriptive statistics, threshold values, and mean and standard deviation comparisons for the representative sample (RS) and nonrepresentative sample (NRS) native speaker groups
Note.
Min = minimum; max = maximum; diff. = difference.
To test whether the incidence rates were affected by the nature of the NS norm group, the range of performances of the NNS sample was compared with a RS and a NRS NS norm. For each task, I determined how many NNSs scores fell inside or outside the RS and NRS range (Table 3). For vocabulary, Table 3 shows that 63 L2 learners were excluded by both norms, and 29 learners were included by both. However, 22 learners (19%) fell outside the NRS norm but inside the RS norm. This suggests that it is easier to meet the norm set by the RS than the norm set by the NRS. Although the numbers differed from task to task, this pattern of results was obtained for all of the tasks. McNemar tests were performed to assess whether the observed distributions may be considered significantly different from one another. This was true for all measures, even for those measures for which I did not observe significant mean or variance differences.
Observed incidence of NNSs falling inside or outside the representative sample (RS) and nonrepresentative sample (NRS) ranges based on the mean +/− 2 SDs
The second hypothesis predicted that fewer L2 learners would fall within the NS norm if the norm were based on the low-frequency task items only. For these analyses, only the results for vocabulary, semantic processing accuracy and speed, and self-paced listening were included in the analyses, as they were the only tasks that were manipulated for frequency. Table 4 shows the descriptive statistics for NSs, split out by norm group and frequency.
Descriptive statistics, threshold values, and mean and standard deviation comparisons for the representative sample (RS) and nonrepresentative sample (NRS) native speaker groups, split by frequency
Note.
Max = maximum; min = minimum; n.a. = not applicable.
Table 5 shows whether the L2 learners fell inside or outside the NS norm as set by low-frequency and high-frequency task items. For vocabulary, 25 learners fell outside and 35 fell inside the RS NS range, irrespective of whether the range was calculated on the basis of high-frequency items only or low-frequency items only. Two learners fell inside the high-frequency range but outside the low-frequency range, whereas 52 learners fell outside the high-frequency range but inside the low-frequency range. The McNemar test confirmed that L2 learners were significantly more likely to fall within the NS range on the low-frequency items. The same result was found for the semantic processing speed task when the NNSs were compared against the NRS norm. All other McNemar p values were not significant. Thus, the second hypothesis was not confirmed. If anything, the opposite was true.
Observed incidence of NNSs falling inside or outside the high-frequency (HF) and low-frequency (LF) ranges, split by norm group
DISCUSSION
This study was conducted to investigate how the selection of NSs in CPH research may affect decisions about the level of nativelikeness of NNSs. The literature review has shown that many CPH researchers do not provide much information about their NS sample or have not sampled NSs from a range of academic and vocational backgrounds. In CPH research, significant effort has been invested in describing the NNS sample, whereas the NS sample has received little attention in terms of text devoted to describing the sample. As a result, for many studies reviewed, it was impossible to determine who the NSs were exactly. In studies that provided more detail, the NSs often turned out to be highly educated. The data presented here showed that means and standard deviations on rather unchallenging measures of language comprehension tended to differ significantly between the nonrepresentative, highly educated sample and a more representative sample in terms of educational level. Native speaker means and standard deviations determine the range against which NNSs are compared, and therefore they affect the observed incidence rate of nativelikeness. In this study, I found consistent confirmation of the hypothesis that the use of a representative NS sample lowers the threshold of nativelike performance. This was true even for measures for which I did not find significant mean or standard deviation differences between the RS and the NRS.
Whereas the first hypothesis was confirmed, the second was not. Incidence rates were not lower when the range was based on low-frequency test items. In fact, insofar as significant effects were observed, they pointed to the opposite. Scores based on high-frequency materials tended to exclude more NNSs from the NS range than scores based on materials of lower frequency. This was probably caused by the fact that there tended to be less variance among the NSs in the scores based on high-frequency item sets. Less variance meant smaller standard deviations, which in turn led to a smaller range and a higher threshold. In the low-frequency item sets, the NSs varied much more and had larger standard deviations. Therefore, the NS range was much wider and the threshold that NNSs had to meet was consequently lower. The effect was strong in the vocabulary task, on which 50 out of 114 NNS participants fell outside the NS RS range based on the high-frequency items but inside the range based on low-frequency items. For this task as well as for the semantic processing speed task, no ceilings were observed. The effect was not observed for semantic processing accuracy, which is probably due to the fact that NS ceiling performance was observed not only on the high-frequency items but also on the low-frequency items. It was also not found for self-paced listening; for this task, there were simply no differences in means and standard deviations between the RS and the NRS or between high- and low-frequency scores.
As stated in the introduction, whether a NNS score falls within the NS range of scores is not just the result of which NSs are selected. It is also determined by which aspects of proficiency are tested as well as by the characteristics of the NNS sample. Given that the NNSs were not selected to be near-native, the observed incidence of nativelikeness was rather high in this study: On several tasks, more than half of the NNSs were able to score within the NS range. This was caused entirely by the comprehension tests that were used. They were developed to gauge the listening comprehension abilities of both NSs and NNSs (Andringa et al., Reference Andringa, Olsthoorn, Van Beuningen, Schoonen and Hulstijn2012; Olsthoorn et al., Reference Olsthoorn, Andringa and Hulstijn2012). As can be judged from the examples given in the Tasks and Materials section, the lexical content of the lower frequency items was probably unproblematic for many relatively advanced L2 learners, with the exception of the vocabulary test, for which no ceilings were observed, not even for the NSs.
A few questions remain to be addressed. For example, would these results have been obtained if tests had been used that are more appropriate to test the claims of the CPH or if truly near-native learners had been selected? These questions are difficult to answer, but the effects of LoE are not unique to this study; they were also observed in many of the studies on grammatical competence reviewed by Dąbrowska (Reference Dąbrowska2012). Thus, it seems likely that they also appear when other language measures are used. One could argue that the LoE effect was due not to differences in language ability but to differences in the ability to deal with the demands of the task. However, given the simplicity of the tasks used in this study, claiming that these findings could be the result of task effects only is probably a bit far-fetched. Similarly, it seems reasonable to assume that NSs will perform more homogeneously on simple, high-frequency language materials, irrespective of what task is used or what domain of language is tested. Whether that will always result in lower incidence of nativelikeness rates for high-frequency materials is an intriguing question that needs to be tested empirically. It also remains to be seen whether this works with highly advanced learners.
Hyltenstam and Abrahamsson (Reference Hyltenstam, Abrahamsson, Doughty and Long2003) have made several recommendations concerning the design of CPH studies. Most important, they recommended that highly advanced learners be subjected to linguistic scrutiny that involves “large sets of elicitation techniques for varied aspects of proficiency” (p. 577). This approach safeguards against claims of nativelikeness on the basis of excellent performance in just one of the many domains of language proficiency. However, they added the recommendation that ceiling effects should be avoided for NSs and NNSs. The results of this study suggest that ceiling effects in NS performance may actually be helpful in identifying nonnative performance. In fact, one might argue that ceiling performance is what constitutes nativelikeness. In CPH research, the objective has been to test whether L2 learners know and can do as much as NSs. However, by selecting highly educated NSs, we may often have been testing at levels of proficiency that not all NSs reach. This approach is illustrated in Abrahamsson and Hyltenstam (Reference Abrahamsson and Hyltenstam2009). They developed highly challenging tests of phonological, grammatical, and lexical ability, tests on which their sample of 15 NSs were not able to perform at maximum levels. The 50-item test assessing knowledge of proverbs—on which the NSs scored a mean of 39 correct answers, ranging from 33 to 46—turned out to be the most difficult for NNSs: No more than 15% (6 speakers) scored within the NS range. Lost in this comparison of total scores is information about who knew what. On average, 11 proverbs were not known by the NSs. Were these always the same proverbs? More important, were there proverbs that were consistently known by all NSs? A comparison of that set of proverbs might constitute a better test of nativelikeness; the results of the present study suggest that even fewer NNSs would fall within the NS range.
Perhaps Hulstijn’s (Reference Hulstijn2011) notions of basic and higher language cognition are helpful tools in solving the paradox outlined in the introduction of how many tests suffice to conclude that someone is or is not nativelike. A tentative answer to Birdsong’s (Reference Birdsong2005) question of which areas of language to consider for falsification of the CPH might be those areas in which NSs do not vary: basic language cognition, which is a finite domain, according to Hulstijn. The results of the present study may be construed as support for Hulstijn’s theory in that they show that there are domains of language cognition in which a fairly heterogeneous group of NSs perform fairly homogenously as well as domains that are characterized by much more variation. As already noted, Hulstijn’s theory has not been empirically tested, but perhaps a practical approach can be adopted in the effort to falsify the CPH. This approach would copy most of the recommendations made by Hyltenstam and Abrahamsson (Reference Hyltenstam, Abrahamsson, Doughty and Long2003), including the recommendation of investigating nativelikeness on many different aspects of L2 proficiency and in domains that are expected to be problematic for the L2 sample (e.g., because of the differences between the L1 and the L2). Crucially, however, the tasks should be demanding for the NNSs only, not for the NSs. As suggested earlier, this can be achieved by eliminating those items on which NSs exhibit variable performance.
The results of the present study have shown that the incidence of NNSs falling within the NS range is affected by the selection of the NSs. Who, then, should be selected? Ultimately, this is a theoretical question, too, not an empirical one. As such, the results of this study do not provide criteria for the selection of NSs. The common practice seems to have been to compare NNSs with NSs who are generally highly educated. This may be fair, but probably only if the NNSs are comparable to the NSs in terms of the linguistic contexts in which they function. This would suggest that NSs and the NNSs should be matched on relevant background characteristics, as was done in Abrahamsson and Hyltenstam (2009). However, this leaves room for the possibility that none of the L2 learners will be found to be fully nativelike, although many of them may actually be more proficient language users than many NSs are. Therefore, there is something to be said for Coppieters’s (Reference Coppieters1987) approach: Select a NS sample that is truly representative of the L1 speech community (although this triggers the sociolinguistic problem of defining a speech community). The NS sample in the present study consisted of speakers from the Amsterdam area, including one or two second-generation immigrants. Should I have included Dutch-Frisian bilinguals and Flemish speakers of Dutch? Such questions are not easy to answer.
In a recent methodological treatment, DeKeyser (Reference DeKeyser2013) acknowledged that NS variation has not been considered much in CPH research. He recommended the use of a homogeneous NS sample: “When there is substantial variation among native speakers, then the native and nonnative ranges of variation are almost bound to overlap substantially, whether for the same reasons or not, yielding trivially predictable results” (p. 58). He also recommended that the L1 and the L2 be distant languages so that one could compare them on more central (rather than peripheral) structural features of the target language. DeKeyser’s recommendations are well worth noting but also raise questions. The NS sample should be homogeneous, but which NSs should be selected? Should they be highly educated and highly experienced language users? When is overlap between NS and NNS performance trivial, and when is it meaningful? Which languages are sufficiently different, and which structures are sufficiently central? Ultimately, these questions cannot be answered on strictly empirical grounds. We still lack a sufficiently comprehensive theory of what it means to master a language as a NS. Without that, we cannot decide on what aspects of the language NNSs should be tested, and we cannot know which NSs should be selected as the norm group. When it comes to selecting the NS norm group, we have seen that the NS samples used in CPH research have been quite small, whereas the NNS samples are often quite large. It should be the other way around. Long (Reference Long1990) has claimed that it may be enough to identify just one NNS who is nativelike in every respect to falsify the CPH. If that is true, then we do not need large NNS samples. What we really need is a sizeable and well-described NS norm group.
