Essential to fluent language use is the automatic execution of various component subskills (McLaughlin, Reference McLaughlin1987). However, skills such as language are cognitively complex. They consist of a hierarchically organized set of subskills, in which automatic and attention-controlled processes can coexist at different levels (McLaughlin, Reference McLaughlin1987; Segalowitz & Segalowitz, Reference Segalowitz and Segalowitz1993). Skill acquisition theory thus views increases in language proficiency, and especially in language fluency, as the transition from more controlled to more automatic processes. Automaticity is a cognitive basis of second language (L2) fluency (Segalowitz, Reference Segalowitz2010); it plays “a significant role in distinguishing fluent from nonfluent abilities” (Segalowitz & Hulstijn, Reference Segalowitz, Hulstijn, Kroll and De Groot2005, p. 373). Thus, this study will investigate the development of L2 automaticity as evidenced in native and nonnative speakers’ fluent performance of two different sentence processing tasks.
The abstract notion of automaticity has attracted the interest of many second language acquisition (SLA) scholars because a high degree of L2 fluency is the ultimate goal for L2 learners. In different SLA theories, the transition from explicit to implicit knowledge (Bialystok, Reference Bialystok1978), from declarative to procedural knowledge (Anderson, Reference Anderson1983), and from nonintegrated to integrated knowledge (Jiang, Reference Jiang2007) all seem to depend on the extent to which L2 linguistic processes become automatized. For instance, implicit knowledge is often described as knowledge accessible by means of automatic processing (e.g., Dörnyei, Reference Dörnyei2009; R. Ellis, Reference Ellis2005), and similarly, procedural knowledge is defined as a resource available for unconscious language use (e.g., Anderson, Reference Anderson1983). Integrated knowledge is also considered synonymous with automatic competence in adult SLA (e.g., Jiang, Reference Jiang2007). Though different scholars have used different terms, the common element that discriminates types of knowledge seems to be the presence or absence of automatic processing in learners’ use of the language.
In view of the prominent role of automaticity in SLA, it is surprising how little literature on learners’ automatic processing in the L2 there is. Many important questions have remained unanswered as to how L2 automaticity develops over the course of language learning, how long it takes for L2 learners to reach the fully automatized phase, how the development of automaticity varies depending on linguistic features, or how automaticity can be validly measured. The present study is particularly interested in a methodological issue: how to assess the development of automaticity in L2 sentence processing. Although automaticity research on L2 word recognition has steadily accumulated since the 1980s, only a few studies have attempted to measure automatic L2 sentence processing (DeKeyser, Reference DeKeyser1997; Hulstijn, Van Gelderen, & Schoonen, Reference Hulstijn, Van Gelderen and Schoonen2009; Jiang, Reference Jiang2007; Robinson, Reference Robinson1997; Rodgers, Reference Rodgers2011; Serafini, Reference Serafini2013). The current study continues and expands the latter line of inquiry by extending a measure typically used to assess automatization of L2 word recognition to the sentence processing level with written stimuli.
BACKGROUND
Automaticity in SLA
In cognitive science and psychology, automatic processing has been defined traditionally as relatively rapid, effortless, unconscious, attention-free, ballistic (unstoppable), uncontrollable, stereotypic, and parallel processing. The study of SLA explores the same properties of automaticity, yet focuses on linguistic processes: how L2 users map forms to meanings during lexical retrieval, how they acquire and apply grammar rules, and how they become skilled at communicating in the L2. Given the complex, hierarchical nature of all of these activities, some linguistic subprocesses may conceivably be more amenable to automatization than others. In reading studies, for instance, the importance of efficient (i.e., automatic) word recognition and syntactic parsing has been accentuated for good reading comprehension (Breznitz, Reference Breznitz2006; Grabe & Stoller, Reference Grabe and Stoller2011; Koda, Reference Koda2005, Reference Koda2007; Van Gelderen, Schoonen, Stoel, De Glopper & Hulstijn, Reference Van Gelderen, Schoonen, Stoel, De Glopper and Hulstijn2007). Segalowitz and Hulstijn (Reference Segalowitz, Hulstijn, Kroll and De Groot2005) posit that “it is especially at the intermediate levels of syntactic, morphological, and phonological encoding/decoding, as well as at the lower levels of articulation and perception of acoustic or orthographic signals, that component processes can become automatic to a large extent” (p. 385). Automatization studies and research on the four communicative skills (i.e., reading, writing, listening, speaking) can thus cross-fertilize each other, as a good understanding of automatization requires a theoretical model of the skill being automatized. However, few researchers have attempted to bridge this disciplinary divide.
One approach that holds potential in this regard is Segalowitz and Segalowitz’ (Reference Segalowitz and Segalowitz1993) coefficient of variation (CV). CV is a measure of processing stability (to be defined below) that can in principle be applied to any type of human or nonhuman behavior. It is important to note that the CV is also compatible with more prevalent approaches to automatization in SLA, such as Anderson's adaptive control of thought (e.g., Anderson, Reference Anderson1976, Reference Anderson1983, Reference Anderson1993, Reference Anderson2007; DeKeyser, Reference DeKeyser1997) and Logan's instance theory (Logan, Reference Logan1988, Reference Logan2002; Robinson, Reference Robinson1997). These theories emphasize that (a) automatization necessitates a qualitative change either from declarative to procedural knowledge (Anderson) or from algorithm-based performance to memory-based performance (Logan), (b) automatization results from intensive and extensive practice, (c) automatization requires strong linguistic knowledge, and (d) automatization follows the power law of practiceFootnote 1 (Anderson, Reference Anderson1992, p. 172; Logan, Reference Logan1988). Although characteristics (b) through (d) seem readily applicable to the study of communicative skills, it is less clear what a qualitative change in processing would entail in this area.
According to Segalowitz (Reference Segalowitz2010; Segalowitz & Segalowitz, Reference Segalowitz and Segalowitz1993) the qualitative change that characterizes automaticity is restructuring: changing and/or organizing the underlying processing routines of a task differently (McLaughlin, Reference McLaughlin1987). Segalowitz (Reference Segalowitz2010) subsumes the many characteristics of skilled processing under three key properties: processing speed, processing stability, and processing flexibility. It is intuitively known that automatic processing is fast processing. However, although processing speed is a prominent feature of automatized processing, Segalowitz (Reference Segalowitz2010; Segalowitz & Segalowitz, Reference Segalowitz and Segalowitz1993) emphasizes that speed alone is not sufficient for genuine automatization. In addition, automatization involves restructuring of the cognitive subcomponents of a task, which will improve processing stability and efficiency. “A process is automatic, even if one cannot specify the exact nature of that automaticity, as long as one can say that the process is functioning more efficiently in some meaningful way compared to how it would have functioned if it were non-automatic” (Segalowitz, Reference Segalowitz2010, p. 79).
Segalowitz and Segalowitz (Reference Segalowitz and Segalowitz1993) proposed the CV as a measure of processing stability. CV takes into account both processing stability (intraindividual standard deviation) and processing speed, or reaction time (RT), by computing the standard deviation to RT ratio. The resulting value reflects a person's performance stability corrected for his or her performance speed, with smaller CVs indicating higher levels of processing stability. Because the validity of the CV as a measure of processing stability is of particular interest to the current study, the relevant literature will be discussed in the next two sections.
Finally, as far as grammatical processing is concerned, Segalowitz (Reference Segalowitz2010) argues that flexibility of attentional control is also essential to skilled performance. Unlike processing speed and stability, processing flexibility is an attention-based mechanism that complements and supports the automatic aspects of processing (Segalowitz, Reference Segalowitz2010). Fluency entails the need to switch one's attentional focus from one task to another quickly and accurately in response to the linguistic information encountered in the input. For example, conjunctions such as because or despite carry information about how the speaker wishes the recipient to construe the link between two events, as a causal or noncausal relation (Segalowitz, Reference Segalowitz2010). Skilled language users are flexible in adapting their attentional focus accordingly and in integrating this new formation into the meaning representation they are constructing in real time. This type of specifically linguistic cognition has been termed language-directed attention (not to be confounded with attention to language; Fauconnier, Reference Fauconnier1994; Talmy, Reference Talmy, Robinson and Ellis2008). It is one of the factors underpinning fluent sentence processing.
CV
Although the use of CV in SLA has not spread much beyond Segalowitz and his collaborators, it is a common measure in the behavioral sciences to assess performance variability by virtue of its natural immunity to different units of measurement or widely different means (Bekoff, Reference Bekoff and Hazlett1997; Carlton, Robertson, Carlton, & Newell, Reference Carlton, Robertson, Carlton and Newell1985; Dawkins & Dawkins, Reference Dawkins and Dawkins1973). In particular, some psychologists show a strong preference for CV as a measure of intraindividual variability, pointing out that unlike the standard deviation, the CV is rarely affected by repeated testing or practice (Flehming, Steinborn, Langner, Scholz, & Westhoff, Reference Flehming, Steinborn, Langner, Scholz and Westhoff2007). CV applications can be found in a wide range of fields of study: from measuring cardiac autonomic function (e.g., Toichi, Suguiura, Murai, & Sengoku, Reference Toichi, Suguiura, Murai and Sengoku1997) and predicting risk sensitivity of humans and other animals (e.g., Weber, Shafir, & Blais, Reference Weber, Shafir and Blais2004) to dissociating between speed and regularity in tapping behaviors (e.g., McManus, Kemp, & Grant, Reference McManus, Kemp and Grant1986).
Segalowitz and Segalowitz (Reference Segalowitz and Segalowitz1993) were the first to advocate the use of CV as a measure of L2 automatization. Central to Segalowitz and Segalowitz’ proposal is that automatization involves a qualitative change in cognitive processing; that is, a restructuring (deletion and/or suppletion) of processing components above and beyond simple speedup. They argued that only this scenario results in a disproportionate decrease in standard deviation and concomitant CV reduction.
Let us take L2 visual word recognition as an example. Suppose that word recognition training helps to decrease the RTs associated with each subcomponent of the word recognition process (e.g., orthographic processing, phonological processing, translation into the native language, first language [L1]-mediated concept retrieval) and thus reduces the total RT for the word recognition behavior as a whole. Given that the default is for standard deviations and RTs to be linearly related (Anderson, Reference Anderson1992; Logan, Reference Logan1988; Wagenmakers & Brown, Reference Wagenmakers and Brown2007), standard deviations will decrease as RTs decrease. Because CV equals standard deviation/RT, this means that CV will remain unchanged if standard deviations drop in proportion to RTs for the word recognition behavior as a whole. In Segalowitz and Segalowitz’ view, no change in CV indicates speedup: each subcomponent has become accelerated but nonetheless functions in the same manner, just like when somebody becomes faster at cooking after having done it many times but still goes through all the same steps he or she did originally.
Alternatively, if extended practice leads to the elimination or dramatic reduction of subcomponents that involve much attentional effort (e.g., L1 translation, metalinguistic analysis), the proportional reduction in standard deviations exceeds the proportional reduction in RTs, and CVs decrease as a result. Slower, effortful processes tend to be more variable (i.e., have larger standard deviations); therefore, removing them will have a disproportionate effect on the standard deviation of the total RTs (for a numerical example, see Segalowitz & Segalowitz, Reference Segalowitz and Segalowitz1993, their table 1). Segalowitz and Segalowitz assert that a CV reduction evidences the development of automaticity. Finally, as CV, RT, and the standard deviation are all assumed to decrease over the course of automatization, a positive correlation between CV and RT is a corollary of a simultaneous reduction in CV and RT.
The CV analysis has proven a valuable tool in SLA studies with diverse populations and for different research purposes (for a review and critique, see Hulstijn et al., Reference Hulstijn, Van Gelderen and Schoonen2009). However, all of the acquisition-related CV research until then focused on the processing of single words. Paradis (Reference Paradis2004, Reference Paradis2009) questioned the findings of such studies on grounds of ecological validity and the special status of vocabulary knowledge within language: “Single words are the least likely candidates for investigating ‘language’ representation, given that what makes language most specific as a cognitive function, that is, the language system (phonology, morphology, syntax), is supported by procedural memory, whereas isolated words are supported by declarative memory” (Paradis, Reference Paradis2004, p. 173). Hulstijn et al. (Reference Hulstijn, Van Gelderen and Schoonen2009) were the first to extend the scope of CV investigations beyond the word level. They investigated the automatization of word- and sentence-level processing, as evidenced by CV, based on a massive data set of 397 Dutch high school learners of English who were followed over 2 years.
Hulstijn, Van Gelderen, and Schoonen's experiment 1
Hulstijn and colleagues (Reference Hulstijn, Van Gelderen and Schoonen2009) carried out a 2-year longitudinal study in a number of high schools in The Netherlands. The students (about 200 at each time of measurement) performed four computerized RT tasks in L1 Dutch and in L2 English in Grades 8, 9, and 10. The tasks consisted of a lexical decision and retrieval task and a sentence verification and production task. As this was a longitudinal study, time was used as an independent index of automaticity; that is, it was assumed that learners’ fluency in the L2 would develop as they advanced to a higher grade and that, consistent with Segalowitz and Segalowitz’ (Reference Segalowitz and Segalowitz1993) hypothesis, this would be reflected in a within-subject decrease in CV over the 2 years. Contrary to this prediction, however, Hulstijn et al. (Reference Hulstijn, Van Gelderen and Schoonen2009) found that “in all eight tasks the CV remained more or less constant”Footnote 2 (p. 567). Similarly, the CV-RT correlation was not always in the expected direction (i.e., some correlations were negative) and not all positive correlations were significant.
Compared to previous CV studies, Hulstijn and colleagues (Reference Hulstijn, Van Gelderen and Schoonen2009) submitted their RT data to a more thorough cleaning procedure before statistical analysis. It is important to discuss the different options for data cleaning in CV research in some detail because the choices one makes can affect the results (for an illustration, see Hulstijn et al., Reference Hulstijn, Van Gelderen and Schoonen2009, exp. 2).
The purpose of data cleaning is to minimize the noise inherent in RT data (e.g., Jiang, Reference Jiang2012). First, in many RT tasks such as lexical decision, it is standard practice to analyze correct trials only because knowledge retrieval on incorrect trials either failed or was nontarget-like. However, if accuracy rates do not approach 100% this may mean discarding a significant portion of the data and hence important information loss. In the analysis reported for the longitudinal data set (exp. 1), Hulstijn et al. (Reference Hulstijn, Van Gelderen and Schoonen2009) circumvented this problem by coding the incorrect trials as missing values and subsequently estimating them with the expectation-maximization algorithm.Footnote 3 Second, RT experiments also require dealing with extremely slow or extremely fast responses, as “such RTs cannot be considered to be a valid measure of the processes under investigation” (Hulstijn et al., Reference Hulstijn, Van Gelderen and Schoonen2009, p. 578). A common practice is to “pull in” outliers; that is, to replace them by a less extreme value such as the item mean ± 2, 2.5, or 3 SD. This approach, however, seems undesirable for CV analyses because “the empirical evidence of investigations on CV reduction is formed by the standard deviation, which, first and foremost, is an index of variability. By cutting off the tails of the distribution, the researcher artificially reduces the variability in the raw data” (Hulstijn et al., Reference Hulstijn, Van Gelderen and Schoonen2009, p. 578, original emphasis). The solution that these authors propose is again to treat the outliers as missing values and have the algorithm estimate them. Third, and perhaps most crucially, Hulstijn et al. argued that only items with high hit rates and participants with high accuracy rates should be included for any given task “because the speed tests [are] intended to measure speed of processing and should not be confounded with accuracy of linguistic knowledge” (p. 566). Herein lies the tricky part, because inclusion criteria can be made more or less stringent. By way of illustration, Hulstijn et al. (Reference Hulstijn, Van Gelderen and Schoonen2009, exp. 2) computed CVs on a more restricted data set that was prepared in six different ways. Only if participants who performed at least at 95% accuracy were included did the CV decrease significantly, but in one analysis, this amounted to eliminating 40% of the data! Although some data cleaning may be indispensable, treatment should be used sparingly so as to minimize the amount of information loss and ensure the data's validity.
For their first experiment, the longitudinal study, Hulstijn and colleagues (Reference Hulstijn, Van Gelderen and Schoonen2009) generally did not find a decrease of CV over time. The authors attributed this null result to a potential confounding of knowledge accumulation and skill acquisition in CV measurement. The argument is that with increasing exposure, L2 learners not only consolidate existing linguistic knowledge but also acquire new, declarative knowledge (e.g., new words). While the former results in faster RTs, the latter typically does not, because novel information tends to be accessed slowly and effortfully (McLaughlin, Reference McLaughlin1987). Hence, “RT tests . . . inherently suffer from confounding measuring improvement in response efficiency with measuring improvement in declarative knowledge” (Hulstijn et al., Reference Hulstijn, Van Gelderen and Schoonen2009, p. 579). Given that high school students are still learning both L1 and L2 vocabulary and grammar knowledge, this problem may have been particularly acute in the Hulstijn et al. study (see p. 580, note 1).
Because of the speed–accuracy trade-off, Hulstijn et al. questioned the reliability of the CV index as a tool to distinguish between speedup and automatization. Although we agree that the mathematical and conceptual bases of the CV deserve close scrutiny, we believe that it is premature to discredit the measure based on the observed null findings. A number of alternative accounts of the findings come to mind. First, we cannot rule out the possibility that Dutch secondary school students failed to automatize their English processing skills in 2 consecutive years. When the students completed the tests for the first time in Grade 8, most had “just started L2 reading” and “their L2 vocabulary and grammar knowledge as well as their efficiency in processing L2 words and sentences [were] at a low level, relative to their L1 skills” (Van Gelderen et al., Reference Van Gelderen, Schoonen, Stoel, De Glopper and Hulstijn2007, p. 478). Because automatization is known to take place relatively late in language development (Anderson, Reference Anderson1983; Segalowitz & Hulstijn, Reference Segalowitz, Hulstijn, Kroll and De Groot2005; Wickens, 1992), more advanced students, for instance from the final years of high school or postsecondary education, might be more suitable study subjects. Results from independent proficiency tests may also be helpful to interpret an apparent lack of automatization. Second, issues of positive transfer of orthographic and syntactic processing from L1 Dutch to L2 English, which are two closely related languages, may have obscured potential gains in automaticity in other linguistic domains. Third, it is possible that automatization did occur but that some properties of Hulstijn et al.'s stimuli made it difficult to detect through a CV analysis.
For all of these reasons, we felt that the time for a conceptual replication of Hulstijn et al. (Reference Hulstijn, Van Gelderen and Schoonen2009) was ripe. We acknowledge at the outset that this study is not as ambitious in scope as the original study, yet we believe that because it addresses the issues pointed out above, it may still make a valuable contribution to the nascent CV literature on L2 sentence processing. Using relatively advanced English as a second language (ESL) learners who have a typologically different L1 (Korean), we sought to answer the following questions:
1. Does the CV index the degree of automatization in L2 sentence processing?
a. Does CV decrease as the automaticity of sentence processing develops?
b. Does the correlation between CV and RT become statistically positive as the automaticity of sentence processing develops?
2. Does the usefulness of CV as an index of the degree of automatization differ across different tasks (e.g., sentence verification task vs. sentence production/construction task)?
Two important assumptions underlie this study: first, sentence processing, which entails grammar processing, is amenable at least partly to automatization. If grammar use is truly an automatic behavior like lexical access (e.g., DeKeyser; Reference DeKeyser1997; Robinson, Reference Robinson1997; Rodgers, Reference Rodgers2011; Serafini, Reference Serafini2013), the principles of CV should work equally well for sentence processing in the L2. Second, L2 automaticity increases with proficiency level. Just as a longitudinal study assumes the gradual development of fluency over the course of time, this cross-sectional study assumes that advanced learners have a higher level of fluency than intermediate learners and that fluency is a behavioral index of automatization.
METHOD
Participants
Forty Korean ESL learners and 20 native speakers of English were recruited in Michigan and New York. Most of the participants were either graduate or undergraduate students. A small number of participants were enrolled in an English for Academic Purposes program at a university, in which they were readying themselves to attend regular university courses in the following semester.
Participants’ proficiency level was assessed through DIALANG, an online diagnostic language test established by Lancaster University. Due to time restrictions, only vocabulary size and grammar knowledge were assessed, amounting to an overall test duration of 10–15 min. DIALANG follows the common European framework (Council of Europe, 2001), which divides learners’ proficiency into three broad levels: the A level for basic users, the B level for independent users, and the C level for proficient users. Each level further branches into two sublevels, such as A1 (beginner), A2 (elementary), B1 (intermediate), B2 (upper intermediate), C1 (advanced), and C2 (proficient). The present study classified ESL learners as advanced if they scored at the C level and intermediate if they scored at the B level. By requiring at least intermediary-level mastery of English, we hoped to minimize the effect of learners’ declarative knowledge on processing skills (see above). Table 1 summarizes the demographic information of the participants.
Description of participants
Note: Onset of learning refers to the age at which learners first started learning English. Onset of arrival is the age at which learners first arrived in the United States. EAP, English for academic purposes; LOR, length of residence in the United States.
Materials
Three RT experiments were administered to measure participants’ L2 word recognition and L2 sentence processing skills: a semantic classification task, a sentence construction task,Footnote 4 and a sentence verification task. For measures of sentence processing efficiency to be maximally informative, all participants should ideally have equally automatized lexical knowledge, so any differences in sentence processing skills can be attributed to the sentence level of language processing. Therefore, the purpose of the semantic classification task was to assess any preexisting differences between participants’ automatization of lexical knowledge.
All tasks were conducted in English only. DMDX was used for the semantic classification and the sentence verification tasks, and the sentence construction task was run on Linger. RTs and accuracy rates were recorded for all three tasks. The original sentences from Hulstijn et al. (Reference Hulstijn, Van Gelderen and Schoonen2009) were made available to us by the authors and were revised to be suitable for Korean ESL speakers studying at US universities. Major modifications are further described in the next three subsections. The experimental stimuli used in each task are given in Tables A.1, A.2, and A.3 in Appendix A.
Semantic classification task
Participants were asked to decide as quickly and as accurately as possible if a word on the computer screen referred to a living being (e.g., boy) or a nonliving artifact (e.g., book). Such animacy judgment tasks have been intended to replace traditional lexical decision tasks, in which participants are asked to discriminate between real words and nonwords (Phillips, Segalowitz, O'Brien, & Yamasaki, Reference Phillips, Segalowitz, O'Brien and Yamasaki2004; Segalowitz & de Almeida, Reference Segalowitz and de Almeida2002; Segalowitz & Frenkiel-Fishman, Reference Segalowitz and Frenkiel-Fishman2005). Segalowitz and his associates posit that the animacy judgment task is a more authentic way of measuring learners’ word recognition skills than a lexical decision task because it invokes relatively strong semantic processing.
All word stimuli were extracted from the subsequent sentence-level tasks in order to expose the participants to the words before the sentence processing tasks. By doing so, we hoped to make sure that individual differences in sentence processing efficiency did not stem from differences in either participants’ lexical processing or their word knowledge. Word frequencies for all the stimuli, based on the trimmed version of the CELEX database MC Word: An Orthographic WordForm Database (Medler & Binder, Reference Medler and Binder2005), are reported in Table A.1 in Appendix A. The task began with six practice items, after which 46 actual trials followed.
Sentence construction task
In this task, the beginning part of a sentence was given on the first screen (e.g., After some time . . .), and two possible options followed on the next screen (e.g., A. works, B. she). Participants were asked to read the sentence beginnings in a self-paced fashion but then to quickly choose the word(s) that best continued the phrase that they just read. Participants were told that the given options would not necessarily complete the sentence but that one of them was a good continuation.
The sentence construction task required participants to process a sentence fragment and to select the continuation that best fit into the syntactic structure they created. Unlike in Hulstijn et al. (Reference Hulstijn, Van Gelderen and Schoonen2009), we kept the stimuli (both the stem and the options) as short as possible; sentence beginnings ranged from one to three words and each answer option constituted a single word or two words, either a function word or a content word. Because the stimuli were so simple, it limited the amount of higher level semantic analysis that participants could bring to the task, focusing instead on participants’ parsing skills. The task started with six warm-up trials, after which 40 real items followed.
Sentence verification task
This is a sentence plausibility judgment task, an extended form of the semantic classification task at the word level. Participants were asked to read a sentence and decide quickly whether it made sense. Unlike in Hulstijn et al. (Reference Hulstijn, Van Gelderen and Schoonen2009), sentence length was controlled for, with each sentence consisting of 7–10 words. The content of each sentence was also revised to fit the geographical location of the current participants. Some sentences contained semantic errors instantiated at the lexical level as in Most theaters in the US have only one chair, and others entailed structural violations at the phrasal level (e.g., My uncle made me for a snowman) or at the clausal level (e.g., He went inside, though it started to rain).
Current event-related potentials (ERP) studies justify incorporating structural violations in the sentence verification task for the purpose of examining automatic L2 sentence processing. During syntactic processing, phrase structure building and syntactic relational processing are known to take place in a relatively automatic manner (Gunter, Hahne, & Mauth, Reference Gunter, Hahne and Mauth2004; Hasting & Kotz, Reference Hasting and Kotz2008). Thus, those who can process a sentence structure automatically should be able to verify the implausibility of the sentence quickly. Furthermore, varying the focus of the sentence verification task seems appropriate for tapping into learners’ automatic processing skills. Recall that the third attribute of skilled processing (Segalowitz, Reference Segalowitz2010) was the flexibility with which learners can shift their focus of attention while dealing with a variety of linguistic input. In this task, some sentences asked participants to utilize their factual knowledge (e.g., Most theaters in the US have only one chair), while others required them to invoke grammatical knowledge (e.g., My mother was cooked by a cake). Learners who possess automatic processing skills are expected to adjust their attentional focus quickly to the adequate level of representation, as directed by the linguistic input, and thus react equally fast to different sorts of sentences. Participants were instructed to focus on meaning, rather than grammar, during reading. The task began with 6 warm-up trials, after which 60 real trials followed. Thirty-four sentences were nonsensible and 26 were sensible.
Procedure
All participants visited the language lab individually. First, the researcher explained the entire procedure to a participant and collected a consent form. The three speeded tasks described above were administered in the following order: (a) the semantic classification task, (b) the sentence construction task, and (c) the sentence verification task. Upon completing the main tasks, participants took the English diagnostic language test, DIALANG. Finally, they filled out a language background questionnaire that included questions about their educational background, age of first arrival in an English-speaking country, length of residence in the United States, and the like (see Appendix B). Participants were all volunteers and were paid $10 for their efforts.
Data analysis
Different statistical tests were used depending on whether the data met the normality assumption and the homogeneity of variances assumption. In the parametric option, we submitted the RTs, standard deviations, and CVs to one-way analyses of variance, followed by Tukey post hoc tests (equal variances) or Games–Howell tests (nonequal variances). The nonparametric solution consisted of a Kruskal–Wallis test followed by Mann–Whitney U tests. For the correlation analyses, we used Pearson's correlation (parametric) or Spearman's rho (nonparametric). It is important that all RT data were analyzed twice: once without data cleaning and the second time with data cleaning. The significance level α was set at .05 for all analyses.
For the sentence verification task, we conducted two analyses at each step. For the first analysis, we included all the correct responses, as we did for the other two tasks. This means that we considered both “hits” (sensible sentences judged sensible) and “correct rejections” (nonsensible sentences judged nonsensible) together. However, because automatic processing may be interrupted when individuals encounter something unexpected or problematic (Baars & Franklin, Reference Baars and Franklin2003; N. C. Ellis, Reference Ellis2005), it could be argued that the correct rejections do not reflect automatic processing but a (correct) reversion to attention-controlled processing. Therefore, we also looked at the hits separately. No such division between yes and no responses was necessary for the semantic classification or sentence construction tasks given the somewhat different nature of these tasks.
RESULTS
Raw data analysis: Clearest evidence of automatization in sentence construction task
For the first data analysis, we included only correct responses but did not do any additional data cleaning. We found that RTs, standard deviations, and CVs decreased in all tasks as English proficiency level increased (see Table 2). In the word-level semantic classification task, however, reduction in CVs was not statistically significant, F (2, 57) = 2.16, p = .13, η2 = 0.207, meaning that we did not find any between-group differences in lexical processing efficiency. Although the decrease in RTs and standard deviations was statistically significant, such differences can be interpreted as reflecting differences in processing speed rather than degree of automaticity.
Statistical results for the three speeded measures before data cleaning
Note: The exact p values are given below each statistic. RT, reaction time; CV, coefficient of variation.
aA significant difference from the native speakers.
bA significant difference from the advanced group.
cA significant difference from the intermediate-level group.
*p = .05. **p = .01. ***p = .001.
In both the sentence construction and verification tasks, RTs and standard deviations decreased significantly. The decrease in CVs in the sentence construction task was significant, F (2, 57) = 18.66, p < .001, η2 = 0.396. Specifically, Games–Howell post hoc tests indicated that the CV of the native speaker group was statistically smaller than that of the advanced group (p < .001, d = 0.93) and the CV of the advanced group was statistically smaller than that of the intermediate group (p = .004, d = 0.96). The decrease in CVs in the sentence verification task was not significant: for both sensible and nonsensible items, F (2, 57) = 1.71, p = .190, η2 = 0.057; for the sensible items only, F (2, 57) = 0.16, p = .859, η2 = 0.005. Nonetheless, participants’ CVs on the verification and construction tasks were positively related (with all items, r = .343, p = .007; with only sensible items in the sentence verification task, r = .280, p = .030).
In terms of the CV-RT correlation analyses, we found a positive and significant correlation between RTs and CVs for the semantic classification task (r = .672, p < .001) and the sentence construction task (r = .751, p < .001). The significant correlation for the lexical task is not very informative because it was not accompanied by a significant CV reduction, but the correlation for the sentence construction task is significant: it may indicate individual differences in L2 automaticity. Finally, the CV-RT correlation in the sentence verification task was not significant: for the entire items (r = .066, p = .619) and for the sensible items only (r = .125, p = .342).
The results of the first data analyses suggest that all participants were comparable at the lexical level, which was consistent with the authors’ prediction but diverged in terms of sentence-level processing skills. An interesting finding was that the two sentence tasks yielded different outcomes. To reduce the potentially confounding effect of participants’ linguistic knowledge on the CV computations (Hulstijn et al., Reference Hulstijn, Van Gelderen and Schoonen2009), we now submitted all RTs from all tasks to the data cleaning procedure described in Hulstijn et al. (Reference Hulstijn, Van Gelderen and Schoonen2009, exp. 1). We then reanalyzed these tailored data using the same statistics.
Data cleaning
Whereas only correct responses were considered for the original analysis, we now estimated RTs for incorrect responses and outliers using multiple imputation and included these data as well. In addition, a number of low-performing items were removed from the analysis.
The data cleaning procedure consisted of four steps. First, RTs of more than 3 SD above the group mean for each item were considered as extremely slow and therefore defined as missing values. To set the fast extremes of nonnative speakers’ RTs, the smallest RT that native speakers achieved was used as the point of reference. Then, nonnative speakers’ RTs faster than the fastest RTs of native speakers were again defined as missing values. Second, to minimize the effect of linguistic knowledge on the processing measure, the items with high error rates were taken out. As Hulstijn et al. (Reference Hulstijn, Van Gelderen and Schoonen2009) suggested, only the items with higher than 87.5% correct responses were included. This represents “a 75% correct score, with the chance level for guessing added” (Hulstijn et al., Reference Hulstijn, Van Gelderen and Schoonen2009, pp. 566–567). As a consequence, we excluded 16 sentence verification items out of 60 (2 plausible sentences, 14 implausible sentences) and 9 sentence construction items out of 40. The excluded items are listed in Appendix C. Third, in addition to these two steps, Hulstijn et al. treated the students who scored less than 62.5% correct answers on any speed test as missing for the whole test. In the current study, only one participant scored 52% correct on the plausible sentences in the sentence verification task (but above threshold on the task as a whole); he or she was thus excluded from the second data analysis for this task. Fourth, all the erased RTs (of incorrect, or extremely slow or fast responses) were estimated by means of multiple imputation. Multiple imputation is the substitution of some value for a missing data point in multivariate analysis (Rubin, Reference Rubin1977).Footnote 5 We calculated the maximum and the minimum RTs for each individual item and defined these as the boundary points for the imputed values to fall in. In terms of the missing data patterns, 3.01% of the values were missing in the word recognition task, 4.96% in the sentence verification task (4.53% for the “yes” response items only), and 5.86% in the sentence construction task.
Cleaned data analysis mostly confirmed findings
The data cleaning procedure did not change the descriptive statistics substantially. All RT measures reduced slightly as a result of data cleaning as did their corresponding standard deviations (compare Table 2 and Table 3). As expected, the native speaker group recorded the smallest RTs, standard deviations, and CVs in all three tasks, whereas the intermediate group showed the largest RTs, standard deviations, and CVs (see Table 3).
Statistical results for the speeded measures after data cleaning
Note: The exact p values are given below each statistic. RT, reaction time; CV, coefficient of variation.
aA significant difference from the native speakers.
bA significant difference from the advanced group.
cA significant difference from the intermediate-level group.
*p = .05. **p = .01. ***p = .001.
A one-way analysis of variance indicated that all three measures for the semantic classification task decreased significantly as proficiency level increased (p = .013, η2 = 0.142 for mean RT, p = .004, η2 = 0.174 for mean standard deviation, and p = .027, η2 = 0.119 for mean CV). The significant decrease for CV was new compared to the earlier analysis. Tukey post hoc tests revealed that the intermediate ESL group was significantly slower, more variable and less efficient than the native speaker group: the differences between the two groups were 123 ms for mean RT (p = .009, d = 0.96), 73 ms for mean standard deviation (p = .011, d = 0.99), and 0.05 for mean CV (p = .046, d = 0.77). The advanced ESL group did not differ from the intermediate learners or the native speakers on any measure. In terms of the CV-RT correlation, a medium-size, positive correlation was found in the entire group (r = .549, p < .001). Within groups, CVs of the intermediate and the advanced group were highly correlated with RTs (intermediate: r = .772, p < .001; advanced: r = .619, p = .004). In contrast, the CV-RT correlation in the native speaker group was virtually zero (r = .031, p = .878).
The results for the sentence construction task were consistent with the results before the data cleaning: the groups differed significantly from one another for all three measures. The native speakers showed the smallest RTs, standard deviations, and CVs, whereas the intermediate learners recorded the largest RTs, standard deviations, and CVs. In terms of the correlation analyses, the entire set of RTs were significantly correlated with the entire set of CVs (r = .642, p < .001). At the group level, the CV-RT correlation was positive and significant for the advanced learners (r = .541, p = .001) but neither for the intermediate learners (r = .191, p = .419) or the native speakers (r = .049, p = .838). By deleting nine items with high error rates, accuracy rose to 95% for the intermediate, 98% for the advanced, and 99% for the native group. The differences in accuracy between the three groups were not statistically significant, F (2, 57) = 2.28, p = .112.
Finally, general patterns for the sentence verification task did not change after the data cleaning: mean RT, standard deviation, and CV decreased as proficiency level increased. For mean RTs and standard deviations, all groups differed significantly from one another. It is interesting that the results of the CV analyses differed somewhat depending on whether only the plausible sentences or both the plausible and the implausible sentences were analyzed. For the plausible sentences only, CVs did not differ between groups (p = .913, η2 = 0.003) and did not correlate with participants’ CVs on the sentence construction task (r = –.035, p = .795). However, when plausible and implausible sentences were combined, CVs did correlate strongly with those for the sentence construction task (r = .540, p < .001). Nonnative speakers’ CVs were also significantly larger than native speakers’ (p = .001, d = 1.23 for intermediate vs. native, and p = .013, d = 0.92 for advanced vs. native), yet the CVs for the two ESL groups did not differ significantly (p = .629).
As regards the CV-RT correlation for the sentence verification task, correlations for the plausible sentences only were all close to zero (across the entire group: r = –.066, p = .621; native speakers: r = –.113, p = .635; advanced learners: r = .061, p = .797; intermediate learners: r = –.156, p = .523). When we included both plausible and implausible sentences, a negative CV-RT correlation was statistical in the intermediate group (r = –.516, p = .020) but not so among the native speakers (r = –.352, p = .128). CV-RT correlations for the advanced learners (r = .361, p = .118) and for the entire sample (r = .203, p = .120) were positive but nonsignificant. After deleting 2 plausible and 14 implausible items with high error rates, accuracy increased to 96% for the intermediate, 97% for the advanced, and 98% for the native group. As a result, group differences became insignificant, F (2, 57) = 3.08, p = .054. Accuracy differed significantly as a function of item plausibility among native speakers, t (19) = –2.84, p = .010, d = 0.077, but not in the ESL groups, intermediate: t (19) = –1.70, p = .106; advanced: t (19) = –1.14, p = .267. The native speakers misjudged the implausible items significantly more often than the plausible items, whereas this was not the case for the nonnative speakers.
Summary
The results of the current study supported Segalowitz and Segalowitz’ (Reference Segalowitz and Segalowitz1993) hypothesis (CV decreased as speakers developed automatic processing skills), regardless of whether the data were cleaned (see Table 4). Data cleaning did not affect overall patterns of RTs, standard deviations, and CVs in any task, yet it boosted accuracy rates and caused some contrasts that had initially been nonsignificant to become significant, notably in the semantic classification and sentence verification tasks. Thus, only after data cleaning were the intermediate learners found to retrieve word meanings with more difficulty than the native speakers (p = .009 for RTs, p = .011 for standard deviations, p = .046 for CVs). No such difference was found for the advanced learners versus the native speakers. Similarly, after data cleaning, but not before, nonnative speakers appeared to be less efficient at sentence verification than native speakers, at least when both plausible and implausible sentences were considered together. Finally, the findings of the sentence construction task seemed to be the most robust, given that CV decreased with increasing proficiency level both before and after data cleaning.
Comparisons of findings in Hulstijn et al. (Reference Hulstijn, Van Gelderen and Schoonen2009) and the present study
Note: The much greater than symbol » indicates a significant difference between groups based on the post hoc analyses. NS, native speaker. RT, reaction time; CV, coefficient of variation.
Table 4 summarizes the important findings of the study and compares them with those of Hulstijn et al. (Reference Hulstijn, Van Gelderen and Schoonen2009). Note that Hulstijn et al. did not report the results of the post hoc analyses. Thus, we do not know which differences between measurement times in Hulstijn et al. were significant.
DISCUSSION
Comparison of findings with Hulstijn et al.'s experiment 1
In this study, we reported the findings for two sets of statistical analyses that were conducted before and after cleaning participants’ RTs for three speeded tasks. The general finding was that RTs, standard deviations, and CVs all decreased as a function of participants’ English proficiency level: from intermediate ESL learners (L1 Korean), to advanced ESL learners, to native English speakers. This finding was in stark contrast to Hulstijn et al.'s (Reference Hulstijn, Van Gelderen and Schoonen2009) study, who failed to find a decrease in Dutch high school students’ CV over three consecutive grade years, in which the students received ESL instruction in school and plentiful exposure to English (e.g., through the media) outside the classroom. The unvarying CV in Hulstijn et al. (Reference Hulstijn, Van Gelderen and Schoonen2009) should not be taken as an index that the Dutch high school students did not progress in their knowledge of the English language (see the appendix in Van Gelderen et al., Reference Van Gelderen, Schoonen, Stoel, De Glopper and Hulstijn2007); however, it might suggest that they were still in a relatively early stage of their language learning trajectory, in which declarative knowledge accumulation (vocabulary and grammar learning) dominated.
Participants in the present study were intermediate and advanced ESL learners enrolled at a US university. Except for a small subset of English for Academic Purposes program students, none of the participants were taking dedicated English language courses anymore. They were immersed in an English-speaking environment in which they were functioning as regular undergraduate and graduate students. The communicative pressures that they encountered in their daily lives may thus have pushed them to practice and proceduralize or automatize their existing English knowledge. Furthermore, participants seemed to have a sound knowledge base of English, which is a prerequisite for automatization (Anderson, Reference Anderson1992), as shown in the high accuracy scores on the three speeded tasks (range = 90%–98%).
Given their findings, Hulstijn et al. (Reference Hulstijn, Van Gelderen and Schoonen2009) questioned the validity of the CV as a measure of automatization in sentence processing. “Although conceptually skill acquisition can be distinguished from knowledge accumulation, in reality, knowledge accumulation forms part of skill acquisition because, in real L2 learning, exposure to new words goes hand in hand with exposure to words encountered before” (Hulstijn et al., Reference Hulstijn, Van Gelderen and Schoonen2009, p. 576, original emphasis). They argued that response efficiency and knowledge accumulation are inherently confounded in RT tasks and that data cleaning is therefore strongly recommended. While the usefulness of data cleaning will depend on the specifics of the study, we emphasize that, in this study, data cleaning was not necessary. Had we not been replicating Hulstijn et al. (Reference Hulstijn, Van Gelderen and Schoonen2009) conceptually, we would have stopped after the first, raw-data analysis of the correct responses.
The raw-data analysis indicated that the ESL learners were indistinguishable from native speakers in terms of their automatic processing of individual words but not so in their processing of phrasal and clausal units. Particularly, processing efficiency on the sentence construction task decreased significantly for the lower proficiency levels. CVs on the sentence verification task did not differ significantly between groups, yet the numerical pattern was the same as for the sentence construction task. As we explain in the next section, we believe that this interpretation may be the most solid account of our data; CV proved to be a useful measure of automatization of this sample's L2 sentence processing skills.
Why data cleaning may not be necessary
On the semantic classification task, any data cleaning beyond the removal of incorrect responses seemed unnecessary because both groups of ESL learners performed near ceiling (98% accuracy), which leaves very little room for a knowledge confound to influence findings. For the sentence construction task, the results did not change before or after data cleaning. Likewise, the CV-RT correlation analyses accorded well with the CV analyses in both cases (compare Tables 2 and 3), and we would argue that the simpler, raw-data analysis is to be preferred. However, for the sentence verification task the outcome did change as a result of data cleaning. Removal of 16 test items (i.e., 27% of the data) and outlier substitutions caused the difference in CV between native and nonnative speakers (both advanced and intermediate) to become significant. However, we believe that this effect should be interpreted with caution, as it coincided with a significant, negative CV-RT correlation in the intermediate ESL group (see Table 3). This signals that intermediate learners who responded more slowly tended to produce less variable RTs and that faster responders tended to show greater variability. Such patterns are at odds with the principle of automatization that fast and reliable processing go hand in hand (Segalowitz, Reference Segalowitz2010).
Sentence construction proved a better task than sentence verification
Following Hulstijn et al. (Reference Hulstijn, Van Gelderen and Schoonen2009), we adopted a sentence verification and a sentence construction task to examine participants’ automatization of L2 sentence processing. The results of the two tasks were compatible (as the positive correlation coefficients between CVs in the two tasks indicate) but not identical. In the construction task, the advanced group exhibited more automatized processing than the intermediate group, whereas in the sentence verification task, the advanced group only responded faster but not more automatically (see Table 4). These results were qualified by an interesting item effect: when only plausible items in the sentence verification task were analyzed, the correlation with the sentence construction task disappeared and the mean CV for the different groups became very similar. At this juncture, a natural question arises: which underlying processes does each task and item type tap into?
Sentence-level processing is more complex than word-level processing because in addition to identifying the individual words that make up the sentence (lexical access), readers should establish how they belong together structurally (syntactic parsing) and draw meaning from them. (At the clause level, the third process is called semantic proposition formation; Grabe & Stoller, Reference Grabe and Stoller2011.) Grabe and Stoller defined lexical access, syntactic parsing and semantic proposition formation as lower level linguistic processes.Footnote 6 They may differ from higher level processes, such as the use of background knowledge to draw inferences, in that they “occur relatively automatically for the fluent reader” (Grabe & Stoller, Reference Grabe and Stoller2011, p. 19). Grabe and Stoller do not really discuss the possibility of automatizing higher level processes, but we believe that these processes cannot be automatized for two reasons: first, they require conscious processing, and second, they are highly context dependent. For instance, an inference one makes in one reading context will seldom be reusable in a different reading context, and so it is difficult to practice and automatize that specific procedure. There is no doubt that inferencing, using background knowledge, building a text model, and so on, are integral components of skilled reading (e.g., Grabe & Stoller, Reference Grabe and Stoller2011; Koda, Reference Koda2005; Schmalhofer & Perfetti, Reference Schmalhofer and Perfetti2007); however, we question that the CV is as suited for measuring these skills as it is for measuring participants’ lower level processing skills.
The implication of this for CV studies on sentence processing is that tasks that reduce the influence of higher level processes will likely produce clearer findings. In our study, this was the sentence construction task. We would argue that besides word recognition, the construction task primarily assessed participants’ parsing, which is a “very [rapid process] without much effort or conscious attention” (Grabe & Stoller, Reference Grabe and Stoller2011, p. 17) in fluent language users. Predicting the next word after a given phrase likely forced participants to build a sentence structure incrementally. Then, to predict the next word, participants had to make use of word order and probabilistic information expeditiously so they could select the response option that belonged to the syntactic category that they predicted would follow. Given that the syntactic structure of Korean is distinct from that of English (e.g., head-final vs. head-initial language) and that, therefore, L1 syntactic transfer will be negative in the beginning stages of learning (Grabe & Stoller, Reference Grabe and Stoller2011; Koda, Reference Koda2005), syntactic parsing may have been a particularly informative subskill to investigate in relation to Korean speakers’ automatization of L2 English. In general, there is a case for studying automatization in bilingual speakers of typologically distinct languages as it will avoid issues of positive orthographic, lexical, and syntactic transfer (for insightful discussion, see Koda, Reference Koda2005, Reference Koda2007).
Whereas the sentence construction task made heavy demands on syntactic parsing, the sentence verification task may have invited the use of top-down, concept-driven knowledge (i.e., world knowledge, background knowledge). Participants were asked to evaluate the plausibility of sentences, which often required their use of factual knowledge or personal experience. For instance, one participant commented that the sentence Jim passed the exam because he was happy could be either sensible or nonsensible. Though experience tells us that passing an exam makes people feel happy, and not the other way around, a reader might make the additional inference that Jim passed the exam because he was happy and happy people are more productive. It is important to note that this kind of additional step in the interpretation process would only have occurred in items that were intended to be nonsensible (because the sensible items were inherently straightforward). That only the native speakers made disproportionately more errors on these nonsensible items (by judging them as sensible) attests to the mental flexibility that skilled readers bring to the meaning making task (e.g., Grabe & Stoller, Reference Grabe and Stoller2011; Koda, Reference Koda2005; Schmalhofer & Perfetti, Reference Schmalhofer and Perfetti2007).
Given that default processing did not yield a congruent interpretation for the nonsensible items, it is all the more surprising that these items were indispensable in discriminating between the different groups in terms of their automaticity. Recall that when we analyzed the sensible items (hits) only, all the patterns expected under an automatization account disappeared. However, when RTs for both hits and correct rejections were considered, CVs decreased with increasing proficiency level and correlated strongly with the CVs on the sentence construction task. Although further research would be needed to clarify this issue, this may suggest that the sentence verification task benefitted from having sentences that tested participants’ processing flexibility (i.e., the nonsensible sentences, which included different types of violations) and that, perhaps, processing flexibility factors into the CV when computed for more complex tasks such as sentence verification.
The discussion of plausible and implausible items resonates with a similar question in the area of implicit and explicit knowledge, namely, whether grammatical and ungrammatical grammaticality judgment test items measure the same or different types of knowledge. Researchers who have found that grammatical and ungrammatical items perform differently (R. Ellis, Reference Ellis2005; Godfroid et al., Reference Godfroid, Loewen, Jung, Park, Gass and Ellis2013; Gutiérrez, Reference Gutiérrez2013; Zhang, Reference Zhang2013) have taken this to mean that grammatical items measure implicit knowledge and ungrammatical items measure explicit knowledge. Therefore, we had not anticipated that the implausible sentence verification items, which are conceptually similar to ungrammatical grammaticality judgment test items, would help to differentiate our three groups’ CVs, as automatic processes (captured by the CV) are said to operate on implicit or procedural, rather than analyzed explicit, knowledge (e.g., Anderson, Reference Anderson2007; DeKeyser, Reference DeKeyser, Williams and Van Patten2007b; R. Ellis, Reference Ellis2005; Paradis, Reference Paradis2004, Reference Paradis2009; Segalowitz & Hulstijn, Reference Segalowitz, Hulstijn, Kroll and De Groot2005).
Longitudinal versus cross-sectional studies of automatization
Ortega and Iberri-Shea (Reference Ortega and Iberri-Shea2005) noted that “many questions concerning L2 learning are fundamentally questions of time and timing” (p. 27). The development of automaticity is no exception, as it requires months or years of deliberate practice to occur (DeKeyser, Reference DeKeyser2007a). Because automatization is a developmental phenomenon, it lends itself naturally to being studied with a within-subject design, longitudinally, or in a training study (for examples see Akamatsu, Reference Akamatsu2008; DeKeyser, Reference DeKeyser1997; Hulstijn et al., Reference Hulstijn, Van Gelderen and Schoonen2009; Robinson, Reference Robinson1997; Segalowitz & Segalowitz, Reference Segalowitz and Segalowitz1993, exp. 2; Segalowitz, Segalowitz, & Wood, Reference Segalowitz, Segalowitz and Wood1998; Segalowitz, Watson, & Segalowitz, Reference Segalowitz, Watson and Segalowitz1995; Serafini, Reference Serafini2013). However, given the complexities involved in sentence-level processing, the time span needed to observe significant progress in L2 automatization may well exceed a couple of years,Footnote 7 which raises the question of how feasible it is for many researchers, especially in early career, to trace the development of automaticity longitudinally. As stated in the introductory section, a possible reason why Hulstijn et al. (Reference Hulstijn, Van Gelderen and Schoonen2009) did not find a significant within-subject CV reduction in high school students at Grades 8, 9, and 10 is that 2 years may not have sufficed for processing efficiency in these intermediate-level learners to develop.
The present study examined the development of L2 automaticity by using a cross-sectional design (see also Harrington, Reference Harrington2006; Phillips et al., Reference Phillips, Segalowitz, O'Brien and Yamasaki2004; Rodgers, Reference Rodgers2011; Segalowitz & Freed, Reference Segalowitz and Freed2004; Segalowitz & Segalowitz, Reference Segalowitz and Segalowitz1993; Serafini, Reference Serafini2013). An advantage of cross-sectional studies is that it is easier to sample participants from more clearly distinct developmental stages; however, between-subject experiments do rest on two assumptions. First, all participants are assumed to follow a common developmental path in terms of automatization. Second, a higher proficiency level means a higher level of automaticity. Regarding the first assumption, Anderson's ACT-R theory suggests a common trajectory of L2 skill development, whereby all learners move from the declarative, through the procedural, to the automatic stage (e.g., Anderson, Reference Anderson2007). The distinction between speedup and automatization (Segalowitz & Segalowitz, Reference Segalowitz and Segalowitz1993) also carries implications for the developmental aspect of skill acquisition, namely, mere speedup precedes automatization (Wickens, 1992). Crucial in between-subject designs is the second assumption that level of proficiency indexes level of automaticity. Although the RT data of the current study satisfied this assumption, it might not be always the case. As documented in Swain (Reference Swain2000), a French immersion program attracted much criticism because immersion students consistently had problems with accuracy, while showing nativelike fluency in their oral production. In this case, the immersion students had achieved a high level of L2 automaticity, whereas their proficiency level, which takes both accuracy and fluency into account, was lagging behind. The reverse scenario of high accuracy and low fluency also exists, for example, in many ESL programs in East Asia, which prepare students for passing high-stakes English language tests but do not give them enough opportunity to put their linguistic knowledge into practice. Thus, the link between proficiency level and degree of automatization ought to be evaluated within the context in which language learning takes place.
It is important to keep in mind that automatization occurs in a relatively late stage of L2 development (Anderson, Reference Anderson1983; Segalowitz & Hulstijn, Reference Segalowitz, Hulstijn, Kroll and De Groot2005; Wickens, 1992). This is also illustrated in Anderson's skill acquisition theory where learners in the earlier phases primarily focus on acquiring rules and facts, while those in the later phases are more likely to fine tune their processing skills. Because beginning or even intermediate learners are likely to work on both knowledge accumulation and skill development (Hulstijn et al., Reference Hulstijn, Van Gelderen and Schoonen2009), the development of automatic processing skills might be more visible among relatively advanced learners.
CV as a measure of automatization in reading
The use of CV as a measure of automatization rests on a subtle distinction between a proportional reduction of processing variability (speedup) and a disproportional reduction (automatization) relative to an increase in processing speed. Segalowitz argued that a disproportional reduction in standard deviation, and hence a smaller CV, will only occur if the underlying processes are restructured (Segalowitz, Reference Segalowitz2010; Segalowitz & Segalowitz, Reference Segalowitz and Segalowitz1993). Thus, in Segalowitz’ view, the restructuring of cognitive processes that enable skilled performance is key to automatization.
However, in an early account of automaticity, McLaughlin (Reference McLaughlin1987) observed that “[cognitive] theory is not precise enough about when and how restructuring occurs” (p. 152). Despite the advances that have been made since then, we argue his general assessment still holds true, especially as regards the restructuring of communicative competence in the four primary skills (i.e., speaking, listening, writing or reading) rather than the restructuring of rule knowledge. A reason for this is that communicative skills are so complex. They comprise a variety of subskills, which draw on distinct linguistic (and sometimes nonlinguistic) knowledge. Therefore, in theory automatization could entail reorganizing the different subskills involved (e.g., by eliminating a subcomponent) or it could consist of the automatization of one particular subskill only (e.g., more efficient word recognition in reading). Finally, because the subskills themselves differ in the knowledge sources they require, some component processes likely cannot be automatized at all.
If we look at the automatization of reading, which was the main skill targeted in this study, it appears that the phenomenon can be understood best in terms of a component skills approach (Carr & Levy, Reference Carr and Levy1990; Koda, Reference Koda2005, Reference Koda2007) and, optionally, the assumption of limited attentional capacity (LaBerge & Samuels, Reference LaBerge and Samuels1974). The component skills approach to reading holds that reading is “the product of a complex information-processing system, involving a constellation of closely related mental operations” (Koda, Reference Koda2005, p. 19; see also Carr, Brown, Vavrus, & Evans, Reference Carr, Brown, Vavrus, Evans, Carr and Levy1990, p. 5). The different mental operations that make up reading are widely believed to be organized hierarchically (e.g., Breznitz, Reference Breznitz2006; Grabe & Stoller, Reference Grabe and Stoller2011; Koda, Reference Koda2005; Stanovich, Reference Stanovich2000), from lower level, data-driven processes to higher level, conceptual processes. On the assumption that attention is a limited resource (LaBerge & Samuels, Reference LaBerge and Samuels1974), the inefficient, effortful execution of lower level processes (e.g., word decoding) will negatively affect the amount of attention available for higher level reading processes. “Because inefficient decoding is resource demanding, it severely restricts readers’ involvement in higher order comprehension operations, such as text-information integration, inference, and reasoning” (Koda, Reference Koda2007, p. 23).
Although not all accounts of automatization in reading assume that attention is capacity limited (see Stanovich, Reference Stanovich2000), there does seem to be a consensus that reading subprocesses differ in whether they can be automatized. Likewise, there is theoretical agreement that some reading subprocesses may be automatized before others. From a component skills perspective, then, it seems important to analyze the specific cognitive demands of one's sentence processing tasks, as it is the automatization of these subskills that CV will measure primarily. In effect, this amounts to measuring the automatization of a particular reading subskill; as of yet, we do not see how CV computations for sentence-level processing tasks could address the question of how the different subskills are organized (and potentially reorganized or restructured) within the information-processing system as a result of practice.
Finally, because of the high temporal sensitivity of CV we believe it may work best with tightly controlled tasks such as lexical decision, semantic classification, and sentence construction. Given that other fluency measures are available (see Andringa, Olsthoorn, Van Beuningen, Schoonen, & Hulstijn, Reference Andringa, Olsthoorn, van Beuningen, Schoonen and Hulstijn2012; De Jong, Steinel, Florijn, Schoonen, & Hulstijn, Reference De Jong, Steinel, Florijn, Schoonen and Hulstijn2011; Schoonen et al., Reference Schoonen, Gelderen, Glopper, Hulstijn, Simis and Snellings2003; Van Gelderen et al., Reference Van Gelderen, Schoonen, De Glopper, Hulstijn, Simis and Snellings2004) a challenge for future researchers is to determine for what tasks CV is a good measure of cognitive fluency and in which cases other, more global fluency measures may be more informative.
Limitations and future prospects
The current study did not use all of the tasks that Hulstijn et al. (Reference Hulstijn, Van Gelderen and Schoonen2009) did. Because our participants performed word and sentence processing tasks in L2 English only, we cannot make any claims about the automaticity of their L1 Korean processing skills. It is conceivable that the ESL learners transferred some of their L1 reading skills to the L2 sentence processing tasks, even though the typological distance between Korean and English would seem to preclude large amounts of positive transfer. Nonetheless, it would be prudent to include the corresponding L1 tasks in future studies so as to obtain measures of L2 specific cognitive fluency.
In the preceding discussion sections, we emphasized the importance of the task and task characteristics in CV studies of automatization. Although we analyzed the sentence construction and sentence verification tasks at some length, these analyses may benefit from additional, independent support. For example, RT experiments can be triangulated with other psycholinguistic measures, such as ERPs, to obtain converging evidence. As a case in point, Phillips et al. (Reference Phillips, Segalowitz, O'Brien and Yamasaki2004) validated the use of CV by comparing the CV analysis with ERP measures that were recorded during a semantic priming task measuring word recognition skills. To justify the use of CV in sentence processing, similar comparisons need to be made at the sentence level.
CONCLUSION
The purpose of the study was to investigate the development of L2 automaticity at the sentence level and to validate the use of CV as a measure of L2 automatization in sentence processing. The study was a partial conceptual replication of Hulstijn et al. (Reference Hulstijn, Van Gelderen and Schoonen2009), using the same analyses on a subset of the same speeded tasks. Compared to Hulstijn et al., however, the current study made important changes to the data collection and data analysis procedures: (a) participants’ L1 background was restricted to Korean, which is typologically distinct from English; (b) learners’ English proficiency was assessed independently via an online English diagnostic test; (c) participants were likely more advanced than the high school students in Hulstijn et al.; and (d) the sentence stimuli in the speeded tasks were made shorter.
As a result, the CV in the sentence-related tasks did decrease as learners’ proficiency level increased, suggesting that the different ESL groups in this study differed in the automatization of their L2 processing skills. This result was found regardless of whether the data were cleaned; it lends support to Segalowitz and Segalowitz’ (Reference Segalowitz and Segalowitz1993) hypothesis. Hence, the data from the Korean college-level ESL students seem to justify the expansion of the CV measure to the sentence processing level.
However, whether a smaller CV really indicates increased automaticity is still an open question. The assumptions on which Segalowitz and Segalowitz’ (Reference Segalowitz and Segalowitz1993) logic is based require further theoretical elaboration and empirical support; in particular, the jury is still out on whether the elimination of initially slow subcomponents really causes a disproportionate reduction in standard deviation and whether the argument also goes the other way. By the same token, it seems to be difficult to falsify Segalowitz and Segalowitz’ hypothesis. If CV did not decrease, it is taken to indicate mere speedup, whereas if CV decreased, it is a sign of automatization. As the theoretical foundation for CV (a smaller CV means a higher level of automaticity) becomes a reason for data interpretation, the argument for the validity of CV appears to be somewhat circular. In addition to crunching numbers, we will need psycholinguists’ insights into how L2 learners process linguistic information, what cognitive subcomponents can be actually removed through practice, and what aspects of L2 processing can or cannot be automatized. Although CV experiments are far from solving the automaticity puzzle, they certainly provide interesting behavioral pieces that will keep psycholinguists busy for some time to come.
APPENDIX A
Word stimuli for the semantic classification task
Stimuli for the sentence verification task
Stimuli for the sentence construction task
APPENDIX B
English Language Learning Background Questionnaire
1. Age: ______
2. Gender: Male □ Female □
3. Mother tongue (First language): _______________
4. Other language spoken at home as a child:_______________
5. Academic status (please choose all applied to you)
□ Freshman □ Sophomore □ Junior □ Senior □ MA □ PhD □ EAP (advanced ESL)
□ Other (please specify) _______________
6. How long have you been studying in the English-speaking countries (including previous education, if applicable)?
For ______ years ______ months.
7. At what age did you begin studying English? ______
8. At what age were you first exposed to English communication environment? ______
For how long? ______
9. Have you learned (or studied) any foreign languages other than English? □ Yes □ No
If yes, please specify all information including 1) which language, 2) how long, and 3) in what context(s)
I have learned (or studied) _______________ for ______ month(s) context: _______________
I have learned (or studied) _______________ for ______ month(s) context: _______________
[Example] I have learned French for 3 years as a part of my coursework in high school.
I have learned German for 2 year because my family moved to Germany during that time
10. English Proficiency Test Score (if you remember)
Test: _______________ (e.g., TOEFL, TOEIC, ILETS, TEPS, MSUELT) Total score: ______
Test Date: _____________________ (i.e., summer 2007) (Reading score:______)
11. How much time do you usually spend using English from outside of the classroom?
Your privacy will be protected. Thank you for your participation.
APPENDIX C
Deleted items from the sentence verification task (16 out of 60 items)
6. The boy whose sister plays soccer is tall. (sensible)
8. Good teachers are believed to be helpful. (sensible)
11. Jim passed the exam because he was happy. (nonsensible)
15. Jim was borrowed the note from his friend. (nonsensible)
16. A mailman is delivered by many postcards. (nonsensible)
20. If you study hard, you will fail the exam. (nonsensible
24. Americans go to work seven days a week. (nonsensible)
27. The student who came late left very early. (sensible)
28. Take off your jacket if you feel cold. (nonsensible)
30. Why don't you study harder if you feel tired? (nonsensible)
42. English was taught by the new teacher. (sensible)
48. April comes right before March. (nonsensible)
50. Jane expects a church to go to Sunday. (nonsensible)
52. He went inside, though it started to rain. (nonsensible)
56. Jane had the desk fix the broken legs. (nonsensible)
57. My uncle made me for a snowman. (nonsensible)
Deleted plausible items from the sentence verification task (2 out of 25 items)
6. The boy whose sister plays soccer is tall. (sensible)
42. English was taught by the new teacher. (sensible)
Deleted items from the sentence construction task (9 Items)
ACKNOWLEDGMENTS
We thank Jan Hulstijn, Amos Van Gelderen, and Rob Schoonen for sharing their stimuli with us and kindly taking the time to answer our questions.
