In the present study, we investigate what knowledge types native speakers (NSs) and nonnative speakers (NNSs) of English draw on to make judgments regarding the acceptability of English sentences. Unlike previous studies, we employ a methodology new to this line of research—namely, eye-tracking. Specifically, we compare participants’ eye-movement patterns as they perform judgment tasks with different design features (i.e., time pressure and item grammaticality) hypothesized to elicit the use of different types of linguistic knowledge, in this case, explicit and implicit knowledge.
BACKGROUND
Implicit and Explicit Knowledge
The distinction between implicit and explicit knowledge is now widely accepted in both cognitive psychology (Cleeremans & Jiménez, Reference Cleeremans, Jiménez, French and Cleeremans2002; Dienes & Perner, Reference Dienes and Perner1999; Reber, Reference Reber1976) and SLA (N. C. Ellis, Reference Ellis1994; R. Ellis, 2005; Rebuschat, Reference Rebuschat2013; Williams, Reference Williams, Ritchie and Bhatia2009); it is also supported by neurobiological evidence (Paradis, Reference Paradis2004). The fundamental difference between the two types of knowledge lies in whether individuals are aware of what they know. In the case of explicit knowledge, individuals are aware of and thus can consciously apply what they know; in the case of implicit knowledge, individuals are not aware of what they know and thus use their knowledge without being conscious of it. Linked to this fundamental difference is the issue of the accessibility of the two types of knowledge. Explicit knowledge consists of declarative representations that, in general, can be accessed only through controlled processing. Implicit knowledge involves procedural representations and thus can be accessed through automatic processing. The distinction between controlled and automatic processing is attentional in nature (Bialystok, Reference Bialystok1994; Shiffrin & Schneider, Reference Shiffrin and Schneider1977). We seek to exploit this distinction in the current study by looking for evidence of controlled and automatic processing in participants’ eye-movement records, the assumption being that automatic processing reflects access to implicit knowledge (or automatized explicit knowledge; DeKeyser, Reference DeKeyser, Doughty and Long2003)Footnote 1 and controlled processing draws on analyzed explicit knowledge (see R. Ellis, 2005).
With regard to SLA, second language (L2) learners’ deployment of their linguistic knowledge depends in part on the extent of their implicit and explicit representations of the L2 and in part on the type of language use involved. Second language learners who have experienced only traditional form-focused instruction may have to rely mainly on explicit knowledge because that may be the main type of knowledge they possess. In contrast, L2 learners who have learned a language naturalistically may draw primarily on implicit knowledge. Furthermore, tasks that require a focus on meaning and fluency (with the time pressure that comes from participating in a communicative event) favor the use of implicit knowledge, whereas tasks that entail a focus on linguistic forms and exert no time pressure allow for the use of explicit knowledge. However, tasks only predispose learners toward the use of one type of knowledge or the other; they cannot determine which type of knowledge is used, as, in part, what knowledge is used depends on the knowledge sources available to the individual learner.
Grammaticality Judgments as Measures of Implicit and Explicit Knowledge
Grammaticality judgment test (GJT) is the common term used for the elicitation methodology adopted in this study.Footnote 2 In the current study, GJTs are used with reference to target language normsFootnote 3 with an eye toward understanding processing differences associated with different GJT items (grammatical, ungrammatical, timed, and untimed) and how these differences relate to the linguistic knowledge measured (implicit or explicit). Time pressure may constrain what knowledge representations are available to the test taker because it suppresses reflection and makes it more difficult to access explicit knowledge (R. Ellis, 2005; Loewen, Reference Loewen, Ellis, Loewen, Elder, Erlam, Philp and Reinders2009). Grammatical and ungrammatical sentences differ in that only ungrammatical sentences have a clear critical area (i.e., the error) that may invite the use of explicit, declarative knowledge (see Bialystok, Reference Bialystok1979; R. Ellis, Reference Ellis1991; Hedgcock, Reference Hedgcock1993).
Because L2 learners and educated NSs are widely assumed to possess both implicit and explicit knowledge of the target language (see, e.g., the contributions in N. C. Ellis, Reference Ellis1994; R. Ellis et al., Reference Ellis, Loewen, Elder, Erlam, Philp and Reinders2009; Hulstijn & Ellis, Reference Hulstijn and Ellis2005; Rebuschat & Williams, Reference Rebuschat and Williams2012; Sanz & Leow, Reference Sanz and Leow2011), it is unclear as to which knowledge sources—implicit, explicit, or both—individuals draw on when making judgments. The types of knowledge that participants use could also be influenced by certain task design features, including item grammaticality and the presence or absence of time pressure. Hence, the aim of the present article is to employ eye-tracking methodology as we seek to refine our understanding of the construct validity of GJTs.
In two pioneering studies conducted to explore learners’ use of explicit (analyzed) versus implicit (unanalyzed) knowledge,Footnote 4 Bialystok (Reference Bialystok1979, Reference Bialystok1982) investigated participants’ performance on GJTs with different task characteristics and instructions. The 1979 study looked at the effects of time pressure: L2 French learners performed auditory GJTs under 3-s and 15-s time limits. Because having more time did not improve participants’ performance on the ungrammatical items (for which they could presumably do a mental search for a rule), Bialystok concluded that, overall, auditory GJTs encourage the use of implicit knowledge. However, manipulating task instructions, such as asking participants to identify, correct, or describe the error, could push test takers to rely on explicit or analyzed knowledge, as shown by strong correlations between scores on GJTs that included an additional error identification, error correction, or rule description component (Bialystok, Reference Bialystok1979, Reference Bialystok1982). Task modality also mattered, with written GJTs arguably allowing for controlled, nonautomatic knowledge retrieval, unlike their auditory counterparts (Bialystok, Reference Bialystok1982).
More recently, there have been multiple factor-analytic studies investigating the construct validity of GJTs and the roles of time pressure and item grammaticality in influencing the types of knowledge used by learners to make their judgments. In these studies, participants were given a battery of three to five tests, and the accuracy scores were submitted to factor analyses to determine the underlying relationships among the tests. Most studies have found that time pressure, rather than item grammaticality, was the influencing factor in L2 learners’ test performance (Bowles, Reference Bowles2011; R. Ellis, 2005; R. Ellis & Loewen, Reference Ellis and Loewen2007; Han & Ellis, Reference Han and Ellis1998; Zhang, Reference Zhang2014). This finding has been interpreted as showing that timed and untimed written GJTs provide relatively separate measures of implicit and explicit knowledge, respectively. In contrast, Gutiérrez (Reference Gutiérrez2013) identified grammaticality, rather than time pressure, as the crucial variable in his factor analyses, leading him to propose that grammatical items better measured implicit knowledge whereas ungrammatical items better measured explicit knowledge. Partial support for the influential role of grammaticality comes from R. Ellis (2005), who found that ungrammatical items on the untimed GJT correlated more strongly with measures of explicit knowledge than grammatical items did.
Evidence for the validity of GJTs and other tests in the battery as measures of implicit and explicit knowledge has consisted primarily of participants’ accuracy scores. Factor analyses can reveal which tests pattern together (exploratory factor analysis) or whether the proposed patterning can explain the data better than a competing model (confirmatory factor analysis). However, what the patterning means—that is, how to interpret the latent variables—falls to the researcher, and, at this time, alternative interpretations (e.g., test difficulty) cannot be ruled out. In an attempt to respond to this criticism and to illuminate the processes that take place during GJT performance, the current study utilizes eye-tracking methodology to test whether previous accuracy-based claims, which rest on a product measure, can be supported by processing data. To this end, we recorded participants’ eye movements and determined whether GJT items hypothesized to measure different types of knowledge are also processed differently.
Eye Movements and Grammaticality Judgments
Eye-movement registration, or eye-tracking, refers to the recording of an individual’s eye movements as he or she performs a particular task. Eye movements are a useful data source in cognitive research because the point of gaze serves as an index of overt attention (Wright & Ward, Reference Wright and Ward2008) that can be used to make inferences about participants’ corresponding covert attentional processing, or mental focus. This assumption of an eye-mind link (Reichle, Pollatsek, & Rayner, Reference Reichle, Pollatsek and Rayner2012) underlies most psycholinguistic research using eye-tracking.
Eye-movement data have been analyzed extensively in first-language (L1) and L2 sentence processing research (see Clifton, Staub, & Rayner, Reference Clifton, Staub, Rayner, van Gompel, Fischer, Murray and Hill2007, for a review of the L1 literature, and Dussias, Reference Dussias2010; Frenck-Mestre, Reference Frenck-Mestre2005; Roberts & Siyanova-Chanturia, Reference Roberts and Siyanova-Chanturia2013, for L2-focused reviews). The bulk of this research has investigated issues in parsing, which is the online computation of syntactic structure, by presenting participants with temporarily or globally ambiguous sentences. Comparatively less eye-movement research has dealt with semantically or syntactically anomalous sentences (see Warren, Reference Warren, Liversedge, Gilchrist and Everling2011), and, to our knowledge, only a few eye-tracking studies have employed grammatical and ungrammatical sentences to investigate readers’ knowledge of the violated structures (e.g., N. C. Ellis, Hafeez, Martin, Chen, & Boland, Reference Ellis, Hafeez, Martin, Chen and Boland2014; Keating, Reference Keating2009; Lim & Christianson, Reference Lim and Christianson2014; Sagarra & Ellis, Reference Sagarra and Ellis2013). Evidence for the availability of targetlike knowledge representations during reading comes from so-called sensitivity effects. Readers are said to show sensitivity when they fixate significantly longer on or regress more frequently from the error than from its grammatical counterpart in a sentence.Footnote 5
In a study on gender agreement, Keating (Reference Keating2009) found that native Spanish speakers and advanced L2 learners looked longer at (in total) and regressed more from incongruous adjectives than congruous ones. Keating interpreted these effects as showing that the advanced L2 speakers had acquired grammatical gender, even though they were not able to process gender as efficiently as NSs. Unlike natives, the advanced L2 speakers detected mismatches in agreement only in adjacent noun-adjective pairs, whereas NSs could also compute agreement across syntactic boundaries. For our purposes, it is important to note that Keating used processing data (eye-movement measures) to make inferences about participants’ underlying grammatical representations of Spanish gender agreement. Because Keating conceived of gender as an abstract feature belonging to Universal Grammar (p. 504), he may have assumed (in line with that theoretical framework) that knowledge of grammatical gender, as reflected in a real-time, meaning-oriented processing task, is by definition implicit. This assumption may also hold true for other researchers adopting grammatical sensitivity as a measure of grammar acquisition (for a review, see Lim & Christianson, Reference Lim and Christianson2014). Therefore, if the goal is to differentiate between implicit and explicit knowledge using eye-movement data, analyses other than word-based comparisons of fixation times may be needed (see Godfroid & Winke, Reference Godfroid, Winke and Rebuschatin press). Consequently, in this study we examine reading patterns across an entire sentence, rather than sensitivity data for individual words, using what is known as a scanpath analysis (see the Method section for a description).
A scanpath or eye-movement pattern is “a trace of a participant’s eye-movements in space and time” (Holmqvist et al., Reference Holmqvist, Nyström, Andersson, Dewhurst, Jarodzka and Van de Weijer2011, p. 253). Scanpaths are useful tools in reading research because they consider the entire sequence of eye movements in a trial rather than focusing on a localized subset of events. We used scanpath analysis (with appropriate modifications to fit our data) to address the following research questions:
1. Do written, timed and untimed GJT items elicit different scanpaths among NSs and NNSs?
2. Do written, grammatical and ungrammatical GJT items elicit different scanpaths among NSs and NNSs?
As previous studies have found effects of both time pressure (Bowles, Reference Bowles2011; R. Ellis, 2005; R. Ellis & Loewen, Reference Ellis and Loewen2007; Han & Ellis, Reference Han and Ellis1998; Zhang, Reference Zhang2014) and grammaticality (Gutiérrez, Reference Gutiérrez2013) on GJT accuracy scores, we expect the answer to both questions to be affirmative for NNSs. Because NS data have most commonly been excluded from previous factor analyses, we do not make specific predictions about the effects of time and grammaticality on NS processing.
We explore whether scanpaths differ depending on GJT properties. In general, the presence or absence of regressions (i.e., right-to-left eye movements) is hypothesized as a key distinctive feature of participants’ reading behavior under different task conditions. Regressions likely indicate controlled processing (e.g., Clifton & Staub, Reference Clifton, Staub, Liversedge, Gilchrist and Everling2011; Clifton et al., Reference Clifton, Staub, Rayner, van Gompel, Fischer, Murray and Hill2007; Reichle, Warren, & McConnell, Reference Reichle, Warren and McConnell2009) and may therefore reflect attempts to retrieve explicit knowledge. Conversely, fluent, left-to-right reading is “the ‘default’ reading process” (Reichle et al., Reference Reichle, Warren and McConnell2009, p. 7) for NSs showing good text comprehension and is, thus, suggestive of automatic processing (see the Analysis and Statistical Analysis sections for further details).
METHOD
Participants
The study took place at a large midwestern university in the United States. Forty NNSs and 20 NSs participated in the study. The NSs were undergraduate students enrolled in language teaching courses. Eighteen were female and two male. Nineteen of them had studied a foreign language before. The NNSs had various L1 backgrounds, including Chinese (n = 21), Arabic (n = 12), Thai (n = 4), Korean (n = 2), and Japanese (n = 1). Their average age was 22 (range: 18–47), and all of them had moved to the United States after puberty (range: 17–46). No participants had studied at a university for more than 2 years at the time of data collection. The NNSs were enrolled in two different programs at an English language center. Twenty students were enrolled in an intensive English program, consisting of four levels of full-time, preuniversity English as a second language (ESL) courses. Eleven students were at Level 3, and nine were at Level 4. The other 20 students were from Level 5 English for academic purposes (EAP) classes. These students were taking part-time ESL courses as well as regular undergraduate courses at the university. Preliminary analyses did not find significant or meaningful differences in GJT scores based on institutional proficiency levels; consequently, all the NNSs were treated as one group for the analysis.
Tasks
Two GJTs were administered: one GJT under time pressure (timed GJT) and then an identical GJT without time pressure (untimed GJT). Both tasks were integrated into the EyeLink 1000 eye-tracking system to collect participants’ eye movements and judgment responses during reading. The GJTs consisted of 68 English sentences adapted from Loewen (Reference Loewen, Ellis, Loewen, Elder, Erlam, Philp and Reinders2009) and R. Ellis (2005), which involved 17 different linguistic structures with two grammatical and two ungrammatical sentences per structure (see the Appendix). The structures were intended to cover a variety of grammatical structures at various levels of L2 proficiency, and the GJTs were administered to learners at similar levels of proficiency to those in R. Ellis (2005) and Loewen (Reference Loewen, Ellis, Loewen, Elder, Erlam, Philp and Reinders2009). In addition to using the original GJT items, for purposes of counterbalancing, we created another version containing the respective grammatical and ungrammatical counterparts of the original 68 sentences. For example, the counterpart of the original item He plays soccer very well became *He plays very well soccer. Only the targeted linguistic structure was manipulated, and the rest of the sentence was unchanged. Thus, the overall length of the sentence as well as the size of the interest areas (see the Interest Areas section) was approximately identical for each pair of sentences. Participants read 34 grammatical and 34 ungrammatical sentences, with grammaticality counterbalanced among participants. However, each individual participant received the same version of the test in both the timed and untimed conditions. The amount of time allocated for each item in the timed GJT was adapted from R. Ellis’s (2005) and Loewen’s (Reference Loewen, Ellis, Loewen, Elder, Erlam, Philp and Reinders2009) studies, in which 20% was added to the median response times provided by NS pilot data to account for L2 learners’ slower processing speed. The time limits calculated in this manner ranged from 1,800 ms to 6,240 ms across the test items, whereas the length of the sentences ranged from 5 to 12 words (M = 7.9 words).
Procedures
The participants completed the timed GJT first, followed by the untimed GJT. At the end of the experiment, participants filled out a language learning background questionnaire.
Timed GJT
Participants read one test sentence at a time on the computer while their eye movements were recorded by a desk-mounted eye-tracker with head support. Each sentence was presented for the predetermined fixed time limit and then disappeared automatically. Participants were instructed to read and make a grammaticality judgment on the sentence by pressing the designated button (either grammatical or ungrammatical) on a controller within the given time limit. For the last 34 of the 68 items, participants were asked to provide source attributions and certainty ratings for their judgments (see Rebuschat, Reference Rebuschat2013); however, these data are not included in the current analysis.
Untimed GJT
After a 15-min break, participants completed the untimed GJT, which was exactly the same as the timed GJT, except that participants were allowed unlimited time to make their judgments. For the second half of the test items, participants once again provided source attributions and confidence ratings.
The NSs spent an average total time of 2 min 24 s and 3 min 5 s reading the sentences in the timed test and untimed test, respectively. The average reading times for the NNSs showed greater differences between the two tests: 2 min 46 s in the timed test and 6 min 16 s in the untimed test. Note that the average reading times reported here reflect only the reading times on the test items. The total duration of one test (i.e., timed or untimed GJT) ranged between 17 and 25 min due to the additional components in the experiment procedure (i.e., source attributions and certainty ratings, eye-tracker setup, and practice session).
Data Analysis
Inspired by previous scanpath analyses in reading research (e.g., Holmqvist, Holsanova, Barthelson, & Lundqvist, Reference Holmqvist, Holsanova, Barthelson, Lundqvist, Hyönä, Radach and Deubel2003; Hyönä, Lorch, & Kaakinen, Reference Hyönä, Lorch and Kaakinen2002; Von der Malsburg, Kliegl, & Vasishth, Reference Von der Malsburg, Kliegl and Vasishth2014; Von der Malsburg & Vasishth, Reference Von der Malsburg and Vasishth2011) and in reading and writing research (e.g., Johansson, Wengelin, Johansson, & Holmqvist, Reference Johansson, Wengelin, Johansson and Holmqvist2010; Wengelin et al., Reference Wengelin, Torrance, Holmqvist, Simpson, Galbraith, Johansson and Johansson2009), we investigated participants’ sequential eye-movement patterns during GJT performance. An advantage of scanpath analysis is that “long effects that are spread over several words can be captured in whole…and do not have to be chopped up into pieces” (Von der Malsburg et al., Reference Von der Malsburg, Kliegl and Vasishth2014, p. 3). Furthermore, because every trial is represented by its own scanpath, there is no risk of blending several distinct and sometimes competing fixation patterns into a single analysis (Von der Malsburg & Vasishth, Reference Von der Malsburg and Vasishth2011).
Von der Malsburg and Vasishth (Reference Von der Malsburg and Vasishth2011) performed a scanpath analysis of existing data (regression sequences) from Meseguer, Carreiras, and Clifton’s (Reference Meseguer, Carreiras and Clifton2002) study on the processing of temporarily ambiguous sentences. Von der Malsburg and Vasishth (Reference Von der Malsburg and Vasishth2011) quantified the similarity among scanpaths of individual trials and identified three representative “scanpath signatures” (p. 109): (a) a regression to the beginning of the sentence followed by rereading, (b) a regression to the beginning of the sentence without rereading, and (c) a regression from the disambiguating region (the verb) to the ambiguous region (the adverbial clause). Of these three patterns, only a regression followed by rereading occurred significantly more often when the sentence was disambiguated in the nonpreferred manner (i.e., a garden-path sentence that required reanalysis). This led Von der Malsburg and Vasishth to conclude that regression plus rereading is most closely associated with the process of syntactic reanalysis.
In the present study, we computed scanpaths only for items that were correctly judged (i.e., grammatical sentences judged grammatical and ungrammatical sentences judged ungrammatical). This resulted in the elimination of about 32% of our dataset. In analyzing correct responses only, we followed the convention of reading time analyses in previous processing studies (e.g., L1: Garnsey, Pearlmutter, Myers, & Lotocky, Reference Garnsey, Pearlmutter, Myers and Lotocky1997; Phillips, Reference Phillips2006; L2: Dussias & Piñar, Reference Dussias and Piñar2010; Juffs, Reference Juffs2005; Omaki & Schulz, Reference Omaki and Schulz2011; White & Juffs, Reference White, Juffs, Flynn, Martohardjono and O’Neil1998). Furthermore, reading patterns associated with correct responses are more likely to reflect targetlike knowledge retrieval, whether explicit or implicit, and are therefore directly relevant for our purposes. In contrast, for incorrect responses it is difficult to differentiate nontargetlike knowledge, which could be implicit or explicit, from indeterminate knowledge or no knowledge. In the case of indeterminate or no knowledge, the question of implicit or explicit knowledge is moot.
Scanpaths can be represented in different ways, including as symbol strings, vectors, and attention maps (Holmqvist et al., Reference Holmqvist, Nyström, Andersson, Dewhurst, Jarodzka and Van de Weijer2011). In this study, we opted for a string-based representation, in which string symbols (e.g., A, B, C, D) represent visits by the eyes to functional interest areas, also termed semantic interest areas by Holmqvist and colleagues (see Figures 1, 2, and 3, for examples). We defined four functional interest areas based on the relative position of the error in the ungrammatical version of a sentence (see the examples in [1] and [2] and the Appendix). Using functional interest areas allowed us to equalize the differences among our target structures. Thus, although the sentences for different structures differed in length, all of them had a sentence-initial region, a primary interest area, and, when the structure allowed it, a spillover region. (Tag questions and sentences with dative alternation do not allow for spillover regions due to the sentence-final location of the structures; see the Appendix.) The number of words within each interest area varied among the structures; however, this was inconsequential for our scanpath analysis because we defined scanpaths on the basis of readers’ eye movements between, rather than within, different interest areas.
Interest Areas
All sentences began with a sentence-initial region, which preceded the primary interest area (PIA). The PIA was defined as the part of the sentence where readers could notice the ungrammaticality for the first time during forward reading, if they were sensitive to it. Grammatical sentences were assigned the same PIA as their ungrammatical counterparts, except that the word(s) now appeared in grammatical form (see the examples in [1] and [2] and the Appendix). A spillover region immediately following the PIA was designated to capture spatially delayed effects in participants’ registration of the grammatical error (see Reichle, Reference Reichle, Liversedge, Gilchrist and Everling2011). We extended the PIA or the spillover region by one or two words when the original region consisted of fewer than five letters, to minimize the amount of skipping and ensuing data loss. Any words following the spillover region were designated as the sentence-final region.
Interest areas are indicated in the examples in (1) and (2), which represent a pair of test sentences for indirect questions; A is the sentence-initial region, B is the PIA, C is the spillover, and D indicates the sentence-final region.
(1) She wanted to know whyhe hadstudiedGerman. (grammatical)
A B C D
(2) She wanted to know whyhad hestudiedGerman. (ungrammatical)
A B C D
Analysis
Fixation durations as well as the number and direction of eye movements were extracted for each interest area in each GJT item. We then conducted an inductive qualitative analysis of a subset of these data to identify relevant scanpath categories. As a first step, we looked at 20% of the participants’ trials for a lexical, a morphosyntactic, and a syntactic structure—namely, since/for, the indefinite article, and the unreal conditional, respectively. We identified recurring patterns in participants’ reading paths for the timed and untimed and grammatical and ungrammatical test items for these structures. This process resulted in an initial classification system with nine different scanpath types. Through an iterative process, involving examination of trials and discussion of their similarities and differences, the original classification system was reduced to four categories, one of which (skipping the PIA and spillover) was eventually discarded (see the final paragraph of the Finished Reading with Regression section for details). In applying the final three categories to NSs’ and NNSs’ reading of all 17 target structures, we found that the scanpath categories captured the data well in that the different scanpaths were represented in all the applicable target structures.Footnote 6
Each scanpath category was composed of different scanpath patterns with one or more common features. The defining features of each scanpath category are described in the following three sections.
No regression
This category consisted of sentences containing no regressions out of the PIA or subsequent parts of the sentence, meaning that the lettering in the symbol strings must occur in alphabetical order. Sentences could be read in a single pass, without any regressions, or sentences could contain regressions within the sentence-initial region. Most importantly, however, there was no regression out of the PIA or a region after that, indicating that participants did not go back to reanalyze previous parts of the sentence once they reached the critical area. Figure 1 shows a participant’s eye movements while reading a sentence with no regressions. Circles represent eye fixations, and arrows represent eye movements between fixations, known as saccades. In Figure 1, all arrows point right, with the exception of a refixation within the PIA on nicer, and they represent forward, as opposed to regressive, saccades. The pattern can be represented symbolically as ABCD.
No regression. The sentence is *I think that he is more nicer and more intelligent than James.
Unfinished reading with regression
Sometimes participants read up until the PIA or spillover area and then regressed to an earlier part of the sentence. They then made their judgments without ever reading the end of the sentence, as shown in the absence of fixations on the sentence-final region (D) in Figure 2. This scanpath is represented as ABCBC. Trials were classified as unfinished reading with regression if (a) the sentence had a sentence-final region but the corresponding symbol string did not have a D and (b) at least one earlier letter in the alphabet followed a later letter in the alphabet (i.e., revisiting A after B or B after C).
Unfinished reading with regression. The sentence is *I think that he is more nicer and more intelligent than James.
Finished reading with regression
The third scanpath category is similar to the second in that participants regressed at least one time while reading. In contrast to the previous category, however, participants also read the sentence-final region.Footnote 7 Any symbol strings containing D (or, more generally, the letter that denoted the last region in that sentence) as well as at least one nonalphabetically ordered sequence (e.g., C before B) were included in this category. For instance, the symbolic representation of the scanpath in Figure 3 is ABCBCDA.
Finished reading with regression. The sentence is *I think that he is more nicer and more intelligent than James.
Our original statistical analysis also included a fourth category of scanpaths in which participants skipped the PIA and spillover region entirely. However, because such trials were rare (3% of the data for both NSs and NNSs) and arguably anomalous, we removed this category from the analysis. Doing so did not influence our results. Finally, 43 trials (0.8%) did not fall into any of the preceding categories, mainly due to participants’ performance error during the tasks (e.g., a judgment was made before any reading occurred); these data were also excluded from the analysis.
Statistical Analysis
To answer the research questions, we tallied the number of items for each scanpath category and examined the frequencies according to the independent variables of L1-L2 status, time pressure, and grammaticality. We then analyzed the effects of these predictors statistically using a mixed-effects multinomial logistic regression. We entered two length measures as control variables into the regression model. To account for the fact that unfinished reading with regression could only occur if there was a sentence-final region, the length of the sentence-final region (range: 0–4 words) was entered as a covariate. We also added total-sentence length (range: 5–12 words) because this could influence the likelihood of a reader regressing (Kliegl, Grabner, Rolfs, & Engbert, Reference Kliegl, Grabner, Rolfs and Engbert2004; Rayner, Reference Rayner1998, Reference Rayner2009; Reichle et al., Reference Reichle, Warren and McConnell2009; Vitu & McConkie, Reference Vitu, McConkie, Kennedy, Radach, Heller and Pynte2000) and, therefore, could potentially affect participants’ reading patterns.
Multinomial logistic regression is an extension of binary logistic regression to categorical dependent variables with more than two possible outcomes (e.g., Agresti, Reference Agresti2013; Hosmer & Lemeshow, Reference Hosmer and Lemeshow2013; Skrondal & Rabe-Hesketh, Reference Skrondal and Rabe-Hesketh2003). It produces a set of statistics for each possible outcome separately (see the scanpath categories in Tables 3–4), except for one outcome category, which is the baseline. In this study, we chose no regression as the baseline because no regression, or straight-pass reading, arguably represents the clearest evidence for fluent, automatic processing of the critical part of the sentence (see also Reichle et al., Reference Reichle, Warren and McConnell2009). In comparison, regressions out of the critical part of the sentence indicate reanalysis and controlled processing (compare Clifton & Staub, Reference Clifton, Staub, Liversedge, Gilchrist and Everling2011; Clifton et al., Reference Clifton, Staub, Rayner, van Gompel, Fischer, Murray and Hill2007; Frazier & Rayner, Reference Frazier and Rayner1982; Meseguer et al., Reference Meseguer, Carreiras and Clifton2002; Mitchell, Shen, Green, & Hodgson, Reference Mitchell, Shen, Green and Hodgson2008; Reichle et al., Reference Reichle, Warren and McConnell2009; Von der Malsburg & Vasishth, Reference Von der Malsburg and Vasishth2011). Comparing the amount of finished and unfinished reading with regression, on the one hand, to reading with no regression, on the other, was informative because it showed the relative amount of controlled and automatic processing in a given condition.
Because logistic regression is performed on a mathematic transformation of the data, results are interpreted in terms of odds and odds ratios (ORs; see Liberman, Reference Liberman2005, for further details). Important for our purposes is the observation that both odds and ORs are ratios. Therefore, their quotient—whether larger or smaller than 1—indicates whether the tested scanpath (in the numerator) occurred more often or whether it was no regression (in the denominator) that occurred more often. Thus, if we interpret finished and unfinished reading with regression as controlled processing and no regression as automatic processing, odds and ORs that are greater than 1 indicate more controlled processing, and odds and ORs that are less than 1 signal more automatic processing.
Odds ratios are effect size measures (Ferguson, Reference Ferguson2009). They measure the effect of a particular independent variable on the baseline odds. For instance, for finished reading with regression, OR (timed∣untimed) indicates how the relative amount of finished reading with regression versus no regression changes with time pressure. Odds ratios that are greater than 1 suggest that adding time pressure promotes controlled processing, whereas ORs that are less than 1 indicate that the timed GJT results in more automatic processing than the untimed test. For L1-L2 status, we examined how being a NNS influenced reading behavior compared to NS processing: OR (NNS∣NS). For grammaticality, we tested how reading patterns changed for ungrammatical compared to grammatical items: OR (ungrammatical∣grammatical).
Fixed effects are significant at α = .05 if the 95% confidence interval for the odds ratio does not span 1. Ferguson (Reference Ferguson2009) suggested that an OR of 3.0 (or 0.33 when the baseline is the more frequent pattern) is a moderate effect, whereas an OR of 4.0 or 0.25 represents a strong effect. We also included random intercepts for subjects in the model to account for individual differences in participants’ reading behavior.Footnote 8 All analyses were run in the Statistical Analysis Software (SAS; Version 9.4) using the glimmix procedure.
RESULTS
Accuracy Scores
The percentages of correctly judged items and corresponding raw frequencies are shown in Table 1. Native speakers scored 87% or above for all test sections except for the timed ungrammatical items. The scores for the NNSs varied much more and ranged from a low of 26% on the timed ungrammatical items to a high of 82% for the untimed grammatical ones. Because we analyzed only correctly judged items, NSs had a somewhat similar number of items for each category, ranging from 518 for the timed ungrammatical items to 662 for the untimed grammatical items. For the NNSs, there was a considerably greater range; however, the proportion of accurate judgments roughly mirrored that of NSs. The timed ungrammatical section had the fewest items, at 355, whereas the untimed grammatical had the most, with 1,123.
Raw frequencies and percentages of correctly judged items
Reading Patterns: Descriptives
Table 2 displays the raw frequencies for the reading patterns found in the data, whereas the pie charts in Figure 4 display percentages, which allow for comparison across L1-L2 status, grammaticality, and time pressure.
Raw frequencies of items according to reading patterns
Note. Gram = grammatical; UnG = ungrammatical.
Percentages of reading patterns for timed versus untimed and grammatical versus ungrammatical GJT items.
The pie charts show that for all conditions (i.e., regardless of L1-L2 status, grammaticality, or time pressure), the most frequent reading pattern was finished reading with regression. Nevertheless, there are some differences in the percentages of items read with regressions compared to the other patterns across the data. For the NSs, the percentage of finished reading with regression is roughly similar across all four categories. The lowest percentage is 61% for untimed ungrammatical, whereas the highest is 70% for the untimed grammatical items. In addition, NSs read between 21% and 28% of the items with no regression, whereas unfinished reading with regression occurred in 8% to 13% of the items.
For the NNSs, the distribution of patterns is more varied. Although finished reading with regression remains the predominant pattern in all contexts, there are considerable differences for this category between the timed and untimed conditions, with NNSs regressing on roughly half of the items in the timed condition, whereas in the untimed condition, the proportion rises to roughly three quarters of items. Furthermore, there is a difference in the number of finished reading with regression patterns between the ungrammatical and grammatical items, with the grammatical items having the highest percentage of regressed items. In the untimed condition, the finished reading with regression pattern for the grammatical items is 9% greater than for the ungrammatical ones, a difference that is the same for the NSs.
As for the other patterns for the NNSs, there were more than twice as many straight-pass items (i.e., no regression) on the timed test as on the untimed test, whereas unfinished reading with regression occurred most frequently in the ungrammatical items yet was virtually absent from the grammatical conditions.
Reading Patterns: Inferentials
Finished Reading with Regression
Table 3 presents the odds and ORs for the largest category, finished reading with regression. The reference category for the odds in this table and Table 4 is no regression.
Regression output (fixed effects) for finished reading with regression
Note. The intercept represents the odds of regression over no regression for a NS reading a grammatical, untimed sentence with an average total length (7.81 words) and with a sentence-final region of average length (0.61 words).
Although there are two statistically significant main effects, the more interesting and important effects are the two significant two-way interaction effects: Time × L1-L2 status and Time × Grammaticality. To understand the interplay of these variables, we broke the comparisons down into four post hoc tests and adjusted the significance threshold for each post hoc test to control for the overall type I error rate. Because eight post hoc tests were performed for this scanpath category and the next, we set α at .01 for each post hoc test, so the familywise α was .08.
First, we compared NS and NNS processing for the timed GJT and the untimed GJT separately. The post hoc analyses revealed that, on the untimed test, NNSs tended to read more items to the end and regress than did NSs (OR = 1.75, 95% CI [1.13, 2.72], p = .013), whereas, under time pressure, NNSs did less regressing than NSs (OR = 0.51, 95% CI [0.33, 0.80], p = .003). Native speakers read and regressed on about the same percentage of sentences in the timed GJT (68%) as the untimed GJT (69%), but NNSs’ finished reading with regression rose from 57% on the timed test to 81% on the untimed test.Footnote 9 Conversely, NNSs’ straight-pass reading fell from 39% on the timed GJT to 15% on the untimed GJT, whereas NSs read in largely the same manner on the two GJTs (timed: 24%; untimed: 23%).
For the second interaction, we compared timed and untimed reading of grammatical items and ungrammatical items separately. We found that item grammaticality influenced reading patterns in the untimed GJT (OR = 0.74, 95% CI [0.61, 0.90], p = .002) but not the timed GJT (OR = 1.22, 95% CI [0.99, 1.50], p = .063). Whereas the amount of finished reading with regression was essentially the same for grammatical and ungrammatical items in the timed GJT (i.e., 62% of trials), it occurred on 70% of ungrammatical items and 80% of grammatical items in the untimed GJT. Thus, more finished reading with regression occurred in the untimed than in the timed test, and the increase was disproportionately strong for the grammatical items.
Unfinished Reading with Regression
For unfinished reading with regression, the three-way interaction L1-L2 status × Time × Grammaticality was borderline significant (see Table 4). Therefore, we contrasted NNS with NS processing for each of the four time-by-grammaticality combinations separately.
Regression output (fixed effects) for unfinished reading with regression
Note. The intercept represents the odds of regression over no regression for a NS reading a grammatical, untimed sentence with an average total length (7.81 words) and with a sentence-final region of average length (0.61 words).
The only condition in which NNSs and NSs engaged in statistically comparable amounts of unfinished reading with regression is the untimed, ungrammatical items (OR = 1.28, 95% CI [0.76, 2.18], p = .354). Nonnative speakers regressed and did not finish reading 9% of these sentences, and NSs performed similarly, with an unfinished-reading rate of 11%. The estimates for timed, ungrammatical sentences are the same, with an unfinished-reading rate of 9% by NNSs and 11% by NSs. However, because of concurrent shifts in the baseline category (no regression), the resulting OR for unfinished reading over no regression for NNSs compared to NSs was significant (OR = 0.43, 95% CI [0.23, 0.78], p = .006).
For grammatical items, NNSs engaged in comparatively less unfinished reading than NSs in both the untimed (OR = 0.28, 95% CI [0.15, 0.53], p < .001) and the timed (OR = 0.22, 95% CI [0.12, 0.41], p < .001) conditions. In these conditions, NNSs rarely did not finish reading a sentence after making a regression (2% of trials), whereas NSs sometimes did (7% of trials).
DISCUSSION
The aim of this study was to use processing data from eye-tracking to advance our understanding of the types of linguistic knowledge measured by written GJTs. We compared findings from previous factor-analytic research, which relied primarily on accuracy scores, with eye-movement data. Considering how different both measures are, the convergence in findings is rather remarkable. Similar to Bowles (Reference Bowles2011), we found that NSs were essentially unaffected by the experimental manipulations (i.e., time pressure and grammaticality). In contrast, NNS processing changed rather drastically under time pressure (see Bowles, Reference Bowles2011; R. Ellis, 2005; R. Ellis & Loewen, Reference Ellis and Loewen2007; Han & Ellis, Reference Han and Ellis1998; Zhang, Reference Zhang2014), whereas the untimed, grammatical condition stood out as a locus of potential task effects (see R. Ellis, 2005).
Time Pressure Affected NNS Processing
Adding time pressure reduced the amount of NNSs’ reading with regression from heavy to moderate (see Figure 4). Post hoc analyses indicated that no regression was the main scanpath category to benefit from this shift: The percentage of trials with no regression increased from 15% in the untimed GJT to 39% in the timed GJT. Thus, when under time pressure, NNSs regressed less and read more sentences in one straight pass.
An important question is what, if anything, we can infer from this shift in processing strategy with regard to participants’ underlying knowledge representations. Surely, it is to be expected that when people are given less time to read, they will indeed read less, and, consequently, eye-movement records will reflect this. However, what form such a reduction in reading will take is not determined by the task conditions. In other words, participants might have coped with the time pressure by skipping more words or giving up trying to read the entire GJT item—yet this is not what we found. In what follows, we present two arguments to explain why the observed shift towards straight-pass reading could reflect a qualitative change in the knowledge representations participants draw on. The first argument is that independent factor-analytic research has found that timed and untimed GJTs measure different things, using different data and a different approach than the present study (Bowles, Reference Bowles2011; R. Ellis, 2005; R. Ellis & Loewen, Reference Ellis and Loewen2007; Han & Ellis, Reference Han and Ellis1998; Zhang, Reference Zhang2014; but see Gutiérrez, Reference Gutiérrez2013). Researchers in these studies factor-analyzed accuracy scores for a number of tests in addition to GJTs. In the present study, we analyzed scanpaths, a processing measure, of correctly judged GJT items only. Despite these different approaches, the same general pattern of findings emerged.
The second point in favor of a processing-knowledge connection is more theoretical and argumentative. Forward reading is the default mode of fluent, skilled reading. It “occurs when the reader is proceeding through the text without any problems”—unlike regressions, which are assumed to “reflect difficulty with higher level language processing” (Reichle et al., Reference Reichle, Warren and McConnell2009, p. 4). For knowledge to be retrieved quickly and efficiently during fluent reading, it needs to be stored in a highly accessible form. Such knowledge representations could consist of production rules (e.g., DeKeyser, Reference DeKeyser and DeKeyser2007); subsymbolic, connectionist networks (e.g., Rebuschat & Williams, Reference Rebuschat and Williams2012); or yet other types of representation. Although competing accounts exist (e.g., Bialystok, Reference Bialystok1994; McLaughlin, Rossman, & McLeod, Reference McLaughlin, Rossman and McLeod1983), skill acquisition theory and contemporary psychological models posit that automatic, fluent processing operates on automatized explicit knowledge (DeKeyser, Reference DeKeyser and DeKeyser2007) or implicit knowledge (R. Ellis, 2005). Consequently, the processing differences found in the current study could be accounted for by the timed GJT not allowing NNSs to engage in more controlled processing or to access explicit knowledge.
Uniform Processing by NSs across Conditions
In contrast to the moderately strong effects of time pressure in the NNS group, the distribution of NSs’ processing patterns remained stable across conditions. Native speakers made at least one regression in the majority of sentences (69% finished reading), with no regression being a clear second (24%). Native speakers also made some fast decisions, including on grammatical sentences (7% unfinished reading). The unfinished reading of grammatical sentences is problematic and could reflect some degree of probabilistic judgment, because strictly speaking one needs to read a whole sentence to be able to tell if it is grammatical. However, the number of such instances is relatively small. In contrast, NNSs engaged in unfinished reading only on ungrammatical sentences, in which the error does indeed license the possibility of an early decision.
The lack of time effects on NS processing likely reflects NS-NNS proficiency differences. Following R. Ellis (2005) and Loewen (Reference Loewen, Ellis, Loewen, Elder, Erlam, Philp and Reinders2009), participants received 20% extra response time, based on NS pilot data. Whereas this approach worked in pressuring NNSs, our data suggest that NSs require shorter response times to pressure their reading. Future researchers should reduce the allotted time to explore how processing changes when NSs experience actual time pressure. As a case in point, in his fourth experiment, Hopp (Reference Hopp2010) strongly increased the time pressure in a written GJT administered to native German speakers. The NSs’ performance broke down in a manner similar to near-native speakers, who had taken the same test under less demanding, yet also speeded, conditions. Hopp concluded that “non-native and native grammars and processing systems are fundamentally identical” (p. 901), but their processing efficiency differs.
Grammaticality and Task Effects
Consistent with Gutiérrez (Reference Gutiérrez2013), our analyses also suggested a role for item grammaticality. However, unlike Gutiérrez’s study, we observed an effect of grammaticality only in the untimed condition. Compared to untimed, ungrammatical items, both NSs and NNSs showed a further increase in finished reading with regression on the untimed, grammatical items (NSs: from 63% to 71%; NNSs: from 75% to 84%). An inspection of the other scanpath probabilities showed that this increase was at the expense of NNSs’ unfinished reading with regression, whereas in the NS group the losses were divided evenly between unfinished reading with regression and no regression.
One possible explanation for this phenomenon is that grammatical and ungrammatical sentences provide different types of evidence as to the grammaticality of a sentence. Whereas the presence of an ungrammatical element is sufficient evidence that a sentence is ungrammatical, the absence of an ungrammatical element is essentially a lack of evidence. The sentence could be grammatical, or a closer reading might reveal that it is ungrammatical after all. Therefore, learners may wish to reconfirm their initial impression of a sentence by rereading it. Importantly, the rereading of grammatical sentences when time is available is likely an artifact of the task, which is to judge the grammaticality of the sentence. Therefore, we believe that the reading patterns for untimed, grammatical sentences in a written GJT may show the clearest departures from natural reading.
The special status of untimed, grammatical items is consistent with Loewen’s (Reference Loewen, Ellis, Loewen, Elder, Erlam, Philp and Reinders2009) finding that learners took an average of 1 s longer to judge grammatical sentences compared to ungrammatical ones. However, it could be at odds with R. Ellis’s (2005) finding that untimed, ungrammatical items provided a clearer measure of explicit knowledge than untimed, grammatical items. If finished reading with regression is indicative of controlled processing, we would expect the untimed, grammatical items to be the more valid measure of explicit knowledge because that is where this reading pattern occurred most often. The two positions can be reconciled if we take the additional amount of finished reading with regression of untimed, grammatical sentences to reflect task effects and—consequently—untimed, grammatical sentences to be less valid as measures of linguistic knowledge of any type.
More generally, the results of this study highlight the need for more comparative SLA research on the effects of reading goals or task instructions on reading (see, e.g., Lim & Christianson, Reference Lim and Christianson2014; Holmqvist et al., Reference Holmqvist, Holsanova, Barthelson, Lundqvist, Hyönä, Radach and Deubel2003; Kaakinen & Hyönä, Reference Kaakinen and Hyönä2005, Reference Kaakinen and Hyönä2008). Across all conditions, the amount of reading with regression was high or very high, begging the question of what the distribution of reading patterns would look like when participants read for comprehension. Leeser, Brandl, and Weissglass (Reference Leeser, Brandl, Weissglass, Trofimovich and McDonough2011) investigated the effects of task instructions on reading times using self-paced reading.Footnote 10 They compared intermediate L2 learners’ reading times for words (or phrases) in grammatical and ungrammatical Spanish sentences. In one task, the participants answered a comprehension question after each sentence, whereas in the other task they made untimed grammaticality judgments. Leeser and colleagues found that participants read more slowly in the GJT task (at least up to the point of the ungrammaticality) than in the comprehension task. Furthermore, readers were sensitive to violations in one of two structures in the GJT task but not in the comprehension task. Leeser and colleagues argued that the sensitivity effect in the GJT showed that “participants were activating a different kind of knowledge (i.e., conscious, metalinguistic knowledge) in addition to the implicit linguistic knowledge that is assumed to guide sentence comprehension” (p. 194; see also Lim & Christianson, Reference Lim and Christianson2014, for a similar claim regarding the effects of translation on reading). Because learners could not regress in Leeser et al.’s study, the comparison with the current study is not straightforward; however, both studies suggest that the question of task effects certainly warrants further research.
Limitations and Future Research
In this study, we performed a scanpath analysis to address a SLA-related question. We analyzed sentence reading patterns rather than eye-movement data for individual interest areas because we felt this analysis was more revealing about the nature of GJTs (see Von der Malsburg & Vasishth, Reference Von der Malsburg and Vasishth2011). Scanpaths have many other potential applications in SLA, including in research on computer-based assessment, computer-assisted language learning, task-based language teaching, dictionary use, subtitle processing, and reading. Indeed, the concept of a functional interest area—that is, a region on the screen with a particular function (Holmqvist et al., Reference Holmqvist, Nyström, Andersson, Dewhurst, Jarodzka and Van de Weijer2011)—is flexible and allows tailoring to different research areas. In addition, future researchers could refine the scanpath measure and analysis so as to optimize the approach.
One limitation of the present study is that we defined scanpaths in spatial terms only and essentially disregarded the temporal information contained in the eye-movement data. Scanpath algorithms now exist that consider both location and duration of eye movements (Von der Malsburg & Vasishth, Reference Von der Malsburg and Vasishth2011), yet they are computationally more intensive. A second potential limitation is that we identified scanpath categories inductively, by visually inspecting trials and referring to previous literature on eye-movement behavior. Cluster analysis could provide a statistical means to identify scanpath categories. Finally, we made the decision to segment our sentences into interest areas (i.e., the sentence-initial region, primary interest area, spillover region, and sentence-final region). Although this approach took care of length differences among sentences, it would also have been possible to treat space continuously and to work with spatial coordinates instead of symbol strings.
Finally, there are several limitations due to the fact that this study replicated GJT items and procedures from previous studies that were not designed specifically for eye-tracking technology. For example, the inclusion of sentences with errors at the end of the sentence meant that there was no sentence-final region that readers could skip in these cases, ruling out the possibility of unfinished reading with regression. Additionally, as one reviewer correctly pointed out, the use of the same items in both the timed and untimed GJTs resulted in the former test containing new items, whereas the latter contained old items that had already been read, a fact that could have affected reading patterns. However, it might be expected that item familiarity on the untimed test would reduce the occurrence of regressions during reading because the learners already knew the answers, in this case from the timed version of the GJT. This is not what we found; in contrast, reading with regression stayed consistent or increased from the timed to untimed tests. Nevertheless, future research could benefit from including different sentences for the timed and untimed GJTs, as Bowles (Reference Bowles2011) did.
CONCLUSION
This eye-tracking study reconciled previous claims about the psychometric properties of written GJTs by showing that both time pressure and grammaticality play a role in the construct validity of GJTs. However, time pressure is the variable that appears to limit NNSs’ ability to engage in more controlled processing, and thereby to access declarative, explicit L2 knowledge. Grammaticality, on the other hand, appears to exhibit the influences of task effects as NSs and NNSs engage in additional processing, time permitting, in an attempt to identify ungrammatical elements in grammatical sentences.
The eye-movement data we presented were methodologically innovative for SLA in that they consisted of scanpaths (sentence reading patterns) rather than eye fixations or regressions for individual regions in the sentence. As such, our data suggested that timed and untimed GJTs (particularly ungrammatical, untimed GJTs) measure different things. How this finding is interpreted with reference to the underlying test constructs will depend on one’s theoretical orientation. Specifically, timed GJTs, on the one hand, and ungrammatical items on untimed GJTs, on the other, could be argued to measure implicit and explicit knowledge, respectively (R. Ellis, 2005); automatized explicit and analyzed explicit knowledge (DeKeyser, Reference DeKeyser, Doughty and Long2003); or automatic versus controlled access of either knowledge type (Bialystok, Reference Bialystok1994). The need to distinguish among these competing accounts will likely push methodological innovation in SLA forward and give rise to more data triangulation and additional techniques such as brain imaging studies. Understanding the interface between processing and knowledge remains an important, and challenging, task.
APPENDIX
EXAMPLE SENTENCES AND INTEREST AREAS
