Second language acquisition researchers (including the authors of this article) use the term “acquisition” on a regular basis, without being able to point to a universally accepted guideline for measuring the acquisition of any particular grammatical form or structure. Instead, acquisition has by necessity been defined or operationalized in a variety of ways over decades of changing methodologies: for example, as a particular percentage of accurate suppliance in oral production (e.g., White, Valenzuela, Kozlowska-MacGregor, & Leung, Reference White, Valenzuela, Kozlowska-MacGregor and Leung2004), or as nativelike eye movements in response to ungrammaticalities (e.g., Foucart & Frenck-Mestre, Reference Foucart and Frenck-Mestre2012). But for many researchers, especially those working from a linguistic framework, the real target of inquiry is the more abstract concept of linguistic competence: That is, do learners have an appropriate or nativelike representation of the structure in their grammar?
Because representation cannot be measured directly, many recent studies draw conclusions about representation by observing learner behavior in both production and reception. The reasoning is that if a particular error extends both to production and reception, it is likely to be a representational problem (see McCarthy, Reference McCarthy2008, for a clear example). Indeed, this reasoning is consistent with many theoretical models, which link production and reception through a single representation. For instance, White (Reference White and Eubank1991), working in a Chomskyan framework, keeps input-processing systems and production mechanisms separate, but connects them through the grammar. Levelt’s (1993) model of speech links production and reception through the lexicon, which also contains grammatical information.
Psycholinguistic research, by contrast, has not emphasized linguistic competence per se, but rather has tended to assume that productive and receptive skills involve different mechanisms, and therefore has investigated them separately. However, some recent psycholinguistic and neurolinguistic work finds evidence for overlap between the two systems (e.g., Kempen, Olsthoorn, & Sprenger, Reference Kempen, Olsthoorn and Sprenger2012; Menenti, Gierhan, Segaert, & Hagoort, Reference Menenti, Gierhan, Segaert and Hagoort2011; Pickering & Garrod, Reference Pickering and Garrod2007, Reference Pickering and Garrod2013; Segaert, Menenti, Weber, Petersson, & Hagoort, Reference Segaert, Menenti, Weber, Petersson and Hagoort2012), suggesting that there may be a shared “grammatical workspace” (Kempen et al., Reference Kempen, Olsthoorn and Sprenger2012) between them.
This interest in determining which systems are shared and which are distinct in production and reception has recently been extended to processability theory (PT) (Pienemann, Reference Pienemann1998, Reference Pienemann and Pienemann2005, Reference Pienemann, VanPatten and Williams2007), a theory of language development that predicts an order of acquisition for particular grammatical forms and structures, based on processing procedures that must be acquired sequentially. The basic conceptual question for PT is the same as for other theoretical frameworks, although processing routines are the focus here rather than representations per se: That is, what does it mean when we say that processing procedures are fully acquired? Are processing procedures part of a shared workspace that is linked to both production and reception (see Lenzing, ms.), or must they be acquired separately for production and reception?
Several recent studies have explored this question, with mixed results. For example, both Spinner (Reference Spinner2013) and Buyl (ms.) used grammaticality judgment tasks (GJTs) to investigate whether the order of acquisition predicted by PT for production also extends to receptive tasks. Neither study found the PT order to be borne out. However, Buyl and Housen (Reference Buyl and Housen2015) used a picture-selection task to examine six target structures in stage 2 and stage 5 of the processability hierarchy, and found that French-speaking learners of English acquired the structures in an order that is compatible with PT. Similarly, both Keatinge and Keßler (Reference Keatinge, Keßler, Keßler and Keatinge2009) and Lenzing (ms.), using picture-selection tasks and other interpretation tasks, found a clear correspondence between production and comprehension of the passive, a construction that is included in extended PT (Pienemann, Di Biase, & Kawaguchi, Reference Pienemann, Di Biase, Kawaguchi and Pienemann2005).
Methodological issues may be a factor in the different findings in these studies. Fortunately, PT (Pienemann, Reference Pienemann1998) provides one way to examine these issues: the procedural skills hypothesis (PSH). The PSH states that L2 learners who have acquired particular processing procedures should respond similarly to native speakers on relevant structures, including with online receptive measures. In this study, the PSH is tested with a self-paced reading (SPR) task. The purpose is to determine whether PT can account for emergence of grammatical forms and structures in an online receptive task as well as production, and thus to clarify the nature and scope of the procedures proposed by PT. The findings also speak to broader questions about the organization of the linguistic system.
PROCESSABILITY THEORY
PT proposes that learners can only produce and comprehend grammatical forms and structures that their language processor can manage at any given time. Learners can be observed to move through stages of development with the emergence of processing procedures, which deal with information transfer within phrase structure. This information transfer is modeled with lexical functional grammar (Kaplan & Bresnan, Reference Kaplan, Bresnan and Bresnan1982), specifically, with the notion of feature unification. Feature unification is based on the idea that every entry in the lexicon is annotated for particular grammatical features. For example, the lexical entry “runs” is assigned the lexical class “verb” with a singular NUMBER feature, a third-PERSON feature, a present TENSE feature, and nonprogressive ASPECT feature. Similarly, a lexical entry such as “John” is assigned the lexical class “noun” and a singular NUMBER feature. Feature unification is achieved when the feature values of two lexical entries are matched. For example, for subject-verb agreement to occur, the singular NUMBER feature and third-PERSON feature on “John” need to be matched to the singular NUMBER feature and third-PERSON feature on “runs.” This is accomplished by passing the grammatical information to the noun phrase procedure and verb phrase procedure; the information is then passed along to the sentence procedure for unification.
The processing procedures develop in the following order:
1. No procedure
2. Category procedure
3. Noun phrase procedure
4. Verb phrase procedure
5. Sentence procedure
6. Subordinate clause procedure
Based on these procedures, a developmental trajectory for any language can in principle be created. PT has found support with a variety of languages (Bonilla, Reference Bonilla2015; Di Biase & Kawaguchi, Reference Di Biase and Kawaguchi2002; Glahn et al., Reference Glahn, Hakansson, Hammarberg, Holmen, Hvenekilde and Lund2001; Håkansson & Norrby, Reference Håkansson and Norrby2010; Kawaguchi, Reference Kawaguchi2000, Reference Kawaguchi and Pienemann2005; Mansouri, Reference Mansouri and Pienemann2005; Pienemann, Reference Pienemann1998; Sakai, Reference Sakai2008; Spinner, Reference Spinner2011; but see Alhawary, Reference Alhawary2009 and Dyson, Reference Dyson2009 for mixed evidence or counterevidence). Typically, these studies involve extensive interviews with learners that elicit relevant forms and structures in a natural way.
Importantly, accuracy is not considered when determining whether a learner has reached a particular stage. Instead, the emergence of productive use of a form or structure, even if it is variable, is taken as evidence of a learner having reached a particular stage because logically it could not be produced without the associated procedure. In principle, emergence is “(minimally) one occurrence in a sample” (Pienemann, Reference Pienemann1998, p. 133), but in practice it can be difficult to determine which uses are productive. Various criteria have been suggested over the years, including finding contrasting forms (e.g., table and tables, kick and kicked), and multiple occurrences (e.g., tables, cars, and books) (e.g., Pienemann, Johnston, & Brindley, Reference Pienemann, Johnston and Brindley1988).
Several recent studies directly investigate PT with receptive rather than productive data. The first was Keatinge and Keßler (Reference Keatinge, Keßler, Keßler and Keatinge2009), who focused on the passive voice in L2 English. Ten German-speaking adolescents completed a story-telling task, a film description, sentence-completion tasks, and a semantic decision task in which learners matched sentences with pictures. Learners who could produce passives scored 70% or higher on the comprehension task, while those who could not produce them had extremely low scores.
Lenzing (ms.) also examined the acquisition of passive, arguing that it is a gradual process that is best accounted for by PT and Lenzing’s extension of PT, the multiple constraints hypothesis (Lenzing, Reference Lenzing2013, Reference Lenzing2015). Lenzing uses a narration task to elicit production data and an interpretation task involving the arrangement of figurines to examine the acquisition of several different aspects of the passive. She determines that there are no cases of learners producing passive without also comprehending it, although there are a few cases of comprehension without production, which she attributes to semantic factors. Importantly, Lenzing argues that the processing procedures of PT are shared by a grammatical encoder (for production) and decoder (for reception), and therefore it should be expected that there be a correspondence between production and reception of the grammatical elements in PT.
Unlike the previously described studies, Spinner (Reference Spinner2013) included all PT stages, other than stage 1. She used an audio GJT with two groups of ESL learners of varied proficiency (n = 114). The GJT contained 150 grammatical and ungrammatical items representing PT stages 2 through 6. Various analyses of the data found no clear pattern of acquisition, and the PT hierarchy was not supported for comprehension. Similarly, Buyl (ms.) also used a GJT to investigate a variety of grammatical elements in PT, focusing on morphology. This study also failed to find evidence for the PT order in the receptive data. Note, however, that GJTs are potentially problematic for measuring second language knowledge and abilities, for a variety of reasons (e.g., Ellis, Reference Ellis1991; Schütze, Reference Schütze1996); in particular, they may measure explicit rather than implicit knowledge (e.g., Ellis, Reference Ellis2005).
Finally, Buyl and Housen (2015) examined the predictions of PT for reception data with 72 French-speaking children learning English. This study has several advantages in that it considers six different grammatical phenomena from stages 2 and 5, including both morphology and syntax, and uses picture-selection tasks instead of grammaticality judgments. As with other receptive tasks, a particular cutoff point needed to be established as the point of acquisition or emergence. Buyl and Housen examined a variety of cutoff points and determined that learners who demonstrate stage 5 knowledge also demonstrate knowledge of at least some of the stage 2 structures. The authors conclude that PT may be able to account for the order of acquisition in reception as well as production. They acknowledge, however, that leaving out structures from stage 3 and stage 4 may obscure important details. They suggest that future studies should explore online processing such as self-paced reading, a call that we attempt to answer with this study.
THE PROCEDURAL SKILLS HYPOTHESIS: PT AND COMPREHENSION
Pienemann’s work in PT has provided an example of measuring comprehension/reception in the PSH, which states that learners who have acquired particular processing procedures should respond similarly to native speakers on relevant structures. Pienemann (Reference Pienemann1998) argues that reaction time experiments with grammatical and ungrammatical linguistic stimuli are well suited to examine the PSH. The assumption is that the acquisition of grammatical forms and structures entails the automatization of processing routines; because ungrammatical structures or novel words have no ready-made routines available, they take longer to process.
Pienemann (Reference Pienemann1998) tested the PSH with a sentence matching task. First, oral interviews were used to place 14 nonnative speakers of German into two groups: those for whom subject-verb agreement had emerged in production (n = 7), and those for whom it had not (n = 7). Both groups, along with native speaker controls (n = 7) completed a sentence matching task with statements such as those in (1) and (2), which were taken from Clahsen and Hong (Reference Clahsen and Hong1995).
(1) Du fliegst nach Korea am nächsten Sonntag
You fly-2nd sg. to Korea next Sunday
(2) *Du fliegt nach Korea am nächsten Sonntag
You fly-3rd sg. to Korea next Sunday
Participants pressed a button indicating whether the sentences were the same or different. If there is sensitivity to the ungrammaticality, it should take longer to decide “same” or “different” on ungrammatical items. Pienemann’s predictions were explained as follows (Pienemann, Reference Pienemann1998, pp. 222–223):
A test of the procedural skill hypothesis therefore requires:
(1) NNSs [non-native speakers] with the given L2 skill;
(2) NNSs without the given L2 skill; and
(3) NSs [native speakers] (who will naturally have the given L2 skill).
Predictions:
Groups (1) and (3) will perform in a similar way.
Group (2) will perform unlike groups (1) and (3).
Grammaticality effects will show only in groups (1) and (3).
Pienemann reasons that when subject-verb agreement is acquired in production, it must be the case that a learner has acquired the routine that processes such agreement. Thus, the NNSs for whom subject-verb agreement has emerged should perform similarly on the matching task to the native speakers, while those for whom it has not yet emerged should perform differently. Indeed, this is what Pienemann found.
Despite these arguments in Pienemann (Reference Pienemann1998) and elsewhere, the focus of PT has almost exclusively been on language production rather than reception (Hulstijn, Reference Hulstijn2015), including in Pienemann and colleagues’ own work. To our knowledge, no one has investigated this issue further with online methodologies. That is the goal of this study. Like Pienemann, we investigate stage 5 structures and conduct oral interviews to divide learners into groups of those who produce stage 5 structures and those who do not. We also include a control group. We then conduct a self-paced reading task to determine whether the L2 groups perform in a nativelike or nonnativelike way.
Self-paced reading has been used extensively to measure language learners’ sensitivity to grammatical anomalies, including third-person singular –s and various syntactic structures (e.g., Jiang, Novokshanova, Masuda, & Wang, Reference Jiang, Novokshanova, Masuda and Wang2011; Sagarra & Herschensohn, Reference Sagarra and Herschensohn2010; Shibuya & Wakabayashi, Reference Shibuya and Wakabayashi2008; for recent reviews and methodological guide see Jegerski, Reference Jegerski, Jegerski and VanPatten2014; Keating & Jegerski, Reference Keating and Jegerski2015). While many researchers have used self-paced reading or listening in combination with other methods to get a clearer picture of learners’ knowledge and abilities (e.g., Marinis, Roberts, Felser, & Clahsen, Reference Marinis, Roberts, Felser and Clahsen2005; Roberts & Liszka, Reference Roberts and Liszka2013), the comparison often focuses on online versus off-line performance rather than production versus reception per se. Currently the relationship between oral production and self-paced reading performance is not clear, and we hope to contribute to this body of research with this study.
THE CURRENT STUDY
Our study is guided by the following research question: Do ESL learners who are in stage 5 or over of the PT scale based on oral interviews also show sensitivity to stage 5 structures in a self-paced reading task?
Participants
Participants in this study were 61 ESL learners (42 males and 19 females) enrolled in an intensive English program at a large American university. They represented a variety of L1 backgrounds, including Arabic (n = 30), Chinese (n = 16), Portuguese (n = 10), Japanese (n = 2), Korean (n = 2), and Mongolian (n = 1). Given the variety of participant L1s, it should be noted here that PT predicts some first language transfer. However, transfer is developmentally moderated; that is, transfer can only occur when the L2 processor is developed enough to handle it (the developmentally moderated transfer hypothesis of Pienemann et al., Reference Pienemann, Di Biase, Kawaguchi and Pienemann2005). Thus, the universal order of acquisition is predicted to hold for speakers of any first language, although the rate of acquisition may vary. (For further discussion of L1 influence in PT, see Pienemann et al., Reference Pienemann, Di Biase, Kawaguchi and Pienemann2005 and Vainio, Pajunen, & Hyönä, Reference Vainio, Pajunen and Hyönä2014.)
There was a range of age (M = 22.7 years old, SD = 4.8, range: 18–39) and length of residence (M = 10.33 months, SD = 10.57, range: 1–60). The learners’ English proficiency also varied, from low intermediate to advanced level, based on the classes in which they were placed in the ESL program at the time of participation in the study. Also included were 22 native English speakers (5 males and 17 females) who were all undergraduate students except for one college graduate. They were 21.4 years old on average (SD = 4.04) and served as a control group for the self-paced reading task. Most participants were compensated with extra credit.
Procedure
All data collection took place on campus in a quiet laboratory setting equipped with a Mac laptop connected to a 23-inch LCD monitor and a Cedrus button box. Upon their arrival, the ESL learners participated in an oral interview for about 30 minutes, then completed the self-paced reading task. Native speaker controls only completed the self-paced reading portion of the study.
Oral Interview
Either a native speaker of English or one of two advanced nonnative speakers of English (L1 Korean) conducted the oral interviews. Each interview was conducted one on one in a small room. After a short warm-up conversation, language games, picture descriptions, and role plays were used to elicit forms and structures of PT, including plural –s, past –ed, possessive -s, wh-copula questions, third-person singular –s, auxiliary second questions, and “cancel sub/aux inversion” structures. For example, to elicit plural –s, interviewers asked learners to compare two cards with pictures and describe the difference. Some of the pairs of cards showed single items that differed in color or shape and some showed groups of items. The entire interview took approximately a half hour and was audio-recorded. An effort was made to ensure that each learner produced enough language to enable a clear decision on each of the structures.
A particular effort was made to elicit structures from stage 5. Both a syntactic element (specifically, auxiliary second questions) and a morphological element (third-person singular –s) were selected from this stage because PT research has suggested that there may be a dissociation in their acquisition (Buyl, ms.; Dyson, Reference Dyson2009). Auxiliary second questions are those with an auxiliary verb following a wh-word, such as “What do you want?” or “When can we leave?” To elicit these questions, the interviewers and participants role played various scenarios. Additionally, interviewers invited the learners to ask questions about a family photo, their schedules during the day, and their preferences for activities. To elicit third-person singular, interviewers asked learners to describe an important person in their life and members of their family.
The recordings were then transcribed. Because it was crucial to examine both use and nonuse of each structure, we noted any instances of the form or structure, any contrasting examples (e.g., singular instead of plural), any context where the structure could have been used, but was not (e.g., my friend go), and novel uses (e.g., foots). The criteria for emergence was three different tokens of the form or structure. That is, for morphology we required at least three uses on different words in the appropriate context. Additionally, to be considered productive it was necessary for the learner to use contrasting morphological forms (e.g., singular to contrast with plural). For a particular question type to be considered productive, there had to be three uses, with unique verbs, auxiliaries, and question words (see Lenzing, ms., for a similar approach to determining emergence).Footnote 1
For instance, one learner produced the following auxiliary second questions, that is, questions with auxiliaries such as do, can, or will in the appropriate position. The learner did not produce questions that included different verbs and different auxiliaries or wh-forms; therefore, the learner was judged to not have productive use of these questions. Note the unvarying phrase “What do you do.”
(3) Learner 229
What do you do last weekend?
What do you do with your friend in the weekend?
What do you do … for the weekend?
What do you routine yesterday?
What do you do?
What about evening?
When you go the bedroom?
How many sister?
How many you have auntie or not have auntie?
By contrast, another learner demonstrated a variety of verbs and auxiliaries and wh-words. She was judged to have auxiliary second questions in her grammar.
(4) Learner 202
What did you do last Saturday?
What did you made?
What do you study here?
How can they have two kids?
Why did you take the picture?
What did you do in your grandfather’s birthday?
When you wake up and what time do you go to school?
How can you go to school?
What did you eat the breakfast?
How long you take shower?
How can you guys marry?
Note that there are ungrammaticalities in some of the questions; this is to be expected because emergence does not entail accuracy. Additionally, some forms are repeated and may be chunked, such as “What did you do,” but because a variety of verbs and question words are used, it was judged that this learner uses some productive auxiliary second questions.Footnote 2
Self-Paced Reading Task
Materials
The test materials in the reading task consisted of 8 practice and 72 target sentences, including six different grammatical structures representing five PT stages, from stage 2 through stage 6, with 12 (6 grammatical and 6 ungrammatical) trials for each structure type: possessive –s, object pronouns, wh-copula questions, third-person singular –s, auxiliary second questions, and cancel sub/aux inversion structures. To answer the research question, only stage 5 structures were needed; however, structures from stages 2, 3, 4, and 6 were included to provide insight as to whether learners acquired the target items in the order predicted by PT. Two grammatical features, one morphological and one syntactic (auxiliary second questions and third-person singular –s), were used to represent stage 5.
The test was designed so that both a grammatical and ungrammatical version of each sentence was included, but each participant only saw one or the other. The order of the trial presentation was pseudorandomized so that items examining the same structure did not appear next to each other. Additionally, three different orders of presentation were created for each version of the test. Examples of the target sentences and areas of interest are provided in Table 1.
Examples of the target sentences in the self-paced reading task
Note: Critical region is bolded; spillover region is underlined.
To avoid any effects of word frequency and vocabulary knowledge on reading times (RTs) and comprehension, the vocabulary in the target regions was controlled. Based on the American English Subtitles (SUBTLEXus) corpus (Brysbaert & New, Reference Brysbaert and New2009), vocabulary items with frequencies of at least 17.86 per million and ranking within the top 4% of high-frequency words were selected. Generally, however, mean frequencies were well above this minimum requirement. There were no significant frequency differences in the vocabulary used across the target structures, F (5, 66) = 1.293, p = .278, and across the different stages, F (4, 67) = 1.635, p = .176. The mean sentence lengths, as measured by the word counts, ranged on average from 7.67 to 9.33 words per sentence across the structures. Lastly, sentences were designed based on the order of acquisition hypothesized by PT. That is, target sentences in one stage included only the grammatical properties that are hierarchically in lower-level PT stages. For instance, items representing stage 5 could include structures from stages 1 through 4 but not those from 6 or 5 (other than the structure being tested).
Experiment
The self-paced reading task was programmed using Superlab software (version 4.0, Cedrus Corporation). Each sentence was displayed in a noncumulative segment-by-segment fashion (Just, Carpenter, & Woolley, 1982), with the number of segments ranging from four to six across the trials.Footnote 3 All text was presented in Helvetica with the font size 36, in black letters on a white background. The sentences all fit on one line.
The participants were instructed that they should read the sentences for meaning at their normal reading speed. After a fixation cross appeared at the beginning of the sentence, the segments appeared in a sequential order from left to right as participants pressed the button at their own pace. Upon the button press, the next part of the sentence appeared while the previously presented segment disappeared. Lines indicated the location of the words of the sentence. The end of the sentence was indicated with a period at the end of the last segment.
Individual participants’ RTs on each segment were recorded. After reading the last part of each sentence, participants answered a simple true/false comprehension question on the screen by pressing one of the two designated buttons, a green button (left) for true and red (right) for false. The purpose of the comprehension questions was to ensure that participants read for meaning. The questions did not target information in the critical region of the sentences. For example, for the stage 3 object pronoun sentence The little boys miss me a lot, the comprehension check was The boys are very big (False).
RESULTS
Oral Interview
The oral interviews were generally successful at eliciting the target structures. However, there was difficulty with eliciting questions with two learners, who were reluctant to participate in the language games designed to encourage use of questions. These learners were eliminated from all further analysis.
Each of the remaining learners’ transcribed data was examined to determine whether forms and structures had emerged for each of the PT stages. However, for the current study, the focus was on the presence or absence of stage 5 structures, rather than on the presence or absence of structures in stages 1 through 4 and 6. The reasoning is that the main goal is to categorize the learners into two groups: stage 5 and higher, and stage 4 and lower, a methodology followed by Pienemann in his 1998/2001 study.
Categorizing learners based on the emergence of two stage 5 structures—third-person singular –s and auxiliary second questions—raised a few interesting issues. In many interviews, the learners produced one of these structures, but not the other. According to PT, if a learner demonstrates emergence of any one structure of a stage, he has acquired the processing procedure for that stage; thus, all the learners with one stage 5 structure should be classified as stage 5. However, we were left with two groups of learners: those that produced both elements of stage 5, and those who produced only one. Notably, of the learners who produced only one of the two structures in stage 5, there was a distinct pattern. Out of 18 learners in this group, only three of them produced third-person singular –s but not auxiliary second questions; for the rest, the questions were produced but the morphology was lacking. We return to this issue in the discussion.
For the other group, stage 4 and lower, it was critical to find learners who were unable to produce both questions and auxiliaries. Logically, there are plenty of learners who fit this profile. However, it was not enough to simply find beginning learners whose proficiency was very low. Many of the learners who are low enough on the PT scale to have neither third-person –s nor auxiliary second questions would also be of such low general proficiency that they would be unable to perform the tasks required for the study, in particular reading the sentences in the self-paced reading task. Participants were recruited until learners were found who fit the desired profile—that is, failing to demonstrate productive use of both stage 5 structures, but who appeared to be of a high enough general proficiency to understand the directions and to read the sentences in the self-paced reading task. To make this determination, we used intuition only; that is, we gauged whether they could communicate with us, to follow our directions, and so forth; we did not attempt to measure their proficiency in any formal way.
Based on the criteria explained earlier, we categorized learners as either low level, meaning stage 4 and below (n = 16), or high level, meaning stage 5 and over (n = 25). Only learners who produced both stage 5 structures were included in the high-level group. The reason for this decision was to give the best possible chance for the predictions of PT and the PSH to be demonstrated. That is, we assume that this group is likely to have the level 5 procedures robustly because both a morphological and a syntactic element are used productively. We also included in a separate group the 18 learners who produced one but not both level 5 structures. For ease of discussion, we label them “mixed,” even though according to PT they are at stage 5 or over. Recall that two learners had to be eliminated for producing too little oral data; therefore, the total number of participants retained for analysis was 59.
The results of the oral interviews are presented in Table 2. In this table, the number of tokens that each learner produced for each form or structure is provided. Following Bonilla (Reference Bonilla2015), Lenzing (Reference Lenzing2013), and others, we also provide the total number of contexts for morphological forms (to the right of the /) but not for syntactic forms.
Oral interview results
Note: Bolded entries meet the criteria for emergence. 2PL = plural –s, 2PST = Past –ed, 2POS = Possessive -s, 4WHQ = Wh-copula questions, 5SAGR = third-person singular –s, 5AUX2 = auxiliary second questions, and 6CAUX2 = cancel aux 2nd clauses.
Self-Paced Reading
Comprehension Accuracy
Participants’ overall accuracy on the comprehension questions was checked to ensure that participants paid attention to the task and processed the sentences for meaning.Footnote 4 Both the native English controls and three subgroups of ESL learners were accurate on comprehension questions, with accuracy higher than 80%: M = 95.18 for native English; M = 84.90, 89.39, 87.73, respectively, for the low-level, high-level, and mixed ESL. There was a significant accuracy difference between the English controls and the three ESL subgroups as revealed by a one-way-ANOVA (F 3, 77 = 7.520, p < .001) with a follow-up scheffe post-hoc analysis (low-level, p < .00l; high-level p = .053; and mixed ESL, p = .014.). There was no reliable difference among the three ESL groups. Crucially, however, the three ESL groups’ accuracy scores were all statistically well above chance level (two-tailed one-sample t-test, t (15) = 13.290, p < .001, t (24) = 33.126, p < .001, t (17) = 20.555, p < .001, respectively for the low-level, high-level, and mixed group). Therefore, we concluded that not only the native speakers, but also the ESL learners processed the sentences for meaning to a large extent.
Reading Times
Any RTs that were two standard deviations above or below a participant’s mean RT on the same structure in the same grammaticality condition were replaced with the means, which affected 2.72% and 1.60% of the data on the critical and spillover region, respectively. Then sets of paired-sample t-tests were carried out on the two target regions (see Table 1) to compare participants’ RTs in the ungrammatical condition to those in the grammatical condition separately for each structure and each group.
Native English Speakers
In the self-paced reading task, the native English speakers were expected to have significantly longer RTs in the ungrammatical conditions than in the grammatical conditions because the detection of the ungrammaticalities should interfere with automatic processing routines. Indeed, the native English speakers had longer RTs in the ungrammatical conditions than in the grammatical conditions, consistent for all six structures across the PT stages. Consequently, grammatical sensitivity scores (i.e., RTs on ungrammatical segments minus RTs on corresponding grammatical segments) consistently yielded positive values reflecting sensitivities in the right direction. See Table 3.
Summary of the RTs in milliseconds (ms) on the critical and spillover region—Native English controls
Sets of paired-sample t-tests were carried out for both by-subject (t 1) and by-item (t 2) analyses. First, the analyses on the critical region revealed that native speakers’ RTs in the ungrammatical condition were significantly longer than the RTs in the grammatical condition for all structures except the object pronoun in the by-subject analysis (p 1 = .113). We also examined the spillover regions, the segments immediately following the critical regions, because sometimes slowdowns due to grammatical anomalies are slightly delayed. RTs were consistently longer in the ungrammatical than in the grammatical condition in this region. The by-item analyses found that such differences were significant for all structures except the wh-copula questions. The by-subject analyses also confirmed significantly longer RTs in the ungrammatical condition on three structures: possessive -s, object pronoun, and cancel sub/aux inversion structures. Full details on the by-subject and by-item analyses in the critical and spillover region are provided in Table 4.
Summary of the inferential statistics on the critical and spillover region—Native English controls
In summary, the native speakers demonstrated sensitivity to all grammatical structures tested in the self-paced reading task as expected, including the crucial stage 5 structures: auxiliary second questions and third-person singular –s.
ESL Learners
The ESL learners were divided into three groups. High-level learners demonstrated emergence in production of both target stage 5 structures; low-level learners demonstrated emergence in production of neither of them. We also included for comparison a group of 18 “mixed” learners who demonstrate emergence of one stage 5 structure, and who according to PT are also in stage 5 or higher.
According to the PSH, only the high-level ESL group (and perhaps also the mixed group) should demonstrate grammatical sensitivity on the auxiliary second questions and third-person singular –s sentences in stage 5. If they are of a high enough level, they may also demonstrate sensitivity to cancel sub/aux inversion structures in stage 6, although no prediction can be made for that level. However, the low-level ESL learners may demonstrate sensitivity to level 2, 3, or 4 structures, but they should not demonstrate sensitivity to level 5 structures. Descriptive statistics of the three groups are presented in Table 5.
Summary of the mean RTs on the critical and spillover region (ms)—ESL learners
Note: Grammatical sensitivity (RT [ungrammatical] – RT [grammatical]) scores that are negative indicate longer RTs on grammatical items, thus indicating reversed sensitivity.
Results demonstrated that the high-level ESL learners showed no sensitivity on the auxiliary second questions in either region in the by-subject and by-item analyses: by-subject: critical region, p 1 = .782, and by-item: p 2 = .424; spillover region by-subject: p 1 = .682, and by-item: p 2 = .361. They also failed to show sensitivity to the third-person singular –s structures, both on the critical (p 1 = .063, and p 2 = .254) and spillover regions (p 1 =.229, and p 2 = .181). In fact, the learners had longer RTs on grammatical than ungrammatical third-person singular verbs, at a level near significance (see Table 5). In other words, the learners processed the ungrammatical verbs lacking morphological marking (e.g., she run) more easily than the grammatical verbs including morphological marking (e.g., she runs).
The low-level ESL learners performed similarly on the crucial stage 5 region, showing sensitivity to auxiliary second questions neither on the critical region (p 1= .836, and p 2 = .701), nor on the spillover region (p 1= .296, and p 2 = .395). For the third-person singular –s structures, RTs were longer on the critical region in grammatical sentences than ungrammatical sentences, a difference that was statistically significant in the by-subject analysis, p 1. = .024, indicating that they processed the sentences with verbs lacking morphological marking more easily than those with grammatical marking. Results on the spillover region were not significant, p 1 = .225. In the by-item analysis, no significant differences in RTs were found: critical region, p 2 = .120; spillover region, p 2 = .296.
Finally, the mixed group also failed to show any sensitivity to the stage 5 ungrammaticalities. The summary of the statistical analyses on the two stage 5 structures are provided in Table 6.
Summary of the inferential statistics of the two PT stage 5 structures—ESL learners
Note: Negative t-values indicate longer RTs in grammatical than in ungrammatical sentences.
Table 7 provides the summary of the inferential statistics for structures in stages 2, 3, 4, and 6. Note that the learners demonstrated sensitivity to some structures in the self-paced reading task, indicating that the task was suitable to register sensitivity to ungrammaticality in the learner groups. However, some of the significant results appear only in the by-item analysis, indicating that there is variation in the performance of the learners.
Summary of the inferential statistics of the stages 2 through 4 and 6: ESL learners
Note: p-values in bold indicate significantly longer RTs in ungrammatical than in grammatical sentences.
The low-level group showed a delayed effect of ungrammaticality on the spillover region in the possessive –s structure in stage 2, both in by-subject and by-item analyses, by-subject: p 1 = .026, d 1 = .617 and by-item: p 2 = .034, d 2 = .487. They also demonstrated sensitivity to the level 4 structure in a by-item analysis of the spillover region: p 2 = .011, d 2 = .946. Additionally, for this group the results for the stage 6 structure in the by-subject analysis approached significance in the critical region, p 1 = .061, d 1 = .418. Finally, the low-level learners read the stage 3 object pronoun grammatical sentences significantly more slowly in the target region in the by-item analysis (in other words, a “backward” result), p 2 = .037, but not in the spillover region. In the spillover region, RTs were longer on ungrammatical items at a level approaching significance, p 2 = .065, d 2 = .507, a rather puzzling set of results.
The high-level ESL learners demonstrated delayed sensitivity on the spillover region in stage 2 and 3: stage 2 possessive –s, by-subject, p 1 = .025, d 1 = .520; stage 3 object pronoun, p 1 = .003, d 1 = .779 and p 2 = .010, d 2 = .710. The by-item analysis demonstrated sensitivity to the stage 6 structure in the critical region (p 2 = .011, d 2 = .987). However, learners spent longer on grammatical sentences when they read the stage 4 wh-questions, showing reversed sensitivity patterns, which was at a level approaching significance in the by-item analysis (p 2 = .068).
The mixed group demonstrated delayed sensitivity to the stage 3 structure only, both in by-subject (p 1 = .036, d 1= .743) and by-item analyses (p 2 = .043, d 2 = .723) of the spillover region. In the by-subject analysis, the result approached significance for the stage 6 structure, but in reverse—that is, grammatical sentences were read more slowly than ungrammatical ones: p 1 = .085. Table 8 summarizes the findings on the self-paced reading task.
Summary of findings on the self-paced reading task
DISCUSSION
The research question that motivated this study is: Do ESL learners who are in stage 5 or higher on the PT scale based on oral interviews also show sensitivity to stage 5 structures in a self-paced reading task, as predicted by the PSH? The results of the self-paced reading task indicate that the answer to the research question is no. That is, learners who used stage 5 structures in a productive way in an oral interview failed to demonstrate sensitivity to those same structures in a receptive task. Instead, they performed similarly to the stage 4 and under group on these structures. This result contrasts Pienemann’s (1998, 2001) findings for learners of German, and fails to support the PSH. The finding relates to the acquisition of both a syntactic structure, namely auxiliary second questions, and a morphological form, third-person singular –s. That is, learners who produced auxiliary second questions in their speech did not demonstrate sensitivity to ungrammaticalities in these questions in an online receptive task, and learners who produced third-person singular –s in speech did not demonstrate sensitivity to ungrammatically unmarked third-person singular verbs in the online receptive task.Footnote 5 By contrast, learners demonstrated sensitivity to other structures in the task, which indicates that it is adequate to detect some kinds of grammatical sensitivity in nonnative speakers.
The fact that production and reception do not appear to be acquired at the same time for these structures raises some important issues about PT, primarily regarding its place in the linguistic system. According to Pienemann, processing procedures are not part of a declarative store of information, such as the lexicon; rather, they are separate mechanisms: “In this perspective, the language processor is seen … as the computational mechanisms that operate on (but are separate from) the native speaker’s linguistic knowledge” (1998, pp. 4–5). Only recently have researchers addressed the question as to whether these procedural systems can be accessed during both language production and comprehension, perhaps in a similar manner to the lexicon in Levelt’s (1993) model, which is centrally connected to both productive and receptive systems. Lenzing (ms.), for example, suggests that processing procedures are part of a central workspace called the “grammatical coder” that is shared by both the grammatical encoder and the grammatical decoder. However, the dissociation between production and reception found in the current study implies that the procedures might not be shared by both systems in a straightforward way.
An alternative explanation is that the same basic procedures could function in comprehension after a delay. That is, receptive skills could in principle emerge in the same implicational order as productive skills, but some additional factors relating to listening or reading tasks could cause them to appear more slowly. There is perhaps some mixed evidence for this when the results from the other stages are considered. The low-level learners demonstrate some sensitivity to stages 2, 3, and 4. This group demonstrated fairly robust sensitivity to stage 2 structures, demonstrating longer RTs on ungrammatical structures in both the by-subject and by-item analysis. However, they had conflicting results on stage 3 (significantly longer RTs on grammatical items in the target region, but longer RTs approaching significance on ungrammatical items in the spillover region, both in the by-item analysis) and a somewhat weak result for stage 4 (a significant difference between grammatical and ungrammatical items only in the spillover region in the by-item analysis). The weak and inconsistent results could feasibly point to a transitional period before the acquisition of stage 5 knowledge, at least for some of the learners; recall that this group was categorized only as stage 4 or under, so they are of mixed developmental levels. The high-level learners demonstrated fairly robust sensitivity to stage 2 and 3 structures but not to stage 4. In fact, RTs were longer on grammatical items in stage 4. These findings give the impression of a possible implicational order, although it is not clear why the high-level learners would not be more clearly advanced than the low-level group. The odd group here is the “mixed” learners, who are only sensitive to stage 3 structures, seemingly “skipping” stage 2. This result is puzzling because according to PT, this group should perform similarly to the high-level learners. Their performance indicates that the emergence of receptive skills may not be clear-cut.
The findings for stage 6 structures at first glance seem to provide further evidence against the predictions of the PSH, but it is not clear that this is the best interpretation of the results. First, note that both the low-level and the high-level learners demonstrated sensitivity to stage 6 structures, but not stage 5. It is likely that the particular way that wh-noun clauses are expressed in English may be the root of the issue. Consider these example sentences:
(5) I knew / what / my friend was / thinking / yesterday.
(6) *I knew / what / was my friend / thinking / yesterday.
To demonstrate grammatical sensitivity to this structure, learners must read the grammatical sentence (5) more quickly than the ungrammatical (6). Note that the grammatical sentence in this case uses subject verb object (SVO) word order rather than the order typical of English questions, which involves subject-auxiliary inversion. It is therefore possible that the low-level learners could be getting the right result for the wrong reason. That is, they may prefer SVO word order because they are not yet completely familiar with subject-auxiliary inversion in these types of questions.Footnote 6 This analysis is supported by the fact that they do not produce auxiliary second questions in their speech. However, the mixed learners demonstrated sensitivity to stage 6 structures at a near-significant level but in a backward way; that is, they read the ungrammatical sentences more quickly. In this case, the ungrammatical sentences have subject-auxiliary inversion similar to questions. Note that most of the mixed learners also produced questions with subject-auxiliary inversion in their speech, so it is possible that they are familiar with English word order in questions but have not yet acquired the rule that “cancels” subject-auxiliary inversion in certain embedded clauses. The problem with this argument is that the group should be sensitive to stage 5 structures in the SPR if this is the case. Another part of this story that is not easily explained is the high-level learners’ sensitivity to stage 6 structures. If they have acquired questions but not stage 6 structures yet, they should behave more similarly to the mixed learners. However, if they have acquired the stage 6 structures, then they should also demonstrate sensitivity to stages 4 and 5. Therefore, we have no clear explanation—and unfortunately no explanation that is consistent with PT—for their results.
At this point, it should be noted that there are several factors that could have influenced these results. One possible factor is the variety of L1s among the participants. PT predicts that L1 transfer should not alter the order of acquisition of the grammatical stages. However, transfer is possible once the processor is developed enough to handle a particular grammatical structure. It is conceivable, therefore, that transfer could influence the level of sensitivity that learners have to a particular structure once they reach that stage. Future research could examine this issue by considering learners’ responses on productive versus receptive tasks, while controlling for the L1.
A second issue is the selection of appropriate measures of acquisition in production and reception. PT has a long-established measure for acquisition—that is, the productive emergence of a grammatical element. It is possible that this criterion is simply too low for this comparison. That is, learners may need to be able to use a grammatical element in a more accurate way before they are sensitive to it in receptive tasks. Indeed, Pienemann (Reference Pienemann1998) makes a similar suggestion. He states that processing routines in learners and native speakers should be alike once these routines are automatized—presumably, that is, once they proceed rapidly, without conscious awareness, unstoppably, and without much interference from other processes (Bettoni & Di Biase, Reference Bettoni, Di Biase, Bettoni and Di Biase2015; Segalowitz, Reference Segalowitz, Doughty and Long2003). Unfortunately, PT provides little guidance as to how to measure automatization of PT procedures, and Pienemann does not attempt to measure it in his (1998, 2001) study. Like Pienemann, we did not measure automatization here, but it is reasonable to assume that more advanced learners would have demonstrated greater sensitivity on the SPR task.
If automatization is the key factor in receptive performance, there are several important questions relating to these issues, such as (a) How should automatization be measured in relation to PT?; (b) Should automatization be measured separately for production and reception?; and (c) Does automatization of grammatical elements proceed according to the PT order? It is not clear whether work in this area would benefit by making connections between approaches such as skill acquisition theory (e.g., DeKeyser, Reference DeKeyser, VanPatten and Williams2015), which more directly deals with these types of issues. The influence of automatization in PT would be a highly fruitful area for future research, as Bettoni and Di Biase (Reference Bettoni, Di Biase, Bettoni and Di Biase2015) note.
One other factor that could have affected the results of this study is the methodology. Self-paced reading tasks measure exactly that: reading. If learners’ literacy skills are poor, self-paced reading tasks will not be an accurate measure of grammatical sensitivity (see Reference Smith, Benati and YamashitaSmith, 2016). Note that the L1s of many participants in the study have different reading directionality than English, different writing systems and orthographies, all of which could have affected their performance on this task. One way to address these issues in future studies is to employ self-paced listening or other measures that do not employ reading. Unfortunately, one downside of self-paced reading, listening, and eye-tracking measures is that they cannot (as far as we can see at this point) directly address emergence in comprehension per se. That is, it is not possible with these methodologies to measure the beginning moments of comprehension. Indeed, it is likely that the use of a variety of methodologies will be necessary to better understand how emergence in comprehension can be measured (see Buyl & Housen, Reference Buyl and Housen2015 and Spinner, Reference Spinner2013 for further discussion on measuring emergence with receptive tasks).
One final point of interest from this study is the finding from the oral interviews that several learners used the syntax of stage 5 productively in speech but not the morphology. This has been a consistent finding of recent work on PT; both Bonilla (Reference Bonilla2015) and Dyson (Reference Dyson2009) found that morphology lagged behind syntax in learners’ production. Dyson explained the dissociation between syntax and morphology with her universal properties hypothesis. Her proposal is that properties of Universal Grammar (UG) interact with the morphological procedures of PT, in that the syntax creates contexts for the morphology. As far as we are aware, there is no other proposed explanation, and no explanation within PT, as to why syntax might precede morphology in the PT stages. This issue should be further addressed. A useful starting point might be conducting a complete distributional analysis of each learner’s production data to examine more precisely the state of the learner’s system.
Of course, it should be noted that the emergence of syntax before morphology is not evidence against PT. As Bonilla (Reference Bonilla2015) and Buyl and Housen (Reference Buyl and Housen2015) point out, according to PT, not all elements of a given stage need to emerge before a learner moves on to the next stage. Unfortunately, the curious consequence of this statement is that, in principle, the only way to provide unequivocal evidence against PT when looking at sequences of acquisition is to examine every possible grammatical form or structure from a stage because it would be crucial to pick the right one—the one that emerges first—or else risk drawing a false conclusion against PT (see Mansouri & Håkansson, Reference Mansouri, Håkansson and Mansouri2007 and Bettoni & Di Biase, Reference Bettoni, Di Biase, Bettoni and Di Biase2015 on intrastage sequencing). Obviously if this were necessary, it would present a formidable challenge to researchers, and would weaken the predictions of the theory. But in fact, it appears not to be necessary because many researchers have found evidence for PT while examining only a selection of forms from various stages. The extent to which all forms and structures of a particular stage emerge after the first emergence in that stage could be another fruitful subject of future research.
CONCLUSION
This study examined whether the PT order of acquisition is upheld in a self-paced reading task. We examined the PSH, which predicts the behavior of L2 learners in online receptive tasks. While the PSH was not supported, there was some support for an order of acquisition in receptive data as predicted by PT, albeit one that follows production after a delay. However, given that this support was quite mixed, it warrants future research.
We conclude here by noting that these results have raised several questions that should be pursued further. First, it is not clear whether some of the results here are due to methodological limitations of the self-paced reading task; other methodologies such as eye-tracking, self-paced listening, and so on need to be employed. Additionally, because there is evidence that syntax precedes morphology in the PT order, and because it has been suggested that only one form or structure from a particular PT stage may be acquired before a learner moves to the next stage, a variety of structures should be examined. Finally, the role of both L1 influence and automatization in production and reception needs to be explored. We are hopeful that consideration of these issues will serve to better develop PT.
