1. Introduction
In second language (L2) teaching, a common practice is for teachers to provide WCF in response to students’ production, a practice aimed at rectifying learners’ errors. Language teaching methodologies that have gained popularity in the last 15–20 years, such as focus on form approaches and task-based language teaching, support this practice. Among L2 researchers, however, WCF is a divisive issue. While many contend that WCF can serve as a catalyst for L2 development (cf. Sheen, Reference Sheen2010; Sheen & Ellis, Reference Sheen, Ellis and Hinkel2011; Bitchener & Ferris, Reference Bitchener and Ferris2012; Bitchener & Knoch, Reference Bitchener and Knoch2015; Nasaji & Kartchava, Reference Nassaji and Kartchava2017), others have called for researchers to examine critically the practice of grammar correction and additional mediating factors rather than to accept without question its effectiveness (e.g., Truscott, Reference Truscott2007; Truscott & Hsu, Reference Truscott and Hsu2008; Kang & Han, Reference Kang and Han2015; Liu & Brown, Reference Liu and Brown2015).
Complementing WCF research in L2 writing, the field of instructed second language acquisition (ISLA) has produced a considerable number of empirical studies on the effectiveness of WCF, in response to, on the one hand, Truscott's (Reference Truscott2007) unfavorable review of WCF research and, on the other hand, the ongoing call for evidence objectively verifying instructional gains in L2 development (Long, Reference Long2017). While this vast body of research has led to complex, if not seemingly conflicting findings, the immense productivity of corrective feedback researchers in the last decade attests to an ongoing interest in this ubiquitous L2 classroom practice. The general agreement among ISLA researchers appears to be that corrective feedback plays a crucial role in L2 learning (Nassaji & Kartchava, Reference Nassaji and Kartchava2017), although the questions of how and when to provide effective feedback and what the potential side effects may be remain unresolved.
One way to address these pressing questions is to reproduce previous WCF studies. An important aspect that has not yet received enough attention in the literature is the choice of target errors in WCF interventions. So far, the discussion of WCF targets, aptly summarized by Lee (Reference Lee2013), has largely focused on a number of dichotomies: between ‘errors’ and ‘mistakes’—leading to a conclusion that only errors need correcting; between global and local errors—with only the former needing attention; and between treatable (rule-governed) and untreatable errors (non-rule-governed)—suggesting WCF is most effective when directed at treatable errors. With little empirical evidence to support these dichotomies, ISLA researchers working on WCF have elected to target a somewhat limited set of linguistic structures, with much attention given to two complementary structures from the set of English articles—first mention a and subsequent mention the (e.g., Sheen, Reference Sheen2007; Ellis, Sheen, Murakami, & Takashima, Reference Ellis, Sheen, Murakami and Takashima2008; Bitchener, Reference Bitchener2008; Bitchener & Knoch, Reference Bitchener and Knoch2008, Reference Bitchener and Knoch2009, Reference Bitchener and Knoch2010).
Dubbed ‘the article studies,’ these investigations have provided widely-cited empirical evidence that feedback targeting two single-rule English article functions is effective for L2 learning and that WCF can lead to greater grammatical accuracy in L2 writing. While many have interpreted the findings of the ‘article studies’ as counter-evidence to Truscott's (Reference Truscott1999, Reference Truscott2007) claims that WCF has no effect, or even a negative effect (cf. Polio, Fleck, & Leder, Reference Polio, Fleck and Leder1998), on learners’ accuracy, the nature of the target feature and the selection of only two functions of a and the raises questions as to the strength of the conclusions that can be drawn from these studies. For example, to our knowledge, when reporting improvements in accuracy after participants received some type of WCF on the two article functions, none of the existing studies considers the potential effects of WCF on the remaining functions of the article system. The question remains as to what effects the provision of WCF on these two article functions may have on the remaining functions of articles and their use in L2 writing.
In view of this unresolved issue, it is critical to replicate and expand on the studies that have begun to examine how WCF contributes to overall L2 development. By undertaking a replication of Bitchener and Knoch (Reference Bitchener and Knoch2010), the present investigation reproduces one of the most cited and influential studies in the field of WCF. The 2010 study by Bitchener and Knoch culminates the work of the first research team to investigate the potential of WCF with a tightly controlled methodology including a pre-, post-, and delayed post-test design, a control group, and a specific linguistic error category. In their work, Bitchener and Knoch also tested the effects of WCF in the writing of new texts immediately after the feedback was provided, thereby enabling researchers to claim that it was the feedback rather than instruction that facilitated the change as well as durable effects. Thus, the work of Bitchener and Knoch expanded primary research and broke new ground for the subsequent research meta-analyses.
The replication described here was undertaken with a view to confirm or otherwise qualify the results of the 2010 study and to shed more light on the issue of validity of Bitchener and Knoch's interpretations of their findings. Specifically, the present replication seeks to address the question of the suitability of the chosen linguistic targets (i.e., two single-rule English article functions, first mention a and subsequent mention the) for WCF techniques and potential side effects that WCF interventions may have on larger linguistic systems (i.e., the English article system). Answers to these questions would, in turn, contribute to a more nuanced understanding of the contribution of WCF to L2 development of grammatical accuracy in writing as well as the variables that mediate such development.
2. WCF research in ISLA
ISLA's interest in corrective feedback has largely been motivated by Schmidt's (Reference Schmidt1990, Reference Schmidt and Robinson2001) noticing hypothesis, which holds that it is only what the learner attends to and notices in the input that can potentially assist L2 learning:
Learning in the sense of establishing new or modified knowledge, memory, skills, routines is … largely, and perhaps exclusively, a side effect of attended processing.
(Schmidt Reference Schmidt and Robinson2001, p. 29)The issue of whether conscious, focal attention is necessary for L2 acquisition is therefore of particular significance to the body of research concerned with corrective feedback. It is hypothesized that provision of corrective feedback may help L2 learners notice that there is a mismatch between their interlanguage and the target language system. Many ISLA researchers contend that the resolution of this mismatch may trigger destabilization and consequent restructuring of the learner's L2 grammar. Oral corrective feedback techniques, such as recasting, clarification requests, comprehension checks, and repetition, have been the most investigated to date, based on the argument that implicit strategies are more beneficial in terms of engendering the acquisition of form-meaning mappings than explicit strategies (e.g., Doughty, Reference Doughty, Doughty and Long2003). By contrast, WCF studies in ISLA rely on the notion that learners draw upon both explicit and implicit linguistic knowledge during the writing process. As pointed out by Bitchener & Knoch (Reference Bitchener and Knoch2009), L2 learners are likely to reflect upon the form or structure that should be used in their writing, relying either on their explicit, metalinguistic knowledge or making decisions about which structure to use without conscious reflection.
Of particular interest to ISLA studies on WCF is the question of potential mediating factors. For instance, there has been a line of research focusing on the efficacy of WCF as a function of its scope—focused vs. unfocused—in improving L2 learners’ overall accuracy (cf. Sheen, Reference Sheen2007; Ellis et al., Reference Ellis, Sheen, Murakami and Takashima2008; van Beuningen, de Jong, & Kuiken, Reference van Beuningen, de Jong and Kuiken2012). Earlier WCF studies (cf. Ferris, Reference Ferris2002, Reference Ferris2003, Reference Ferris, Hyland and Hyland2006) often targeted a wide-range of linguistic forms and structures, sometimes as many as 15. In ISLA studies relevant to the present replication (cf. Sheen, Reference Sheen2007; Bitchener & Knoch, Reference Bitchener and Knoch2008, Reference Bitchener and Knoch2009, Reference Bitchener and Knoch2010), the researchers decided to limit the focus of WCF. Ellis (Reference Ellis2009) reports that focused feedback targeting fewer grammatical errors is more effective than unfocused feedback that targets more errors, arguing that the former focuses, rather than disperses, learner attention, thereby enabling learners to more readily recognize the difference between their own use and the target-like use of forms.
As mentioned earlier, a mediating factor of utmost importance to the present replication concerns the selection of linguistic target features in the investigation of focused WCF. The question of which errors to correct, or the choice of the target feature in WCF studies, has been hypothesized as an important consideration, most likely contributing to the outcomes of the WCF interventions. Research evidence amassed in the past decade is still inadequate to answer the question originally posed by Williams & Evans (Reference Williams, Evans, Doughty and Williams1998) as to whether certain linguistic features are more amenable to feedback than others. Thus, the choice of errors for WCF should be considered ‘absolutely crucial, since WCF given to certain errors may facilitate deeper processing and uptake than WCF given to others’ (Lee Reference Lee2013, p. 112).
The English articles a and the (first and subsequent mention uses) have, to date, received a great deal of attention in WCF studies (e.g., Sheen, Reference Sheen2007; Bitchener, Reference Bitchener2008; Bitchener & Knoch, Reference Bitchener and Knoch2008; Ellis et al., Reference Ellis, Sheen, Murakami and Takashima2008; Bitchener & Knoch, Reference Bitchener and Knoch2009, Reference Bitchener and Knoch2010). While this choice may seem odd given the complexity of the English article system (cf. Master, Reference Master2007; Ekiert & Han, Reference Ekiert, Han and Alonso2016), in many respects, a focus on articles was considered ideal as articles are unavoidable in English, difficult for learners across proficiency levels, and often rule-governed, specifically the indefinite article a used for first mentions of singular count nouns and the definite article the used for subsequent mentions of the same noun. Sheen (Reference Sheen2007) also notes that with articles being generally overlooked by L2 instructors, the effect of error correction could be easily isolated from any potential effect of grammar teaching in general.
WCF studies targeting English articles provisionally suggest that focused WCF has a positive effect on developing learners’ accuracy, at least with regard to the two uses examined, first mention a and subsequent mention the. For example, examining the effects of WCF on the two article uses, Sheen (Reference Sheen2007) included three groups: one group received direct WCF, another received direct WCF and metalinguistic explanations pertaining to article usage, and the third was a control group. Both of Sheen's WCF groups outperformed the control group. Expanding upon Sheen's design, Bitchener (Reference Bitchener2008) added a fourth group who received direct WCF with both written and oral metalinguistic explanations. Similar to Sheen's (Reference Sheen2007) results, all treatment groups outperformed the control group. In another follow-up, Ellis et al. (Reference Ellis, Sheen, Murakami and Takashima2008) again examined WCF with three groups, this time distinguishing focused from unfocused WCF in the two treatment groups. As with the other studies, both experimental WCF groups outperformed the control group.
3. The original research study: Bitchener & Knoch (2010)
Following Bitchener (Reference Bitchener and Knoch2008) and Bitchener and Knoch (Reference Bitchener and Knoch2008), the original study set out to investigate the extent to which provision of WCF was effective for targeting two rule-based uses of the English article system (the referential indefinite article a for referring to something the first time and the referential definite article the for referring to something already mentioned). The ten-month study involved 52 low-intermediate English as a second language (ESL) learners divided into four groups: direct error correction; direct error correction and written metalinguistic feedback; direct error correction and a combination of written and oral metalinguistic feedback; and no feedback. The study measured WCF effectiveness by comparing the level of improved accuracy in immediate and delayed post-tests with that of a pre-test. Accuracy on each occasion was calculated as a percentage of correct usage on all obligatory occasions.
The study found that all treatment groups outperformed the control group in all post-tests, leading to the conclusion that focused WCF was effective in helping L2 learners significantly improve their accuracy using the two article functions, both short- and long-term. It must be noted that the treatment groups in the study sharply increased their accuracy between the pre- and the immediate post-tests. The subsequent three delayed post-tests, while extending the study to ten months, showed no meaningful changes in the participants’ performance on the two target articles. Moreover, Bitchener and Knoch (Reference Bitchener and Knoch2010) found no difference in effect for the different WCF types. In other words, the most explicit feedback combination (i.e., direct error correction with written and oral metalinguistic explanation and example) was no more effective than the least explicit type of feedback (direct error correction only).
Of particular importance to the present replication was the method of WCF provision with regard to the target article functions. Noting that exceptions exist for the article rules included in their study, Bitchener and Knoch (Reference Bitchener and Knoch2010) decided not to provide WCF in certain cases, explaining:
Because there are occasions when the definite article is required for referring to something for the first time…or for referring to mass nouns…, WCF was not provided on such occasions.
(Bitchener & Knoch, Reference Bitchener and Knoch2010, p. 10)That is, the feedback provided to the experimental groups was intentionally incomplete so as to avoid inaccuracies resulting from the provision of just two rules. This feedback practice was also found in other ‘article studies,’ prompting Ellis et al. (Reference Ellis, Sheen, Murakami and Takashima2008) to note, ‘it is possible that the correction [in the study] led to overuse of articles in contexts that were not the focus of [WCF]’ (Ellis et al., Reference Ellis, Sheen, Murakami and Takashima2008, p. 369). We followed this line of thinking in the present replication, asking if the provision of such feedback impacts accuracy in all functions of articles, and, specifically, how such feedback affects the subset of articles not addressed in the original study. In doing so, the current conceptual replication study (Porte, Reference Porte2012) aims to address the issue of potential side effects that was identified, but underreported, in the original study. In other words, we attempt to shed more light on the validity of the interpretation of Bitchener and Knoch's (Reference Bitchener and Knoch2010) results. By modifying the original study methodology, we also hope to generalize the findings to other contexts where focused WCF is provided with a view to describe its impact in a more complete way.
4. The replication study
4.1 Research questions
The research questions pursued in the present study incorporated the original research questions (closely reproduced in question 1), and added new ones to reflect the aims of the current conceptual replication (presented in question 2). We asked:
1. Does learners’ accuracy in the use of two target functions of the English article system improve as a result of focused written feedback given on two functions? If so, does accuracy vary according to the written corrective feedback options provided? (the two original study questions combined)
2. Does learners’ accuracy in the use of the remaining non-target functions of the English article system change as a result of the focused written corrective feedback given on two functions? If so, does it vary according to the written corrective feedback options provided?
4.2 Design
We kept the original study's design but altered several elements of its research methodology. We replicated Bitchener and Knoch's (Reference Bitchener and Knoch2010) quasi-experimental research design with a pre-test—immediate post-test—delayed post-test structure, but used just one, not three delayed post-tests and three, not four groups, excluding the oral modality in WCF. In the replication study, the 3 × 3 design assessed three groups at three distinct points in time in a semester-long study: pre-test in Week 1 followed by three treatment sessions in Weeks 3 through 5, immediate post-test in Week 5, and delayed post-test in Week 11 (see Table 1 for the study schedule). We replaced the ‘one shot’ treatment in the original study with the ecologically valid sequence of three treatments (Ellis et al., Reference Ellis, Sheen, Murakami and Takashima2008; Liu & Brown, Reference Liu and Brown2015). We also departed from Bitchener and Knoch (Reference Bitchener and Knoch2010) in not extending the replication beyond one semester, as the additional delayed post-tests in the original study did not show meaningful changes in the participants’ performance.
Study Schedule
4.3 Participants
Sixty-three participants from three intact intermediate, college-based, writing-intensive ESL classes in the United States participated in the study. The three classes had the same instructor. As in Bitchener & Knoch's (Reference Bitchener and Knoch2010) study, the students were drawn from both international and immigrant ESL populations and represented various language and ethnic backgrounds. In the original study, the students were predominantly from East Asian countries studying English in New Zealand. Reflecting U.S. immigration trends, in the current study the three major first languages (L1s) were Spanish, Chinese, and Bengali, accounting for between 50% and 70% of participants across the study groups. Other L1s represented were Arabic, Nepali, Tibetan, Urdu, Hindi, Korean, Burmese, Greek, French, Polish, Turkish, and Pashtoo, providing a similarly diverse participant pool as in Bitchener & Knoch's study. Importantly, as in the original study, equal numbers of participants in the three groups came from L1s that do not employ articles. There were 31 male and 32 female students aged 18–45 (median = 22, mean = 23, SD = 5.8). Following the original study, the three intact classes were arbitrarily assigned to the direct corrective feedback (DF) group (n = 22), the direct corrective feedback + the metalinguistic explanation (DF+ME) group (n = 23), or the control group (n = 18).
4.4 Target structure
The complex, multi-functional English article system has been widely described for its acquisitional difficulty for L2 learners. A relatively well-defined aspect of that system was chosen as the WCF target structure in Bitchener & Knoch (Reference Bitchener and Knoch2010): the complementary use of the indefinite and definite articles a and the to express first mention and anaphoric (subsequent mention) reference. Taken from the data collected for the present study, the examples of the two uses are as follows:
1. First mention or referential a (e.g., I was sitting at a café with my friend).
2. Subsequent mention or anaphoric the (e.g., I think the café was being renovated).
In line with Bitchener and Knoch (Reference Bitchener and Knoch2010), the two uses of a and the constituted the treatment and accuracy assessment targets in the present study.
Extending Bitchener and Knoch (Reference Bitchener and Knoch2010), our investigation also assessed learners’ accuracy in the use of the remaining functions of articles found in the data. Those article functions included:
1. Cataphoric the, where the definite article is used with a first mention noun that has a modifier (e.g., Its fabric can't stand the power of their pull).
2. Situational the, where the definite article is used when the referent of a first mention noun can be sensed directly or indirectly by the interlocutors, or where the referent is known to members of a local community, such as the only bookstore in a town (e.g., When it was 7 am, they opened the doors).
3. Nonreferential a, where a nonspecific reference applies to any token of its type (e.g., My neighbor is an attractive woman).
4. Idiomatic uses of a and the, where articles are ‘locked’ in formulaic chunks (e.g., once upon a time or all the time).
In accounting for the articles which were not the target of correction in Bitchener and Knoch (Reference Bitchener and Knoch2010), our replication intended to assess the impact that the selective instructional focus on just the two targeted uses might have on the remaining functions of a and the. An additional consideration was provided by corpus studies (cf. Yoo, Reference Yoo2009) which report that, contrary to general belief, situational and cataphoric uses of the are very common in conversation, newspaper, and academic prose. A post-hoc analysis of the free production data in our study revealed that, as in Bitchener and Knoch (Reference Bitchener and Knoch2010), first and subsequent mention a and the appeared regularly in the texts produced by our participants. They accounted for two-thirds of articles in our free production writing task. However, the proportion of other uses was hardly negligible and amounted to one-third of all uses, unlike in the original study which reported that only 8% of occasions in which articles were used were non-targeted a and the.
4.5 Treatment
The present study used the two WCF techniques utilized in Bitchener and Knoch (Reference Bitchener and Knoch2010)—direct feedback (DF) and metalinguistic explanation (ME). Following the original study, DF was operationalized as a strategy that consists of indicating the location of an error on the student's text and providing the correct form by deleting and/or replacing the error or by adding a form. In the replication study, as in the original study, direct correction was limited to a for the first mention and the for a subsequent mention of the noun. WCF was not provided for other occasions when a or the were needed. A sample direct correction is shown in the following example from our data set, with our insertions in bold:
One night, a handsome man and a beautiful woman at a restaurant to eat dinner. They order a meal and nice wine.
ME, taken directly from Bitchener and Knoch (Reference Bitchener and Knoch2010), included an explanation of the two targeted functional uses of a and the, together with an example of their use, contextualized for each testing occasion. In providing ME, we used written modality, unlike Bitchener and Knoch, who employed both written and oral modalities. Participants who received ME were given direct in-text correction and the following explanation with an appropriate illustration:
1. Use ‘a’ when referring to something for the first time.
2. Use ‘the’ when referring to something that has already been mentioned.
Example: A man and a woman went to a restaurant for dinner. The man ordered a bottle of wine and the woman drank the wine.
Two experienced raters, each with a background in applied linguistics, provided the corrections. Group 1 (DF) received in-text direct error correction only and Group 2 (DF+ME) received in-text direct error correction complemented by a written metalinguistic explanation. As a departure from the original study, Group 3 (Control) received ‘summary end notes’ (Ellis et al., Reference Ellis, Sheen, Murakami and Takashima2008; Ferris, Reference Ferris2012) on the overall quality of their writing, the rationale being that the study should include a true control group if the WCF efficacy is to be effectively addressed. Neither in-text corrections were provided nor were any references to article use in the end notes made.
4.6 Instruments
In the original study, only one task—a picture description of different social gatherings for each data collection point (a beach, a picnic, a campsite, a family celebration, and a sporting event)—was used to elicit written data. The researchers chose pictorial input for its authentic value in the context of a language class as well as its potential to create opportunities for the use of the two target functions of articles. Each picture was accompanied by a vocabulary gloss in order to lower the testing anxiety level for the students. In the present replication, a set of two tasks was introduced. In designing the two instruments, we followed the well-heeded recommendation to examine free as well as controlled production to assess learners’ L2 proficiency (Ellis, Reference Ellis2005) and employed two tasks at each testing juncture tapping into both free and controlled production. Similar to the original study, the two tasks in the replication study were based on pictorial input and called for elements of description in student writing.
The two instruments in the replication study employed narrative pictorial prompts, following the suggestion that ‘in creating tasks for developing knowledge of articles, the task designer… [should] consider using narrative tasks for the definite/indefinite distinction’ (Loschky & Bley-Vroman, Reference Loschky, Bley-Vroman, Crookes and Gass1993, p. 133). The narrative genre was also considered ecologically valid in the context of an L2 writing class. The two narrative tasks created an authentic textual context for the interplay of form-meaning mappings expressed by articles which can be fully conveyed and understood only at the discourse level (cf. Ekiert, Reference Ekiert, Ortega, Tyler, Park and Uno2016). Given the limited context for the interplay of different functions of English articles provided by the static pictures used in Bitchener and Knoch (Reference Bitchener and Knoch2010), the two tasks designed for the replication study included sufficient context for the introduction and reintroduction of objects and people into a connected narrative, calling for frequent explicit referent identification and tracking with articles.
1. Writing task
Free written production was elicited with cartoon narratives. Five different cartoons were used, each of them consisting of six pictures (see Appendix A for an example). The cartoons involved multiple individuals and objects introduced and re-introduced in the stories, thus creating ideal contexts for uses of the target structures. Following Ellis et al. (Reference Ellis, Sheen, Murakami and Takashima2008), each cartoon was accompanied by a written story stimulus of approximately 300 words (see Appendix A for an example) to ensure that all participants processed the story line in the same way and completed the same task. The written stimulus story was not available to participants during the writing time. In sum, the replication study replaced the one-picture task input and vocabulary glosses found in the original study with a six-picture cartoon and written stimulus story. To guard against the possibility that the stories posed differing levels of difficulty for the learners, they were counterbalanced in each group at each testing time. The task was not timed and took 30 minutes, on average, to complete.
2. Correction task
The correction task used a structured input designed to control for specific linguistic contexts requiring the application of target articles. Three narratives adapted from Aesop's fables served as the stimulus text of approximately 300 words (see Appendix B for an example). The test items were embedded in sentences that formed a coherent text. Based on the Flesch-Kincaid Reading Ease test, the difficulty of the fables was found to be at 95, indicating a fairly easy-to-read text. All instances of targeted article use (nonreferential a, referential a; anaphoric the, cataphoric the, situational the, and idiomatic uses of a and the) were deleted in the text. The forms were piloted and only unambiguous contexts were included in the final versions. Following Liu and Gleason (Reference Liu and Gleason2002), no blanks were given for the missing items, as learners might otherwise feel obligated to fill every blank, thereby compromising the validity of the data. To guard against the possibility that the fables posed differing difficulty levels, they were counterbalanced in each group at each testing time. The task was not timed and, on average, took 20 minutes to complete.
4.7 Data collection setting and procedures
Data collection took place during class time within a regular semester. The classes met for two hours daily and were focused on academic writing and reading. Care was taken to ensure that, during the period of study, no explicit attention was paid to articles. During the 12-week semester, the three groups received three treatment sessions, following Ellis et al. (Reference Ellis, Sheen, Murakami and Takashima2008), and were tested at three different points in time, as presented in Table 1. On day one in Week 1, the pre-test was administered. In Weeks 3 and 5, treatment was provided in three consecutive sessions. The immediate post-test was also completed in Week 5, and the delayed post-test in Week 11.
The writing task involved reading and then rewriting cartoon-based narratives. The procedure was adopted from Ellis et al. (Reference Ellis, Sheen, Murakami and Takashima2008) and was as follows. The written stimulus story was handed out with each cartoon story but collected before the participants wrote their own narratives. The participants were told to retell the story with as much detail as possible, using the cartoon pictures. During the treatment phase of the study, feedback was provided on the cartoon-based narratives in all groups in accordance with the treatment guidelines described earlier. In the following week, the participants received their narratives back and were given time to examine their individual feedback. The instructor did not comment further on the errors or give any additional explanation; nor were the participants asked to revise their writing. Instead, they wrote the next story in the same lesson.
The correction task involved reading and correcting a fable. A picture stimulus was included with the written text, and the participants were shown a short practice sentence involving non-article functors (i.e., pronouns and prepositions) before the beginning of the test. The instructor did not comment on the insertions and no feedback was provided.
4.8 Data coding and scoring
Two raters independently coded the data elicited from the writing and correction tests, identifying unnecessary uses of articles and inserting missing yet obligatory ones. In doing so, the replication study departs from the original study in which data coding and scoring was limited to a for the first mention and the for a subsequent mention of the noun.
In view of limited information on how the appropriateness of article use was determined in the original study, in the replication study the judgments were based on the context provided by each text produced by L2 participants (cf. di Gennaro, Reference Di Gennaro2016). Disagreements were resolved by discussion and ambiguous cases were disregarded (e.g., in several cases the intended reading was ambiguous as in, The woman felt a shame; The bartender went to exit; The woman who was a company with him). In the next step, each obligatory and nonobligatory incidence was coded for type (referential or nonreferential uses of a; anaphoric, cataphoric, or situational the; idiomatic a and the). Twenty percent of the data was randomly selected and subjected to inter-rater reliability analysis. Inter-rater reliability calculations revealed a 96% agreement on the identification of article errors and a 91% on the assignment of errors to the article categories.
To examine the effects of the two types of treatment on the participants’ use of first and subsequent mention articles (henceforth ‘treated’ articles) as well as the remaining article uses (henceforth ‘untreated’ articles), scores for each of the writing tests (pre-test, immediate post-test, and delayed post-test) and also for the three administrations of the correction test were obtained. As in Bitchener and Knoch (Reference Bitchener and Knoch2010), the scores in both the writing and the correction tests were calculated by means of obligatory occasion analysis (Ellis & Barkhuizen, Reference Ellis and Barkhuizen2005). All obligatory occasions for article use were identified and each occasion was inspected to determine whether the correct article had been supplied. An accuracy score was then calculated for each learner by dividing the total number of correctly supplied articles by the total number of obligatory occasions and expressed as proportions of 1. None of the participants scored above .9 on the pre-test (the threshold of 90% accuracy is commonly taken as the criterion level for acquisition of a grammatical feature).
4.9 Data analysis
Descriptive statistics for the pre-test and the two post-tests on the two subsets of articles, treated versus untreated, were calculated separately for the three groups. In a departure from the original study and to help guide interpretation of instructional effects in both studies, we calculated effect sizes by comparing treatment groups with the control group in the original and replication studies (Li, Reference Li2010). We used Hedges's g, a conservative version of Cohen's d, which corrects for bias due to small sample size (<20), to assess the size of differences. Following the recent meta-analysis of WCF by Kang and Han (Reference Kang and Han2015), we adopted ISLA-specific benchmarks proposed by Plonsky and Oswald (Reference Plonsky and Oswald2014) to interpret effect size magnitude. For mean differences between groups, g and d values of .40 were considered small, .70 medium, and 1.00 large. For mean differences within groups, values of .60, 1.00, 1.40 were interpreted as small, medium, and large, respectively. In addition, the effect sizes found in the present replication study were compared to the benchmarks observed for classroom-based/ESL setting studies in corrective feedback (Li, Reference Li2010) for which the average effect was found to be .50.
The scores were then analyzed statistically with alpha set at .05. The choice of statistical tests was determined by the original study. Similar to Bitchener and Knoch (Reference Bitchener and Knoch2010), no statistical differences on the pre-test scores were found and a mixed-model analysis of variance (ANOVA) was employed to compare pre-test, immediate post-test, and delayed post-test scores for accuracy on treated articles (Research Question 1) and untreated articles (Research Question 2). There was one within-subject factor (time) and one between-subject factor (group). The advantage of a repeated measures design is that, due to more closely correlated measures (the same people tested three times), the statistical power of a test is higher despite a small sample size—13 in Bitchener and Knoch (Reference Bitchener and Knoch2010) and close to 20 in our study. In a departure from the original study, eta-squared values η 2 were calculated to provide estimates of effect sizes for the ANOVAs with values of .06, .16, .36 (Norouzian & Plonsky, Reference Norouzian and Plonsky2018) considered small, medium, and large, respectively. Post-hoc tests were used to isolate the exact points in time where differences between the groups occurred.
5. Results
Descriptive statistics obtained in the replication study are presented in direct comparison to the original study results. To make the side-by-side comparison straightforward, results from two additional delayed post-tests and the fourth group in Bitchener and Knoch (Reference Bitchener and Knoch2010) are omitted in our presentation. In other words, for comparison's sake, the original study's 4 × 5 design is rendered in the 3 × 3 format. Table 2 presents the descriptive statistics for the group mean accuracy scores on articles targeted by the WCF at the pre-test, immediate post-test, and delayed post-test in the original and replication studies. These scores are complemented by the summary of descriptive statistics for the subset of articles not targeted by the WCF and measured only in the replication study.
Descriptive statistics for accuracy on treated (original vs. replication study) and untreated articles by group over time (replication study)
On the first mention a and the subsequent mention the, the three groups were comparable at the pre-test in both studies and improved over the course of the intervention. Thus, the results replicate Bitchener and Knoch's (Reference Bitchener and Knoch2010) findings for the treatment groups. However, unlike in the original study, in the replication study students in the control group also increased their accuracy mean scores in the 12-week intervention. It is also worth noting that in the replication study, the group means on the pre-test, immediate post-test, and the delayed post-test were approximately 10–20 points lower, on average, than in the original study. This may be explained by the use of more sensitive instruments than in the original study.
With regard to the subset of articles which were neither targeted by the WCF nor measured in the original study, the DF group's performance displayed a steady decline in accuracy over time. The DF+ME group also sharply decreased in accuracy initially, but then recovered by the delayed post-test and recorded slight accuracy gains overall. The end result for the control group was similar to that of the DF+ME group, but the trajectory was markedly different. The control group gained in accuracy on the immediate post-test.
Figure 1 provides a visual representation of the similarities and differences in the performance over time on treated articles between the original and replication study participants. The graphs show that the two WCF treatment groups improved from the pre-test to the post-test in their accuracy on the two articles targeted by the intervention, first mention a and subsequent mention the. The visual also includes the comparison of the accuracy on treated vs. untreated articles in the replication study. It highlights the losses in accuracy exhibited by the two treatment groups at the immediate post-test as well as the gains of the control group in the same period of time.
Accuracy scores on treated (original vs. replication study) and untreated articles (replication study) by group over time
Table 3 summarizes the instructional effects of WCF type as effect size represented by Hedges's g in the original and replication study, the metric not reported by Bitchener and Knoch (Reference Bitchener and Knoch2010). The first striking difference between the two studies is concerned with the better than expected performance of the control group in the replication study—the negative g values indicate that the means of the control group were larger than those of the experimental groups in the replication. The second striking difference lies in the magnitude of effect sizes. While in the original study WCF appears to have had a moderate to large effect on grammatical accuracy in the two targeted article functions (Hedges's g ranging from .64 to 1.20), the effects in the replication study are small (Hedges's g ranging from −.15 to −.55). The same conclusion of small instructional effect can be drawn for the influence of treatment type on accuracy in the remaining articles, except in one case. The provision of focused WCF on just two functions of articles (first mention a and subsequent mention the) had a large effect (g = −1.06) on the performance of the group that received the most explicit feedback (DF+ME) as recorded at the immediate post-test. In sum, we were unable to replicate the effect sizes of the original study concerning the gains in accuracy on the two target articles; however, our data replicate the upward trend of increased accuracy on the two target articles, first mention a and subsequent mention the.
Effect sizes for WCF on treated (original vs. replication study) and untreated articles (replication study) by group over time
As in Bitchener and Knoch (Reference Bitchener and Knoch2010), a series of ANOVAs were computed with regard to the effect of WCF type over time on the accuracy of the first mention a and subsequent mention the, the two targets of WCF. The results of the mixed-model ANOVA in the original study indicated a statistical interaction between time and WCF type. In the replication study, the mixed-model ANOVA yielded a statistical difference for time (F (2, 120) = 17.5, p < .001, η 2 = .21), but not for WCF type (F (2, 60) = .63, p = .54, η 2=.02) or the time-WCF type interaction (F (4, 120) = 1.23, p = .30, η 2 = .03). In other words, in the replication study a statistical change over time took place across all groups on the two article types, including the control group. Time explained 21% of variation in accuracy on the first mention a and subsequent mention the, indicating a medium effect size. This means that time had a moderate, positive impact on accuracy on the two article types, while WCF type led to small differences in accuracy over time on the first mention a and subsequent mention the.
We performed post-hoc dependent samples t-tests to understand the nature of the differences attributable to time. The results of the dependent t-tests are presented in Table 4. On average, all groups in the replication study increased their accuracy on the first mention a and subsequent mention the. Specifically, between the pre-test and the two post-tests all groups recorded gains in accuracy on the two target articles, independent of group status, but the effect sizes indicating the magnitude of change were small (.60 and .62, respectively). No statistical change occurred between the immediate and delayed post-tests.
Summary of post-hoc dependent sample t-tests for testing time comparisons on the subset of treated articles
Note. Pre = pre-test; Post = immediate post-test; Delayed = delayed post-test
These findings are consistent with Bitchener and Knoch's (Reference Bitchener and Knoch2010) findings for their treatment groups—there was improvement in accuracy on the two functional uses of a and the over time, but no difference in the effectiveness of any given treatment. However, the replication study's results for the control group performing on par with the two experimental groups at both post-tests are in conflict with the findings of the original study in which treatment groups consistently outperformed the control group.
For the subset of articles that were neither the target of the WCF treatment nor measured by the original study, a mixed-model ANOVA indicated a statistical interaction between time and WCF type (F (4,120) = 7.989, p < .001, η 2 = .20), confirming that on the untreated articles the three groups developed differentially over time. The interaction between time and group membership accounted for 20% of the variance in accuracy scores, indicating a medium effect size. The main effects for time (F(2,120) = .199, p = .82, η 2 = .002) and group (F(2,60) = 1.05, p = .36, η 2 = .03) were not statistically different.
To explore the interaction effect between time and WCF type on accuracy scores in the subset of untreated articles, we performed post-hoc one-way ANOVAs in which we compared the performance of the three groups at each testing time. As shown in Table 5, there were no statistical group differences on the pre-test or delayed post-test. At the immediate post-test, however, the groups differed statistically from each other (F(2, 63) = 7.781, p < .001, η 2= .20). Tukey's post-hoc pair-wise comparison was performed to isolate the statistical differences among the groups. The test indicated that the DF+ME group differed significantly from both the control and DF groups at the immediate post-test.
Summary of one-way ANOVAs with time predicting accuracy on the subset of untreated articles
6. Discussion
The present replication study addressed the same research problem as the original study, investigating the extent of impact of focused WCF on learners’ written production of English articles and the effectiveness of differentiated treatment types. As in the original study, WCF targeted two rule-based uses of English articles, the first mention a and subsequent mention the. To compare our findings with the findings of Bitchener & Knoch (Reference Bitchener and Knoch2010) first, we asked in Research Question 1 whether learners’ accuracy in the use of the two target functions of the English article system improved as a result of focused written feedback given on these two functions, and, if so, whether it varied according to the WCF options provided. The original study found that treatment groups outperformed the control group on all post-tests, with no difference in effectiveness found among the treatment groups. Bitchener & Knoch concluded that ‘the enduring effect on accuracy…is clear evidence of the role that WCF can play in helping learners raise the level of accuracy with which they use simple linguistic forms/structures like the two functional uses of the English article system’ (Reference Bitchener and Knoch2010, p. 14). The study also brought into question Truscott's (Reference Truscott2007) claim that WCF is ineffective and ended with a call for further research to determine the extent to which WCF is effective in helping learners acquire more complex structures.
The results of the present study replicate Bitchener and Knoch's (Reference Bitchener and Knoch2010) findings in that the treatment groups improved over the course of the intervention. Our replication study found that for articles which were the target of the intervention (first and subsequent mention a and the), there was a statistical change over time for treatment groups in the study. However, while in the original study WCF appeared to have had a moderate to large effect on grammatical accuracy in the two targeted article functions, the WCF effects in the replication study were small. Moreover, unlike in the original study, the control group in the replication study also improved its performance on the first mention a and subsequent mention the in the span of the study. These findings challenge the original study's conclusion that focused WCF alone is responsible for the observable difference in learners’ accuracy on English articles.
The replication study followed the stipulation of the original research to look beyond the ‘simple’ rules. As in the original study, in the present replication study, participants received some type of focused treatment on two select article forms. However, the present investigation differed from Bitchener and Knoch (Reference Bitchener and Knoch2010) in how the effects of treatment on the target form were measured in that article use accuracy was captured in all contexts created by the participants in their writing, both free and controlled types. Thus, in Research Question 2 we asked whether learners’ accuracy in the use of the remaining non-target functions of the English article system changed as a result of focused WCF given on two select functions of the system (first mention a and subsequent mention the) and whether the accuracy in the use of the remaining articles varied according to the WCF options provided. In doing so, our conceptual replication study addressed the potential side effects that were identified by other WCF research (cf. Ellis et al., Reference Ellis, Sheen, Murakami and Takashima2008) but ignored in the original study's analysis.
In investigating the effect of focused WCF on the subset of articles not addressed in the original study, we found, once again, that the control group outperformed the two experimental groups, both short- and long-term. The two experimental groups’ performance on the non-targeted uses of a and the is the most interesting and revealing in terms of the impact of focused WCF on the remaining parts of the system. Both experimental groups recorded losses in accuracy on the article uses that were not the focus of the intervention, immediately following the treatment. The provision of focused WCF on just two functions of articles had a large effect on the performance of the group that received the most explicit feedback (DF+ME) as recorded at the immediate post-test. The same group also experienced a temporary negative effect of the focused feedback and recorded a virtually unchanged performance on the non-targeted uses of a and the on the delayed post-test. The negative effect on the group that received less explicit feedback (DF) appeared enduring and their accuracy on the non-targeted uses of articles did not improve between the pre- and the delayed post-test. Some explanation of why this may be the case is found in the original study which reports accuracy losses in the metalinguistic groups’ performance on the target first mention a and subsequent mention the at Time 3 (first delayed post-test). Bitchener & Knoch (Reference Bitchener and Knoch2010) explain that in a post-hoc review of the students’ scripts they found a higher article overuse. Similar to the current replication study, the groups receiving metalinguistic information in the original study were found to ‘recover’ from this phenomenon. At the end, neither the original nor the replication study could confirm the overriding effect of treatment type.
The unique performance of the control group in the replication study raises questions whether the improvements in accuracy were the function of treatment or whether having students engage in tasks in which certain grammatical structures arise naturally and frequently may be both necessary and sufficient to improve L2 learners’ performance with those structures (cf. Loschky & Bley-Vroman, Reference Loschky, Bley-Vroman, Crookes and Gass1993). Along the same lines, the control group's behavior could be explained by the role task repetition played during the testing and intervention times. The repetition of encounters with two tasks that re-used the same content with the same overall purpose (five encounters with the cartoon-based narrative writing task and three with the cartoon-based narrative correction task) may be reflective of improved task performance with the growing mastery of both the task and of relevant language (Bygate, Reference Bygate2018). The original study concluded that while ‘no advantage was evident for any of the treatment groups,’ as long as some form of WCF is provided ‘it may have a positive effect on accuracy and the acquisition of simple functional uses of the English article system at least’ (Bitchener & Knoch Reference Bitchener and Knoch2010, p. 15). The present replication problematizes this assessment by finding a more consistent picture of accuracy gains on the part of the control group and the troubling side effects of the fragmentary metalinguistic explanation provided to one of the WCF treatment groups.
7. Conclusion
The larger question addressed by Bitchener and Knoch (Reference Bitchener and Knoch2010) and the present conceptual replication study is whether WCF has the potential to facilitate L2 learning and acquisition. Neither the original nor the replication study confirm the advantage for any of the WCF options. The present replication of Bitchener & Knoch brings forth new and more complete evidence on the extent of impact that WCF focusing on only two functions of the complex English article system may have on the remaining functions of the system. The findings presented here problematize the reports of the enduring target form mastery and explicate the impact of partial metalinguistic explanation given to learners.
It appears that WCF focusing on only two functions of the article system inadvertently impacts the remaining functions of the system. While the WCF provided in Bitchener and Knoch (Reference Bitchener and Knoch2010) led to increased accuracy in some functions of articles, the same WCF led to increased inaccuracy in other functions. In sum, WCF studies may need to take into account potential negative effects of focused WCF on participants’ accuracy, even when these effects are limited to the form under investigation.
Given the complex picture of findings presented by this conceptual replication of Bitchener and Knoch (Reference Bitchener and Knoch2010), further replication work would be helpful to ascertain the generalizability of the findings of the original study. A useful avenue for approximate replication research would be to address potential mediating factors contributing to the degree of improvement or decline following the provision of focused WCF while considering the role that repeated practice/task repetition and the utility/essentialness of the targeted structure in a task play in ensuring L2 accuracy gains. Another line of replication research might take the form of a close replication, keeping the study design the same but selecting new linguistic structures in order to determine whether targeting two select functions of a complex structure inadvertently impacts the remaining functions of the system. Identifying another linguistic system, besides English articles, would complement the tentative understanding of which errors should be the target of WCF in L2 classrooms. In turn, this would also help to determine whether selective targeting of linguistic features by WCF can potentially shed any light on the issue of attention in L2 learning. Finally, a fruitful area for close replication research would be to keep the design of the original study, but alter the way in which metalinguistic information was given to the participants, considering how strong the effects of such feedback are. In sum, additional replications would enable ISLA researchers to assess the validity of some of the explanations offered by Bitchener and Knoch (Reference Bitchener and Knoch2010) in the original study regarding the contribution of WCF to language development, both short- and long-term.
Acknowledgements
The authors would like to thank K. Philip Choong for his assistance with instrument design. We are grateful to the anonymous reviewers for their constructive comments and to the journal editor, Graeme Porte, and to Andrea Révész, for their helpful suggestions on the manuscript. Any inaccuracies that remain are our own. This research was supported in part by PSC-CUNY Research Award TRADA-46-174.
Monika Ekiert is Associate Professor in the Department of Education and Language Acquisition at LaGuardia CC, City University of New York, where she teaches undergraduate courses in bilingualism, linguistics, and academic writing. Her research interests lie at the intersection of SLA and pedagogy. Currently, she is involved in research projects focusing on task-based language teaching, crosslinguistic influence in SLA, and WCF. Her work appeared in various journals including Applied Linguistics and in the book series Task-based language teaching (John Benjamins) and Second language acquisition (Multilingual Matters).
Kristen di Gennaro is Associate Professor of English and Director of Composition at Pace University (NYC) where she teaches courses in writing, composition theory, and language and linguistics. Her scholarly interests include composition pedagogy, writing assessment, generation 1.5 writers, sociolinguistic aspects of writing, and the interaction of language and gender. Her work has appeared in various journals including Assessing Writing, Composition Studies, Language Testing, Writing & Pedagogy, and Journal of Basic Writing.
Appendix A
Input for the Writing Task: In a Restaurant (adopted from Author 1)
Last night, I was having a lovely dinner with a friend from out of town who is a chef in Chicago. She asked me to choose a restaurant so I made a point of selecting a very chic place downtown that might impress her. We were having a really nice time chatting, drinking wine, and nibbling at our appetizers.
I was about to ask my friend to pass the salt when our attention was shifted to a loud and rude man sitting at the table next to us. The rude man was dining with a beautiful woman. She appeared to be enjoying her meal and the wine they were drinking, but he had the most annoyed expression on his face. Out of nowhere, the man got very upset and started pointing with his index finger at the bowl of dip sitting on the table. I tried to see what he was pointing at and saw that a small fly had gotten stuck in the dip. Very loudly and rudely the man called over a young waiter to complain.
At this point, all the patrons were disturbed by the scene he was making. He reprimanded the waiter, standing up to yell even louder at the poor young man. The waiter, who looked like he could be an actor, appeared very upset by the man's actions; he couldn't take being yelled at anymore. He threw off his apron, walked out, and quit his job on the spot. After this happened, the rude man's companion had had enough. She got up and left the table, leaving him to finish the dinner alone. The man, with his mouth open, looked totally shocked and just sat there. I don't know why he was surprised—he had acted terribly.
Appendix B
Input for the Correction Task: The Fox and the Goat (adopted from Clark Reference Clark1995)
A fox was chasing a bird when it fell into a well. Fortunately, the well was not very deep, and there was not much water in the bottom of the well. The fox tried to jump out, but the well was just a little too deep. The fox was stuck. The animal looked up at the sky and listened. Unfortunately, all the fox could hear was the sound of the wind.
In a little while, the fox heard the tinkling bells of goats. A young goat came near the well for a drink, and the fox cried out, “Go away!” Standing on the edge, the curious goat looked down into the well and saw the fox sitting in the bucket used to bring up water. “What are you doing there?” asked the goat. “Haven't you heard?” asked the fox. “There is a great drought coming, and I jumped down here so I will have water. But there is enough for two, if you want to jump in.”
The foolish goat jumped in, and as it did so, the fox jumped on the back of the goat, pulled up the rope, and jumped out of the well. “Where are you going?” cried the goat. “For an animal of your age, you are very naïve,” said the fox. “But, I'm a friendly fox and I'll return when the drought comes,” said the fox as it ran away into the woods. On hearing that, the goat thought to itself, “I'm such a foolish animal!” and started crying for help hoping that the goats wandering in the fields would come to its rescue.
