Assessing feedback effectiveness

A key question in feedback research has always been whether, and to what extent, feedback is effective. Studies have used various measures to assess feedback effectiveness ranging from various forms of learner responses to feedback (including uptake and learner repair) and individualized tailor-made tests in descriptive research to different kinds of pretest-posttest measures in experimental research. The question that arises in this context is what these tests measure and to what extent they provide evidence for L2 learning. In the subsequent sections, I will discuss some of the most widely used outcome measures in corrective feedback research and the issues associated with them. I will begin with measures used in descriptive research and then those in experimental research.

Measures used in descriptive research

Individualized posttests

To provide more direct evidence for the relationship between feedback and learning in descriptive research, some studies have used what is called individualized posttests. Such tests are tailor-made and designed based on the feedback each learner receives on an erroneous target form and are then administered to the same learner after the interaction. The aim is to examine the effects of feedback on forms targeted incidentally during interaction. Individualized posttests are constructed based on the focus on form episodes (FFEs) in which a target language form has occurred. The construction of individualized posttests begins with the identification of such episodes and then the creation of the test items based on those episodes. The following from Nassaji (Reference Nassaji2010, p. 919) provides an example of an FFE and a test item developed based on that episode.

Example 4 Focus on form episode (FFE)

S1: I don't know wolf.

S2: Wolf is a . . . wolf is a [gesturing with hands] . . .

T: Wolf [opening and closing hand like the mouth of a dog]. Ah . . . similar to a dog. A wild dog is a wolf.

Individualized posttest item based on FFE

Which animal is a wolf?

Several studies have used individual posttests to assess the effectiveness of interactional feedback and FonF (Loewen, Reference Loewen2002, Reference Loewen2005; Nabei & Swain, Reference Nabei and Swain2002; Nassaji, Reference Nassaji2010; Kim & Nassaji, Reference Kim and Nassaji2018). Loewen (Reference Loewen2005), for example, used such tests to investigate the effectiveness of incidental feedback in communicative English as a second language (ESL) classrooms. He found a significant relationship between feedback and the learners' correction of the targeted form after the interaction. Nassaji (Reference Nassaji2010) used individualized posttests to examine the effects of preemptive and reactive FonF including interactional feedback. He found higher individualized posttest scores for preemptive FonF than reactive FonF (see also Kim & Nassaji, Reference Kim and Nassaji2018).

Hawkes and Nassaji (Reference Hawkes and Nassaji2016) used an innovative form of individualized posttests (i.e., video-based stimulated correction tests) to assess the effectiveness of incidental recasts. The method involved identification and extraction of oral episodes in which a student had made an error. Two types of such episodes were identified (i.e., +recast and −recast episodes). These episodes were then edited to produce clips ending with the student's erroneous utterance. After the interaction, each student was shown both types of clips on a computer screen and was asked to identify and orally correct his or her own errors. The tests were individualized since each student was shown exchanges that contained his or her own utterances. The results showed that students successfully modified a greater proportion of their errors on +recast episodes than on –recast episodes.

Issues with individualized posttests

The advantage of individualized posttests is that they allow for the assessment of incidental feedback. They also enable the researcher to analyze the extent to which each individual learner has benefited from the feedback. One limitation, however, is the lack of pretests (Loewen & Philp, Reference Loewen and Philp2006). This is problematic because, without a pretest, it is hard to know whether the learner has really learned the targeted form. Although the error made by the learner during the interaction can be considered as an indication that the learner did not know the correct form, it is still unclear whether the error is due to a lack of knowledge, partial knowledge, or simply a slip of the tongue during spontaneous interaction (Nassaji, Reference Nassaji2009). Another issue is that the test simply assesses the learner's recall of the targeted form involving receptive skills, and not learners' production. Individualized posttests are also often written whereas the interaction is oral. Thus, if learners do not provide a correct response on the test, it could also be partly due to the mismatch between the modality of the test and the context of the feedback (this can apply to all written tests used to assess oral interaction). One way of addressing this issue would be through using tests in which students both hear the stimuli and then respond orally such as in Hawkes and Nassaji's (Reference Hawkes and Nassaji2016) aural-oral individualized posttests.

Individualized posttests are often created based on errors made in oral spontaneous discourse. Although the occurrence of such errors may be because the learner has not acquired the form, as noted earlier, it may also be due to random mistakes in the production of known forms. Thus, relying on spontaneous oral errors to measure L2 acquisition can be problematic (see Nassaji, Reference Nassaji2009). The fact that the learner makes oral errors could of course be also because the learner has declarative and conscious knowledge about the form but the form has not yet been fully internalized or proceduralized.

To address the pretest limitation of individualized posttests, Nassaji (Reference Nassaji2009) used a pretest-posttest design for individualized tests, which attempted to assess the learners' prior knowledge of the forms that were targeted by the feedback during interaction. The data involved adult ESL learners who participated in task-based dyadic interactions and received various forms of interactional feedback. To provide a measure of the learner's previous knowledge, in addition to the dyadic oral interaction, the research design included three other components: a pre-interaction scenario description component, an immediate post-interaction error identification/correction component, and a delayed error identification/correction component. Initially the pre-interaction scenario descriptions were examined and the learners' erroneous utterances were identified. The erroneous utterances were then tracked and compared to those occurring during oral interaction to identify errors in common. Finally, the post interaction scenarios were examined to determine the extent to which learners were able to correct the errors that occurred prior to interaction and then received feedback during interaction. Table 1 provides an example of such an analysis. In this example, in the pre-interaction scenario description task, the learner produced an utterance with a lexical error: guy for girl. Then during the dyadic interaction, the learner produced the same error and received a recast. When analyzing the scenario description after the interaction, the learner identified and successfully corrected the error by changing the word guy to girl. This correction was then taken as evidence for the effect of feedback during the interaction.

Table 1.

An example of a learner's pre-interaction description, during-interaction description, teacher feedback, and post-interaction modification

(Nassaji, Reference Nassaji2009, p. 431)

Experimental pretest-posttest research

To assess feedback effectiveness more directly, many studies have adopted an experimental research design using various pretest-posttest measures (Doughty & Varela, Reference Doughty, Varela, Doughty and Williams1998; Han, Reference Han2002; Ishida, Reference Ishida2004; Lyster, Reference Lyster2004; McDonough, Reference McDonough2005; Ammar & Spada, Reference Ammar and Spada2006; Ellis, Loewen, & Erlam, Reference Ellis, Loewen and Erlam2006; Ellis, Reference Ellis and Mackey2007; Loewen & Nabei, Reference Loewen, Nabei and Mackey2007; Yang & Lyster, Reference Yang and Lyster2010; Saito & Lyster, Reference Saito and Lyster2012; Nassaji, Reference Nassaji2017). In experimental research, usually a particular type of feedback is selected and used as a treatment and its effects are assessed on learning some preselected target forms. Learners' pretest performance is then compared to their posttest performance and any increase is often taken to be due to the treatment. The findings of experimental pretest-posttest research have been reported in several narrative reviews and meta-analytic studies that combine results across studies (Russell & Spada, Reference Russell, Spada, Norris and Ortega2006; Mackey & Goo, Reference Mackey, Goo and Mackey2007; Li, Reference Li2010; Lyster & Saito, Reference Lyster and Saito2010; Lyster, Saito, & Sato, Reference Lyster, Saito and Sato2012; Goo & Mackey, Reference Goo and Mackey2013; Nassaji, Reference Nassaji2015, Reference Nassaji2016; Brown, Reference Brown2016). The findings of these studies have indicated that feedback in general has a positive effect on L2 learning. Table 2 summarizes the findings of some of these meta-analyses, all of which report a medium to a large effect size for feedback.

Table 2.

Overall effectiveness of feedback in different meta-analyses

Although experimental research has generally shown positive effects for feedback, there are a number of issues/limitations with pretest-posttest experimental research. One is that most of these studies have focused mainly on the changes in the pretest and posttest scores with much less concern with the details of what goes on during interaction. Some experimental studies have analyzed learner repair during interaction (e.g., Yang & Lyster, Reference Yang and Lyster2010). However, these studies have often reported the overall frequency of the repair without showing who, or how many of the students who received the feedback, produced the repair. This can be problematic if all or most instances of the repair reported come from only a few of the participants.

Experimental studies have been mostly dealing with planned feedback or feedback provided intensively and repeatedly on certain predetermined target forms. The study of preplanned feedback has provided important insights into the nature of effective feedback. However, because of their controlled focus, it is hard to know whether the effects shown are because of the feedback, its intensity, or their combination. For example, the positive effects shown for recasts in experimental research could be partly due to the intensity of the feedback on specific target forms (Nicholas, Lightbown, & Spada, Reference Nicholas, Lightbown and Spada2001; Han, Reference Han2002; Lyster et al., Reference Lyster, Saito and Sato2012).

There is also the problem of ecological validity of intensive feedback in experimental research as teachers often provide feedback broadly on a wide range of forms instead of certain preselected forms. For example, in the context of discussing the role of recasts, Ellis and Sheen (Reference Ellis and Sheen2006) argued that although intensive recasts (i.e., recasts that occur repeatedly in response to preplanned forms) may be viewed to be more effective than incidental recasts, such claims are ‘of little practical significance for teachers because they are not in a position to know whether their learners are developmentally ready or whether the targeted feature is one amenable to treatment through recasts’ (p. 597). These researchers believe that recasts that occur incidentally and in response to a wide range of errors might have greater practical significance.

Issues with assessment tasks and testing instruments

A variety of measures have been used in experimental research to determine the effects of feedback or instruction. Norris and Ortega (Reference Norris and Ortega2000) classified them into four types: (a) free constructed responses, (b) constrained-constructed responses, (c) selected responses, and (d) metalinguistic judgments. Free constructed-response measures are those in which learners produce the linguistic feature in tasks that involve free production. Constrained-constructed response measures require learners to produce a target form in controlled tasks such as in fill-in-the-blank tests. Selected-response measures need learners to select the correct response among a number of alternatives such as in multiple-choice tests. In metalinguistic judgments, learners should decide whether an utterance is grammatically correct or not. Norris and Ortega's classification is mainly based on the test format. For the purpose of this article, I will categorize them based on their function or the kind of knowledge measured. In doing so, I distinguish four broad categories: (a) explicit knowledge tests, (b) implicit knowledge tests, (c) functional/pragmatic knowledge tests, and (d) measures of noticing, perception, and L2 processes.

Explicit knowledge tests

In the field of SLA, a distinction has been made between explicit and implicit knowledge. Williams (Reference Williams, Ritchie and Bhatia2009, p. 319) defined explicit knowledge as ‘knowledge that we know that we know . . . and that we are aware of using.’ Ellis (Reference Ellis2004) defined it as declarative knowledge about language and numerated a number of characteristics for it such as being held conscious, learnable, verbalizable, and typically accessed through controlled processing when learners experience some kind of linguistic difficulty in using the L2. Implicit knowledge, on the other hand, involves little conscious awareness. It is also procedural and ‘is available for use in rapid, fluent communication’ (Ellis, Reference Ellis2005b, p. 214).

Although some researchers do not posit an essential role for explicit knowledge in SLA (e.g., Krashen, Reference Krashen1985; Reber, Reference Reber1993), many believe that explicit knowledge contributes importantly to L2 acquisition (e.g., Sharwood Smith, Reference Sharwood Smith1981; Anderson, Reference Anderson1983; Hulstijn, Reference Hulstijn2002; Ellis, Reference Ellis2004, Reference Ellis2005a; DeKeyser, Reference DeKeyser and Hinkel2005). For example, using information processing theories and skills acquisition models of learning, some researchers have argued that language learners learn language initially through processing information available in the input and then accessing it for subsequent comprehension and production (Ellis, Reference Ellis2008). Skills acquisition theories (see Anderson, Reference Anderson1982, Reference Anderson1983) claim that learning is a movement from declarative knowledge (i.e., explicit knowledge of rules and systems) to procedural knowledge (i.e., knowledge of how to use the system). This perspective proposes that learners can learn the new target rules and structures first through the development of conscious knowledge and then practice them to gain control of them (e.g., McLaughlin, Reference McLaughlin1990; DeKeyser, Reference DeKeyser and Hinkel2005). A variety of measures have been used to assess explicit knowledge both in SLA and L2 feedback research. Three most commonly used measures are metalinguistic knowledge tests, untimed grammaticality judgment tests, and error correction tests.

Metalinguistic knowledge tests (MKTs). MKTs assess learners' metalinguistic knowledge. The operationalization of the notion of metalinguistic knowledge has varied across studies. However, this kind of knowledge has often been defined as the learners' ability to describe and explain errors using their knowledge of grammatical rules, with an important component of this knowledge being learners' analytic ability (Bialystok, Reference Bialystok1982, Reference Bialystok, Phillipson, Kellerman, Selinker, Sharwood Smith and Swain1991). Roehr (Reference Roehr2007, p. 174) defined metalinguistic knowledge as ‘explicit knowledge about [grammatical] categories as well as explicit knowledge about relations between categories’. MKTs can take different forms but in general learners are presented with a series of ungrammatical sentences and are asked to explain their incorrectness using metalinguistic terms. For example, in a study of explicit and implicit corrective feedback, Ellis et al. (Reference Ellis, Loewen and Erlam2006) presented learners with five ungrammatical sentences containing the target structure (i.e., English past –ed) and asked learners to correct the errors and explain why the sentence was ungrammatical. Since such tasks require the learner to analyze the sentence and also use grammatical concepts, they are considered to encourage the use of explicit knowledge.

Untimed grammaticality judgment tests (GJTs). GJTs assess learners' ability to judge the grammaticality of sentences. Such tests are often composed of a series of grammatically correct and incorrect sentences and require participants to indicate which is correct and which is incorrect. Learners' responses are often in the form of categorical judgments but learners may also sometimes be given choices such as ‘clearly grammatical’, ‘probably grammatical’, ‘probably ungrammatical’, and ‘clearly ungrammatical’ (Ellis & Barkhuizen, Reference Ellis and Barkhuizen2005).

GJTs can tap into either explicit or implicit knowledge depending on the time condition or the grammaticality of the sentences. For example, untimed GJTs are considered to provide a measure of explicit knowledge (e.g., Ellis, Reference Ellis2005a; Ellis & Barkhuizen, Reference Ellis and Barkhuizen2005; Ellis et al., Reference Ellis, Loewen and Erlam2006; Erlam, Reference Erlam2009) as learners have time to monitor their language using their conscious knowledge. Several feedback studies have used untimed GJTs to examine the effect of feedback on the development of explicit knowledge (e.g., Ellis et al., Reference Ellis, Loewen and Erlam2006; Loewen & Erlam, Reference Loewen and Erlam2006; Loewen & Nabei, Reference Loewen, Nabei and Mackey2007; Kamiya, Reference Kamiya2015; Nassaji, Reference Nassaji2017; Thomas, Reference Thomas2018). For example, in a recent study, Nassaji (Reference Nassaji2017) used written grammaticality judgment tasks to examine the effects of extensive versus intensive recasts. The task consisted of sentences that provided obligatory contexts for the use of the target structure (i.e., English articles). Half of the sentences were grammatical and the other half were ungrammatical. Learners were asked to read each sentence and decide whether it contained an error and also correct the error.

At times, learners may be asked not only to judge and correct the sentence, but also to say what their judgment was based on (e.g., feel, rule, guess). In Ellis et al.'s (Reference Ellis, Loewen and Erlam2006) study, for example, in addition to judging the grammaticality of the sentences, learners were asked to indicate the degree of their certainty and also indicate whether they used a language rule or their intuition to make a judgment. The latter requirements were used to determine to what extent learners were sure of their judgments or relied on their explicit versus implicit knowledge to make a decision.

Error correction tests (ECTs). ECTs measure explicit knowledge by asking learners to correct erroneous sentences (see Muranoi, Reference Muranoi2000; Sheen, Reference Sheen2008). The difference between ECTs and GJTs is that in the latter learners need to judge what sentence is grammatical and what sentence is not but in the former the erroneous part is identified and the learner needs to correct it. The format is often written and the learner is asked to correct the error in writing. Since such tests require learners to correct errors, they may pay attention to the language structure in the tests using their conscious language knowledge. Also, since learners can perform the task under no time pressure, they are assumed to have time to monitor their performance using their explicit analytic knowledge. In a study of the relationship of recasts and learner anxiety, Sheen (Reference Sheen2008), for example, used an ECT consisting of two related statements, one of which contained an error, which had been underlined. The learners were required to correct the underlined erroneous part. The study found no effects of recasts on learners' performance on this test, which was interpreted as being possibly due to the difference between the nature of the treatment (involving oral input) and the nature of the test (involving written analytic knowledge).

Implicit knowledge tests

Although explicit knowledge can be helpful and, as assumed by some theories, can become implicit knowledge through practice, most researchers agree that the kind of knowledge underlying spontaneous and fluent language use is implicit knowledge because accessing explicit declarative knowledge is a slow process whereas accessing implicit knowledge is fast. As noted earlier, explicit knowledge is declarative and involves conscious awareness whereas implicit knowledge involves little conscious awareness, is proceduralized, and can be accessed fluently and spontaneously. It has been argued that the ultimate goal of feedback and instruction should be the development of implicit knowledge, and therefore, research in this area should also include measures that tap into implicit knowledge (Ellis, Reference Ellis2004). To assess implicit knowledge, a number of measures have been used in SLA and feedback research (e.g., Ellis et al., Reference Ellis, Loewen and Erlam2006, Reference Ellis, Loewen, Elder, Erlam, Philp and Reinders2009; Loewen & Erlam, Reference Loewen and Erlam2006; Loewen & Nabei, Reference Loewen, Nabei and Mackey2007; Révész, Reference Révész2012; Kamiya, Reference Kamiya2015; Nassaji, Reference Nassaji2017). Four most widely used ones are timed GJTs; oral elicited imitation tests (OEITs), speeded dictation (SD), and free oral production tests (FOPTs). In what follows, I will discuss these measures and the issues associated with them.

Timed GJTs. As noted earlier, the GJT is composed of a series of grammatically correct and incorrect sentences and the learners are asked to judge which of them is grammatical and which is ungrammatical. These tests are suggested to measure either explicit or implicit knowledge depending on the time condition or the grammaticality of the sentences (i.e., whether the learners' responses are to grammatical or ungrammatical sentences). As discussed previously, untimed GJTs are assumed to assess explicit knowledge. Timed GJTs, however, are assumed to measure implicit knowledge because the time pressure in such tests gives the learner little opportunity to rely on explicit knowledge (e.g., Ellis, Reference Ellis2005a; Ellis & Barkhuizen, Reference Ellis and Barkhuizen2005; Ellis et al., Reference Ellis, Loewen and Erlam2006; Erlam, Reference Erlam2009). Responses to grammatical sentences of GJTs are considered to tap into implicit knowledge whereas responses to ungrammatical sentences tap into explicit knowledge (Ellis, Reference Ellis2004, Reference Ellis2005a; Ellis et al., Reference Ellis, Loewen and Erlam2006). Ellis (Reference Ellis2005a) showed some support for the latter by finding that while learners' scores on ungrammatical sentences were more correlated with tests of explicit knowledge (e.g., metalinguistic knowledge tests), responses on grammatical sentences were more associated with tests assumed to measure implicit knowledge, such as free oral production tests and timed GJTs (see also Gutiérrez, Reference Gutiérrez2013). GJTs can be written or aural. Aural GJTs are more cognitively demanding than written tests as the learners need to process the language under real operating conditions. Therefore, they might provide a better measurement of implicit knowledge (Kim & Nam, Reference Kim and Nam2017).

OEIT. The OEIT is a particular elicitation technique used initially in first language (L1) acquisition research to assess L1 development (see Slobin & Welsh, Reference Slobin, Welsh, Ferguson and Slobin1973, for a review). It has also received considerable attention in L2 research as a measure of general L2 proficiency (e.g., Hamayan, Saegert, & Larudee, Reference Hamayan, Saegert and Larudee1977; Wu & Ortega, Reference Wu and Ortega2013; Tracy-Ventura, McManus, Norris, & Ortega, Reference Tracy-Ventura, McManus, Norris, Ortega, Leclercq, Edmonds and Hilton2014) and also implicit knowledge (Ellis, Reference Ellis2005a; Ellis et al., Reference Ellis, Loewen and Erlam2006; Erlam, Reference Erlam2009; Erlam & Loewen, Reference Erlam and Loewen2010; Spada, Shiu, & Tomita, Reference Spada, Shiu and Tomita2015). The technique consists of asking learners to listen to an utterance that usually includes a target structure and then repeat it as exactly and correctly as possible. The assumption is that learners are able to repeat an utterance correctly if they have internalized the grammar and the lexicon needed to comprehend, parse, and reproduce the utterance (Rebuschat & Mackey, Reference Rebuschat, Mackey and Chappelle2013). Such tasks are assumed to be constructive in that learners need to process the input and then reconstruct it using their own grammatical knowledge (Erlam, Reference Erlam2009). The OEITs that ask learners to repeat correctly ungrammatical sentences are suggested to be more constructive because in order to correctly and spontaneously reproduce ungrammatical sentences, learners must have already internalized the target structure as part of their linguistic system (Munnich et al., Reference Munnich, Flynn, Martohardjono, Tarone, Gass and Cohen1994). A number of studies have used OEITs to investigate the role of feedback in the development of implicit knowledge. Ellis et al. (Reference Ellis, Loewen and Erlam2006), for example, used an OEIT to investigate the effects of recasts and metalinguistic feedback on the acquisition of the implicit knowledge of English past tense –ed. Results showed an advantage for metalinguistic feedback.

Speeded dictation tests (SDTs). SDTs present learners with a series of sentences containing the target structure and ask learners to write down quickly the sentences read to them by the teacher or the researcher. Because of the time pressure and their reconstructing nature, these tasks are also assumed to minimize learners' ability to use their explicit knowledge and therefore are considered to tap into implicit knowledge. Sheen (Reference Sheen2008), for example, used an SDT to examine the effects of recasts among low- and high-anxiety learners. The test consisted of 15 sentences containing the target structure (English definite and indefinite articles). Learners were asked to listen to each sentence and write it down as quickly as they could and as exactly as they heard it. The study found that the low-anxiety learners outperformed the high-anxiety learners on this test, which was taken to indicate that recasts had an effect on the development of implicit knowledge among these learners.

Free production tasks (FPTs). Other tests used to measure implicit knowledge are FPTs. These include various kinds of unplanned oral or written tasks that provide opportunities for free and meaningful language production, such as story-telling tasks (e.g., Sheen, Reference Sheen2008; Nassaji, Reference Nassaji2017), picture description tasks (e.g., Nassaji, Reference Nassaji2017), narrative tasks (see Yang & Lyster, Reference Yang and Lyster2010, for an example of a written task and Ellis, Reference Ellis2005a and Nassaji, Reference Nassaji2009, for examples of oral tasks), and story-retelling tasks, in which student read and rewrite a story (e.g., Sheen, Reference Sheen2008) or retell a story orally (e.g., Yang & Lyster, Reference Yang and Lyster2010). Various forms of information gap tasks such as spot-the-difference tasks, picture-sequencing tasks, and picture-drawing tasks have all been used to elicit and also create a context for interaction and meaning-focused language production.

Since FPTs require learners to produce the language without any preplanning while their focus is on meaning, they are considered to tap into implicit knowledge (Ellis, Reference Ellis2005a; Ellis et al., Reference Ellis, Loewen and Erlam2006; Révész & Han, Reference Révész and Han2006; Révész, Reference Révész2012; Nassaji, Reference Nassaji2017). The assumption is that no planning time will prevent the activation of explicit knowledge and will trigger implicit knowledge.

Functional/pragmatic knowledge tests

All the tests discussed so far focus on learners' accuracy of grammatical forms. Even when communicative tasks are used, their scoring focuses almost exclusively on linguistic accuracy. Functional/pragmatic tests emphasize not only learners' ability to produce grammatically accurate sentences but also utterances that are contextually or pragmatically appropriate. These tests are usually context-specific and their scoring considers both accuracy and contextual appropriacy. They have often been used in pragmatic research to assess learners' pragmatic ability. Examples of such tests include role plays designed to examine how learners understand and produce language appropriately in social interaction. Examples of more controlled forms of such tasks are dialogue and discourse completion tasks (DCTs) used to examine learners' ability to use a particular speech act, such as requests, refusals, and apologies, in simulated communicative contexts (Hinkel, Reference Hinkel1997; Rose & Kasper, Reference Rose and Kasper2001; Kasper & Blum-Kulka, Reference Kasper and Blum-Kulka2003; Koike & Pearson, Reference Koike and Pearson2005; Martínez-Flor & Usó-Juan, Reference Martínez-Flor and Usó-Juan2010; Bardovi-Harlig & Vellenga, Reference Bardovi-Harlig and Vellenga2012). In DCTs, learners are provided with a description of a situation and are asked to describe what they would say in that situation. In a dialogue completion task, they are presented with an incomplete dialogue and are asked to complete it. These tasks can also be both written and oral. Their oral form may tap more into implicit knowledge whereas their written form may also encourage explicit knowledge. DCTs may also differ in the degree of control, ranging from those that are close-ended such as multiple-choice DCTs to those that require learners to say whatever they would say in a specific situation.

Several feedback studies have used various forms of pragmatic knowledge tests including DCTs or dialogue completion tasks to assess the effect of feedback on learners' socio-pragmatic skills (Fukuya & Zhang, Reference Fukuya and Zhang2002, Reference Fukuya and Zhang2006; Koike & Pearson, Reference Koike and Pearson2005; Takimoto, Reference Takimoto2006; Ajabshir, Reference Ajabshir2014; Nguyen, Do, Pham, & Nguyen, Reference Nguyen, Do, Pham and Nguyen2017). Fukuya and Zhang (Reference Fukuya and Zhang2002), for example, investigated the effects of implicit recasts on Chinese learners' learning of English requests. Using DCTs, they found a significant effect of recasts on learning this pragmatic feature.

Although functional/pragmatic tests such as DCTs might be an effective way of gathering socially appropriate responses from the learner, one limitation is that they do not elicit naturally occurring speech (Beebe & Cummings, Reference Beebe, Cummings, Gass and Neu1996). Another is the difficulty of designing such tests in ways that can elicit only the speech act targeted.

Assessing noticing, perception, and L2 processes

While the role of learner attention or the extent to which learners need to be consciously aware of certain linguistic forms in order to learn them is a contentious issue in the field of SLA, many researchers believe that learning cannot occur without some degree of attention to form or noticing (Schmidt, Reference Schmidt and Robinson2001; Doughty, Reference Doughty, Doughty and Long2003; Robinson, Reference Robinson, Doughty and Long2003; Leow, Reference Leow2015; Ellis, Reference Ellis2016b). SLA research also suggests that the effectiveness of corrective feedback depends largely on the degree to which learners successfully notice and process the feedback (Ellis & Sheen, Reference Ellis and Sheen2006; Ellis, Reference Ellis2016a; Nassaji, Reference Nassaji2016).

Mental activities including noticing or L2 processing are not subject to direct observation. It is impossible to directly observe what goes on in a learner's mind when performing a task. However, there are indirect ways of getting some evidence. To this end, research has used a number of tools, including, for example, various forms of retrospective and introspective methods. In the following section, I will discuss five of such methods which have been used in feedback research to find out what learners notice or how they process feedback during interaction. These include immediate recall, stimulated recall, think-aloud protocols, eye-tracking methods, and other retrospective methods such as questionnaires and interviews.

Immediate recall. Immediate recall is a retrospective technique in which learners are asked immediately after the feedback to recall what they noticed. The assumption is that such information is accessible for further recall. One of the studies that used immediate recall is Philp (Reference Philp2003) to examine learners' noticing of recasts in dyadic task-based interaction. Noticing was operationalized as remembering recasts following the feedback and was measured by cued immediate recall, where, after each recast, the learner's response was interrupted by the sound of two knocks to cue the learner to recall the feedback. The results showed that learners were able to notice a substantial number of the recasts (60–70%). The amount of noticing was also found to be affected by learners' level of language proficiency as well as the length and the number of changes in the recast. Learners with a higher level of L2 proficiency were more likely to notice recasts than lower-level learners, and shorter recasts were better noticed than longer recasts.

Stimulated recall. Another research tool used to collect noticing or processing data in feedback research is stimulated recall (e.g., Mackey, Gass, & McDonough, Reference Mackey, Gass and McDonough2000; Mackey, Reference Mackey2006; Egi, Reference Egi2007, Reference Egi2010; Bao, Egi, & Han, Reference Bao, Egi and Han2011). Stimulated recall is a retrospective method in which learners are required to recall their thought processes after a task is completed. It is called stimulated recall because learners are presented, after the interaction, with a clue or stimulus to recall their thoughts at the time they were doing the task (see Gass & Mackey, Reference Gass and Mackey2017). In corrective feedback research, learners are usually shown a video clip of their feedback exchange and are asked to pause the clip when they hear the feedback move and recall what they were thinking at the time they received the feedback. One of the feedback studies using such a method is Mackey et al. (Reference Mackey, Gass and McDonough2000), which examined learners' noticing of the feedback in task-based dyadic interaction. The researchers found that the learners perceived the target of the feedback differently depending on error types and feedback types (see also Carpenter, Jeon, MacGregor, & Mackey, Reference Carpenter, Jeon, MacGregor and Mackey2006; Egi, Reference Egi2007, Reference Egi2010).

Think-aloud. The think-aloud is a concurrent strategy used to capture learners' thoughts when completing a task (Ericsson & Simon, Reference Ericsson and Simon1993). When doing a think-aloud, learners are often asked to verbalize their thought processes while completing the task. The think-aloud reports are then recorded, transcribed, and coded. Think-alouds might be difficult to use at the time of oral feedback. However, they can be used to assess the effect of feedback during a subsequent task. Nipaspong and Chinokul (Reference Nipaspong and Chinokul2010), for example, employed think-aloud protocols during a succeeding discussion task to examine the effect of explicit and implicit feedback on learners' noticing of a pragmatic feature (English refusals). Their results showed an advantage for implicit over explicit feedback.

Eye-tracking methods. Eye-tracking has recently attracted increasing attention among L2 researchers as a tool to assess how L2 learners notice and process L2 input. This is an online technique using computer-based technology to gather real-time data about how and where learners allocate their attention by recording their eye movements and the length of their fixations using video-based eye trackers (see Winke, Godfroid, & Gass, Reference Winke, Godfroid and Gass2013; Conklin & Pellicer-Sánchez, Reference Conklin and Pellicer-Sánchez2016, for a discussion). Since the technique does not interrupt the task performance, it does not have the limitations of introspective tools that require learners to do two tasks at the same time. Eye-tracking is used when learners are processing the input naturally and hence ‘provides a very rich moment-to-moment data source’ (Conklin & Pellicer-Sánchez, Reference Conklin and Pellicer-Sánchez2016, p. 453). This method has been used in a number of L2 studies to investigate text processing, including reading (Kang, Reference Kang2014), vocabulary learning (Godfroid, Boers, & Housen, Reference Godfroid, Boers and Housen2013; Godfroid et al., Reference Godfroid, Ahn, Choi, Ballard, Cui, Johnston and Yoon2018), syntactic processing (Frenck-Mestre, Reference Frenck-Mestre2005), and also written feedback (Shintani & Ellis, Reference Shintani and Ellis2013). Relatively fewer studies have employed this technique to examine oral feedback, but some are emerging. For example, in a recent study of recasts, McDonough, Crowther, Kielstra, and Trofimovich (Reference McDonough, Crowther, Kielstra and Trofimovich2015) examined the relationship of learners' eye-gaze with their responses to recasts. The eye gaze lengths of both the feedback provider and the learner were tracked. Their results showed a significant relationship between the learner's and the interlocutor's mutual eye gaze and the learner's target-like responses to recasts.

Retrospective questionnaires and interviews. Retrospective questionnaires and interviews are other means that can gather information about learners' perception of the feedback, though very indirectly. In a classroom study of interactional feedback, Fu and Nassaji (Reference Fu and Nassaji2016), for example, used a retrospective questionnaire in which students were asked to indicate how often they received such feedback. The findings showed that the perception of the feedback frequency reported was very close to the actual frequency of the feedback. Tools such as reflective journals and post-interaction uptake sheets have also been used to assess learners' perception of the feedback (see Basturkmen, Loewen, & Ellis, Reference Basturkmen, Loewen and Ellis2004; Sheen, Reference Sheen2008; Loewen et al., Reference Loewen, Li, Fei, Thompson, Nakatsukasa, Ahn and Chen2009; Junqueira & Kim, Reference Junqueira and Kim2013, for examples).

Other issues and challenges

In addition to the issues discussed so far, there are a number of other issues and challenges when it comes to assessing the effectiveness of feedback. I will discuss these under the following two categories: acquisition definition challenges and measurement challenges.

Acquisition definition challenges. One of the challenges in L2 acquisition research including feedback research is how to define acquisition. This issue is important because if there is no consensus on how to conceptualize this construct, there will be no agreement on how to measure it. Ellis (Reference Ellis2006b) provided three definitions for acquisition: (a) internalization of a new linguistic form, (b) increased control over a known form, and (c) progress along an acquisitional path. As he pointed out, most studies have examined acquisition in terms of increased accuracy over a language form. However, there are also studies that have defined acquisition in terms of sequence of stages (e.g., Mackey & Philp, Reference Mackey and Philp1998; Mackey, Reference Mackey1999; Mackey & Oliver, Reference Mackey and Oliver2002; McDonough, Reference McDonough2005). Although these definitions suggest that acquisition is complex and multifaceted, it is clear that different conceptualizations of the same construct could lead to different research results, which makes comparison across studies difficult.

There are also different theoretical perspectives in SLA, which provide different views about what acquisition is and how it should be measured. Interactionists, for example, define acquisition in terms of learners' ability in using language in social interaction. Good outcome measures are then those that assess learners' skill to use the language in communicative interaction. Cognitivists distinguish between two types of knowledge: knowledge of what, or declarative knowledge and knowledge of how, or procedural knowledge. They then define acquisition as a movement from controlled to free production or declarative to procedural knowledge. From this perspective, good tests are those that measure both types of knowledge and in particular learners' progress from controlled to automatized language use. Generativists consider language knowledge as idealized decontextualized knowledge residing in the learner's mind. They make a distinction between competence and performance and argue that performance is affected by extra-linguistic factors. Therefore, tests should measure competence not performance.

While these different views broaden our perspective of what acquisition is, they also make it difficult to agree upon approaches to research and data collection, and for the same reason make it hard to synthesize the results of different studies and arrive at a conclusive evaluation.

Measurement challenges. Another main challenge is that of measurement. As discussed before, there are theoretical distinctions among different types of knowledge and attempts to measure them. However, it is very difficult to know what kind of knowledge learners use to perform a task. It is also difficult to control the kind of knowledge learners use when doing a test. In addition, learners' performance on any test would vary depending on a number of variables, including the complexity of the test, learners' language proficiency, and the various strategies learners may use in doing the test. Some authors have also pointed out (Tarone, Reference Tarone1983, Reference Tarone1985) that language competence is not uniform but highly variable. For the same reason, depending on the social context in which learners operate and its complex interaction with other learner- and context-related variables, learners may perform differently on the same test or respond differently to the same test item. Ellis (Reference Ellis, Burmeister and Rounds1991), for example, found that learners' performance varied significantly when they were asked to do the same grammaticality judgment task a second time.

Conclusions and recommendations

In this paper, I have discussed the different assessment measures frequently used to evaluate the effectiveness of corrective and interactional feedback. Based on this, I would like to make some final remarks and offer some recommendations.

First, most measures, particularly those used in experimental research, have focused primarily on the product rather than the process of learning. That is, they have focused on what is learned rather than how it is learned. Testing the process of learning is also crucial as it helps us understand why L2 learners who are similar in many other respects learn differently from the same feedback type. It can also explain some of the variability seen in research results. As noted earlier, various retrospective and introspective verbal reports such as concurrent think-aloud, stimulated recall, and eye-tracking techniques have been used to capture learners' mental processes during a task. Although these measures have been used in feedback research, few studies have used a combination of both product- and process-oriented measures. Such a combination can provide more comprehensive information regarding the effect of feedback by showing not only what, but also how, learners have learned from the feedback.

As discussed earlier, acquisition is a complex and multidimensional construct. Thus, it is important to use multiple assessment measures. This could be not only tests of different types of knowledge (e.g., explicit and implicit) but also different skills (receptive and productive) as well as tests of accuracy and appropriacy. It is important to use multiple measures as they allow us not only to capture different levels of acquisition but also cross-validate findings across studies and measurements. Using multiple measures is also important because learners have been shown to display variability in their performance depending of the type of outcome measures used. Ellis (Reference Ellis2006a), for example, found that learners exhibited different degrees of difficulty with the use of regular past tense (–ed) depending on the kind of tests used. For example, they showed a high score for this target form on explicit tests but a relatively low score for it on implicit tests. For adverb placement, however, they showed a moderately high implicit score but a low explicit score.

Although it might be difficult to use many measures in one study, we cannot conclude that learners have acquired a linguistic form successfully unless they are able to use it across a range of tasks and situations. Therefore, we need to be cautious when interpreting our results. In a study on the role of task-based instruction, Shintani (Reference Shintani2012, p. 269), for example, concluded that her results showed ‘that the listen-and-do tasks led to the acquisition of both nouns and adjectives’. Such claims about acquisition are not warranted in the absence of tests that measure learners' ability in a variety of contexts. As Norris and Ortega (Reference Norris, Ortega, Doughty and Long2003) highlighted, in assessing SLA, ‘a multiplicity of behavioral observations’ can assist ‘and triangulate interpretations' (p. 731).

As reviewed earlier, feedback studies have started to use multiple measures including tests of explicit and implicit knowledge (e.g., Ellis et al., Reference Ellis, Loewen and Erlam2006; Erlam & Loewen, Reference Erlam and Loewen2010; Révész, Reference Révész2012; Nassaji, Reference Nassaji2017). Such research should continue. Other worthwhile attempts would also be including tests of not only accuracy but also fluency and complexity in feedback research. It may be considered that with a focus on accuracy, corrective feedback may have the potential to disrupt the flow of communication and thus may negatively affect learner fluency during meaning-focused interaction. Little research, however, has assessed the impact of feedback on L2 fluency. There may also be a trade-off between fluency and accuracy, and also accuracy and complexity. Based on the notion that humans have limited cognitive resources and that these resources compete between attention to form and attention to meaning (e.g., Skehan, Reference Skehan2009; VanPatten, Reference VanPatten2015), we can assume that the more the learner pays attention to producing a language form accurately, the less attention he or she may have to process the form fluently. It is also suggested that as learners become more fluent, an improvement may be noticed in the complexity of their production (Larsen-Freeman & Strom, Reference Larsen-Freeman and Strom1977). Given these assumptions, it is useful to examine the extent to which corrective feedback with emphasis on accuracy would affect learner fluency and complexity.

As for the acquisitional effect of feedback, it is quite clear the question cannot be satisfactorily answered without valid and reliable measures. As we saw earlier, there are currently several validation studies to examine the construct validity of various explicit and implicit knowledge tests (Ellis, Reference Ellis2005a; Ellis & Loewen, Reference Ellis and Loewen2007; Godfroid et al., Reference Godfroid, Loewen, Jung, Park, Gass and Ellis2015; Spada et al., Reference Spada, Shiu and Tomita2015; Kim & Nam, Reference Kim and Nam2017). There is clearly a need for more studies in this area. Also as Prutting, Gallagher, and Mulac (Reference Prutting, Gallagher and Mulac1975) pointed out, a true test of acquisition is a test that is representative of learners' ability in real-life situations. Thus, there is a need to systematically examine and show how well the tests claiming to measure acquisition meet this criterion. There is also a need for studies to examine the different factors that may affect what a test measures. In a recent study, for example, Kim and Nam (Reference Kim and Nam2017) offered evidence that OEITs and GJTs provided stronger measures of implicit knowledge when the OEIT was performed under time pressure and when the timed GJT used aural modality.

Finally, the various challenges associated with reliably assessing SLA underscore the significance of replication studies, including those on corrective feedback. The need for replication has recently been emphasized in several publications in the field of applied linguistics (see Porte, Reference Porte2012, for a detailed discussion). There are many reasons why replication is needed, including confirming the outcomes of previous research, adding new perspectives on the results of the original studies, or examining the generalizability of findings in new contexts. Another reason could be the inadequacy of assessment measures. In this respect, replication informs us of the relevance of the outcome measures used and also increases our understanding of how accurately they assess the intended outcomes across different settings and learners.