Use of Verbal Reports in L1 and L2 Research

Verbal reports have a long history in language research. Since the 1980s, verbal reports have been used systematically in L1 reading (Alvermann, Reference Alvermann1984; Cohen, Reference Cohen1986, Reference Cohen1987; Earthman, Reference Earthman1992; Fehrenbach, Reference Fehrenbach1991; Folger, Reference Folger2001; Gordon, Reference Gordon1990; Harmon, Reference Harmon2000; McGuire & Yewchuk, Reference McGuire and Yewchuk1996; Rosenshine & Meister, Reference Rosenshine and Meister1992), writing (Breetvelt, Reference Breetvelt1994; Durst, Reference Durst1987; Green & Sutton, Reference Green and Sutton2003; Zellermayer & Cohen, Reference Zellermayer and Cohen1996), and language testing research (Anderson, Bachman, Perkins, & Cohen, Reference Anderson, Bachman, Perkins and Cohen1991; Gavin, Reference Gavin1989; Wijgh, Reference Wijgh, Cumming and Berwick1996).Footnote ¹

Similarly, introspective methods, including verbal reports, have been used extensively as a data elicitation technique in SLA. Since SLA was first studied systematically in the early 1970s, there has been some debate over whether learners' introspections, or their production alone, should be used for theory building (Corder, Reference Corder and Svartik1973; Selinker, Reference Selinker1972). Although the debate over the use of verbal reports continues today, researchers in virtually all strands of SLA are using both concurrent and retrospective verbal reports increasingly, regardless of their theoretical positions, to investigate such phenomena as L2 reading and writing (Cavalcanti & Cohen, Reference Cavalcanti and Cohen1990; Cohen, Reference Cohen1987; Cohen & Cavalcanti, Reference Cohen and Cavalcanti1987; Hosenfeld, Reference Hosenfeld1976, Reference Hosenfeld1977, Reference Hosenfeld and Borne1979, Reference Hosenfeld, Alderson and Urquhart1984), L1 and L2 strategies (Chamot & El Dinary, Reference Chamot and El Dinary1999; Davis & Bistodeau, Reference Davis and Bistodeau1993; Nevo, Reference Nevo1989; Yamashita, Reference Yamashita2002), L2 test-taking strategies (Cohen, Reference Cohen, Ekbatani and Pierson2000; Norris, Reference Norris1992; Warren, Reference Warren, Burston, Monville-Burston and Warren1996), translation (Enkvist, Reference Enkvist1995; Færch & Kasper, Reference Færch, Kasper, House and Blum-Kulka1986; Jaaskelainen, Reference Jaaskelainen, Condit and Jaaskelainen2000; Kern, Reference Kern1994; Ronowicz, Hehir, Kaimi, Kojima, & Lee, Reference Ronowicz, Hehir, Kaimi, Kojima and Lee2005), interlanguage pragmatics (Cohen, Reference Cohen1998a, Reference Cohen1998b; Cohen & Hosenfeld, Reference Cohen and Hosenfeld1981; Cohen & Olshtain, Reference Cohen and Olshtain1993; Félix-Brasdefer, Reference Félix-Brasdefer2004; Kasper, Reference Kasper1999; Kasper & Blum-Kulka, Reference Kasper and Blum-Kulka1993; Kasper & Rose, Reference Kasper and Rose2002), conversational interaction (Egi, Reference Egi2004; Mackey et al., Reference Mackey, Gass and McDonough2000; Nabei & Swain, Reference Nabei and Swain2002; Philp, Reference Philp2003; Polio, Gass, & Chapin, Reference Polio, Gass and Chapin2006), noticing during interaction with L2 input (Bowles, Reference Bowles, Kempchinsky and Piñeros2003; Leow, Reference Leow1997, Reference Leow1998a, Reference Leow1998b, Reference Leow and Gutiérrez-Rexach1999, Reference Leow2000, Reference Leow2001a, Reference Leow2001b; Rosa & Leow, Reference Rosa and Leow2004a, Reference Rosa and Leow2004b; Rosa & O'Neill, Reference Rosa and O'Neill1999), and the relationship between explicit and implicit L2 knowledge (Ellis, Reference Ellis2004; Hu, Reference Hu2002). In his recent article on the definition and measurement of explicit knowledge, Ellis stated that “collecting verbal explanations … would appear, on the face of it, to provide the most valid measure of a learner's explicit knowledge” (p. 263). SLA researchers use verbal reports in an effort to gain information about learners' cognitive processes as they interact with the L2. As Gass and Mackey (Reference Gass and Mackey2000) stated, “understanding the source of second language production is problematic because often there are multiple explanations for production phenomena that can only be assessed by exploring the process phenomena” (p. 26). Verbal reports are one means of accessing such process phenomena.

Categorization of Verbal Reports

All verbal reports are not identical; rather, there are important differences among verbal reports based on the conditions under which they are collected. Ericsson and Simon (Reference Ericsson and Simon1984, Reference Ericsson and Simon1993) proposed a typology to categorize verbal reports based on both the temporal frame in which they are collected and the level of detail of reporting. Concurrent reports are those collected as subjects verbalize while performing the task in question, whereas retrospective reports are collected when subjects verbalize some time after performing the task. In addition to categorizing verbal reports in terms of temporal space, Ericsson and Simon (Reference Ericsson and Simon1984, Reference Ericsson and Simon1993) also distinguished between reports that require subjects to verbalize their thoughts per se and those that require subjects to verbalize additional information, such as explanations and justifications. Following from recent research by Bowles and Leow (Reference Bowles and Leow2005), verbalizations of thoughts per se will be referred to as nonmetalinguistic and verbalizations of explanations and justifications will be referred to as metalinguistic.

Validity of Verbal Reports

Despite the frequency with which verbal reports have been used as a methodological tool to gain insight into L2 learners' cognitive processes, systematic research on their validity in SLA is just beginning. Such research is clearly warranted, as Ericsson and Simon (Reference Ericsson and Simon1993), among others, have identified potential threats to the validity of both retrospective and concurrent verbal reports. For retrospective reports, because participants verbalize some time after completing a task, there is a potential for veridicality, or memory decay. In other words, retrospective verbal reports might not accurately reflect thought processes because participants simply might not recall what they were thinking as they completed the given task. However, this threat can be minimized if there is only a short delay between task performance and verbalization. Similarly, if learners are provided with some stimulus, such as a video or audiotape of their performance, as described in Gass and Mackey's (Reference Gass and Mackey2000) account of stimulated recalls, the possibility of veridicality is also lessened. For concurrent verbal reports, the threat to validity is not veridicality because verbalization and task performance are concomitant. Rather, the validity of concurrent reports has been questioned because it is not known whether the act of verbalizing while completing a task is reactive, acting as an additional task and altering cognitive processes rather than providing a true reflection of thoughts, as Ellis (Reference Ellis2001) and Jourdenais (Reference Jourdenais and Robinson2001) have suggested in the SLA literature. The current study investigates only concurrent verbal reports and therefore only the issue of reactivity, operationalized as an observed alteration of learners' cognitive processes due to verbalization during task completion.

In addition to categorizing verbal reports, Ericsson and Simon's (Reference Ericsson and Simon1993) model made predictions about their validity. Specifically, it predicted that nonmetalinguistic verbalizations (which Ericsson and Simon refer to as Type 1 verbalizations) would be largely nonreactive; that is, they would reflect the nature of cognitive processes fairly accurately, merely slowing processing slightly. Furthermore, the model predicted that metalinguistic verbalization (Type 2 and 3 verbalizations in Ericsson and Simon's model) might be more reactive.

Previous Non-SLA Studies

Whereas investigations into the validity of verbal reports in SLA are just beginning, reactivity has been studied in the psychological literature on problem solving for more than 50 years. The earliest studies in this vein (Ewert & Lambert, Reference Ewert and Lambert1932; Katona, Reference Katona1940) investigated the effects of experimenter-provided verbalizations on participants' task performance. In these studies, the experimenter would identify a series of principles necessary for successful completion of a problem-solving task and would then verbally state these essential principles to one group of participants, hypothesizing that the performance of the participants who received this verbalization would improve, whereas the performance of the participants who did not receive such verbal guidance would remain unchanged. By the 1950s, however, an ideological shift had begun to take place in psychology, whereby the focus of experimentation was redirected, moving away from the study of external modifications made to the input and toward participants' own internally driven cognitive processes. It was at this point that researchers started to ask participants to verbalize their thoughts as they completed tasks and to investigate the reactivity of participants' verbalizations on task performance (Brunk, Collister, Swift, & Stayton, Reference Brunk, Collister, Swift and Stayton1958; Gagné & Smith, Reference Gagné and Smith1962; Hafner, Reference Hafner1957; Marks, Reference Marks1951).

Over the years, as the use of verbal reports has increased, numerous studies—primarily in cognitive psychology—have investigated their potential reactivity. A synthesis of the studies that included verbalization as an independent variable indicates that nonmetalinguistic verbalization did not seem to affect task performance (i.e., a finding of nonreactivity; Ericsson, Reference Ericsson2002; Ericsson & Simon, Reference Ericsson and Simon1993), which suggests that nonmetalinguistic verbalization might be a valid method of capturing internal thought processes.Footnote ² However, results indicated that, overall, verbalization tends to increase the overall solution time. Additional studies that investigated the reactivity of nonmetalinguistic reports but were not described in Ericsson and Simon (Biggs, Rosman, & Serfenian, Reference Biggs, Rosman and Serfenian1993; Brinkman, Reference Brinkman1993; Knoblich & Rhenius, Reference Knoblich and Rhenius1995; Williams & Davids, Reference Williams and Davids1997) found a similar lack of reactivity.

However, when participants are required to provide explanations or justifications (metalinguistic verbalizations), the opposite pattern of findings emerges. Most studies that examined the effects of metalinguistic verbal reports compared to a silent control group (Berry, Reference Berry1983; Berry & Broadbent, Reference Berry and Broadbent1984; Bower & King, Reference Bower and King1967; Davis, Carey, Foxman, & Tarr, Reference Davis, Carey, Foxman and Tarr1968; McGeorge & Burton, Reference McGeorge and Burton1989; Short et al., Reference Short, Schatschneider, Cuddy, Evans, Dellick and Basili1991; Stanley, Mathews, Buss, & Kotler-Cope, Reference Stanley, Mathews, Buss and Kotler-Cope1989; Wilder & Harvey, Reference Wilder and Harvey1971; Wilson & Schooler, Reference Wilson and Schooler1991) found metalinguistic reports to be reactive; that is, provision of metalinguistic verbal reports seems to have an effect on task performance when compared to silent controls.Footnote ³ In general, time on task (latency) was not reported, but Ericsson and Simon's (Reference Ericsson and Simon1993) prediction that metalinguistic verbalization would require extra processing time in comparison with other types of verbalizations or with a control holds true for almost all of the studies that reported latency (Ahlum-Heath & di Vesta, Reference Ahlum-Heath and di Vesta1986; Allwood, Reference Allwood1990; Carpenter, Just, & Schell, Reference Carpenter, Just and Schell1990; Fidler, Reference Fidler, Humphreys, Svenson and Vari1983; Gagné & Smith, Reference Gagné and Smith1962; Robinson, Reference Robinson2001). These findings were also largely confirmed by a review of additional empirical studies examining the reactive effects of metalinguistic reports (Berardi-Coletta, Buyer, Dominowski, & Rellinger, Reference Berardi-Coletta, Buyer, Dominowski and Rellinger1995; Robinson; Schooler, Ohlsson, & Brooks, Reference Schooler, Ohlsson and Brooks1993).

Nevertheless, caution should be taken in interpreting these results, because most of the studies reviewed employed nonverbal tasks, which limits their relevance to SLA. In fact, of all the studies on reactivity, only six used verbal tasks (Lass, Klettke, Lüer, & Ruhlender, Reference Lass, Klettke, Lüer, Ruhlender, Schmidt and Zambarbieri1991; Mathews et al., Reference Mathews, Buss, Stanley, Blanchard-Fields, Cho and Druhan1989; Rhenius & Deffner, Reference Rhenius and Deffner1990; Russo, Johnson, & Stephens, Reference Russo, Johnson and Stephens1989; Short et al., Reference Short, Schatschneider, Cuddy, Evans, Dellick and Basili1991; Stratman & Hamp-Lyons, Reference Stratman, Hamp-Lyons and Smagorinsky1994). Furthermore, some research (e.g., Deffner, Reference Deffner1989) has provided evidence that “the causes of reactivity are not general but due jointly to the demands of the task and to verbalization” (Russo et al., p. 763).

Table 1 summarizes the five studies that expressly set out to investigate the effects of task type on reactivity. Three of the five (Russo et al., Reference Russo, Johnson and Stephens1989; Short et al., Reference Short, Schatschneider, Cuddy, Evans, Dellick and Basili1991; Stratman & Hamp-Lyons, Reference Stratman, Hamp-Lyons and Smagorinsky1994) found differential effects based on the type of task, and, perhaps most relevant to the present study, Stratman and Hamp-Lyons found differential effects for thinking aloud on aspects of a verbal task (L1 writing). Although the sample size was too small to permit statistical analyses, the authors indicated that thinking aloud appeared to hinder participants' ability to detect and fix organizational errors in the text but at the same time to improve their ability to detect pronoun errors. Based on these findings, it is clear that the effects of verbalization do not apply universally to all types of tasks.

Table 1.

Reactivity studies examining the effects of task type

* Indicates that time on task was not reported.

^a Differences between solution times for think-aloud and silent groups were significant for all tasks except sentence assembly, for which they approached significance.

^b Thinking aloud appeared to hinder participants' ability to detect and fix organizational errors in the text and, conversely, improve their ability to detect pronoun errors.

Previous SLA Studies

The only two SLA studies on reactivity investigated the effects of verbalization during L2 reading (Bowles & Leow, Reference Bowles and Leow2005; Leow & Morgan-Short, Reference Leow and Morgan-Short2004). Leow and Morgan-Short conducted the first SLA study to empirically investigate the reactivity of verbal reports. In their study, 77 beginning-level Spanish learners read a text in Spanish, either silently or while verbalizing nonmetalinguistically. Posttests measured text comprehension as well as recognition and written production of formal imperative verbal morphology. Results showed nonreactivity for concurrent nonmetalinguistic verbalization: The two groups did not differ significantly on either text comprehension or posttest scores on the target structure. Because time on task was not one of the study's dependent variables, latency was not addressed.

Expanding on Leow and Morgan-Short's (Reference Leow and Morgan-Short2004) study, Bowles and Leow (Reference Bowles and Leow2005) sought to investigate the reactivity of both metalinguistic and nonmetalinguistic reports on text comprehension and item and system learning of a syntactic structure, the pluperfect subjunctive.Footnote ⁴ Participants were 45 fifth-semester Spanish learners who were randomly assigned to either a control group or to one of two verbalization groups (nonmetalinguistic or metalinguistic). All participants read a text that included tokens of the target structure and then completed a series of comprehension and written production tasks. The groups differed only in the verbalization instructions they received; participants in the control group were silent, whereas those in the nonmetalinguistic group were instructed to say whatever passed through their minds, and those in the metalinguistic group were instructed to verbalize what information they were looking at, what thoughts they were having about any piece of information, how they evaluated different pieces of information, and why. Similar to Leow and Morgan-Short, results indicated that, compared to a control group, nonmetalinguistic verbalization did not significantly affect either comprehension or written production of the target form. However, metalinguistic verbalization caused a significant decrement in text comprehension but no significant difference for production when compared to either the control or the nonmetalinguistic group. Furthermore, results indicated that both verbalization groups took significantly more time to read the text and complete the posttests than the control group, and no significant difference was found between the nonmetalinguistic and metalinguistic groups in terms of time on task.

Motivation for the Current Study

Although verbal reports have been used in a number of SLA studies and stand to provide invaluable insight into the language learning process, their validity as a methodological tool in SLA research is just starting to be examined. Two SLA studies (Bowles & Leow, Reference Bowles and Leow2005; Leow & Morgan-Short, Reference Leow and Morgan-Short2004) have investigated type of verbal report as an independent variable during L2 reading. Although these studies' findings certainly shed some light on the validity of verbal reports, future studies are warranted to investigate the reactivity of such reports on different types of tasks. Because verbal reports have already been used in SLA research during a range of L2 tasks, including picture drawing and spot-the-difference tasks in oral interaction research (Nabei & Swain, Reference Nabei and Swain2002; Philp, Reference Philp2003), role-play (Félix-Brasdefer, Reference Félix-Brasdefer2004) and discourse completion tasks in interlanguage pragmatics (Robinson, Reference Robinson and Kasper1991), and task-essential problem-solving tasks targeting morphological and syntactic L2 structures (Leow, Reference Leow1997, Reference Leow1998a, Reference Leow1998b, Reference Leow2000, Reference Leow2001a, Reference Leow2001b; Rosa & Leow, Reference Rosa and Leow2004a, Reference Rosa and Leow2004b; Rosa & O'Neill, Reference Rosa and O'Neill1999), investigations into reactivity might be even more crucially needed, given that several studies in cognitive psychology have found verbal reports to be reactive for some tasks but not for others (e.g., Rhenius & Deffner, Reference Rhenius and Deffner1990; Russo et al., Reference Russo, Johnson and Stephens1989).

In addition to the potential effect of task type, another factor that has not been examined in studies on the reactivity of verbal reports in SLA is feedback. Some studies in cognitive psychology (Brehmer & Lindberg, Reference Brehmer and Lindberg1971; Hagafors & Brehmer, Reference Hagafors and Brehmer1983; Hammond & Summers, Reference Hammond and Summers1972) have observed the separate and combined effects of immediate explicit feedback and concurrent verbalization on task performance. Hagafors and Brehmer's study is of particular interest because it investigated the effects of metalinguistic verbalization and explicit feedback on performance of a cue-probability learning task. Sixty-four high school students were randomly assigned to one of four groups: (a) an explicit feedback/justification group, (b) an explicit feedback/no justification group, (c) a no feedback/justification group, or (d) a no feedback/no justification group. All participants completed a series of learning trials followed by a similar experimental trial. Results indicated that verbal justification (metalinguistic verbalization) had a significant facilitative effect on task performance when no feedback was provided. However, when explicit feedback was provided, no difference was found between justification and nonjustification groups (metalinguistic verbalization and silent control groups). In this case, metalinguistic verbalization was found to be reactive in the absence of explicit feedback but nonreactive in the presence of such feedback.

To date, no SLA studies have been conducted to determine whether there is an interaction between the provision of feedback and verbalization. Given the prevalence of tasks involving some type of feedback (whether oral or written) in SLA studies and the increasing use of verbal reports as a methodological tool to gain insight into learners' cognitive processes and access to explicit L2 knowledge (Ellis, Reference Ellis2004, Reference Ellis2005; Gass & Mackey, Reference Gass and Mackey2000; Schmidt, Reference Schmidt and Robinson2001), research appears to be essential not only to ensure the validity of the measure but also to help SLA researchers make progress in defining and determining the role of explicit knowledge in SLA.Footnote ⁵

Additionally, there is no conclusive evidence about whether verbalization affects time on task. Many studies in cognitive psychology neglect to report times for their verbalization groups and, in SLA research, only Bowles and Leow (Reference Bowles and Leow2005) have investigated that question for L2 reading only.

Target Structure

The target linguistic structure is the Spanish syntactic construction of the psych verb gustar “to like,” which requires a dative experiencer obligatorily doubled by a dative clitic. In English, however, experiencers can only have the nominative or accusative case (Montrul, Reference Montrul, Pérez-Leroux and Glass1997, Reference Montrul1998, Reference Montrul2004). Compare (1) and (2):

1. (1)

   

2. (2)

   

Psych verbs like gustar “to like” are an ideal structure to compare the effects of feedback and verbalization because SLA research (Cazzoli-Goeta, Masullo, & Young-Scholten, Reference Cazzoli-Goeta, Masullo and Young-Scholten2003; LoCoco, Reference LoCoco, VanPatten, Dvorak and Lee1987; Montrul, Reference Montrul1998; Van Patten & Cadierno, Reference Van Patten and Cadierno1993; VanPatten & Sanz, Reference VanPatten, Sanz, Eckman, Highland, Lee, Mileham and Weber1995) has shown them to be particularly difficult for L1 speakers of English. Sanz (Reference Sanz, Lee and Valdman1999) suggested that these sorts of difficult structures are the most appropriate targets for feedback because learners are unlikely to acquire them easily and completely without such instructional interventions.

Experimental Tasks

The experimental tasks consisted of 24 three-dimensional computerized mazes for which the target structure was task-essential. Participants entered each maze and moved through it virtually from a first-person vantage point using the mouse. Spanish pronouns and verb forms were placed along the paths of each maze, and only by choosing the correct forms to complete a grammatical L2 sentence could learners exit the maze. Specifically, at each forking path in the maze, there were either two pronouns or two verbs from which learners could choose. One form was correct, and one was a distracter that beginning-level learners were likely to choose (based on Gascón's, Reference Gascón1998, analysis of common learner errors with gustar-type verbs). When learners chose an ungrammatical form, they ran into a wall (and had to choose another form). However, when they chose a grammatical form, the path continued and eventually led them to an exit. See Appendix A for a visual representation of the decision paths in the mazes.

Feedback and Verbalization Condition

In the implicit feedback condition, this intrinsic feedback is all the learners received. In the explicit feedback condition, learners received explicit grammatical information about the target structure (in English) whenever they chose any pronoun or verb in the maze, whether correct or incorrect. Specifically, this explicit information provided learners with (a) a statement of whether the choice was correct (Right!) or incorrect (Sorry!) and (b) an explanation of why the choice was correct or incorrect (i.e., a metalinguistic grammatical explanation). This maze format was chosen for several reasons. Because the study was designed to investigate the validity of verbal reports used with L2 problem-solving tasks, it was important that the task be similar to problem-solving tasks used in previous SLA studies that required participants to think aloud (cf. the multiple-choice jigsaw puzzle in Rosa & Leow, Reference Rosa and Leow2004a, Reference Rosa and Leow2004b). Furthermore, to address the relationship between feedback and reactivity, it was essential that both implicit and explicit feedback versions of the task be created. The maze provided intrinsic, implicit feedback seamlessly and enabled a comparison with explicit feedback.

Participants were randomly assigned to one of the three verbalization conditions: metalinguistic, nonmetalinguistic, or silent (control). All participants received general task instructions, and those in the metalinguistic and nonmetalinguistic groups were also given instructions for verbalization (Appendixes B and C).

Pretests and Posttests

A controlled written production test was designed to measure participants' ability to produce the target structure. The test required participants to write Spanish equivalents of English sentences, as illustrated by the distracter item in (3) and the target item in (4):

1. (3) Pedro writes letters. _____________________ cartas.

2. (4) Luis and Marco like baseball. _____________________ el béisbol.

Vocabulary was carefully controlled and limited to proper names and common nouns that had been presented in the learners' first-semester textbook. As a further safeguard, the last noun of each sentence was provided at the end of the response line to ensure that the test measured learners' syntactic knowledge rather than their vocabulary.

Participants completed three different versions of the written production test: a pretest, an immediate posttest, and a delayed posttest. The pretest was the shortest, with 24 sentences, of which 12 required the target structure and 12 were distracters. Because the posttests needed to accommodate enough tokens of the target structure to enable a comparison of item and system learning, they were slightly longer, with 35 sentences, of which 24 required the target structure (12 exemplars from the experimental task and 12 novel exemplars) and 11 were distracters.

To allow for a comparison with the existing body of research on reactivity (Bowles & Leow, Reference Bowles and Leow2005; Ericsson & Simon, Reference Ericsson and Simon1993; Leow & Morgan-Short, Reference Leow and Morgan-Short2004), only immediate posttest scores are analyzed and reported here.

Testing Procedure

This study was completed in three phases. On the first day, participants completed the language background questionnaire and the written production pretest. Two days later, participants completed the experimental tasks and the immediate posttest. Finally, 3 weeks after the treatment, participants completed the delayed posttest and a debriefing questionnaire that informed them of the target of the study and asked them to report any outside exposure to the target form during the period of the experiment.

On the day of the treatment, participants reported to the language laboratory in intact classes but were randomly assigned to one of the six experimental conditions (see Table 2) when they logged on to the testing computers. All participants completed a familiarization session before starting the computerized experimental tasks. Additionally, participants in the two verbalization conditions received instructions on how to verbalize and then watched a demonstration in which a verbal report (either metalinguistic or nonmetalinguistic, according to condition) played as the cursor moved through a sample maze. Finally, the participants in the verbal reports groups were given the opportunity to warm up by verbalizing while working through one final sample maze.Footnote ⁶ To ensure that participants in all conditions received the same amount of input on the target structure over the course of the treatment, none of the sample mazes in this session contained the target form.

Immediately after the familiarization session, participants completed the experimental treatment on their computers by working through the 24 mazes at their own pace, with either implicit or explicit feedback, according to their verbalization condition. Participants' mouse clicks (detailing the paths that they chose) and time on task were collected and stored in a database for subsequent scoring, coding, and analysis, as were their immediate posttest responses.

Scoring Procedure

Because the target structure was a complex syntactic construction, all tests were scored to allow for partial knowledge. For each item, learners received 1 point for (a) using the preposition a to prepose the experiencer, (b) using the correct clitic pronoun, and (c) correctly conjugating the verb gustar “to like.” Therefore, the maximum possible score on the pretest was 36 points, and the maximum possible score on the posttest was 72 (36 points for previously encountered exemplars and 36 for novel exemplars).

Because one independent variable was the type of verbalization, it was essential that all of the participants' verbalizations be coded accurately as metalinguistic or nonmetalinguistic. To ensure that participants had verbalized according to the initial instructions they received, the researcher listened to all the participants' verbal reports and coded them in a binary fashion (Bowles & Leow, Reference Bowles and Leow2005; Rosa & O'Neill, Reference Rosa and O'Neill1999).Footnote ⁷

The most important feature that distinguishes the metalinguistic group from the nonmetalinguistic group is the provision of justifications. Participants in the metalinguistic condition should have provided a justification or explanation for each path they chose to follow in the maze. Because participants were not likely to go down every path in every maze, it was clear that all participants would not have the same number of justifications to make. To resolve this, the researcher kept track of (a) the paths each participant chose throughout the course of the 24 experimental mazes and (b) whether the participant provided a justification for each path. Despite the fact that manipulation checks were conducted as participants completed the tasks, each participant in the metalinguistic condition might not have provided a justification for every path chosen. If a participant provided justifications for 50% or more of the paths chosen, his or her report was coded as metalinguistic. Conversely, if a participant provided justifications for fewer than 50% of the paths chosen, his or her report was coded as nonmetalinguistic. In practice, however, participants were never near this 50% cutoff. The verbal reports coded as nonmetalinguistic contained justifications for 0–20% of the paths chosen, whereas those coded as metalinguistic contained justifications for 70–100% of the paths.

Of the 125 total verbal reports, only 8 were reclassified, which indicates that overall participants verbalized according to the instructions they received. As a check on reliability, 20% of the verbal reports (n = 25) were selected at random to be coded by a second coder. Intercoder reliability, calculated using Cohen's kappa, was found to be high (κ = .90).

Excerpts from both types of verbal reports are provided in (5) and (6). Note that these excerpts are from participants' verbal reports as they worked on the 12th maze, in which the target sentence was A Ana le gustan las películas “Ana likes movies.” In these excerpts, participants are at the final forking path in the maze, where they have the choice between the third person singular verb form *gusta and the third person plural form gustan. (See Appendix A for a detailed description of this maze.)

1. (5) Excerpt from a nonmetalinguistic verbal report

   It should be gustan and I really hope there's not gonna be any wall here. [Selects the correct form, gustan, and receives explicit feedback.] Yes, I'm right! Takes me straight to the exit. Right! [Participant exits the maze.]

2. (6) Excerpt from a metalinguistic verbal report

   So the ella [third person singular] form of gustar must be gust- gust-a because you always add an -a to the end so choosing gusta. [Selects the incorrect form, gusta, and receives explicit feedback.] No? I guess I got it wrong. OK, it [the feedback] says that the verb agrees with the thing that is liked or películas. So it's gonna be gust-an 'cause there's more than one movie. [Selects gustan.] That's it! [Participant exits the maze.]

Limitations

Because this is the first study to investigate the reactivity of verbal reports on a L2 task other than reading, care should be taken overall in interpreting the findings. Additionally, a number of specific limitations are pertinent.

A first limitation concerns the pool of participants, who were first-semester L2 learners unfamiliar with the target structure at the outset of the experiment. It is not clear what effect learners' proficiency might have had on reactivity and whether learners who had some degree of prior knowledge of the target structure might have performed differently given the requirement to verbalize during the task.

Similarly, the target syntactic structure was task-essential (Loschky & Bley-Vroman, Reference Loschky, Bley-Vroman, Crookes and Gass1993), so learners could not successfully complete the experimental tasks without using the target grammatical structure correctly. Certainly, the task-essentialness of the structure could have had an effect on the results, because the task pushed learners to attend to the grammar as they worked through the experimental tasks.

Finally, just one type of posttest (a highly controlled written production test) was used. This can be seen as a benefit in the sense that such a controlled test enables direct comparison among participants with little individual variation in possible correct responses. However, it can also be viewed as a limitation because no evidence about the potentially reactive effects of verbalization on more open-ended tasks was provided here. For this reason, the results cannot speak to reactivity on less controlled tests of L2 knowledge.

Future Research

The present study contributes to a growing body of research investigating the validity of verbal reports as a SLA research method. Future research is certainly warranted in this area, as verbal reports continue to be used in various strands of L1 acquisition and SLA research. Especially important is an investigation into the reactivity of verbal reports with other types of SLA tasks.

Verbal reports could provide invaluable information to SLA researchers about language learners' cognitive processes, information that would otherwise be unavailable to them. If these measures are shown to be valid and to accurately represent learners' mental processes, they will greatly contribute to our understanding of L2 learning. Metalinguistic reports are of particular interest because, if they are found to be valid, they would allow researchers to gain even more insight into learners' processing and justifications as they process and produce the L2. Data from these protocols would therefore allow researchers to target specifically the areas where students are in need of improvement and to address the faulty reasoning that is behind (at least some) L2 errors. This could prove to be one part of the solution to the perennial problem of identifying, understanding, and addressing L1 transfer errors such as those reported in Spada and Lightbown (Reference Spada and Lightbown1999). Some studies (Butler, Reference Butler2002) have already shown that adult L2 learners seem to be able to consistently use their explicit knowledge to provide justifications of aspects of their L2 use (although these are, of course, not always correct). Further research on methods of measuring attention and awareness can only serve to move the field forward and enable us to learn more about our students' processes as they interact with a L2 or foreign language.

It will be crucial to identify a systematic way to investigate reactivity and to determine which types of tasks, under which conditions, are likely to be reactive. Thus, researchers can begin to formulate principled guidance to those interested in using verbal reports with L2 learners. Only by synthesizing findings from previous SLA and cognitive psychology research and by focusing on issues and task features, such as task complexity (e.g., Deffner, Reference Deffner1989; Russo et al., Reference Russo, Johnson and Stephens1989), already identified to play a role in reactivity, can researchers develop well-informed models. Once created, these models can be tested empirically using orthogonal pairings of L2 tasks with varying task demands and participants at varying levels of language proficiency to assess the separate and combined effects of language and task demands on reactivity.

Additionally, future research in the area of introspective SLA research methods is essential to furthering our understanding of the nature and use of explicit knowledge in SLA (Ellis, Reference Ellis2004). Clearly, explicit knowledge cannot be measured directly, but verbal reports are one option that SLA researchers have for getting a window into their learners' explicit knowledge, cloudy as the pane of glass may be. By collecting and carefully coding and analyzing verbal reports, we can gain insight into “what kinds of explicit knowledge learners exploit and in what ways” (Ellis, p. 268). Regardless of one's theoretical position, as our objective tools for measuring explicit knowledge become sharper, verbal reports can be used as a complement, given that “another entirely different approach based on tasks that elicit verbal reports is [also] needed” (Ellis, p. 268).