PREVIOUS SLA RESEARCH

Verbal reports in SLA studies have been mainly used to explore the cognitive strategies adult learners use while reading an L2 text and their potential effects on subsequent comprehension (e.g., Carrell, 1989; Cohen, 1986; Pritchard, 1990). Pressley and Afflerbach (1995) surveyed 38 studies of first language (L1) reading that use verbal reports and were able to classify strategies as before reading, during reading, after reading components, evaluating, and monitoring. Verbal reports have also provided insight into links between L1 and L2 reading strategies (Davis & Bistodeau, 1993; Nevo, 1989). Likewise, L2 test-taking strategies have been observed using verbal reports (Cohen, 2000; Norris, 1992).

Verbal reports have not only been used to determine L1 and L2 strategies but also in empirical research, typically with problem-solving tasks. These verbal reports provided a more robust type of data that cannot be obtained through pretest + treatment + posttest research methodology. Leow (1997a, 1998a, 1998b, 2000, 2001a, 2001b), Rosa and Leow (in press a, in press b), and Rosa and O'Neill (1999) assessed attention and awareness and found that different levels of attention and awareness appear to correlate with performance. Alanen (1995) employed on-line think-alouds to compare explicit information versus implicit textual enhancement. She reported that participants who noticed the linguistic form, regardless of treatment type, demonstrated acquisition of at least some aspects of the targeted forms. The quantitative use of verbal reports in addition to behavioral data can strengthen an empirical study's support for a hypothesis or theory, given that it offers converging evidence not otherwise available.

Although verbal reports offer many benefits, they are not without controversy. The validity of the use of concurrent think-aloud protocols to elicit metalinguistic on-line data of learners' processes as they interact with L2 experimental data has been critiqued. Nisbett and Wilson (1977) suggested that access to metalinguistic information is limited and that participant reports often do not coincide with behavior. Olson, Duffy, and Mack (1984) suggested that verbal reports results may vary due to at least one of the following variables: instructions given, type of material, and nature of data analysis. Lyons (1986) purported that participants may not fully report information.

As pointed out by Leow (2002), a more serious critique for concurrent verbal reports is the issue of reactivity in SLA methodology: By thinking aloud, participants' internal processes may differ from what they would have been had they not performed the verbalization. In SLA, this critique appears to be leveled against introspective, metalinguistic verbalization that potentially promotes “dual processing—performing some learning task and commenting on their thinking processes” (Ellis, 2001, p. 37, note 14). Similarly, Jourdenais (2001), drawing from the findings of an unpublished doctoral dissertation (Jourdenais, 1998) that employed on-line, nonmetalinguistic verbalizations gathered while learners were engaged in written production but not during exposure to the L2 data, cautioned that “the think aloud data collection method itself acts as an additional task which must be considered carefully when examining learner performance” (2001, p. 373). However, the caution that thinking aloud while completing an L2 reading task may impede learners' natural cognitive processes and result in data that are not rational or logically coherent is based more on anecdotal rather than empirical evidence (see also Seliger, 1983; Stratman & Hamp-Lyons, 1994). The potential impact of reactivity in studies that employ concurrent data elicitation procedures remains to be empirically tested in the SLA field.

PREVIOUS NON-SLA RESEARCH

The validity of problem-solving tasks to elicit concurrent data (e.g., Leow, 1997a, 1998a, 1998b, 2000, 2001a; Rosa & Leow, in press a, in press b; Rosa & O'Neill, 1999) finds empirical support from studies in other scientific fields in which think-aloud protocols have served as a data-elicitation procedure. Ericsson and Simon (1993) reviewed a series of studies that address the issue of reactivity and found that, although participants completing verbal reports may take longer to complete a task, there is no evidence that their internal processes are altered. Three studies reviewed by Ericsson and Simon reported nonreactivity on learners' performances during problem-solving tasks, perhaps due to the nature of the task. In the field of ergonomics, Brinkman (1993) reported nonreactivity when students think aloud while solving fault-diagnosis problems. In speech and cognition research, Deffner (1984, 1989) found no evidence of reactivity for geometric puzzles or N-term series problems. Finally, in psychology, Rhenius and Heydemann (1984) reported nonreactivity for Raven's matrices. These non-SLA disciplines provide evidence that performance immediately after thinking aloud during a problem-solving task is not significantly different from that obtained after a silent version of the task. After an extensive search in the literature, only a few empirical studies directly addressing the issue of reactivity but not reviewed by Ericsson and Simon (1993) were found. The results of these studies, along with key studies reviewed by Ericsson and Simon (see Table 1), will be briefly summarized.

Think alouds (TAs) and reactivity: Overview of empirical research

In the field of memory and cognition, Russo, Johnson, and Stephens (1989) asked 24 university students to complete four tasks under three conditions: concurrent verbalization, retrospective verbalization, and no verbalization. The tasks included (a) a verbal task, anagrams; (b) a numerical task, simple gambles; (c) a pictorial task, Raven's matrices; and (d) a mental task, addition. Russo et al. compared the concurrent and silent conditions for reactivity. No reactivity for accuracy was found for anagrams or for Raven's matrices. Nor was extended time required when thinking aloud on addition and Raven's matrices. However, reactivity was found for accuracy on gambles and addition and for time on gambles and anagrams. These results need to be considered in light of the fact that participants did not perform these tasks only under the two conditions but also under a retrospective condition that prompted students to provide metalinguistic introspection. Because the conditions were randomly generated, participants who were asked for retrospection prior to the concurrent condition may have been more metalinguistic under the concurrent condition. As noted previously (Ericsson & Simon, 1993), introspective, metalinguistic verbalizations are not considered to be reliable. As such, these results may not truly reflect concurrent, nonmetalinguistic verbalizations. Taking into consideration the conflicting results, Russo et al. suggested that, when using verbal protocols, it may be wise to add a silent group to the research design to control for accuracy. Overall, they stated: “We believe that nothing can match the processing insights provided by a verbal protocol. Given the unique benefits, the challenge is to identify and reduce causes of their invalidity” (p. 767).

A study in the field of behavioral decision making (Biggs, Rosman, & Serfenian, 1993) examined the issue of reactivity by asking 15 equity analysts to provide a ratio analysis of the investment quality of three companies. The researchers predicted nonreactivity for their task on the basis of the cognitive theory of verbalization proposed by Ericsson and Simon (1984), which states that “the degree to which verbalizing interferes with the primary task should be negligible if verbalization is limited to items in short-term memory and the task does not require intermediate processing to translate nonverbal stimuli into a verbal form” (p. 190). Participants used computer-based information to rate three firms and were instructed to search for as much information as necessary to be able to make a rating. Biggs et al. found that adding concurrent, nonmetalinguistic verbalization to the task did not prove reactive, as it did not cause changes in the amount of information searched for, the pattern of acquisition of the information, or the accuracy of the final ratings, which were compared to industry-standard ratings. The only evidence of reactivity was the amount of time required to complete the task.

An important study reviewed by Ericsson and Simon (1993) was Brinkman (1993), in the field of ergonomics. In this study, 18 university students were asked to diagnose faults in graphically displayed networks consisting of rows and columns of interconnected logical and gate components for 12 problems. Brinkman carefully followed verbal-report procedures laid out by Ericsson and Simon, including providing participants with a warm-up period. Participants then attempted to diagnose faults under three conditions: (a) concurrent verbalization, (b) retrospective verbalization, and (c) no verbalization. For the concurrent verbalization condition, no evidence of reactivity for processing information was found. Results did, however, indicate reactivity for time. This study demonstrated the typical pattern found for concurrent, nonmetalinguistic verbal protocols.

Given that the studies reviewed thus far used fairly well defined tasks, Stratman and Hamp-Lyons (1994), in a pilot study, asked whether the same conclusions would hold for ill-defined tasks as are typical in L1 writing. Twelve graduate and undergraduate university students were asked to revise two faulty texts at different time periods separated by several weeks. Each participant revised one text while thinking aloud and the other text under a silent condition. Stratman and Hamp-Lyons were unable to carry out any statistical analyses of their results due to the small number of participants and tasks performed. They found no difference in participants' ability to detect phrase- and word-level errors. However, they suggested that the ability to detect and remedy organizational errors decreased and the detection of pronoun errors and the occurrence of word-level errors increased when participants thought aloud. Although these results may suggest that verbal protocols are not valid for some tasks, this study cannot be considered strong empirical evidence for this conclusion. Thus, further research is necessary before conclusions can be made regarding the use of verbal reports and L1 writing.

Williams and Davids (1997) conducted two experiments in sport and exercise science. One of their primary questions was whether verbalization while performing a task had any effect on performance. Twenty participants were divided into two groups of experienced and less experienced soccer players. Each participant then watched a video series of actual 11-versus-11 soccer simulations under two conditions. In the first condition the participants' attention was assessed by measuring their eye movements, and in the second condition the participants concurrently verbalized to which area of the screen they were paying attention. As a final task, participants indicated the anticipated destination of the final pass. Williams and Davids reported no difference in performance for either group across the two conditions and concluded that concurrent verbalization did not result in reactivity. In a second experiment, 24 experienced and less experienced soccer players watched 3-versus-3 soccer simulations under an eye-movement condition and a verbal-report condition. Again, participants indicated where they expected the ball to be passed. However, in this second experiment, under the verbal condition, participants were asked to report where they were extracting information from, not which area of the screen they were focusing on. This design was used because experienced players tend to rely more on their peripheral vision than less experienced players. Results from the second experiment indicated a difference between the two conditions. Williams and Davids proposed that the use of peripheral vision is often subconscious and that asking participants to consciously verbalize this subconscious information results in reactivity. Nonetheless, their results indicated that verbal reports were more veridical than eye-movement measures because the latter did not detect use of peripheral vision. They concluded that both methods are valid measures of selective attention for tasks dependent on the use of central vision (e.g., computer programming, machine operation, industrial inspection, and radiography) but that verbal reports are preferred for tasks requiring the extraction of information through peripheral vision (e.g., driving a car, flying, landing helicopters, and sports simulations).

Taken together, the results obtained by Biggs et al. (1993), Brinkman (1993), Russo et al. (1989), Stratman and Hamp-Lyons (1994), and Williams and Davids (1997) suggest that verbal reports, although extending time-on-task, do not result in altered internal processing. These studies provide further support for Ericsson and Simon's (1993) statement that in general, “thinking aloud does not lead to a reliable change in the cognitive process as reflected in accuracy of response” (p. xx). To be confident that this assumption of nonreactivity is applicable to tasks in SLA, direct empirical evidence is necessary.

The increasing need to address the internal validity of studies conducted within an attentional framework in SLA is clearly manifested in current research designs (e.g., discourse in Alanen, 1995; Leow, 2001b; Rott, 1999; problem-solving tasks in Leow, 1997a, 1998a, 1998b, 2000, 2001a; Rosa & O'Neill, 1999; stimulated recalls in Mackey, 1999; computer-based tasks in Rosa & Leow, in press a, in press b). These research designs have included some form of a concurrent data-elicitation procedure to measure whether attention was indeed paid to the L2 data. However, the critique of the possible reactivity of think-aloud protocols needs to be noted and empirically validated in the field of SLA. Due to the inherent limitations of introspective metalinguistic verbalization, the present study investigates only concurrent think-aloud protocols produced by learners instructed to verbalize their thoughts, without further request to comment or focus on any specific aspect of the L2 data. This study empirically investigates the effects of thinking aloud while reading an L2 text on learners' subsequent comprehension, intake, and controlled written production, guided by the following three research questions:

1. Does thinking aloud while performing a reading task have any detrimental effect on adult readers' comprehension when compared to readers not thinking aloud?
2. Does thinking aloud while performing a reading task have any detrimental effect on adult readers' intake when compared to readers not thinking aloud?
3. Does thinking aloud while performing a reading task have any detrimental effect on adult readers' controlled written production when compared to readers not thinking aloud?