USE OF AND ASSUMPTIONS ABOUT VERBAL REPORTS

Decades before they were first applied in language research, verbal reports were used extensively in cognitive psychology, where they were traditionally used to investigate problem-solving strategies on nonverbal tasks and puzzles (J. H. Davis, Carey, Foxman, & Tarr, Reference Davis, Carey, Foxman and Tarr1968; Gagné & Smith, Reference Gagné and Smith1962). Since the 1980s, “both concurrent and retrospective verbal reports [have become] generally recognized as major sources of data on subjects’ cognitive processes in specific tasks” (Ericsson & Simon, Reference Ericsson and Simon1993, p. xi). The collection of verbal reports has become standard in many fields, such as accounting (Anderson, Reference Anderson1985); care planning (Fowler, Reference Fowler1997); counseling (Bozarth, Reference Bozarth1970); drug and alcohol addiction treatment (Midanik & Hines, Reference Midanik and Hines1991); ergonomics (Brinkman, Reference Brinkman1993); marketing (Biehal & Chakravarti, Reference Biehal and Chakravarti1989); nursing (Greenwood & King, Reference Greenwood and King1995); software engineering (Hughes & Parkes, Reference Hughes and Parkes2003); medicine, where verbal reports are routinely used in the treatment of autism and developmental disorders (Berg, Reference Berg2002; Friedman & Mulhern, Reference Friedman and Mulhern1976); speech pathology (Karsenty, Reference Karsenty2001); neurology (Chan, Hoosain, & Lee, Reference Chan, Hoosain and Lee2002); and cardiology (Bernardi et al., Reference Bernardi, Wdowczyk-Szulc, Valenti, Castoldi, Passino and Spadacini2000). In each of these fields, verbal reports provide insight into the cognitive processes of clients and patients.

Think-alouds have also been used to investigate many aspects of first and second language acquisition and use. In the L1 literature, there have been numerous studies on reading, which have tended to focus on reading strategies used by older and younger, more and less successful readers (Cohen, Reference Cohen1986, Reference Cohen1987; Earthman, Reference Earthman1992; Folger, Reference Folger2001; Gordon, Reference Gordon1990). Think-alouds have also been used to study the L1 writing process (Breetvelt, Reference Breetvelt1994; Zellermayer & Cohen, Reference Zellermayer and Cohen1996). In some cases, think-alouds have been used as one component of programs designed to improve students’ reading (Baumann, Jones, & Seifert Kessell, Reference Baumann, Jones and Seifert Kessell1993; Robertson, Reference Robertson1995; Walczyk, Marsiglia, Bryan, & Naquin, Reference Walczyk, Marsiglia, Bryan and Naquin2001; Wilhelm, Reference Wilhelm2001) and writing ability (Box, Reference Box2002; Cushman, Reference Cushman2002; Fresch, Wheaton, & Zutell, Reference Fresch, Wheaton and Zutell1998; Scardamalia, Reference Scardamalia1984).

In second language acquisition (SLA) research, think-alouds have also been used to study reading (Carrell, Reference Carrell1989; Cohen, Reference Cohen1986, Reference Cohen1987; Cohen & Cavalcanti, Reference Cohen and Cavalcanti1987; Hosenfeld, Reference Hosenfeld1976, Reference Hosenfeld1977, Reference Hosenfeld and Borne1979, Reference Hosenfeld, Alderson and Urquhart1984; Pressley & Afflerbach, Reference Pressley and Afflerbach1995; Pritchard, Reference Pritchard1990) and writing (Qi & Lapkin, Reference Qi and Lapkin2001; Sachs & Polio, Reference *Sachs and Polio2007). However, they have also been used to investigate a wide array of other phenomena, including L1 and L2 strategy use (Chamot & El Dinary, Reference Chamot and El Dinary1999; J. Davis & Bistodeau, Reference Davis and Bistodeau1993; Yamashita, Reference Yamashita2002), lexical organization (Herwig, Reference Herwig, Cenoz, Hufeisen and Jessner2003), the use of translation in L2 learning (Kern, Reference Kern1994), interlanguage pragmatics (Woodfield, Reference Woodfield2008), the role of different levels of awareness in L2 learning (Leow, Reference Leow1997, Reference Leow1998a, Reference Leow1998b, Reference Leow and Gutiérrez-Rexach1999, Reference Leow2000, Reference Leow2001a, Reference Leow2001b; Rosa & Leow, Reference Rosa and Leow2004a, Reference Rosa and Leow2004b; Rosa & O'Neill, Reference Rosa and O'Neill1999), and the relationship between explicit and implicit L2 knowledge (Ellis, Reference Ellis2004; Hu, Reference Hu2002).

As this brief review shows, verbal reports have been used to answer a wide array of research questions in SLA. Verbal reports have been used in cognitivist approaches to SLA, where they are viewed as a window into the minds of learners, that is, as a means of gathering information about their internal thought processes. Crucially, an underlying assumption is that verbalizations can accurately reflect thought processes without altering them and can therefore be a data collection tool.

CONTROVERSY OVER THE USE OF VERBAL REPORTS

Indeed, despite the frequency with which verbal reports are gathered, their use is not uncontroversial. More than three decades ago cognitive psychologists raised questions about whether verbal reports actually capture thought processes accurately, or whether they alter those processes (Nisbett & Wilson, Reference Nisbett and Wilson1977; Payne, Braunstein, & Carroll, Reference Payne, Braunstein and Carroll1978). Similar concern has since been voiced in SLA. For instance, Jourdenais (Reference Jourdenais and Robinson2001) specifically cautioned that “the think-aloud data collection method itself acts as an additional task which must be considered carefully when examining learner performance” (p. 373).

Specifically, the two main validity issues surrounding the use of verbal reports are reactivity and veridicality. Verbalization is said to be reactive if it alters task performance, and it is said to be veridical if it accurately reflects the participant's cognitive processes. Ericsson and Simon's (Reference Ericsson and Simon1993) classic model of verbalization speaks to the validity of think-alouds. The model does not treat all verbal reports alike; rather it distinguishes them along two dimensions: time of reporting (concurrent vs. retrospective) and level of detail. It predicts that concurrent reports will be more complete and accurate (veridical) than retrospective reports, since participants who think aloud during a task are not subject to memory decay, unlike those who are asked to report their thoughts some time after completing a task. The model furthermore makes different predictions based on how detailed the verbalizations are required to be. Verbalizations that are generated as a normal part of the solution process (what Ericsson & Simon refer to as Type 1 verbalizations, or nonmetacognitive verbalizations) will be largely nonreactive; that is, they will reflect the nature of cognitive processes fairly accurately, while slowing processing slightly. Conversely, their model predicts that verbalizations that include additional reasoning or justifications that would not normally figure into the normal solution process (what Ericsson & Simon refer to as Type 2 and 3 verbalizations, or metacognitive verbalizations) are more likely to be reactive, or to affect task performance.

Experimental studies examining the effects of think-alouds on problem-solving and decision-making tasks have been conducted in the field of cognitive psychology since the 1950s. To assess the extent to which verbalization affected participants’ task performance, one group would complete a task silently, while another would complete the same task while verbalizing. Overall, a synthesis of the studies reviewed in Ericsson and Simon (Reference Ericsson and Simon1993) reveals a clear pattern of results. When participants were asked to think aloud nonmetacognitively, their task performance (usually measured as accuracy) was generally not significantly different from that of the participants in the silent groups. On the other hand, when participants were asked to think aloud metacognitively, their task performance was, in most studies, significantly different from that of participants in silent groups. In some cases, participants in verbalization groups outperformed silent participants, and in others, the verbal ones underperformed the silent, suggesting that sometimes verbalization is facilitative and sometimes it is detrimental. Furthermore, participants who verbalized (either nonmetacognitively or metacognitively) tended to take significantly more time to complete the task than silent participants, because the additional time needed for verbalization increases the overall solution time. These findings support the predictions of Ericsson and Simon's model. However, most of the studies used nonverbal and/or problem-solving tasks, limiting their relevance to SLA.

REVIEW OF SLA STUDIES ON REACTIVITY

Leow and Morgan-Short (Reference *Leow and Morgan-Short2004) was the first SLA study to empirically investigate the reactivity of think-alouds. Their study investigated the effects of nonmetacognitive think-alouds on 77 beginning Spanish learners’ text comprehension and intake and written production of formal imperative morphology in Spanish. Learners in the control condition read and completed the tasks silently, while learners in the experimental condition read and completed the tasks while thinking aloud. Results showed nonreactivity, since the two groups did not differ significantly on either text comprehension or posttask assessments of the targeted inflectional morphology. Time on task was not a dependent variable measured in the study, so it was not possible to determine whether thinking aloud affected solution time.

Expanding on Leow and Morgan-Short (Reference *Leow and Morgan-Short2004), Bowles and Leow (Reference *Bowles and Leow2005) sought to investigate the reactivity of both metacognitive and nonmetacognitive think-alouds on text comprehension and item and system learning of the pluperfect subjunctive. Participants were 45 fifth-semester Spanish learners who were randomly assigned to either a control or to one of two verbalization groups (nonmetacognitive or metacognitive). All participants read a text that included tokens of the targeted structure and then completed the same comprehension and written production tasks. The only difference between the groups were verbalization instructions: Participants in the control group were silent; those in the nonmetacognitive group were instructed to “say whatever passed through [their] minds”; and those in the metacognitive group were instructed to comment specifically about their processes. Similar to Leow and Morgan-Short (Reference *Leow and Morgan-Short2004), the results indicated that, compared with a control group, nonmetacognitive verbalization did not significantly affect either comprehension or written production of the targeted form. However, metacognitive verbalization caused a significant decrement in text comprehension but no significant difference for production when compared with either the control or to the nonmetacognitive group. Results also indicated that both verbalization groups took significantly more time to read the text and complete the postassessment tasks than the control group, but that there was no significant difference between the two think-aloud groups in terms of time on task.

Sachs and Polio (Reference *Sachs and Polio2007) examined the reactivity of think-alouds on an L2 writing task. The study was carried out in two experiments. In Experiment 1, 15 adult English as a second language (ESL) learners participated in a three-stage, composition-comparison-revision task. Each learner participated in the three-stage process three times, once receiving written corrections on their compositions, once receiving reformulations of their errors, and once receiving reformulations of their errors and verbalizing while they compared their original composition to the reformulated version. Results of a Wilcoxon signed rank test indicated that when learners were silent during the comparison stage (reformulation), they subsequently revised more errors in their composition than when they verbalized during the comparison stage (reformulation + think aloud), although the effect size (η² = 0.28) was weak, suggesting that the difference in performance between the two groups, though statistically significant, was small. Experiment 2 was then conducted as a nonrepeated-measures study, this time with 54 ESL learners, randomly assigned to one of four conditions: (1) error correction, (2) reformulation, (3) reformulation + think aloud, or (4) control, in which learners received no feedback on their writing. A Mann-Whitney test comparing the reformulation and reformulation + think-aloud conditions revealed no significant differences (p = .77), indicating that in Experiment 2, learners who verbalized during the comparison stage and those who did not corrected a similar number of errors during the revision stage. In summary, verbalization was found to be reactive in Experiment 1, since it appeared to hinder learners’ ability to make revisions to their compositions. In Experiment 2, however, verbalization was found to be nonreactive. In both Experiment 1 and Experiment 2, learners were instructed to verbalize in English (their L2), unlike in the previous SLA studies (Bowles & Leow, Reference *Bowles and Leow2005; Leow & Morgan-Short, Reference *Leow and Morgan-Short2004), which gave learners the option to speak in either their L1 or L2. This difference between Sachs and Polio (Reference *Sachs and Polio2007) and previous studies, combined with the repeated-measures nature of Experiment 1, is cause for the results to be interpreted with caution.

Sachs and Suh (Reference *Sachs, Suh and Mackey2007) investigated the issue of reactivity of think-alouds in the context of synchronous computer-mediated communication (CMC). Thirty ESL learners participated in the study, which targeted sequence of tense in English. All learners interacted individually with the researcher via CMC, first taking a written text completion and interactive story-retelling pretest, followed by a story-retelling task either with or without textually enhanced recasts, and with or without thinking aloud. Each learner then completed an interactive story-retelling posttest and a text completion posttest. Learners were allowed to speak in either their L1 (Korean) or in their L2 (English) while thinking aloud, as they preferred. Results indicated that the + and – think-aloud groups did not differ significantly in the amount of time spent on task (t(28) = −0.113, p = .91), indicating that verbalization was nonreactive for time on task, in contrast to the findings of Bowles and Leow (Reference *Bowles and Leow2005), the one previous SLA study that examined time as a dependent variable. Repeated-measures ANOVAs were then run with ± think-aloud as the between-subjects factor, time as the within-subjects factor, and text completion and production (story retelling) as the dependent variables. There was a significant between-group main effect on text completion (F(1,26) = 6.478, p = .02), which indicates that the group that verbalized performed differently on the text completion test than the group that did not verbalize, regardless of time. No significant interaction effect between time and group was found for either the story retelling or the text completion test, leading the authors to conclude that strong generalizations regarding reactivity should not be drawn from the data.

In a study investigating the role of lexical temporal indicators, such as temporal adverbs, in the incidental acquisition of the Spanish future tense, Rossomondo (Reference *Rossomondo2007) compared groups of first-semester Spanish students who read a passage silently and those who read the same passage while thinking aloud nonmetacognitively. Following recommendations from Ericsson and Simon (Reference Ericsson and Simon1993) and from previous SLA studies, Rossomondo gave participants a sample think-aloud as well as a short warm-up text during which they were instructed to think aloud. After reading the experimental passage, all participants completed a 13-item multiple-choice comprehension test in English, followed by either a 13-item form-recognition task, or a 13-item form-production task. The form-recognition task consisted of 13 target sentences taken from the passage (presented without lexical temporal markers) with a blank where the inflected verb should have been. Participants had to choose the correct inflected form (the future) from among four options. The form-production task consisted of the same 13 target sentences taken from the text, but in this case each blank was followed by the infinitive form of a verb in parenthesis, and participants were instructed to conjugate the verb in the same form as in the passage. No reaction time or time on task data were collected. Results indicated that participants who thought aloud while reading scored statistically similarly on comprehension measures to participants who read the passage silently (F(1,159) = 0.078, p = .781). For recognition and production of the target form, results were quite different, with participants who thought aloud scoring significantly higher on both the form-recognition (F(1,77) = 12.194, p < .001) and form-production (F(1,80) = 7.352, p < .008) tests.

Bowles (Reference *Bowles2008) investigated the effects of completing nonmetacognitive and metacognitive think-alouds while performing an L2 problem-solving task on subsequent written production of previously encountered and new exemplars of a target form. A total of 194 first-semester learners of Spanish were randomly assigned to one of six experimental conditions, which differed in terms of the type of verbalization (metacognitive, nonmetacognitive, or silent) and the type of feedback (implicit or explicit). Learners were instructed to speak in either L1 (English) or L2 (Spanish), as they felt most comfortable. Results showed that metacognitive verbalization significantly increased time on task (F(2,193) = 15.763, p < .0001), with the metacognitive think-aloud group taking significantly more time to complete the task than either the nonmetacognitive or silent group. Metacognitive verbalization also hindered participants’ ability to produce exemplars of the target structure that they had seen during the experimental task (F(2,193) = 3.778, p < .05). However, neither type of verbalization significantly affected participants’ ability to produce novel exemplars of the target structure (F(2,193) = 1.713, p = .183), and there was no interaction between verbalization and feedback. These results as a whole indicate that nonmetacognitive verbalization was nonreactive for both item and system learning, in accordance with Ericsson and Simon's model. However, the results indicate that metacognitive verbalization was reactive on item learning but nonreactive on system learning.

Yoshida (Reference *Yoshida, Bowles, Foote, Perpiñán and Bhatt2008) investigated the role that verbalization plays in an L2 reading task, thereby building on the work of Leow and Morgan-Short (Reference *Leow and Morgan-Short2004) and Bowles and Leow (Reference *Bowles and Leow2005). In her study, 64 Japanese ESL students were randomly assigned to a verbalization condition (think-aloud or non-think-aloud) and to one of three reading conditions (a control passage, the same passage with questions embedded throughout, or the passage with guidance to help students produce an outline). Students were allowed to think aloud in their L1 (Japanese) or L2 (English), so that their verbalizations would not be constrained by their language ability. The instructions required students to produce metacognitive think-alouds, since they were asked to explain how they made each decision provided in their responses to written while-reading tasks. After reading the passage and completing a corresponding while-reading task, each learner completed a written recall of the propositions in the passage. The recall scores were taken as an indication of reading comprehension, and results of a two-way ANOVA found no significant main effect for reading condition (F(1,58) = 0.795, p = .376) or for task type (F(2,58) = 0.392, p = .677), and no significant interaction was found between reading condition and task type (F(2,58) = 0.182, p = .834). These results indicate that learners who verbalized performed similarly to learners who did not verbalize, regardless of the task they engaged in while reading. A two-way ANOVA revealed that the think-aloud group took considerably more time on task than the non-think-aloud group (F(1,58) = 12.76, p = .001).

Sanz, Lin, Lado, Bowden, and Stafford (Reference *Sanz, Lin, Lado, Bowden and Stafford2009) reported on two studies investigating the reactivity of concurrent verbal reports in an L2 instructional lesson. In Experiment 1, 24 L1 English-speaking students, 11 in the think-aloud group and 13 in a silent group, completed a computerized lesson on the Latin case system. After the lesson, participants took an aural interpretation test, a written grammaticality judgment test, and a sentence production test that required them to drag and drop appropriate morphemes to create a sentence describing an on-screen photograph. On all tests, there was a main effect only for time, indicating that students in both the silent and think-aloud groups learned as a result of the lesson, and that verbalizing during the lesson had neither a facilitative nor a detrimental effect on performance. Furthermore, Sanz et al. (Reference *Sanz, Lin, Lado, Bowden and Stafford2009) used a more precise measurement than that used in previous studies to measure time. They measured mean reaction times on correct responses from the three pre- and posttests and calculated a grand mean on which to base their overall time measure for each participant. The think-aloud group had longer reaction times than the silent group on just one of the posttests, the grammaticality judgment, indicating mixed results for latency in this experiment.

In Experiment 2, 24 college-age students, 11 in the think-aloud group and 13 in the silent group, completed a less explicit version of the treatment used in Experiment 1. The only difference between the two instructional treatments was that the treatment in Experiment 1 included an explicit grammar lesson on the case system, whereas the second treatment did not. Instead, students had to rely on knowledge they gained from task-essential practice and explicit feedback. All pre- and posttest measures were the same as in Experiment 1. As in the first experiment, there were main effects for time for all tests, indicating that the treatment caused students to improve in their ability to interpret and produce sentences in Latin. But, in contrast to Experiment 1, there were also reactivity effects because students in the think-aloud group performed significantly better than those in the silent group on both the grammaticality judgment and production tests. In this case, verbalization had a facilitative effect on subsequent performance. As for latency, there were no significant differences in reaction times for the two groups across the three tests, indicating that in this experiment, although verbalization enhanced performance, it did not cause any change in the speed with which participants responded to stimuli in the L2.

Finally, in a study similar to Leow and Morgan-Short (Reference *Leow and Morgan-Short2004), Polio and Wang (Reference *Polio and Wang2005) examined the reactivity of think-alouds with advanced-proficiency learners completing a reading task. The authors hypothesized that the reading strategies of advanced learners might be different enough from those of the beginning learners in the original study to cause a change in reactivity. In Polio and Wang (Reference *Polio and Wang2005), 30 Chinese L1 learners of English were randomly assigned to a think-aloud or to a silent reading group and were instructed to read a passage in English, seeded with both frequent and infrequent phrasal verbs. Upon completing the reading task, participants took a comprehension test (in the L2, English) as well as written production and recognition tests targeting the phrasal verbs found in the passage. Since there were 15 participants per group, less powerful nonparametric statistical tests (Mann-Whitney U tests) were used to compare the performance of the think-aloud and silent groups on the three measures. The only significant difference occurred in comprehension, where participants in the think-aloud group comprehended the text significantly worse than participants in the silent group (p = .01). Polio and Wang then examined the contents of the think-alouds to investigate the nature of the reading process for this group of learners. They determined that whereas in Leow and Morgan-Short, translation was the predominant reading strategy, for these advanced learners who used English for academic purposes, translation was relatively uncommon. The researchers hypothesized that this difference in strategy use may have been largely responsible for the differing findings between the original and the replication. Since learners in Leow and Morgan-Short relied heavily on translation, they were easily able to verbalize the contents of their short-term memory without any effect on comprehension. On the other hand, the learners in the replication were engaging in other, more complex strategies, so it may have been more cognitively demanding for them to verbalize, contributing to the finding of reactivity in comprehension.

META-ANALYSIS OF STUDIES ON REACTIVITY

As the review in the preceding section shows, there is a mounting body of primary research investigating the reactivity of think-alouds on verbal tasks. However, no one study on reactivity, regardless of its size, complexity, or number of participants, can provide reliable answers, since individual study findings are susceptible to chance variability and any number of idiosyncrasies in design and sampling. This section therefore presents the results of a quantitative research synthesis, or meta-analysis, which was carried out on reactivity studies that used verbal tasks. A complete description of the meta-analysis is provided in Bowles (Reference Bowles2010).

Besides providing a summary of existing research in different fields, a meta-analysis compares the outcomes of a range of studies with an array of independent variables, in an attempt to identify patterns. In SLA, meta-analyses have increased in frequency in the past decade, as researchers in the field have started to adopt the method to provide more comprehensive answers to long-standing questions, such as the role of instruction in SLA (Norris & Ortega, Reference Norris and Ortega2000), the efficacy of different types of corrective feedback (Russell & Spada, Reference Russell, Spada, Norris and Ortega2006), the effects of conversational interaction (Keck, Iberri-Shea, Tracy-Ventura, & Wa-Mbaleka, Reference Keck, Iberri-Shea, Tracy-Ventura, Wa-Mbaleka, Norris and Ortega2006; Mackey & Goo, Reference Mackey, Goo and Mackey2007), the impact of attitude and motivation (Masgoret & Gardner, Reference Masgoret and Gardner2003), and visual/textual input enhancement (Lee & Huang, Reference Lee and Huang2008) on language learning.

FUTURE RESEARCH DIRECTIONS

The systematic coding and analysis of reactivity studies that was conducted as part of the meta-analysis (Bowles, Reference Bowles2010) revealed several areas that, despite their theoretical importance, have been underinvestigated. At the same time, the meta-analysis highlighted the role that several previously untested variables unique to the context of language research may have on whether thinking aloud is or is not reactive.

Needed are studies such as Polio and Chiu (Reference Polio and Chiu2007) that examine the effect of using just one language (either L1 or L2) in thinking aloud. This is because most of the studies conducted so far have examined the reactive effects of think-alouds when participants were allowed to select either the L1 or the L2, or to use a combination of the two languages, in their reports. Certainly, this is a gap in the research that should be filled.

Similarly, some task types have been investigated more heavily than others with regard to reactivity. Topping the list are studies that have examined the reactivity of think-alouds on reading (n = 6) and grammar learning tasks (n = 7). But the effect of think-alouds on the writing process has only been investigated in two unique sample studies, making it untenable to draw firm conclusions about reactivity with this task type.

More research is clearly needed overall in examining reactivity in conjunction with verbal tasks, given the paucity of reactivity research with language tasks as compared to nonverbal and problem-solving tasks. Empirical studies are needed to test the sources of between-study variance identified in this meta-analysis to determine to what extent they affect reactivity. Analyses to identify the sources of between-study variance revealed that a portion of the variance could be attributed to expected sources, such as type of report (metacognitive vs. nonmetacognitive), which has been discussed at length in the psychology literature (Ericsson & Simon, Reference Ericsson and Simon1993). The other sources of between-study variance that were identified, L2 proficiency and explicitness of instruction, are unique to the language research context. Future research is clearly needed to determine how verbalizing affects learners (a) at different proficiency levels and (b) in conjunction with more and less explicit instruction.

Although reactivity has been the focus of the majority of research on think-alouds, both in psychology and in SLA, the issue of veridicality should not be forgotten. In research studies that use think-alouds as a data elicitation tool, once the think-aloud group has been shown to perform statistically similarly to a silent control group, it is also important to examine the issue of veridicality. The small number of studies in psychology that have investigated veridicality have demonstrated that concurrent verbal reports tend to be more accurate (Robinson, Reference Robinson2001) and to include more information (Kuusela & Paul, Reference Kuusela and Paul2000) than retrospective reports. In SLA, only Polio and Chiu (Reference Polio and Chiu2007) have addressed veridicality directly, using a within-subjects design to examine concurrent and retrospective reports of the L2 writing process. In the study, advanced ESL learners completed concurrent verbal reports while writing and were then asked to provide a retrospective report (stimulated recall). Results showed that learners were indeed able to provide additional information in the retrospective reports, indicating that although concurrent verbalization seemed to accurately reflect thought processes, they were somewhat incomplete. So in cases where think-alouds are shown not to have been reactive, researchers should next examine the question of veridicality to determine how accurate and complete the reporting was.