THE ISSUE OF INTERNAL VALIDITY IN SLA EMPIRICAL RESEARCH

The robustness of research design is what drives the amount of confidence one may place in the findings of any empirical study. Such robustness lies primarily in the study’s internal and external validity—that is, whether the findings faithfully reflect what the study set out to investigate (i.e., internal validity) and whether the findings can be extrapolated to the general population (i.e., external validity). In studies that empirically address cognitive processes such as learner attention and learner awareness, high internal validity means that (a) the independent variable has been well operationalized (i.e., the stage of operationalization is appropriate), (b) the measure is sensitive enough to provide either empirical evidence or data to establish that the independent variable did play a role during the experimental phase and thus clearly contributed to the results, (c) most variables have been controlled in the study, and (d) appropriate statistical analyses were employed.^{Footnote 2} In contrast, low internal validity indicates that the role of the independent variable was not well established, the measure might have been insensitive, inappropriate statistical analyses were employed, or there might have been other uncontrolled variables involved in and contributing to the interpretation of the data, all of which lead to lower confidence in the findings. The foundation of any empirical study is grounded in the robustness of its research design, and, consequently, any critique must first address research design. Leung and Williams’s (Reference Leung and Williams2011) internal validity is suspect mainly due to three major limitations—namely, (a) the stage of operationalization and measurement of the construct of awareness, (b) the provision of implicit feedback during the treatment phase, and (c) the additional exposure to trained items during the postexposure phase of the study.

OPERATIONALIZING AND MEASURING THE CONSTRUCT OF (UN)AWARENESS IN LEUNG AND WILLIAMS (Reference Leung and Williams2011)

In terms of methodology, Leung and Williams (Reference Leung and Williams2011) employed an offline or nonconcurrent operationalization and measurement of the construct of awareness typically employed in non-SLA fields—such as cognitive psychology—where participants’ awareness is often viewed as implicit or explicit knowledge dependent upon whether they can verbalize (partially or fully) the targeted underlying rule after the experimental treatment. In other words, awareness appears to be associated with some form of explicit, declarative L2 knowledge of the rules associated with what is being learned. This operationalization is quite similar to Schmidt’s (Reference Schmidt1990) notion of awareness at the level of understanding, which has been operationalized in the SLA literature as a learner’s ability to test hypotheses and form underlying rules, which, in turn, leads to deeper learning marked by restructuring and system learning (e.g., Leow, Reference Leow2001; Rosa & Leow, Reference Rosa and Leow2004; Rosa & O’Neill, Reference Rosa and O’Neill1999; Sachs & Suh, Reference Sachs, Suh and Mackey2007). After exposure to the experimental data, learners unable to verbalize any partial aspect of the targeted underlying rules are coded as unaware, and implicit learning is reported to have taken place; that is, the learners are considered to have demonstrated no instance of awareness during the process of learning.

Leung and Williams’s (Reference Leung and Williams2011) operationalization and measurement of awareness as a dichotomous construct (i.e., aware vs. unaware) that occurs at only one level runs counter to several published studies in SLA (e.g., Leow, Reference Leow2001; Rosa & Leow, Reference Rosa and Leow2004; Rosa & O’Neill, Reference Rosa and O’Neill1999; Sachs & Suh, Reference Sachs, Suh and Mackey2007) that have reported additional levels of awareness below the level of understanding (i.e., the one used to operationalize and measure awareness in Leung and Williams’s study)—namely, awareness at the level of noticing and awareness at the level of reporting. Additionally, as reported in Hama and Leow (Reference Hama and Leow2010), awareness can also be viewed as occurring on a continuum that can encompass (a) no awareness ￫ awareness, or even (b) no awareness ￫ awareness ￫ no awareness. Failure to methodologically control for the potential presence of any level of awareness below that of awareness at the level of understanding reduces the internal validity of the study given that no empirically supported statement can be made regarding the presence or absence of these lower levels of awareness during the learning process.

Operationalizing and measuring (un)awareness at this nonconcurrent stage also portrays learning as a product instead of a process and presents several potential methodological problems that include (a) the issue of veridicality or memory decay (see Ericsson & Simon, Reference Ericsson and Simon1993; Leow, Reference Leow2000; Leow & Morgan-Short, Reference Leow and Morgan-Short2004), (b) a mismatch between the actual knowledge employed by the learner to process the L2 data and what is being sought in the offline questionnaire or verbal report^{Footnote 3} (see Shanks & St. John, Reference Shanks and St. John1994), and (c) differential test sensitivity or the withholding of knowledge of which learners may not be very confident to report (see Berry & Dienes, Reference Berry and Dienes1993; Shanks & St. John, Reference Shanks and St. John1994). It is interesting to note that this type of operationalization and measurement, which only addresses awareness at the level of understanding at a postexposure stage, was noted as a limitation in the very study that Leung and Williams extended—namely, Williams’s (Reference Williams2005) study in which the author states, “Clearly, a finer-grained approach to assessing learner awareness would have been preferable, ideally utilizing think-aloud procedures during the training and test tasks” (p. 294).

Leung and Williams (Reference Leung and Williams2011) also employed a concurrent measure—namely, the serial reaction time (SRT) methodology (discussed in more detail in the section Findings from Non-SLA Fields)—to study implicit learning and its effects. Ignoring the fact that this measure is premised on the classification (± awareness) derived from the offline verbal report measure of awareness with its inherent limitations, the prediction that any violation of the mapping between articles and thematic roles should result in slowed response times and an increase in errors for both aware and unaware participants does not appear to be fulfilled in the study. According to the results, the unaware participants demonstrated a significant increase in response time but no significant difference in errors, whereas the aware participants did not demonstrate a significant increase in either response time or errors.^{Footnote 4} However, one plausible interpretation of the results could be that no statistical increase in errors was produced because participants were accessing explicit knowledge instead of implicit knowledge (but see also discussion of mean response times) of the targeted mapping. Given that all the plausible explanations provided by the authors to interpret these findings are assumptions that lack the hard evidence needed to support them, the level of internal validity of the study may be lowered.

Failure to operationalize and measure a thorny construct such as unawareness adequately—without concurrent data on learner processes elicited via process measures (e.g., concurrent think aloud protocols)^{Footnote 5}—logically lends itself to the many assumptions and suppositions made with respect to these very processes, and, ultimately, to the role of unawareness in L2 learning, which may not be valid. Only a few exemplars taken from Leung and Williams (Reference Leung and Williams2011) are provided:

• • “Assuming a close link between language forms and attentional processes, changes in attentional processes (i.e., referent identification) are used over time as indicators of learning” (p. 39).

• • “Therefore, it is assumed that the time pressure induced by the reaction time test reduces the likelihood that participants will become aware of the target regularity (the mapping between articles and thematic roles) and encourages reliance on implicit knowledge” (p. 40).

• • “In this case, participants may have reacted after hearing further cues that did not come until after the proper name” (p. 42).

• • “By doing so, it was hoped that the form and meaning of the articles would be activated concurrently when the participants tried to memorize and recall the sentences” (p. 45).

• • “Twenty out of the 25 participants in the experiment claimed to have no awareness of the mapping between the articles and thematic roles.” (p. 48)

• • “It is likely that these participants began forming and testing hypotheses and this caused a delay in their responses” (p. 49).

PROVISION OF IMPLICIT FEEDBACK DURING EXPERIMENTAL TREATMENT

It is quite worrisome that a study that purports to investigate the construct of unawareness allows implicit feedback to be provided during the experimental treatment but does not appear to have controlled for its potential effect: “If participants gave a wrong answer, the page would reload and they would have to start the audio presentation and reaction time test again” (Leung & Williams, Reference Leung and Williams2011, p. 44). Even though reaction times for second and further attempts were excluded from the data analysis, any type or amount of feedback—be it explicit or implicit—holds the potential for some level of awareness to be raised that something is incorrect and that, perhaps, there is a need for more careful consideration or processing. Several SLA studies that have provided implicit feedback and gathered concurrent data during exposure have reported the potential for deeper processing of the incorrect data and noticing of subsequent targeted forms and structures (e.g., Leow, Reference Leow2001; Rosa & Leow, Reference Rosa and Leow2004; Rosa & O’Neill, Reference Rosa and O’Neill1999). The pertinent methodological question here is: If the learners received implicit feedback that a response was incorrect, what effect would this potentially have on (a) the subsequent processing of the additional incoming data and (b) raising awareness of the additional incoming data? Given the fact that Leung and Williams do not provide any report of whether or not they controlled or accounted for this confounding variable, the issue of whether some level of awareness below that at the level of understanding was promoted due to this provision of implicit feedback reduces the internal validity of the study.^{Footnote 6}

ADDITIONAL EXPOSURE TO TRAINING ITEMS

The exposure to about 10 or fewer of the training items—allocated to measure participants’ unawareness of the targeted mapping during the postexposure phase—might have also undermined the internal validity of the study. This attempt to address the role of awareness during concurrent processing at a later stage after exposure appears quite similar to the nonconcurrent measure of stimulated recalls that also suffer from the issues of veridicality or memory decay and double exposure to the targeted input data. Additionally, Leung and Williams (Reference Leung and Williams2011) do not report whether they controlled for the possibility of participants demonstrating awareness of the mapping during this postexposure stage of 10 or fewer of the training items.^{Footnote 7} In both Williams (Reference Williams2005) and Hama and Leow (Reference Hama and Leow2010) there were reported cases of participants who demonstrated awareness during the testing phase after exposure. If not controlled, the question thus remains as to whether some participants were potentially assigned to the unaware group on the basis of incorrect coding, which might have had an important impact on the level of internal validity of the study.

On the basis of these points, we argue that the offline operationalization and measurement of awareness employed in Leung and Williams (Reference Leung and Williams2011), the provision of implicit feedback during the experimental phase, and the additional exposure to the training items after the experimental phase lower the internal validity of this study and, ultimately, the level of confidence in their findings with respect to whether learning was achieved without awareness.

FINDINGS FROM NON-SLA FIELDS TO MOTIVATE LEUNG AND WILLIAMS’S (Reference Leung and Williams2011) NOVEL METHODOLOGY

Leung and Williams (Reference Leung and Williams2011) correctly caution that empirical findings of implicit learning in non-SLA fields such as cognitive psychology should not be generalized to the SLA field: “However, caution should be exercised before generalizing this contention to SLA because language systems, at the core of which lies meaning, are far more sophisticated than simple form-form mappings” (p. 34; but see also similar cautions in Leow, Reference Leow2000, and Simard & Wong, Reference Simard and Wong2001). Given that experimental conditions, data, measures, and tasks may not be appropriate to extrapolate to the SLA field and to the process of language learning in particular, it is quite surprising that the authors appear to rely on such non-SLA studies to support their novel methodology.^{Footnote 8}

To motivate their novel methodology employed to improve sensitivity to learning effects when compared to previous studies—namely, Hama and Leow (Reference Hama and Leow2010) and Williams (Reference Williams, VanPatten, Williams, Rott and Overstreet2004, Reference Williams2005)—Leung and Williams (2011) report the following regarding their methodology:

To support the use of the SRT task in this study, Leung and Williams cite findings from studies that addressed sequence learning (e.g., Destrebecqz & Cleeremans, Reference Destrebecqz and Cleeremans2001; Jiménez, Vaquero, & Lupianez, Reference Jiménez, Vaquero and Lupianez2006). Furthermore, to justify the use of this task for SLA, they cite R. Ellis’s (Reference Ellis2005) finding that timed versus untimed grammaticality judgment tests performed offline “depended on the learners’ implicit and explicit knowledge, respectively” (Leung & Williams, Reference Leung and Williams2011, p. 40). Citing studies from the vision world and sequence learning paradigms together with assumptions made on the basis of supposed time pressures reported in one SLA study as empirical support for the use of SRT in relation to the construct of unawareness in language learning clearly does not contribute much to the robustness of this measure in a SLA context.

According to the authors, there are three improvements in the methodology employed in their study to address implicit learning and the effects of such learning in SLA. The first improvement is the use of a concurrent experimental and testing task to measure learning. In our opinion, the use of a concurrent testing task is highly recommended to address concurrent performance. However, regarding type of learned data, what specifically do Leung and Williams (Reference Leung and Williams2011) measure? The experimental training and testing task were based on participants’ assumed comprehension of a mapping between thematic roles and a set of novel determiners, or, more specifically, selecting as quickly as possible, one out of two options, which, in our opinion, still provides the opportunity to guess at a 50% rate without any recourse to so-called implicit or explicit knowledge.^{Footnote 9} What appears to be purported is that while performing this task, participants implicitly learned the targeted mapping based on the accuracy of selecting the correct option. However, given that such implicit learning is further based on the robustness of the operationalization and measurement of the construct of awareness, which was offline, it is quite a challenge to claim that participants learned without awareness the targeted mapping while performing this receptive task. The researchers’ statement that their study “has only shown that implicit learning influences a receptive task” (p. 52) is clearly premised on having convincingly provided evidence of implicit learning being achieved in this study. In our opinion, the experimental task employed in this study measured participants’ ability to select one out of two options that included the target connection due to the stage in which data were elicited along the learning process. Assumed in this ability was access to some type of knowledge gained during the training phase to make a form-meaning connection. However, any reference to implicit learning effects is undermined by an inadequate awareness measure that does not find much support in SLA or non-SLA fields due to the methodological limitations previously listed.

The second improvement, according to Leung and Williams (Reference Leung and Williams2011), is the choice to address comprehension given that “knowledge has to reach a higher threshold to be expressed in production, the lesser processing demand in our task makes it more sensitive to implicit knowledge” (p. 39). Leaving aside, once again, the inadequacy of the awareness measure employed in this study, this argument appears arbitrary due to the absence of any strong empirical support for this statement. More specifically, to what does a “higher threshold” of knowledge refer? Although the authors refer to “a varying degree of explicit knowledge” (p. 50), this reference appears to conflate levels of awareness with levels of knowledge.

Finally, the third improvement deals with sensitivity and hinges on the issue of time pressure, which is premised on the assumption that such pressure promotes the activation of implicit rather than explicit knowledge (but see R. Ellis, Reference Ellis2005).^{Footnote 10} In other words, one logical assumption would be that mean response times for unaware learners, when compared to those of aware learners, should be statistically lower if the unaware participants were accessing implicit knowledge (faster processing) and not explicit knowledge (slower processing). Leung and Williams’s (Reference Leung and Williams2011) participants failed to demonstrate this differential performance, and, more interestingly, their unaware learners are reported to have spent more time processing the data when compared to the aware participants (i.e., 1,694 ms vs. 1,592 ms, respectively). These results do not appear to lend empirical support to the claim that time pressure plays a role in accessing type of knowledge, and more robust empirical support is clearly needed in the SLA literature. Additionally, no empirically supported criteria are provided to indicate the amount of time that is required to force learners to access implicit knowledge, or, as Leung and Williams assume, to encourage “reliance on implicit knowledge” (p. 40). An average of 1.5 s spent processing the data appears to be a relatively large amount of time to make any claims about accessing implicit knowledge. Furthermore, the issue of individual differences is not discussed as one important factor in learners’ actual performance on a SRT task, especially with relatively small numbers of participants. Given that the number of the aware learners was only five, individual differences could have played a much more significant role when compared to unaware learners (n = 20), which leads us to doubt any differential SRT pattern demonstrated by these two groups.

As such, Leung and Williams’s argument that their new methodology “makes the current experiment potentially more sensitive to implicit learning effects than earlier studies like Williams (Reference Williams, VanPatten, Williams, Rott and Overstreet2004, Reference Williams2005) and Hama and Leow (Reference Hama and Leow2010)” (p. 39) is moot until a more robust research design with a higher level of internal validity and a more appropriate and empirically supported measurement of awareness is employed to address the role of awareness in implicit learning and its effects in L2 acquisition. We do, however, completely agree with Leung and Williams’s statement that an “important issue in all implicit learning research is whether the tests of awareness are sensitive enough” (p. 36). Nevertheless, it is important to note that sensitivity of testing for learning effects appears to be conflated in this study with the sensitivity of testing the construct of awareness. Therefore, we strongly recommend that these tests of sensitivity designed to measure both the constructs of unawareness and learning cannot be divorced from an acceptable level of internal validity.