COMPARISONS AGAINST CHANCE

Several studies employing ALSs in SLA have used statistical chance as the primary baseline against which to assess learning (e.g., Hama & Leow, Reference Hama and Leow2010; Hamrick & Rebuschat, Reference Hamrick, Rebuschat, Rebuschat and Williams2012; Morgan-Short et al., Reference Morgan-Short, Steinhauer, Sanz and Ullman2012; Williams, Reference Williams2005). The logic is essentially this: Participants have never been exposed to the ALS before, so if they do not learn anything during training, then they should be guessing at test. Guess-based performance should be equivalent to a random selection of answers (i.e., chance), which can be calculated by dividing the number of correct answers by the total number of possible answers. For instance, in a typical four-alternative multiple-choice test, chance is considered to be 25% accuracy. If participants perform above that baseline to a statistically significant degree, they are considered to have developed target knowledge.

Is this assumption justifiable? One way of exploring this empirically is to examine the results of experiments that have employed both chance and human control groups as comparisons. In some of these studies, control participants who were not exposed to the training have performed at chance overall (Hamrick, Reference Hamrick2014a, Reference Hamrick2014b; Rebuschat & Williams, Reference Rebuschat and Williams2012, Exp. 1), enabling the researchers to argue that chance had been validated as a baseline for their particular experiments. However, in very similar studies, control participants have been found to deviate from chance overall (Rebuschat & Williams, Reference Rebuschat and Williams2012, Exp. 2) or on specific items (Hamrick, Reference Hamrick2014a; Rebuschat & Williams, Reference Rebuschat and Williams2012; Rebuschat et al., Reference Rebuschat, Hamrick, Riestenberg, Sachs and Ziegler2015), making it untenable to consider the corresponding experimental groups’ above-chance performance alone as evidence of target learning. Extrapolating out to the area of research as a whole, without a significant body of work establishing the precise circumstances under which participants will simply guess and perform at chance in the absence of training/learning, repeated findings of nonchance performance in human controls underscore that it may be invalid, and therefore irresponsible, to employ chance as the sole baseline without direct experiment-specific evidence to support its use.

Some of the risks associated with using chance as a baseline derive from two assumptions that are likely to be unwarranted in the context of ALS (or L2) learning. The first is that an equivalent group of human control participants would have no systematic biases that might cause them to deviate from chance in the absence of training/learning. This first assumption has two interpretations: (a) participants have no (relevant) systematic biases, or (b) any biases they have are heterogeneous and will wash one another out across items or across participants, leaving overall performance at chance. The second assumption is that participants will not learn any aspect of the linguistic target during the test phase. Importantly, if only comparisons against chance are made in an experiment, there are few ways of checking these assumptions, and (as we discuss in the recommendations section) methods of doing so can be resource intensive.

In relation to the first assumption, it is widely agreed that humans are not bias-free blank slates at birth, much less in adulthood. But how likely is it that their biases will be relevant in ALS experiments? Research conducted within a variety of frameworks, from Competition Model studies (MacWhinney, Bates, & Kliegl, Reference MacWhinney, Bates and Kliegl1984) to statistical learning experiments (Onnis & Thiessen, Reference Onnis and Thiessen2013), has demonstrated that adults’ L1s give them consistent preferences for using particular cues to parse novel linguistic input. For instance, MacWhinney et al. (Reference MacWhinney, Bates and Kliegl1984) found that L1-English speakers systematically relied on word order to identify the actor in English sentences with modified syntax (e.g., NNV: the pig the eraser chases; VNN: chases the pig the eraser), whereas German and Italian speakers relied on animacy and/or agreement cues to interpret sentences with equivalent modifications in their L1s. Considering that ALS experiments often utilize stimuli similar to those of MacWhinney et al., such findings suggest that researchers cannot assume unbiased guessing would occur in the absence of training/learning.^{Footnote 2} Without empirically established, robust, converging evidence of directly relevant processing phenomena in prior research or exact replications sampled from the same population, it may not be possible to predict how human participants would behave, even if researchers are armed with detailed knowledge of how L1 speakers tend to process a language in its unmanipulated form.

To elaborate on the consequences of this problem, if participants’ language-related biases correlate (positively or negatively) with an intended linguistic target, they may perform above or below chance even in the absence of training/learning, making chance meaningless as a sole baseline for establishing evidence of learning. One illustration of this comes from Rebuschat’s (Reference Rebuschat2008; Rebuschat & Williams, Reference Rebuschat and Williams2012) investigation of the implicit learning of syntax in a semi-artificial language consisting of L1-English words arranged in German syntactic structures (e.g., After the instructor a sword brandished, focused Brian more on his defensive stance). In the study’s exposure phase, without having been notified of the rule-governed nature of the stimuli, experimental participants listened to sentences in the ALS and judged whether they made sense; then, a surprise posttest asked them to judge whether new sentences seemed to follow the same rules. Their performance was significantly above chance, suggesting learning. However, by including a control group that was not exposed to any training materials, Rebuschat was able to discover unexpected systematicity in control participants’ responses that could not have been learned during the experiment. The untrained controls’ test performance sometimes departed significantly from chance, and the experimental group sometimes did not differ from the untrained controls, begging the question: What if some of the systematicity in experimental participants’ responses was also due to factors independent of the training? Findings such as these highlight some of the difficulties involved in interpreting apparent learning effects in ALS experiments (see also Reber & Perruchet, Reference Reber and Perruchet2003, p. 113) and raise red flags about the use of chance as the sole baseline in other studies.

Regarding interpretation (b) of the “no systematic bias” assumption, while we agree it is possible that variations in participants’ response biases may cancel one another out, we would argue that it is, at this point, an empirical question requiring direct investigation within the contexts of particular studies. Considering, moreover, that SLA researchers tend to collect data from rather homogeneous samples (Plonsky, Reference Plonsky2015), and that homogeneous samples are more likely to have internal correlations manifesting as systematic group-level behavior (Field, Reference Field2009), it seems reasonable to expect participants to show at least some similar response tendencies independently of training/learning, thereby making chance an inappropriate baseline. Additional examples of this (from Hamrick, Reference Hamrick2013, Reference Hamrick2014a; Rebuschat et al., Reference Rebuschat, Hamrick, Riestenberg, Sachs and Ziegler2015) will be presented in the following text in the sections on untrained and trained controls.

Finally, the other unwarranted assumption underlying the use of chance as a baseline is that participants will not learn any aspect of a linguistic target during the test phase. The fact that participants have never been exposed to a target previously does not exempt ALS research from this potential problem, as noted by Redington and Chater (Reference Redington and Chater1996) in the AG literature some 20 years ago. Human controls in ALS experiments do not always show evidence of learning during the test phase (e.g., Grey et al., 2014; Tagarelli et al., Reference Tagarelli, Borges Mota, Rebuschat, Zhisheng, Borges Mota and McNeill2015), but sometimes they do—as, for example, when features of a posttest unexpectedly spur participants in all groups to search for rules (Rebuschat et al., Reference Rebuschat, Hamrick, Riestenberg, Sachs and Ziegler2015). The critical point is that if researchers are not able to say for certain that participants’ response tendencies are the product of exposure to the training materials (and may be due instead to reactivity during a test, for instance), they cannot be taken to reflect learning in the study (Perruchet & Reber, Reference Perruchet and Reber2003, p. 127).

In sum, using chance as a baseline entails unwarranted assumptions that may invalidate claims about learning in an experimental group. Untrained controls may help to address some risks associated with these assumptions; however, they come with their own limitations, as we discuss next.

UNTRAINED CONTROL GROUPS

In SLA research, participants who are tested without having been exposed to training materials might be referred to as a maturation or testing control group depending on the nature of the study. Within the context of ALS experiments, we call such a group an untrained control group to contrast it with the type of trained control group we discuss in the next section. As illustrated in the preceding text, untrained controls offer certain advantages over the use of chance as a baseline: By allowing researchers to check whether preexisting biases or learning during a test might account for (at least some) systematic response tendencies underlying nonchance performance, they enable better-informed evaluations of whether apparent evidence of learning in an experimental group is genuine. However, the method is not fail-safe. For one thing, as we shall see with trained controls, learning during a test may be influenced by what participants have learned (or fabricated) during training, or even by the simple fact of having experienced a training session. For another, the extent to which untrained controls’ response patterns can be interpreted as reflecting what has likely also happened in an experimental group may depend crucially on whether the researcher has been able to establish similar expectations, motivations, and understandings of the task across groups.

Why might participants’ mind-sets differ across groups? To recap, in research employing ALSs, the goal is to determine whether participants develop new response tendencies based on exposure to training materials containing a linguistic target. The test instructions are then supposed to act as a prompt to rely on these newly acquired tendencies, and such behavior is counted as learning. Logistically, though, untrained controls can be problematic because instructions that make sense for experimental participants may seem decontextualized or may even be uninterpretable for control participants who have not experienced a training phase. The instructions may therefore have to be modified, and this can cause the control and experimental conditions to differ in additional ways beyond exposure to the training. If the groups have different expectations or understandings of the nature of the test, then performance differences between them cannot be said to be exclusively due to the experimental group’s learning of a target that was present in the training, and the validity of using the controls as a comparative baseline diminishes.

Rebuschat’s (Reference Rebuschat2008) aforementioned semi-artificial language study again provides an illustrative example. His inclusion of an untrained control group represented an improvement over using chance as the sole baseline. However, while the experimental participants were told to base their grammaticality judgments on what they had heard in the training phase, the untrained controls, who had not completed the training, could only be asked to perform a grammaticality judgment test on what may have seemed to be scrambled English sentences. Thus, while the experimental group may have known to reject anything too similar to standard English simply by virtue of having heard scrambled sentences earlier and regardless of what they had or had not learned during the training, the controls may have been inclined to answer straightforwardly according to their knowledge of standard English. Notably, some of the “ungrammatical” test sentences in that study actually corresponded to normal English sentences (e.g., Some time ago John filled the bucket with apples), and the untrained controls had extremely high endorsement rates for such sentences in three of the four relevant experiments (Experiments 3, 4, and 6). On these types of test items, it is therefore unclear whether the experimental group can truly be said to have acquired the targeted grammaticality knowledge, as Rebuschat also concluded at the time. An alternative explanation may be that different mind-sets led the groups to judge some sentence types, at least in part, according to their similarity to standard English, producing correct answers for the experimental group and incorrect answers for the control group.

A recent series of experiments by Rogers, Révész, and Rebuschat (Reference Rogers, Révész, Rebuschat and Rebuschat2015) was able to avoid some of these problems. Their study also employed sentences with English lexis and scrambled word orders (e.g., Last month the kasu opened Patrick with the key); however, the target was not syntax, but rather a system of morphological markings on novel (Czech) vocabulary items that were used to replace certain English words. Given that the ALS modified English in multiple ways, and given the novelty of the words to which the morphemes were attached, it may be less likely that the experimental and control groups interpreted the instructions differently and had different expectations about the target morphemes. This does not mean that the groups could be assumed to have the same mind-sets in general, however; after all, by the time the experimental participants were tested, they had already been engaged in processing sentences with novel word orders, whereas for the controls who experienced only the test phase, the novelty of the syntax may have been salient, drawing attention away from morphology. Nonetheless, the instructions, at least, were similarly interpretable across groups.

Achieving similar mind-sets across groups may be an easier task in SLA research where all participants are engaged in learning the target language outside the experiment. In such studies, because all participants already construe themselves as learners of the target language, test instructions are likely to make sense regardless of whether participants have been trained in the experiment. However, even in those cases, interactions with affect and motivation might reduce the comparability of experimental and control groups (Sachs & Weger, Reference Sachs and Weger2011). For instance, if experimental participants are eager to demonstrate what they have learned while controls are confused, frustrated, or less motivated to exert effort on a test due to a perceived lack of helpful training, the validity of the comparison is diminished.

Situations such as these highlight that ALS researchers may need to take extra steps to improve the comparability, and therefore the validity, of control groups by devising creative ways of promoting similar expectations among all participants, while also designing linguistic targets, stimuli, instructions, and test sentences to avoid issues of the sort discussed in the preceding text. While the use of untrained controls represents an improvement over chance as a baseline, researchers should be cognizant of their limitations and consider the ways in which a trained control group may offer greater comparability.

TRAINED CONTROL GROUPS

The use of trained controls has precedent in research using language-like stimuli devoid of semantics (e.g., Onnis, Waterfall, & Edelman, Reference Onnis, Waterfall and Edelman2008) but is a more recent development in research employing ALSs (Hamrick, Reference Hamrick2012, Reference Hamrick2013, Reference Hamrick2014a, Reference Hamrick2014b; Rebuschat et al., Reference Rebuschat, Hamrick, Sachs, Riestenberg, Ziegler, Bergsleithner, Frota and Yoshioka2013, Reference Rebuschat, Hamrick, Riestenberg, Sachs and Ziegler2015). Unlike instructed SLA research, where comparison groups are often exposed to the same linguistic input under different instructional conditions, experiments with ALSs expose experimental and trained control participants to training conditions, materials, instructions, and test items that are identical, except that relevant linguistic features have been pseudorandomized in a frequency-balanced way in the trained control group’s training materials so as to eliminate the systematic syntactic patterns or form-meaning connections that constitute the target regularity for the experimental group. That is, trained controls are exposed to (a) nontarget aspects of the stimuli that replicate exactly what is seen by the experimental group, and (b) would-be target aspects of the stimuli that purposeful randomization has rendered not only uninformative (and therefore impossible to learn), but also protected from unintended additional similarities with real languages due to their lack of systematicity where the target is concerned. This avoids some of the limitations of untrained controls by allowing researchers to give all groups the same instructions as well as equivalent opportunities to learn nontarget regularities (or fabricate them) during the training phase.

Research employing AGs/ALSs has repeatedly demonstrated that it is not only possible but indeed common for experimental participants to contrive nontarget patterns or internalize other unintended but real (e.g., partial) patterns during training. The development of nontarget “knowledge” sometimes makes no difference in participants’ test scores (e.g., Hamrick, Reference Hamrick2013), but it can sometimes boost (e.g., Hamrick, Reference Hamrick2014a; Knowlton & Squire, Reference Knowlton and Squire1996) or harm their performance, such as when participants devise misguided rules of thumb that they reportedly use to override their intuitions (e.g., Rebuschat et al., Reference Rebuschat, Hamrick, Riestenberg, Sachs and Ziegler2015). To illustrate, consider Hamrick’s (Reference Hamrick2013) experiments designed to investigate incidental learning of novel syntactic structures. Experimental participants were exposed to a semi-artificial language with English phrases placed into three structures derived from Persian (e.g., Yesterday Charlie at the supermarket milk bought; Yesterday Charlie milk at the supermarket bought; Yesterday bought Charlie at the supermarket milk), then given a surprise posttest. Although the experimental participants’ level of performance at test suggested target learning, subsequent analysis of their postexperimental verbal reports indicated that many focused specifically on verb finality without learning other targeted aspects of the syntax. Because two-thirds of the training structures and grammatical test sentences happened to be verb-final, a tendency to endorse verb-final test items led to high levels of performance regardless of whether experimental participants had learned any other aspects of the system. Thus, Hamrick may have found some evidence of learning fragmentary, partial patterns, but not of learning the whole linguistic target as defined by the researcher.

Fortunately, by including a trained control group, Hamrick was able to discover an apparently independent inclination for participants to focus on verb finality during training irrespective of whether the regularity actually existed in the training materials. Trained control participants were exposed to sentences containing the same words, phrases, and meanings as those shown to the experimental participants, but the orders of the phrases were pseudorandomized in a frequency-balanced way to remove any phrase-order patterns. In other words, the trained controls saw all possible orders (including ungrammatical ones) an equal number of times, and verbs occurred in all positions with equal frequency. Despite these facts, postexperimental verbal reports indicated that trained controls were (overly) sensitive to instances of sentence-final verb placement, which may have been more salient due to a general human bias toward stimulus edges (Endress, Carden, Versace, & Hauser, Reference Endress, Carden, Versace and Hauser2010), combined with the novelty (for L1-English speakers) of occasionally seeing a verb at the end of the sentence. Whatever the reason for the attentional bias, the results indicated that preexisting proclivities can influence test response patterns in a way that suggests learning even when learning was not possible due to the uninformative (randomized) nature of the training materials. Here again, the use of an additional control group simultaneously illuminated and complicated interpretations of the experimental group’s results. Such insights would have been missed if Hamrick had compared the experimental group’s performance against only chance or untrained controls’ performance.

The possibility that participants will react unexpectedly to features of training stimuli can involve semantics as well as grammar. In Rebuschat et al.’s (Reference Rebuschat, Hamrick, Sachs, Riestenberg, Ziegler, Bergsleithner, Frota and Yoshioka2013, Reference Rebuschat, Hamrick, Riestenberg, Sachs and Ziegler2015) replications of Williams’s (Reference Williams2005) study on implicit learning of form-meaning (determiner-animacy) relationships, for instance, there was an unexpected tendency for both experimental and trained control participants to formulate hypotheses about the relevance of animacy in determiner use. Importantly, this was despite the fact that an animacy regularity was present only in the experimental group’s training materials; for the trained control group, each determiner was used an equal number of times with animate and inanimate referents. If there was no underlying pattern that could have directed trained control participants’ attention to animacy, then what prompted them to formulate such hypotheses? Concurrent and retrospective verbal reports indicated that participants in all groups had negative affective reactions to certain animate entities (e.g., rats, flies) in the training sentences. The inclusion of a trained control group along with triangulation from verbal reports allowed the researchers to discover that some of the experimental group’s success may have been due not to gradually emerging associations between determiners and animacy, per se, but to the noticing of exemplars that struck participants in both groups as salient and led to extrapolations independently of the existence (or learning) of any pattern. Taken together, these findings should motivate researchers to ask whether their design will allow them to be certain that experimental participants have learned the target from training, or whether there is a possibility that participants exposed to similar stimuli minus the regularity might fabricate or assume it irrespective of actual learning. The fact that the latter has been shown to occur in studies with trained controls means that extra caution is warranted in claiming evidence of learning in studies without this design feature.

In highlighting some advantages of trained control group designs, it is important to point out that they too suffer from several limitations. One, to be discussed in greater depth in the recommendations section, is that without additional process measures (e.g., think-aloud data), it may be difficult to identify the source of systematic response patterns that arise due to exposure independently of (opportunities for) learning. Another is analogous to the mind-set-comparability issues presented in relation to untrained controls. Within the AG literature, it has been demonstrated that, compared to untrained controls, trained controls’ performance is less affected by preexisting systematic biases (Reber & Perruchet, Reference Reber and Perruchet2003); however, the very few ALS experiments in SLA that have included trained controls have found at least some systematicity in participants’ responses following pseudorandomized training. Beyond biases of the sorts described in the preceding text, such patterns may also stem from a human tendency to seek coherence when faced with randomness. Indeed, some evidence suggests that exposure to unpatterned stimuli actually increases the likelihood that people will search for patterns and mistakenly report finding them (Whitson & Galinsky, Reference Whitson and Galinsky2009). As such, it is possible that exposing trained controls to pseudorandomized materials might trigger deeper, or more persistent, search processes than are seen in an experimental group. If randomness in the stimuli prompts trained controls to process the materials differently from experimental participants, then their value as a comparison group decreases.

Finally, researchers must bear in mind that scoring methods might unfairly penalize trained controls for learning from pseudorandomized training materials. In Rebuschat et al.’s (Reference Rebuschat, Hamrick, Sachs, Riestenberg, Ziegler, Bergsleithner, Frota and Yoshioka2013, Reference Rebuschat, Hamrick, Riestenberg, Sachs and Ziegler2015) experiments, for instance, we tested all participants on “memory” items that had appeared as exemplars in the experimental group’s training as well as on novel “generalization” items. In the interest of one kind of direct comparability, we scored all items consistently across groups according to what the experimental group was supposed to have learned (i.e., the target regularity), not according to the pseudorandomized exemplars the trained controls had seen. Results indicated that the trained controls performed significantly below chance on the memory items, possibly reflecting accurate recall of pseudorandomized training exemplars. Consequently, the finding that experimental participants performed significantly above chance and better than the trained controls on those items cannot, in itself, be interpreted as evidence of learning. To have confidence that apparent evidence of learning by experimental participants is genuine, assessments should, where applicable, give credit for exemplar learning according to what each group has actually seen. If this is done and experimental participants are still found to perform differently from trained controls on memory items, then their different performance might suggest contributions of other knowledge besides memory for exemplars, perhaps complementing information gained from using generalization items to test rule-based knowledge. In any case, researchers must consider whether their particular combination of pseudorandomized training materials, test format, and scoring method will count trained control participants’ answers as right or wrong for the appropriate reasons as far as internal validity is concerned.

SOME RECOMMENDATIONS FOR ESTABLISHING EVIDENCE OF LEARNING

To this point, we have focused primarily on elucidating some of the tacit assumptions and challenges associated with measuring learning in experiments employing ALSs. Fortunately, there are steps that researchers can take to address some of these challenges and minimize the extent to which experiments are based on unwarranted assumptions. There is no one-size-fits-all solution, and space limitations prevent extensive recommendations here, but we can offer some brief suggestions regarding the benefits of triangulation and the potential for more sophisticated statistical approaches to illuminate trends in the data. Essentially, we recommend that researchers seek as much information as feasible through additional controls and measures, then report the whole complex picture without cherry-picking and interpret results cautiously with the limitations we have reviewed in mind.

It is not uncommon for SLA researchers to recommend triangulation as a means of addressing the limitations of individual methods used in isolation. In light of the problems we have outlined concerning different baselines for establishing evidence of learning, we would strongly recommend that, when possible, researchers include multiple control groups to take advantage of their different benefits and gain a richer understanding of the kinds of learning that are (or are not) occurring in their studies. Because resources are often scarce, however, it may make sense to prioritize the inclusion of a trained control group. In combination with verbal reports, trained controls’ performance can often provide unanticipated and illuminating information about how participants (and their biases) interact with the training stimuli. In the event that trained controls’ performance is difficult to interpret (e.g., due to mind-set-related confounds or scoring problems), researchers may benefit from adding an untrained control group to allow for comparisons unaffected by these issues—bearing in mind, however, that untrained controls are subject to logistical issues and mind-set differences of other sorts.

One might argue that if researchers carefully consider potential biases in advance, they can avoid certain confounds and do not need to use resource-consuming methods to check for them directly. Indeed, it may be possible to predict certain types of biases, such as well-established L1-based processing tendencies. However, a wide variety of other sources of bias exist, from basic perceptual tendencies to knowledge gained in linguistics and foreign language classes. All of these may interact with features of an ALS in unexpected ways, even when the same ALS is used in multiple studies. For instance, Williams’s (Reference Williams2005) implicit learning study and a series of replications by Rebuschat et al. (Reference Rebuschat, Hamrick, Sachs, Riestenberg, Ziegler, Bergsleithner, Frota and Yoshioka2013, Reference Rebuschat, Hamrick, Riestenberg, Sachs and Ziegler2015) produced results that differed from replications by Hama and Leow (Reference Hama and Leow2010) and Faretta-Stutenberg and Morgan-Short (Reference Faretta-Stutenberg, Morgan-Short, Granena, Koeth, Lee-Ellis, Lukyanchenko, Prieto Botana and Rhoades2011). Notably, the former set of studies included linguistics students as participants whereas the latter purposely excluded them—a non-L1-related difference that may help to explain the differing results. In verbal reports, many of Rebuschat et al.’s (Reference Rebuschat, Hamrick, Riestenberg, Sachs and Ziegler2015) experimental and trained control participants demonstrated metalinguistic knowledge of phenomena such as semantic features and morphophonological alternations, which they mentioned had come to mind while processing the stimuli. Because concurrent verbalization (i.e., think-alouds) can be reactive (e.g., Rebuschat et al., Reference Rebuschat, Hamrick, Riestenberg, Sachs and Ziegler2015), it is crucial to include a silent control group as well when they are used. However, we have found the follow-up analyses they allow to be an invaluable source of insights well worth the additional time and effort. Without qualitative data on participants’ approaches to the tasks and reactions to certain aspects of the stimuli and test items, we might have come to very different conclusions or found certain results to be uninterpretable.

In addition to the triangulation of research methods and data-collection instruments, there are some statistical procedures that researchers may find informative. To illustrate, imagine a scenario similar to Hamrick’s (Reference Hamrick2014a) study, described earlier, where chance performance on a grammaticality judgment test is 50% and an experimental group performs significantly above chance (60%), as does a control group (57%), with no statistically significant difference between them. Limited to basic statistical analyses, a researcher might conclude that (a) no learning occurred in either group (because the controls were not exposed to the target regularity and therefore could not have learned it) or (b) the same pseudo-learning effect somehow occurred in both groups (perhaps during the test phase, or due to an unintentionally informative cue in the stimuli). However, there is a third possibility: namely, that the groups performed above chance for different reasons. Perhaps the experimental group did (partially) learn the target pattern, while the controls seized upon some other aspect of the stimuli that happened to be positively correlated with the target.

How can researchers explore these possibilities statistically? One option might be to use predictive multilevel models, which would allow researchers to analyze test results on a trial-by-trial basis. For instance, with Hamrick’s (Reference Hamrick2014a) data, test item number could be used as a predictor to assess whether learning occurred during the test phase and, if so, whether the amount differed across groups. Analyzing further, a researcher could check for interactions between potentially relevant test-item features (e.g., verb finality, grammaticality) and group membership, to determine whether there were different bases for the experimental and control groups’ performances (Baayen, Davidson, & Bates, Reference Baayen, Davidson and Bates2008). Importantly, though, because this would require the researcher to specify covariates of potential or known interest beforehand, it might be unhelpful in situations in which the problem is precisely that unexpected and unknown factors might be driving the groups’ performances. While such a statistical procedure offers potential solutions in principle, and while its use would be consistent with broader methodological reforms taking place in applied linguistics (Cunnings, Reference Cunnings2012), to our knowledge no one has utilized it for these purposes in SLA research employing ALSs. As such, further elaboration of these possibilities and specifications of appropriate procedures will be necessary to move the field forward along these lines.