3.1 N. C. Ellis & Sagarra (Reference Ellis and Sagarra2010a)

Ellis et al., (Reference Ellis, Hafeez, Martin, Chen, Boland and Sagarra2014), published in Studies in Second Language Acquisition, is one of a series of studies (e.g., Ellis Reference Ellis2006; Ellis & Sagarra Reference Ellis and Sagarra2010a, Reference Ellis and Sagarra2010b, Reference Ellis and Sagarra2011; Ellis et al. Reference Ellis, Hafeez, Martin, Chen, Boland and Sagarra2014) that have provided an alternative explanation for the apparent limited attainment (when compared to near native-like competence) typically evidenced in adult L2 acquisition. This perspective explores adults’ difficulty in acquiring L2s in terms of ‘cognitive principles of transfer in the associative learning of form-meaning relations in linguistic constructions’ (Ellis & Sagarra Reference Ellis and Sagarra2010a: 554), a perspective that includes the notions of transfer and learned attention observed in several theoretical underpinnings, for example, the competition model (MacWhinney Reference MacWhinney and Robinson2001), the input processing model (VanPatten Reference VanPatten and VanPatten2004), and the associative-cognitive CREED (i.e., that SLA is construction-based, rational, exemplar-driven, emergent, and dialectic, see Ellis Reference Ellis2006, Reference Ellis, Robinson and Ellis2008). Given adult L2 learners’ limited cognitive resources for processing all incoming L2 data, especially learning form-meaning connections that require much mental effort, learners have ‘to select which aspects of the input to process’ (Ellis & Sagarra Reference Ellis and Sagarra2010a: 554). According to the researchers, one factor in such selection is cue saliency. For example, recall the verbal inflection discussed above such as the past tense morpheme ‘ed’ in ‘walked’. Such inflections are less likely to be paid attention to when compared to, for example, temporal adverbs such as ‘yesterday’, which is quite pronounced in the speech stream. Viewed from an associative learning theory perspective, L2 learners’ prior knowledge of ‘yesterday’ as a cue for past action will block attention to the redundant verbal inflection -ed as a result of prior experience. While studies in this research strand all appear to provide external validity support for previous ones, there are several methodological issues in previous studies that need to be addressed, therefore a detailed report of the first experiment in Ellis & Sagarra (Reference Ellis and Sagarra2010a) is provided below.

Ellis & Sagarra (Reference Ellis and Sagarra2010a) recently explored the phenomenon of learned attention in two experiments that employed the standard blocking experimental paradigm (Kruschke Reference Kruschke2006) but with linguistic content. In the first experiment, conducted in a laboratory setting, the researchers investigated the attention to or, perhaps more appropriately, processing of L2 lexical and morphological cues to temporal reference for its short term effects. Fifty-four native speakers of English learned a small number of Latin expressions and their English translations. They were randomly assigned to one of three groups: adverb pre-training, control, and verb pre-training. In Phase 1, adverb pre-training participants learned two adverbs and their temporal reference, namely, hodie ‘today’ (present) and heri ‘yesterday’ (past); verb pre-training participants learned two verbs and their temporal reference, namely, cogito ‘I think’ (present) and cogitavi ‘I thought’ (past). Both these two groups received feedback indicating whether their selection was ‘correct’ or ‘wrong’, the latter accompanied by the correct response. The control group did not receive any pre-training. In Phase 2, all participants were visually shown on the computer screen six sentences (hodie cogito ‘today I think,’ cogito hodie ‘I think today,’ heri cogitavi ‘yesterday I thought,’ cogitavi heri ‘I thought yesterday,’ cras cogitabo ‘tomorrow I will think,’ and cogitabo cras ‘I will think tomorrow’), which combined the adverb with a verb – three in adverb-verb and three in verb-adverb word order – to counterbalance which cue was experienced first across sentences. They were requested to choose whether these sentences referred to the present, the past, or the future. As in Phase 1, participants were given feedback if incorrect. There were six blocks of these trials to consolidate learning.

In Phase 3 (the reception test), all combinations of adverb (hodie ‘today,’ heri ‘yesterday,’ cras ‘tomorrow’) and verb tense marking (cogito ‘I think,’ cogitavi ‘I thought,’ cogitabo ‘I will think’) were combined and participants were asked to judge, without feedback, whether each sentence referred to the past, present, or future on a scale of 1 (past) to 5 (future). No feedback was provided in this phase. Both word orders were tested, and the block was repeated twice. The average weight of the two cues within each block was then used to statistically address participants’ interpretation of the adverbial and inflectional cues to temporal reference.

Phase 4 was the production testing phase in which participants were asked to translate from English to Latin by typing the Latin equivalents of the various elements to which they had been exposed: I thought, I think, I will think, yesterday, today, tomorrow, yesterday I thought, today I think, or tomorrow I will think. There was no feedback on the responses.

To address participants’ receptive ability, three multiple regression analyses were performed, one for each group, in which the independent variables were the interpretations cued by both the adverbial cue and the verbal inflection, and the dependent variable was the group mean temporal interpretation for each of the 18 two-word strings. To address participants’ productive ability, a two-factor ANOVA (3 groups × 2 cues) was employed. Participants’ production was scored based on correct production of either adverb or verb and intermediate scores assigned to misspellings based on number of letters away from the correct spelling. The results revealed that the three groups’ interpretation and production of the cues were statistically different, providing evidence to support the notion of learned attention in L2 learning of morphological (i.e., verbal inflections) and lexical cues (i.e., adverbs) to temporal reference.

3.2 Approaches to replication

As can be seen, the findings reported in Ellis & Sagarra (Reference Ellis and Sagarra2010a) and other similar studies appear to support theoretical underpinnings that postulate the role of selective attention to L2 data in the input premised on cue strength and activation of prior knowledge or exposure. In addition, practitioners may use these findings to manipulate the input they provide in the classroom setting in an effort to promote more robust learning. However, the research design reported in Ellis & Sagarra (Reference Ellis and Sagarra2010a) raises several issues that a replication study may address in an effort to probe deeper into or control for several variables that could have impacted the findings of the original study.

From a theoretical perspective, for example, a replication of this study could probe deeper into whether it is only learner attention to cues in the L2 input that really matters or whether learners need to process such cues at a certain depth. To this end, replicating this study while gathering concurrent or online data during their interaction with the L2 data would provide more robust evidence of what specifically they were paying attention to and/or processing during exposure. To gather concurrent data to address the issue of where attention is being paid, an eye-tracker can be used if accessible. To gather concurrent data that address the issue of how the cue is processed, think aloud protocols may be employed. The ideal combination may be the type of eye-tracker employed in Smith (Reference Smith2012), which was a MangoldVision portable eye-tracker from EyeTech Digital Systems (model TM3) together with the MangoldVision Gaze Data Analysis Software Suite. This remote eye-tracker is fastened to the bottom edge of any computer screen and an infrared camera follows one's pupil movements and can record one's eye gaze in real time. Unlike earlier models of eye-trackers that did not allow much movement (e.g., from speaking) during data collection, this one also allows for a normal range of head movements (25 cm × 16 cm × 19 cm), which would allow participants to speak their thoughts aloud while being eye-tracked. Concurrent data would clearly provide important information as to whether L2 learners simply need to pay attention, for example, peripheral attention, to the cues or whether they also need to process such cues at a deeper level in order to select the correct response.

A second replication possibility is to address the potential role played by the provision of feedback in the findings. Such a replication study is important given that type or amount of feedback may be conflated with the experimental group and the results may not be directly derived from type of exposure, as reported in the original study. The feedback provided in Phases 2 and 3 are relatively inconsistent in that while a correct selection of one out of two options (a 50% chance of guessing) received a ‘correct’ feedback, incorrect selections not only received a ‘wrong’ feedback but this was accompanied by an explicit type of feedback providing the correct meaning. In addition, amount of feedback per participant and group is not reported, which could have potentially encouraged more attention to and potential deeper processing of the sentences provided. A replication study controlling for both type and amount of feedback would assist in providing important data as to whether feedback (type or amount) is associated with learned attention and the phenomenon of blocking in relation to temporal reference. To control for type of feedback, a simple approach is to keep all feedback provided similar, that is, either provide a ‘correct’ or ‘wrong’ response or provide both feedback responses accompanied by the elaborated correct response. To control for amount of feedback, there are many computerized programs available that can track such data via mouse use (e.g., MouseTracker (Freeman & Ambady Reference Freeman and Ambady2010), a free software package specifically created to design and run mouse-tracking experiments in psychology-related research) or similar devices. These data will then be analyzed to ensure that amount of feedback was statistically similar in the experimental groups. Similar findings in this replication study will provide empirical support not only to the external validity of the original study but also remove any confounding role for the provision of feedback during exposure.

Given that the L2 data were provided visually on a computer screen, a third potential replication study may seek to address the issue of whether the findings of this study can be extrapolated to the aural mode. To address the issue of modality, the researcher can quite simply record the L2 data and present them in this aural mode instead of using a computer screen. The computer can still be used to record responses. The ideal replication study would incorporate both modes (aural and written) in the same design so that statistical comparisons are within the study instead of across studies. Similar findings would support the generalizability of the findings regarding modality while different findings would indicate that there may be cognitive constraints associated with type of mode (see Leow Reference Leow1995; Wong Reference Wong2001).

5.1 Leung & Williams (Reference Leung and Williams2012)

Leung & Williams (Reference Leung and Williams2012), published in Language Learning, is the last (at the time of writing this paper) of a series of studies spanning over a decade that have methodologically operationalized the construct of unawareness in an effort to address whether L2 learners can learn new L2 data implicitly, that is, without awareness. This strand of research to date has reported conflicting findings that leave the role of unawareness in L2 development unresolved in the SLA field (see Leow Reference Leow2000; Williams Reference Williams, VanPatten, Williams, Rott and Overstreet2004, Reference Williams2005; Hama & Leow Reference Hama and Leow2010; Faretta-Stutenberg & Morgan-Short Reference Faretta-Stutenberg, Morgan-Short, Granena, Koeth, Lee-Ellis, Lukyanchenko, Botana and Rhoades2011; Leung & Williams Reference Leung and Williams2011, Reference Leung and Williams2012). A brief chronological review of this strand of research reveals inconsistent findings. Leow (Reference Leow2000) addressed the effect of awareness, or lack thereof, on L2 learners’ subsequent intake and written production of L2 morphological forms. He reported that ‘no disassociation between awareness and learning was found in this study’ (p. 573) and cautioned that the findings reported were only directed to his sample of ‘adult beginning learners of Spanish’ (p. 573). These findings were contradicted by Williams's (Reference Williams, VanPatten, Williams, Rott and Overstreet2004, Reference Williams2005) studies that reported evidence of learning without awareness. Williams (Reference Williams2005), a methodological improvement on Williams (Reference Williams, VanPatten, Williams, Rott and Overstreet2004), employed a unique design to test whether participants were able to learn miniature noun class systems without awareness. While his 41 participants were informed that of the four novel determiners (gi, ro, ul, ne), the first two meant ‘near’ and the latter two ‘far’, they were not informed that these determiners also carried animacy values: gi and ul were animate and ro and ne were inanimate. According to the author, the results showed ‘that, at least for some individuals, it is possible to learn form-meaning connections without awareness of what those connections are’ (p. 293).

Hama & Leow (Reference Hama and Leow2010) extended Williams (Reference Williams2005) by addressing several methodological issues of Williams's research design and, contrary to the original study, revealed that ‘unaware learners, at the stage of encoding, did not appear to demonstrate any significant animacy bias in either their selection or production of the trained or new determiner-noun combinations’ (p. 482). Similarly, Faretta-Stutenberg & Morgan-Short (Reference Faretta-Stutenberg, Morgan-Short, Granena, Koeth, Lee-Ellis, Lukyanchenko, Botana and Rhoades2011), a conceptual replication of Williams (Reference Williams2005), reported no evidence of learning without awareness among their participants, failing to provide empirical support for the external validity of the findings reported in Williams's (Reference Williams2005) original study. However, two recent publications (Leung & Williams Reference Leung and Williams2011, Reference Leung and Williams2012), building on the research agenda and design of Williams (Reference Williams2005), have employed the novel determiners (gi, ro, ul, and ne) in the L1 with different embedded features in their respective studies. Unique in these two studies was the use of reaction times to measure the process of learning concurrently as opposed to the previous offline judgment measure employed in the original study. Both studies have reported that unaware learners can provide evidence of learning form-meaning connections without awareness of what those connections are.

The debate on the role of unawareness in the SLA field is beginning to parallel other non-SLA fields of inquiry and clearly constitutes a worthy strand for future research. In light of the inconsistent findings, replications are needed to provide empirical support for the external validity of these studies and also to probe deeper into this thorny construct, therefore a detailed report on Leung & Williams (Reference Leung and Williams2012) is provided below.

Leung & Williams (Reference Leung and Williams2012) is an extension of Williams's (Reference Williams2005) research agenda and reports on two experiments designed to consolidate, in the first experiment, the original study's results by examining implicit learning of a correlation between determiners and noun animacy using a different methodology and, in the second experiment, to test whether a similar effect will be obtained when determiners correlate with a meaning feature (relative size) that is not a grammaticalized feature in natural languages. While the overall methodology employed in this study follows the one reported in Williams (Reference Williams2005) and Leung & Williams (Reference Leung and Williams2011), this study, like Leung & Williams (Reference Leung and Williams2011), employed a reaction time task as a concurrent measure of learning as opposed to a post-experimental assessment task, the method used in the original study. The reaction time task was assumed to introduce time pressure, which would encourage ‘the use of implicit rather than explicit knowledge’ (Leung & Williams Reference Leung and Williams2011: 40) and, notably, there was no separation between the ‘training’ and ‘test’ phases. In addition, it was postulated in Leung & Williams (Reference Leung and Williams2011) that in the Violation block, in which the mapping between the articles and animacy was reversed, ‘response times should be slowed and errors should increase’ (p. 42), irrespective of presence or absence of awareness.

In Experiment 1, 33 English-speaking participants were introduced to the same four artificial articles employed in Williams (Reference Williams2005): gi, ro, ul, and ne. Before the training phase, participants were provided with the translations of the four novel determiners for ‘near’ (gi, ul) or ‘far’ (ro, ne) and informed that this information would assist them in responding quickly. What they were not given was the information about the animacy rules of these determiners, that is, gi and ro were also associated with animate objects while ul and ne were associated with inanimate objects.

During the training phase, participants were presented with 272 slides with an object either depicted near (placed in the center of the screen) or far (placed in the top left corner of the screen) accompanied by an audio description in the form of a noun phrase (e.g., ro bull ‘the far bull’). For each slide, participants performed three tasks: (1) a reaction task in which they had to decide as quickly as they could whether the object named by the audio description was a living thing (by left-clicking on the serial mouse) or non-living thing (by right-clicking on the serial mouse), (2) an oral repetition of the phrase they heard, and (3) translating the determiner in the phrase they heard into ‘near’ or ‘far’. Reaction time was recorded from the onset of the article. During the first half of training, implicit feedback was provided during the experimental treatment if participants gave a wrong answer but the reaction times for subsequent responses were not considered for statistical purposes. There was a break halfway through the training phase.

The training phase was followed by eight Transition slides to familiarize participants with both old and new objects, then a Control block of 32 slides and a Violation block of 32 slides. While some of the objects were repeated in the training phase, each slide in the Transition, Control, and Violation blocks contained new and unique objects. According to Leung & Williams, the critical data came from the last 32 trials of the experiment, which were divided into Control and Violation blocks with no division between these trials and the training. In the Control block, the sentences respected the same system used in training, whereas in the Violation block, the articles in the auditory descriptions were switched so that ul and ne were used with near and far animate objects, respectively, and gi and ro were used with near and far inanimate objects, respectively.

After the training phase, like in previous studies (Williams Reference Williams, VanPatten, Williams, Rott and Overstreet2004, Reference Williams2005; Leung & Williams Reference Leung and Williams2011), to operationalize learners’ awareness, participants were asked in an interview, in the form of a detailed questionnaire after exposure, whether they had any feelings about when gi versus ul and ro versus ne were used. They were also exposed once more to the training items and asked to make as many guesses as they could and state whether they were confident in their guesses. Based on their reported knowledge of the target form-meaning mapping, participants were classified as being aware or unaware. Learning of the animacy rule was measured by participants’ performances on the reaction time test. In this type of task, participants demonstrated their interpretation of a stimulus by hitting corresponding response keys. No post-exposure learning test was administered. The authors concluded that ‘the implicit learning effect we obtained in Experiment 1 must have been due to learning the correlation between articles and noun animacy’ (p. 23).

Experiment 2 was almost identical to Experiment 1 with the following differences: participants were told that gi and ro were used with animate objects and ul and ne were used with inanimate objects. They were not told that gi and ul were also used with objects that were relatively big and ro and ne were also used with relatively small objects. In other words, in Experiment 2 the target for implicit learning was the association between articles and the relative size of the object referred to by the accompanying noun. The researchers reported no empirical evidence for implicit learning of relative size. Leung & Williams concluded that learning is constrained by the nature of the meaning involved and argued that concepts are differentially available to implicit language learning processes.

5.2 Approaches to replication

As can be seen, the novel methodology patterned after Leung & Williams (Reference Leung and Williams2011) appears to support the role of implicit learning in L2 development that involves novel determiners while taking into consideration constraints based on type of concepts. This information would certainly support theoretical underpinnings that postulate no role for awareness during the initial stage of learning (e.g., Tomlin & Villa Reference Tomlin and Villa1994) while refuting the postulations of others that do place some role for awareness at this stage (e.g., Robinson Reference Robinson1995; Schmidt Reference Schmidt and Robinson2001) or have reported no such role (Leow Reference Leow2000; Hama & Leow Reference Hama and Leow2010; Faretta-Stutenberg & Morgan-Short Reference Faretta-Stutenberg, Morgan-Short, Granena, Koeth, Lee-Ellis, Lukyanchenko, Botana and Rhoades2011). In addition, practitioners could also take comfort that a more meaning-based exposure to the L2 is not detrimental to the learning process. At the same time, the research design reported in Leung & Williams (Reference Leung and Williams2012) raises several issues that a replication study may address in an effort to probe deeper into, or control for, several variables that could have impacted the findings of the original study. From both a theoretical and methodological perspective, the issue of where the construct of awareness is operationalized and measured has been well discussed in both the SLA (e.g., Leow Reference Leow2000; Leow & Morgan-Short Reference Leow and Morgan-Short2004) and non-SLA (e.g., Berry & Dienes Reference Berry and Dienes1993; Ericsson & Simon Reference Ericsson and Simon1993; Shanks & St. John Reference Shanks and John1994) literature. To classify their participants as learning with or without awareness, Leung & Williams (Reference Leung and Williams2012) relied on the results of one offline measure, namely, verbal reports, to first establish type of knowledge (implicit or explicit), which have been critiqued for several potential methodological problems that include (a) the issue of veridicality or memory decay (see e.g., Ericsson & Simon Reference Ericsson and Simon1993; Leow Reference Leow2000), (b) a mismatch between the actual knowledge employed by the learner to process the L2 data and what is being sought in the offline questionnaire or verbal report (see e.g., Shanks & St. John Reference Shanks and John1994), and (c) ‘differential test sensitivity’, that is, the withholding of knowledge of which learners may not be very confident to report (see e.g., Berry & Dienes Reference Berry and Dienes1993; Shanks & St. John Reference Shanks and John1994). A participant unable to remotely mention the underlying rule was assumed to possess implicit knowledge and subsequently coded as unaware. A replication study may address this critique by employing thorough and multiple offline procedures (see Rebuschat (Reference Rebuschat2013) for several measures proposed for the SLA field) that together should provide a more robust and sensitive measurement of participants’ levels of awareness.

A second possibility for a replication study is to address the potential role played by the provision of feedback in the findings. Given that Leung & Williams (Reference Leung and Williams2012) address implicit learning, a process that is usually viewed as effortless with a very low level of processing, it is important to isolate any potential effect of feedback being provided during exposure to the experimental data. The issue here is whether such feedback could have contributed to potential intentional learning in that incorrect responses might have triggered an increase in participants’ awareness that more cognitive effort or processing might be required. This might have led to a type of processing that is usually not associated with implicit learning. To control for the potential role of feedback, the researcher simply needs to include in the research design an experimental group that does not receive feedback during the first half of the training phase but follows all the other phases of the study. This group's performance on the reaction time task will then be statistically compared to the original study's group that received feedback to establish whether the provision of feedback led to similar or differential results. Similar results will indicate that feedback played no role during the process providing empirical support for the original study's findings while different results may warrant the need for caution to be taken when considering the original findings.

A third replication study may address the potential issue of participant fatigue, as raised by the researchers themselves. Controlling for this potential fatigue is important given that the slow down on the Violation block is taken as evidence for rule learning. To control for potential fatigue, a second Control block needs to be inserted after the Violation block in the research design. A reversal to a statistically similar reaction time mean evidenced in the Control block will indicate that fatigue did not play a role and, once again, provide external support for the original study's design and results.

A fourth replication study may be to simply increase the numbers of participants in the study given that the original study did have relatively small numbers of participants in the unaware (20, 20 in the first and second experiments, respectively) and especially in the aware (13, 6 in the first and second experiments, respectively) groups. Replicating the study with larger numbers of participants in both aware and unaware groups would address (1) the relatively inconsistent findings evidenced by reaction times in relation to both aware and unaware groups’ performances on the Control versus Violation blocks and the number of errors reported on the Violation block and (2) allow adequate statistical comparisons to be made since it is assumed that differences do exist between the processes involved in explicit and implicit learning.

To promote robustness of findings, all these issues can be easily incorporated within the research design of a single replication study that ideally includes a productive test for tangible evidence of learning item and/or system).