Research shows that retrieval increases second language (L2) vocabulary learning (e.g., Barcroft, Reference Barcroft2007, Reference Barcroft2013; Folse, Reference Folse2006; Karpicke & Roediger, Reference Karpicke and Roediger2008; Royer, Reference Royer1973). Retrieval refers to the process of accessing information about L2 words from memory. Barcroft (Reference Barcroft2007), for instance, found that presenting L2 word forms 6 seconds after their meanings, which provides learners opportunities for retrieval, increased learning compared with presenting L2 words and their meanings simultaneously, which does not encourage retrieval. Royer (Reference Royer1973) also found that studying of L1 and L2 word pairs printed on different sides of word cards increases learning more than studying of L1 and L2 word pairs printed on the same side of word cards, presumably because the former encourages retrieval while the latter does not. Given that retrieval facilitates learning, one may wonder whether increasing retrieval frequency in one learning session (hereafter referred to as within-session repeated retrieval) increases L2 vocabulary learning. Although it seems reasonable to expect repeated retrieval to increase learning, the distributed practice effect, retrieval effort hypothesis, and desirable difficulty framework suggest that within-session repeated retrieval is not necessarily desirable (for details, see Literature Review). The present study examined the effects of within-session repeated retrieval on L2 vocabulary learning. Findings of this study will be valuable because they may enable us to determine how we can optimize L2 vocabulary learning from retrieval.
LITERATURE REVIEW
Theoretical Perspectives Regarding the Effects of Within-Session Repeated Retrieval
Although it seems reasonable to expect repeated retrieval to increase learning, the distributed practice effect (e.g., Bahrick & Phelps, Reference Bahrick and Phelps1987; Bahrick, Bahrick, Bahrick, & Bahrick, Reference Bahrick, Bahrick, Bahrick and Bahrick1993; Bloom & Shuell, Reference Bloom and Shuell1981; Cepeda et al., Reference Cepeda, Coburn, Rohrer, Wixted, Mozer and Pashler2009), retrieval effort hypothesis (Pyc & Rawson, Reference Pyc and Rawson2009), and desirable difficulty framework (e.g., Bjork, Reference Bjork, Metcalfe and Shimamura1994) suggest that within-session repeated retrieval is not necessarily desirable. First, according to the distributed practice effect, larger spacing between retrievals results in superior long-term retention than shorter spacing or no spacing at all (pure massing). The distributed practice effect is a robust phenomenon that has been found to affect the learning of a wide range of verbal information such as L1 vocabulary, name pairs, face-name pairs, general facts, and text materials (e.g., Cepeda, Pashler, Vul, Wixted, & Rohrer, Reference Cepeda, Pashler, Vul, Wixted and Rohrer2006). Cepeda et al. (Reference Cepeda, Coburn, Rohrer, Wixted, Mozer and Pashler2009), for instance, found that distributing retrieval opportunities of L2 vocabulary over multiple days increases retention more than practicing retrieval repeatedly in one day. Other L2 vocabulary studies (e.g., Bahrick & Phelps, Reference Bahrick and Phelps1987; Bahrick et al., Reference Bahrick, Bahrick, Bahrick and Bahrick1993; Bloom & Shuell, Reference Bloom and Shuell1981) also found long-term benefits of distributing practice over a longer period of time. These findings suggest that practicing retrieval multiple times over a month, for instance, would lead to better long-term retention than practicing retrieval for the same number of times in one learning session. Note that the distributed practice effect states that larger spacing enhances long-term retention more than massed practice only when other variables such as the retrieval frequency or time-on-task are held constant. Hence, the distributed practice effect does not necessarily rule out the possibility that within-session repeated retrieval enhances learning relative to fewer retrievals, at least in the short term. However, because within-session repeated retrieval inevitably involves shorter spacing than retrievals distributed over multiple days, the distributed practice effect suggests that the benefits of within-session repeated retrieval will increasingly diminish and may completely disappear in the long term.
While the distributed practice effect is concerned with the effectiveness of within-session repeated retrieval (i.e., the issue of whether repeated retrieval allows learners to learn a large number of words), the retrieval effort hypothesis (Pyc & Rawson, Reference Pyc and Rawson2009) and desirable difficulty framework (e.g., Bjork, Reference Bjork, Metcalfe and Shimamura1994) are concerned with the efficiency of within-session repeated retrieval (i.e., the issue of whether repeated retrieval allows learners to learn a large number of words when time-on-task is controlled). The desirable difficulty framework states that more demanding tasks result in long-term retention superior to retention achieved through less demanding tasks. The retrieval effort hypothesis, which is derived from the desirable difficulty framework, states that difficult and effortful retrievals increase learning more than easy and effortless retrievals. When learners practice retrieval repeatedly in one learning session, retrieval will be progressively easier (e.g., the second retrieval is easier than the first one, and the third retrieval is even easier than the second one). Because according to the desirable difficulty framework and retrieval effort hypothesis, easy retrievals enhance learning less than difficult retrievals, these two theories predict that the positive effects of retrievals attenuate as a function of the retrieval frequency. The second retrieval enhances learning less than the first one, and the third retrieval enhances learning even less than the second one, and so forth. This suggests that when learners practice retrieval repeatedly in one learning session, most learning occurs in the first few retrieval attempts, and practicing retrieval beyond a certain frequency level does not provide a benefit that justifies its cost (i.e., increased study time). The retrieval effort hypothesis and desirable difficulty framework, as a result, predict that within-session repeated retrieval is not necessarily efficient.
Effects of Input Frequency on L2 Vocabulary Learning
A number of studies have examined the effects of input frequency, or the number of encounters with target words, on L2 vocabulary learning. Most of these studies found that increasing input frequency within one learning session increases vocabulary learning (e.g., Bisson, van Heuven, Conklin, & Tunney, Reference Bisson, van Heuven, Conklin and Tunney2014; Chen & Truscott, Reference Chen and Truscott2010; Webb, Reference Webb2007a). A listening study conducted by van Zeeland and Schmitt (Reference van Zeeland and Schmitt2013), however, constitutes an exception. Van Zeeland and Schmitt compared the effects of four levels of input frequency (3, 7, 11, or 15) in one learning session. Although they found rather small positive effects of repeated input on the immediate posttest, no significant difference was found on the posttest conducted 2 weeks after learning. Their results are consistent with the prediction made from the distributed practice effect. Although increasing frequency in one learning session enhances learning in the short term, its advantage diminishes over time. Another significant finding of van Zeeland and Schmitt is that although 7, 11, and 15 encounters were generally more effective than three encounters, little difference was found among the effects of 7, 11, and 15 encounters. This is consistent with the desirable difficulty framework, according to which the positive effects of repetition attenuate as a function of frequency, and increasing frequency beyond a certain frequency level does not provide further benefit that justifies its cost. A similar pattern of diminishing returns is also reported by Bisson et al. (Reference Bisson, van Heuven, Conklin and Tunney2014). Although the findings of these studies are informative, they manipulated the frequency of input (i.e., number of encounters that do not necessarily involve retrieval), not the frequency of retrieval. As a result, these studies do not necessarily demonstrate the value of repeated retrieval for L2 vocabulary learning.
Effects of Retrieval Frequency on L2 Vocabulary Learning
Although smaller in number, several studies have examined the effects of retrieval frequency, not input frequency, on L2 vocabulary learning. These studies also found that repetition increases vocabulary learning. (Note that because the focus of this study is the effects of retrieval frequency, studies that manipulated the frequency of successful retrievals during learning [e.g., Karpicke & Roediger, Reference Karpicke and Roediger2008] will not be discussed in detail.) Laufer and Rozovski-Roitblat (Reference Laufer and Rozovski-Roitblat2011), for instance, examined the effects of frequency over a 13-week course in two conditions: T+F and T+Fs. In the T+F condition, learners read a text with occasional Focus-on-Form (e.g., asking the instructor for the meaning of unknown words or looking up unfamiliar words in dictionaries). In the T+Fs condition, learners first encountered unknown words in a text, followed by Focus-on-Forms activities in which target words were practiced in a number of exercises (e.g., matching the target words and definitions or choosing the correct meanings of target words), which involved retrieval. The target words were also assigned to one of the three frequency levels: (a) two or three, (b) four or five, and (c) six or seven. Note that the frequency of retrieval was manipulated in the T+Fs condition, whereas the frequency of input was manipulated in the T+F condition. Laufer and Rozovski-Roitblat found that frequency had a larger effect on learning in the T+Fs condition than on learning in the T+F condition. However, because Laufer and Rozovski-Roitblat examined the effects of increasing frequency over a 13-week course, their findings may not necessarily be applicable to within-session repeated retrieval.
Several studies examined the effects of increasing retrieval frequency in one learning session. In Barcroft (Reference Barcroft2013), for instance, 75 Spanish-speaking students read an English text containing six target words, each of which occurred three times in the text. Participants were assigned to one of the four groups: III, IIR, IRI, and IRR, where I denotes input (i.e., viewing the target L2 word form and its L1 translation) and R denotes retrieval (i.e., filling a blank with the target L2 word form). In the III group, target words appeared together with their L1 translations in all three occurrences. In the IIR group, L1 translations were provided in the first and second encounters with each target word. In the third occurrence, participants practiced retrieval. Specifically, target words were replaced with a blank, and participants were asked to write down one of the target words. In the IRI group, L1 translations were given in the first and third encounters with each target word, and learners practiced retrieval in the second encounter. In the IRR group, L1 translations were given only in the first occurrence, and learners practiced retrieval in the second and third occurrences. On the posttests conducted immediately after the treatment, Barcroft found that (a) IRI significantly outperformed III and IIR, and (b) IRR significantly outperformed III. The results suggest that although one (IRI) and two retrievals (IRR) are more effective than no retrieval (III), two retrievals (IRR) do not necessarily increase learning relative to one retrieval (IRI or IIR). It should be noted, however, that because Barcroft was interested in comparing the effects of different patterns of retrieval and input opportunities, higher retrieval frequency was naturally associated with lower input frequency in his study. More specifically, IRR had more opportunities for retrieval, but fewer opportunities for input, than IRI and IIR. As a result, the results of his study may be at least partly due to the input frequency rather than the retrieval frequency per se. (Barcroft’s research design, of course, should not be considered a shortcoming because the purpose of his study was not to compare different retrieval frequency levels.)
In Folse (Reference Folse2006), 154 learners of English studied 18 target words under three conditions: (a) one completion, (b) three completions, and (c) original sentences. In the one completion condition, participants answered one fill-in-the-blank exercise for each target word, which required participants to retrieve the L2 word form once. In the three completions condition, they answered three fill-in-the-blank exercises for each target word, which involved three retrievals. In the original sentences condition, participants wrote one sentence for each target word. Learning was measured immediately after the treatment by a modified Vocabulary Knowledge Scale (Paribakht & Wesche, Reference Paribakht, Wesche, Coady and Huckin1997), which required participants to indicate their knowledge of target words on a three-point scale. Depending on their response, participants were asked to write original sentences using the target words and/or produce the meaning of the target words. Folse found an advantage of the three completions condition over the one completion and original sentences conditions. His result suggests that within-session repeated retrieval (three) increases L2 vocabulary learning relative to a single retrieval. Folse also reports that the three completions condition required 50% more time than the one completion condition. As a result, it is not clear whether the three completions condition would have been more effective than the one completion condition if time-on-task had been held constant.
In Peters (Reference Peters2012), 56 Dutch-speaking students read an English text containing 14 target words under message-oriented and vocabulary-oriented conditions. In the former, participants answered comprehension questions about the text. The questions were created so that the participants had to write the L1 translation of target words. As a result, the message-oriented condition required participants to retrieve the meaning of each target word once. In the vocabulary-oriented condition, participants answered translation and multiple-choice vocabulary questions after reading the text, which required them to retrieve the meaning of each target word twice. Peters found that the vocabulary-oriented condition resulted in significantly higher posttest scores than the message-oriented condition immediately and 2 weeks after the treatment. The result demonstrates the advantage of repeated (two) over a single retrieval in one learning session. In Peters (Reference Peters2014), 35 EFL university students studied 12 single words and 12 collocations. Target items were practiced either one, three, or five times in different exercises such as fill-in-the-blanks or collocation matching. Peters found that items practiced five times were consistently remembered better than those practiced only once immediately, 1 week, and 2 weeks after learning, supporting the value of repetition in deliberate vocabulary learning.
One positive feature of the studies conducted by Peters (Reference Peters2012, Reference Peters2014) is that she administered the posttest not only immediately but also 1 and 2 weeks after learning. As noted previously, the distributed practice effect predicts that increasing within-session retrieval frequency facilitates short-term, but not necessarily long-term retention. Peters’s (Reference Peters2012, Reference Peters2014) findings, however, were inconsistent with this prediction. She found that frequency increased vocabulary learning regardless of the retention interval (i.e., interval between the treatment and posttest). One possible explanation is that the retention interval used for the delayed posttests in her research (1 and 2 weeks) was not long enough. Previous research on the distributed practice effect suggests that short spacing tends to be effective at a short retention interval, whereas long spacing tends to be effective at a long retention interval (e.g., Bird, Reference Bird2010; Cepeda et al., Reference Cepeda, Pashler, Vul, Wixted and Rohrer2006, Reference Cepeda, Coburn, Rohrer, Wixted, Mozer and Pashler2009). This suggests that the benefits of within-session repeated retrieval, which involves shorter spacing than retrievals distributed over multiple days, will increasingly diminish and may completely disappear in the long term. If the posttest had been given after a delay greater than 2 weeks, therefore, benefits of repetition might not have been found. One non-L2 vocabulary study supports this prediction (Rohrer, Taylor, Pashler, Wixted, & Cepeda, Reference Rohrer, Taylor, Pashler, Wixted and Cepeda2005). In Rohrer et al.’s (Reference Rohrer, Taylor, Pashler, Wixted and Cepeda2005) study, American undergraduate students studied city-country pairs (e.g., Chiba – Japan; Experiment 1) or L1 word pairs (e.g., acrogen – fern; Experiment 2). Rohrer et al. found benefits of within-session repeated retrieval 1 week after the treatment, but not after a delay of 3 weeks or greater (3, 4, and 9 weeks after learning). Their findings suggest that (a) the benefits of within-session repeated retrieval are short-lived and (b) when examining the effects of within-session repeated retrieval, it may be useful to give a posttest after a delay greater than 2 weeks. Footnote 1
Although Peters’s (Reference Peters2012) findings are useful, the purpose of her study was to compare the effects of message-oriented and vocabulary-oriented conditions and not to compare different retrieval frequency levels. As a result, she compared the effects of only one and two retrievals, and her research was not intended to allow us to obtain a comprehensive picture regarding the effects of within-session repeated retrieval. (This, of course, does not constitute a limitation of Peters’s study because the purpose of her research was not to compare different retrieval frequency levels.) Unlike her earlier study, Peters (Reference Peters2014) used a wider range of retrieval frequency levels (one, three, and five). However, a different set of items was used for each retrieval frequency level in her 2014 study. For instance, in Peters (Reference Peters2014), the target item turnover was practiced once, whereas absenteeism was practiced five times. As a result, as Peters acknowledges, differential posttest performance for these two items may be partly attributed to item difficulty rather than different frequency levels.
PRESENT STUDY
The preceding review of literature shows that although many studies have investigated the effects of repetition on L2 vocabulary learning, only two studies examined the long-term effects of increasing retrieval frequency in one learning session (Peters, Reference Peters2012, Reference Peters2014). With this in mind, the present study examined the long-term effects of within-session repeated retrieval. The present study is original in four respects. First, this study gave posttests after a delay greater than 2 weeks. The distributed practice effect predicts that increasing retrieval frequency in one learning session facilitates short-term, but not necessarily long-term retention. Indirect evidence for this prediction comes from L2 vocabulary research on within-session repeated input (van Zeeland & Schmitt, Reference van Zeeland and Schmitt2013) and non-L2 vocabulary research on within-session repeated retrieval (Rohrer et al., Reference Rohrer, Taylor, Pashler, Wixted and Cepeda2005), both of which failed to find an advantage of repetition at a retention interval of 2 weeks or greater. Considering that Peters (Reference Peters2012, Reference Peters2014) found benefits of within-session repeated retrieval 2 weeks after the treatment, the present study gave posttests 4 weeks as well as immediately and 1 week after the treatment to investigate the long-term effects of within-session repeated retrieval.
Second, the treatment in this study was conducted in a paired-associate learning (hereafter PAL) format. In PAL, target items are presented outside message-focused tasks, and learners link the L2 word form and its meaning such as the first language (L1) translation or L2 definition (e.g., Barcroft, Reference Barcroft2007; Elgort, Reference Elgort2011; Royer, Reference Royer1973; Steinel, Hulstijn, & Steinel, Reference Steinel, Hulstijn and Steinel2007). Examples of PAL include learning from word lists and flashcards. (Although PAL does not necessarily have to involve retrieval, this study is concerned only with PAL that involves a retrieval component. This is because retrieval is found to significantly increase L2 vocabulary learning, e.g., Barcroft, Reference Barcroft2007; Karpicke & Roediger, Reference Karpicke and Roediger2008; Royer, Reference Royer1973.) Employing PAL is valuable because it allows experimenters to exert strict control over the treatment. For instance, when frequency is manipulated in reading texts, variables such as the contextual richness, importance of the target words in the comprehension of texts, or distribution of the occurrences of target words may affect learning (e.g., Elgort & Warren, Reference Elgort and Warren2014; Webb, Reference Webb2008). Most of these variables, however, are either irrelevant or can be controlled if the treatment involves PAL. At the same time, it should be noted that there are some disadvantages to employing a PAL format. One disadvantage is that this study may have limited ecological validity. This is because although it is common for students to study vocabulary in a PAL format outside of classrooms using word lists, flashcards, or computer software (e.g., Nakata, Reference Nakata2011; Schmitt, Reference Schmitt, Schmitt and McCarthy1997; Wissman, Rawson, & Pyc, Reference Wissman, Rawson and Pyc2012), the repeated use of PAL in a foreign language classroom is perhaps uncommon.
It should also be pointed out that PAL is not without criticisms. One common criticism is that PAL does not allow learners to remember L2 words successfully (e.g., Judd, Reference Judd1978; Lewis, Reference Lewis1993; Oxford & Crookall, Reference Oxford and Crookall1990). Empirical studies, however, suggest that PAL is one of the most effective vocabulary learning strategies (e.g., Laufer & Shmueli, Reference Laufer and Shmueli1997; Nation, Reference Nation2013; Webb, Reference Webb2007b). Vocabulary learned in PAL is resistant to decay (Fitzpatrick, Al-Qarni, & Meara, Reference Fitzpatrick, Al-Qarni and Meara2008) and can be retained over several years (Bahrick & Phelps, Reference Bahrick and Phelps1987). Studies have also shown that in PAL, large numbers of words can be learned in a very short time (e.g., Fitzpatrick et al., Reference Fitzpatrick, Al-Qarni and Meara2008; Nation, Reference Nation2013). Another common criticism is that PAL does not necessarily result in successful use of the target items (e.g., Anderson & Nagy, Reference Anderson and Nagy1992; Larson & Smelley, Reference Larson and Smelley1972; Oxford & Crookall, Reference Oxford and Crookall1990). This criticism is largely true possibly because PAL focuses on linking form and meaning, and other aspects of word knowledge such as the spoken form, associations, and constraints on use are usually not dealt with (Nation, Reference Nation2013). Despite these limitations, most researchers agree that PAL should have a place in L2 vocabulary instruction (e.g., Barcroft, Reference Barcroft2012; Folse, Reference Folse2004; Laufer, Reference Laufer2005; Nation, Reference Nation2013; Nation & Webb, Reference Nation and Webb2011). Considering the role that PAL plays in vocabulary development as well as the methodological advantages that it offers, the present study examined the effects of within-session repeated retrieval in PAL.
Third, the present study considered time-on-task as a factor. Because repeated retrievals naturally take more time than fewer retrievals, the results of earlier research may be partly attributed to the difference in the study time. L2 vocabulary studies underscore the importance of considering time-on-task when comparing the effects of different treatments (Ellis, Reference Ellis1995; Folse, Reference Folse2006; Hulstijn & Laufer, Reference Hulstijn and Laufer2001; Webb, Reference Webb2005). Webb (Reference Webb2005), for instance, compared the effects of receptive (reading) and productive (writing) tasks on L2 vocabulary learning. Although Webb found the superiority of receptive learning when time-on-task was controlled, he found the advantage of productive learning when time-on-task was not controlled. Webb’s findings, along with those of other studies (e.g., Ellis, Reference Ellis1995; Folse, Reference Folse2006; Hulstijn & Laufer, Reference Hulstijn and Laufer2001), suggest that it is of value to take time-on-task into account when comparing the effects of different treatments. The present study considered the effects of time-on-task by examining the efficiency as well as the effectiveness (number of items learned) of different retrieval frequency levels. Efficiency was operationalized as the number of items learned per minute (e.g., Ellis, Reference Ellis1995; Kornell, Reference Kornell2009; Mondria, Reference Mondria2003) and calculated by dividing the posttest score by the study time (hereafter referred to as efficiency scores). Examining the efficiency of different frequency levels is useful because it allows us to test the view that within-session repeated retrieval may not necessarily be efficient according to the retrieval effort hypothesis and desirable difficulty framework (see Theoretical Perspectives Regarding the Effects of Within-Session Repeated Retrieval). Examining both effectiveness and efficiency will also help us to reach a more informed decision regarding whether within-session repeated retrievals are more desirable than fewer retrievals. Fourth, in some previous studies, different target items were used for different retrieval frequency levels (e.g., Laufer & Rozovski-Roitblat, Reference Laufer and Rozovski-Roitblat2011; Peters, Reference Peters2014). With this limitation in mind, the same target items were used for all retrieval frequency levels in this study.
In summary, the present study is significant in that it is the first study to examine the effects of within-session repeated retrieval on L2 vocabulary learning while (a) giving posttests at a retention interval greater than 2 weeks, (b) employing a PAL format to exert strict control over the treatment, (c) considering time-on-task as a factor, and (d) using the same target items for all retrieval frequency levels. Findings of the present study will be useful because they may enable us to determine the value of within-session repeated retrieval for L2 vocabulary learning and instruction.
RESEARCH QUESTIONS AND HYPOTHESES
The following two research questions were addressed:
RQ1: In L2 vocabulary learning, does increasing within-session retrieval frequency increase posttest scores immediately, 1 week, and 4 weeks after learning?
RQ2: In L2 vocabulary learning, does increasing within-session retrieval frequency increase efficiency scores immediately, 1 week, and 4 weeks after learning?
Based on the review of literature, the following two hypotheses were formed:
Hypothesis 1: Within-session repeated retrieval leads to higher posttest scores than fewer retrievals immediately and 1 week after learning, but not 4 weeks.
Hypothesis 2: Within-session repeated retrieval leads to lower efficiency scores than fewer retrievals regardless of the retention interval.
Hypothesis 1 predicts that within-session repeated retrieval is more effective than fewer retrievals immediately and 1 week after learning, but not 4 weeks after learning. This is based on the distributed practice effect and on previous studies (Peters, Reference Peters2012, Reference Peters2014; Rohrer et al., Reference Rohrer, Taylor, Pashler, Wixted and Cepeda2005; van Zeeland & Schmitt, Reference van Zeeland and Schmitt2013). Hypothesis 2 predicts that within-session repeated retrievals are less efficient than fewer retrievals because the retrieval effort hypothesis (Pyc & Rawson, Reference Pyc and Rawson2009) and desirable difficulty framework (e.g., Bjork, Reference Bjork, Metcalfe and Shimamura1994) state that although increasing retrieval frequency in one learning session increases learning initially, its benefits may disappear after a certain frequency level.
METHOD
Participants
The original pool of participants consisted of 116 Japanese first-year engineering students (aged 15–16) from three EFL classes at a technical college in Japan. Eighteen students were excluded because they were absent on the day of the experiment, leaving 98 students. The students from the three classes were randomly assigned to one of the four retrieval frequency groups: Retrieval 1, 3, 5, and 7. Although no data were available regarding their English proficiency, an analysis of learning phase performance found no statistically significant difference among the four retrieval frequency groups in their ability to learn L2 vocabulary (see Appendix S1 in the online supplementary materials for details). Each of the Retrieval 1, 3, and 5 groups consisted of 24 students, whereas the Retrieval 7 group consisted of 26 students. The difference in the number of students resulted from the absence of students.
Target and Filler Items
Sixteen English-Japanese word pairs (e.g., husk – 皮) and three additional word pairs were chosen as target and filler pairs, respectively. The number of target word pairs was determined based on the results of pilot studies conducted with 53 Japanese learners of English. The target and filler word pairs were chosen from three previous experiments (Nakata, Reference Nakata2015a; Reference Nakata and WebbNakata & Webb, in press, Experiments 1 & 2), which involved 301 Japanese college students. To ensure that items would be unfamiliar to the participants, English words that were beyond the most frequent 9,000 word families in the British National Corpus frequency lists (Nation, Reference Nation2006) were used (target items: apparition, billow, citadel, dally, gouge, grig, husk, jibe, levee, loach, mane, mirth, rue, toupee, urn, warble; filler items: lava, tyro, valor). Ten of the target items were nouns, and six were verbs, approximating the 6:4 ratio of nouns to verbs in natural text (e.g., Webb, Reference Webb2005, Reference Webb2009). To lower the learning burden, relatively short words were selected. Filler pairs were used as primacy and recency buffers during the treatment (see Procedure for details). Although filler pairs were studied and tested like target pairs, they were excluded from the analysis.
Pretest
Two types of pretests, productive (i.e., form recall) and receptive (i.e., meaning recall), were administered in that order in this study. The productive pretest required learners to produce the English (L2) word form corresponding to the Japanese (L1) translation given (e.g., 皮 = ____?). So that participants would not supply synonyms for a target word (e.g., skin for husk), one letter in the target word and the number of letters in the word were provided as a hint (e.g., _ u _ _ for husk). The receptive pretest required students to provide the Japanese translation corresponding to the English word form given (e.g., husk = ____?). Sixteen target word pairs and three filler word pairs were tested in the pretest. The order of items in the test was randomized by the computer program for each participant. Footnote 2
Posttest
Productive and receptive posttests were administered in that order at three retention intervals: immediately, 1 week, and 4 weeks after the treatment. Unlike in the pretest, the hints (e.g., _ u _ _ for husk) were not given in the productive posttest. Apart from this, the posttests were exactly the same as the pretests. The same 19 word pairs (16 target and three filler) were tested on the immediate and delayed posttests. The delayed posttests were conducted with no prior notice. As in the pretest, the item order was randomized by the computer program for each participant.
Procedure
The study was administered during regular class hours using computer software developed by the author. Participants studied and were tested individually. Participants first took productive and receptive pretests. After the pretests, students studied 19 English-Japanese word pairs (including three filler pairs) using a computer-based flashcard program. At the beginning of the treatment, the English-Japanese word pairs (e.g., husk – 皮) appeared on the participants’ individual monitors for 8 seconds each (initial presentation). The three filler pairs were presented at the beginning, thus serving as primacy buffers (e.g., Nakata, Reference Nakata2015a). The initial presentation was followed by one or more retrieval cycles. There were one, three, five, and seven retrieval cycles for the Retrieval 1, 3, 5, and 7 groups, respectively. Each word pair was practiced only once in each retrieval cycle. During the retrieval cycles, word pairs were practiced in a productive format. More specifically, participants were required to type the English (L2) word form corresponding to the Japanese (L1) translation given (e.g., 皮 = ____?). Participants were allowed to take as much time as they needed to respond. (Because the treatment was self-paced by participants, the average study time was analyzed after the experiment. See Efficiency, the Results section for details.) A productive format (from L1 to L2) was chosen because research shows that productive learning leads to adequate increases in receptive knowledge as well as large increases in productive knowledge and may be more effective than receptive learning (e.g., Mondria & Wiersma, Reference Mondria, Wiersma, Bogaards and Laufer2004; Steinel et al., Reference Steinel, Hulstijn and Steinel2007; Webb, Reference Webb2005, Reference Webb2009).
Each retrieval attempt was followed by feedback, where the target English-Japanese word pair (e.g., husk – 皮) was displayed for 5 seconds per response. For each participant, for half of the target pairs, feedback was given immediately after each response (immediate feedback). For the other half, feedback was withheld until all other word pairs were practiced (delayed feedback). The timing of feedback was manipulated because the current study was part of a larger study that looked at feedback timing as a variable (Nakata, Reference Nakata2015b). The treatment concluded with the final review, where the English-Japanese word pairs appeared on the screen for 5 seconds each. The three filler pairs were presented at the end, thus serving as recency buffers (e.g., Nakata, Reference Nakata2015a). The treatment was followed by a distractor task, which required participants to complete 10 two-digit additions (e.g., 27 + 66 = ?). The immediate posttest followed the distractor task. As a measure of long-term retention, participants took productive and receptive posttests 1 and 4 weeks after the treatment.
Scoring
Two scoring protocols, strict and sensitive, were employed in this study. In the productive test, strict scoring did not allow any misspellings. Sensitive scoring was a replication of the lexical production scoring protocol (LPSP-written) developed by Barcroft (Reference Barcroft2002). In sensitive scoring, scores of 0.00, 0.25, 0.50, 0.75, or 1.00 were given based on the number of letters that are correct (i.e., a given letter in the target word is placed in the same position in the response) and present (i.e., a letter in the target word is found in the response regardless of the position; for further details, see Barcroft, Reference Barcroft2002, Reference Barcroft2007). The responses on the productive test were scored automatically by a computer program developed by the author. In the receptive test, with strict scoring, responses were marked as incorrect if (a) an intransitive verb was provided for a transitive verb (e.g., 調和させる [transitive] for jibe) and vice versa or (b) responses were the wrong part of speech (e.g., 後悔 [noun] for rue). With sensitive scoring, the part of speech and transitive/intransitive distinction were ignored. To maintain consistency, responses on the receptive test were first scored by a computer program based on answer keys compiled by the author, a native speaker of Japanese. The answer keys were verified by two other native speakers of Japanese. Based on the answer keys, the responses on the receptive test were automatically categorized by the computer software into the following five categories: (a) correct (i.e., correct in both strict and sensitive scoring), (b) partially correct (i.e., correct only in the sensitive scoring system), (c) blank responses, (d) intralist errors (i.e., providing correct or partially correct Japanese translations for other target or filler items), and (e) others. The responses that were classified as (e) others (0.5% of all responses) were manually checked by the researcher and another native speaker of Japanese.
Analysis
The posttest scores and efficiency scores were analyzed by a three-way 4 (retrieval frequency: 1/3/5/7) × 3 (retention interval: immediate/1-week delayed/4-week delayed) × 2 (feedback timing: immediate/delayed) ANOVA. The alpha level was set to .05 for all analyses. When a significant effect was found, the Bonferroni method of multiple comparisons was used to investigate where the significant differences existed. Effect sizes and 95% confidence intervals of difference were also estimated. Because some target pairs were answered successfully on the receptive pretest, relative gains were analyzed for the receptive test. Relative gains were calculated based on the following formula (e.g., Rogers, Webb, & Nakata, Reference Rogers, Webb and Nakata2015):
$${\rm{Relative\ gain\ }}(\% ) = ({\rm{posttest\ score}} - {\rm{pretest\ score}})/(16 - {\rm{pretest\ score)}} \times 100$$
Note that in the above formula, 16 indicates the number of target items.
RESULTS
Pretest
None of the participants answered correctly on any of the target words on the productive pretest. The average receptive pretest score (SDs in parentheses) was 0.13 (0.34), 0.13 (0.34), 0.08 (0.28), and 0.12 (0.33) out of 16 with strict scoring and 0.13 (0.34), 0.17 (0.38), 0.08 (0.28), and 0.15 (0.37) out of 16 with sensitive scoring in the Retrieval 1, 3, 5, and 7 groups, respectively.
Posttest Performance
Table 1 provides the immediate and delayed posttest results for the four retrieval frequency groups. Cronbach’s alpha was .84 or higher (.84–.92) for all dependent measures, which indicates good reliability. The productive and receptive test scores were analyzed by a three-way 4 (retrieval frequency: 1/3/5/7) × 3 (retention interval: immediate/1-week delayed/4-week delayed) × 2 (feedback timing: immediate/delayed) ANOVA. Table 2 provides the results of the ANOVAs. As shown by the table, the ANOVAs found significant main effects of retrieval frequency on both productive and receptive tests regardless of the scoring procedure. None of the interactions involving retrieval frequency was significant. Due to the significant main effect of retrieval frequency, the Bonferroni method of multiple comparisons was used to investigate where the significant differences existed at each retention interval. The results of the multiple comparisons are summarized in Tables S2 and S3 in the online supplementary materials (Appendix S2). The multiple comparisons showed that regardless of the posttest, scoring system, and retention interval, the Retrieval 5 and 7 groups significantly outperformed the Retrieval 1 and 3 groups, producing medium to large effect sizes (0.88 ≤ d ≤ 2.04). However, no statistically significant difference existed between the Retrieval 1 and 3 groups, on the one hand, and between the Retrieval 5 and 7 groups, on the other, yielding no more than small effect sizes (d ≤ 0.44). The findings suggest the following order on all posttests: Retrieval 5 = 7 > 1 = 3.
Proportion of correct responses on the productive posttests and relative gains on the receptive posttests
Note. Standard deviations in italics. Strict = strict scoring; Sensitive = sensitive scoring.
Results of three-way ANOVAs for the posttest scores
Efficiency
Participants spent 6.48 (1.12), 12.03 (1.73), 17.98 (1.59), and 22.82 (3.31) minutes (SDs in parentheses) on average studying the target word pairs in the Retrieval 1, 3, 5, and 7 groups, respectively. To examine the efficiency of the four retrieval frequency levels, efficiency scores were calculated by dividing the posttest score by the study time (minutes; e.g., Ellis, Reference Ellis1995; Kornell, Reference Kornell2009; Mondria, Reference Mondria2003). Table 3 provides the efficiency scores in the four groups. The efficiency scores were entered into a three-way 4 (retrieval frequency: 1/3/5/7) × 3 (retention interval: immediate/1-week delayed/4-week delayed) × 2 (feedback timing: immediate/delayed) ANOVA. Table 4 shows the results of the ANOVAs. The ANOVAs showed significant main effects of retrieval frequency on both productive and receptive tests regardless of the scoring system. Although none of the interactions involving feedback timing were statistically significant, the interaction between the retrieval frequency and retention interval was significant on both productive and receptive tests regardless of the scoring procedure. Due to the significant interaction, the simple main effect of retrieval frequency was tested. The simple main effect of retrieval frequency fell short of significance with strict scoring on the 4-week delayed productive posttest, F (94, 3) = 2.52, p = .062, but was significant on all other posttests, F (94, 3) ≥ 2.92, p ≤ .038.
Average efficiency scores in the four retrieval frequency groups
Note. Standard deviations in italics. Strict = strict scoring; Sensitive = sensitive scoring.
Results of three-way ANOVAs for the efficiency scores
To follow-up the significant simple main effect, the Bonferroni method of multiple comparisons was used. The results of the multiple comparisons are summarized in Tables S4 and S5 in the online supplementary materials (Appendix S3). The multiple comparisons showed the following three findings. First, the Retrieval 1 group was significantly more efficient than the other three groups on all posttests except with strict scoring on the 1-week and 4-week delayed productive posttests, yielding medium to large effect sizes (0.71 ≤ d ≤ 3.15). Second, the Retrieval 3 group was more efficient than the Retrieval 7 group with sensitive scoring on the immediate productive posttest and with strict and sensitive scoring on the immediate receptive posttest, producing large effect sizes (1.16 ≤ d ≤ 1.39). Third, the differences were not statistically significant in all other cases. Overall, the results indicate that (a) the Retrieval 1 group was the most efficient among the four groups, and (b) the Retrieval 3 group was more efficient than the Retrieval 7 group on the immediate posttests except with strict scoring on the productive test.
DISCUSSION
The first research question in this study was concerned with the effectiveness of the four retrieval frequency levels. Based on the distributed practice effect and previous research (e.g., Folse, Reference Folse2006; Peters, Reference Peters2012, Reference Peters2014; Rohrer et al., Reference Rohrer, Taylor, Pashler, Wixted and Cepeda2005; van Zeeland & Schmitt, Reference van Zeeland and Schmitt2013), Hypothesis 1 predicted that increasing retrieval frequency in one learning session increases effectiveness immediately and 1 week after learning, but not 4 weeks after learning. The findings of this study, however, did not support this hypothesis: The Retrieval 5 and 7 groups significantly outperformed the Retrieval 1 and 3 groups not only immediately and 1 week after, but also 4 weeks after the treatment. As shown in Tables S2 and S3 in the online supplementary materials, larger effect sizes were observed on the immediate posttest (0.99 ≤ d ≤ 2.04) than on the 1-week (0.88 ≤ d ≤ 1.34) or 4-week delayed posttest (0.93 ≤ d ≤ 1.37). This suggests that the advantage of within-session repeated retrieval perhaps diminished over time. At the same time, the effect sizes observed on the delayed posttests still reached the threshold of a medium (d = 0.70) or large effect (d = 1.00; Plonsky & Oswald, Reference Plonsky and Oswald2014). The findings of this study are consistent with previous L2 vocabulary studies that found long-term (Peters, Reference Peters2012, Reference Peters2014) as well as short-term benefits of within-session repeated retrieval (Folse, Reference Folse2006).
Why was the advantage of repeated retrieval also found 4 weeks after learning, contrary to the prediction of Hypothesis 1? One possible explanation is that the negative effects of massing practice were outweighed by the positive effects of increased frequency. The findings add to the existing literature suggesting the importance of repetition in L2 vocabulary learning (e.g., Bisson et al., Reference Bisson, van Heuven, Conklin and Tunney2014; Chen & Truscott, Reference Chen and Truscott2010; Elgort & Warren, Reference Elgort and Warren2014; Folse, Reference Folse2006; Peters, Reference Peters2012, Reference Peters2014; Webb, Reference Webb2007a). Another possible explanation is that the retention interval for the delayed posttest was not long enough. Research on the distributed practice effect shows that although massed practice facilitates learning in the short term, it leads to poor retention over time (e.g., Bird, Reference Bird2010; Cepeda et al., Reference Cepeda, Pashler, Vul, Wixted and Rohrer2006, Reference Cepeda, Coburn, Rohrer, Wixted, Mozer and Pashler2009). This suggests that the benefits of within-session repeated retrieval, which involves shorter spacing than retrievals distributed over multiple days, will increasingly diminish and may completely disappear in the long term. If the posttest had been given after a delay greater than 4 weeks (e.g., 9 weeks; Rohrer et al., Reference Rohrer, Taylor, Pashler, Wixted and Cepeda2005), therefore, we might not have observed benefits of within-session repeated retrieval. In future research, it may be useful to identify the retention interval at which the positive effects of within-session repeated retrieval disappear. Another possible explanation is that repetition affects retrieval-based and non-retrieval-based learning conditions differently. Although van Zeeland and Schmitt (Reference van Zeeland and Schmitt2013) failed to find an advantage of repeated input 2 weeks after learning, the present study found that the benefits of within-session repeated retrieval persist at least 4 weeks after the treatment. Because retrieval increases learning more than input (e.g., Barcroft, Reference Barcroft2007; Karpicke & Roediger, Reference Karpicke and Roediger2008; Royer, Reference Royer1973), the positive effects of within-session repeated retrieval may be more durable than those of within-session repeated input.
Although the Retrieval 5 and 7 groups significantly outperformed the Retrieval 1 and 3 groups, no statistically significant difference existed between the Retrieval 1 and 3 groups, on the one hand, and between the Retrieval 5 and 7 groups, on the other. The results suggest that although within-session repeated retrieval generally facilitates vocabulary learning, its benefits may not necessarily be observed depending on the range of retrieval frequency levels used (hence, Retrieval 5 > 1, but Retrieval 3 = 1). Theoretically, the results suggest that learning may not occur incrementally but in an all-or-none fashion (one-trial learning controversy; see Roediger & Arnold, Reference Roediger and Arnold2012, for a review).
One caveat to be considered is that scores of the Retrieval 5 and 7 groups approached the ceiling on the immediate productive and receptive posttests (Table 1). As a result, the lack of statistical significance between these two groups on the immediate posttest could be in part due to a potential ceiling effect. The delayed posttests, however, did not show a ceiling or floor effect. The lack of significant differences between the Retrieval 5 and 7 groups on the delayed posttest, thus, suggests that seven retrievals may not significantly increase long-term retention compared with five. Because scores on the delayed posttest are more important than those on the immediate posttest from a pedagogical standpoint, this study nonetheless has pedagogical significance. At the same time, although the present study found statistically significant differences, the 95% confidence intervals (CIs) of difference in the means were relatively wide. The 95% CI of difference in the means between repeated (five and seven) and fewer retrievals (one and three) for the posttest scores ranged from 7% to 49% (Tables S2 and S3 in the online supplementary materials). Given the relatively wide CI of difference, it may be useful to replicate this research with a larger sample size to gain a more precise estimate of treatment effect.
The second research question was concerned with the efficiency of the four retrieval frequency levels. Based on the retrieval effort hypothesis and desirable difficulty framework, Hypothesis 2 predicted that within-session repeated retrieval is less efficient than fewer retrievals. The findings of this study supported Hypothesis 2. The results showed that the Retrieval 1 group was the most efficient regardless of the retention interval. Theoretically, the findings are consistent with the retrieval effort hypothesis and desirable difficulty framework, according to which the positive effects of repeated retrieval attenuate as a function of the retrieval frequency. Most learning occurred in the first retrieval attempt, and practicing retrieval more than once did not provide a benefit that justified its cost (i.e., increased study time). The findings are also consistent with previous L2 studies showing that the incremental benefit to learning decreases as the number of repetition increases (Bisson et al., Reference Bisson, van Heuven, Conklin and Tunney2014; Pyc & Rawson, Reference Pyc and Rawson2009; van Zeeland & Schmitt, Reference van Zeeland and Schmitt2013). Methodologically, the findings underscore the importance of controlling time-on-task when comparing the effects of different treatments. It should be noted that although the findings of this study showed that within-session repeated retrieval may not be efficient, they do not necessarily deny the value of increasing retrieval frequency over multiple sessions. This is because when spacing between retrievals is long enough to induce forgetting (e.g., 1 month), the subsequent retrieval can be as difficult as or more difficult than the initial retrieval. Because increasing retrieval frequency over multiple sessions may allow learners to benefit from the positive effects of both increased frequency and retrieval difficulty, repeated retrieval over multiple sessions may be efficient unlike within-session repeated retrieval. In future research, it may be valuable to identify the interval after which repeated retrievals become more efficient than fewer retrievals.
Next, let us consider the 95% CI of difference for the efficiency scores. As shown by Table S5 in the online supplementary materials, on the receptive test, the CI of difference between Retrieval 1 and the other three groups ranged from 0.41 to 1.34. This means that one retrieval allows learners to acquire 24.6 (0.41 × 60 = 24.6) to 80.4 (1.34 × 60 = 80.4) more words per hour on the receptive test. Considering that PAL allows learners to learn around 33–166 words per hour on average (e.g., Nation, Reference Nation2013; Thorndike, Reference Thorndike1908), it may be possible to argue that the advantage of one retrieval over repeated retrievals is not only statistically but also practically significant. On the productive posttest, however, the results regarding the CI were not consistent across the retention intervals. On the immediate productive posttest, the CI of difference between Retrieval 1 and the other three groups ranged from 0.27 to 1.27. This corresponds to 16.2 to 76.2 words per hour and should be considered practically significant. With sensitive scoring on the delayed productive posttest, however, the CI ranged from 0.15 to 0.72 (1-week delayed) and 0.06 to 0.63 (4-week delayed), which correspond to 9.0 to 43.2 and 3.6 to 37.8 words per hour, respectively. (Because no significant difference was found in the efficiency scores with strict scoring on the delayed posttest, only results with sensitive scoring are reported for the delayed posttest.) Considering that the lower end of the CI is close to zero (1-week delayed: 0.15; 4-week delayed: 0.06), it may be reasonable to assume that one retrieval is not necessarily efficient 1 or 4 weeks after learning for the acquisition of productive knowledge. This is possibly because the delayed productive posttest was the most demanding dependent measure. Hence, practicing retrieval only once did not, perhaps, guarantee successful performance on this measure, and more retrievals were needed for the long-term retention of productive knowledge.
Pedagogical Implications
One pedagogical implication of this study is that increasing retrieval frequency in one learning session is not necessarily desirable, at least for the short-term retention of productive knowledge and short- and long-term retention of receptive knowledge. This is based on the findings that (a) the Retrieval 1 group led to the largest gain when time-on-task was controlled and (b) one retrieval allows learners to acquire 16.2 to 76.2 more words per hour on the immediate productive test and 24.6 to 80.4 more words per hour on the immediate and delayed receptive tests. When learners practice retrieval repeatedly in one learning session, retrievals will be increasingly easier and more successful. This may increase learners’ confidence and create “an illusion of effective learning” (Kornell, Reference Kornell2009, p. 1302). However, it should be noted that within-session repeated retrieval is not necessarily efficient.
At the same time, this study suggested that although within-session repeated retrieval is not necessarily efficient, it may be desirable in terms of effectiveness. This is because the Retrieval 5 and 7 groups significantly outperformed the Retrieval 1 and 3 groups regardless of the posttest, scoring system, and retention interval, producing medium to large effect sizes. If learners are concerned more with effectiveness rather than efficiency, hence, increasing retrieval frequency in one learning session is advisable. Within-session repeated retrieval may be particularly valuable for the long-term retention of productive knowledge. This is because the CIs of difference showed that the Retrieval 1 group was not necessarily more efficient than the other groups on the delayed productive posttests.
The findings of this study are useful because they may help us to propose guidelines for how vocabulary should be learned from word lists or flashcards (e.g., Elgort, Reference Elgort2011; Royer, Reference Royer1973). Developers of computer-assisted language learning (CALL) materials, most of which include a retrieval component (Folse, Reference Folse2006; Nakata, Reference Nakata2011), may particularly benefit from the findings of this study. For instance, based on the findings of this study, CALL developers may use single retrieval when efficiency is the goal and encourage within-session repeated retrieval when effectiveness is the goal. When repeated retrieval is used, it may be useful to ensure that L2 words are practiced in increasingly difficult formats because gradually increasing retrieval difficulty may maximize learning (Logan & Balota, Reference Logan and Balota2008). For instance, when L2 words are practiced four times, they can be practiced in the following order: receptive recognition (multiple-choice), productive recognition, receptive recall, and productive recall (Laufer & Goldstein, Reference Laufer and Goldstein2004). Although this may sound cumbersome, it may be implemented relatively easily if computer programs are used.
Because retrieval is a major component of most vocabulary learning activities such as fill-in-the-blank exercises, true or false statements, multiple-choice questions, and word-definition matching besides PAL (Nation & Webb, Reference Nation and Webb2011), the findings of this study are also useful for teachers and materials writers. At the same time, the issue of whether the findings of the present study also apply to instructional activities other than PAL awaits future research. It should also be noted that because this study was concerned with the frequency of retrieval, the findings of this study may not necessarily be applicable to the frequency of input, which does not necessarily involve retrieval.
Limitations of the Present Study
Although the results of the present study are useful, the study suffers from some limitations. First, in the present study, the retention interval was manipulated within participants, and each participant took posttests three times (immediate, 1-week, and 4-week delayed). This might have affected posttest performance. It would be useful to manipulate the retention interval between participants in future research to avoid the effects that earlier tests have on later tests. Another limitation is that the effects of retrieval frequency may be influenced by a number of factors such as the difficulty or number of target items, nature of learning tasks, type of posttests, or learners’ memory capacity. As a result, the findings of this study may not necessarily be applicable to contexts other than those of this study. Especially, it should be noted that the treatment in this study was restricted to PAL. Although the use of PAL is beneficial in that it allows the experimenter to exert strict control over the treatment, it may have limited ecological validity. Further research investigating the effects of within-session repeated retrieval on vocabulary learning during other kinds of tasks such as reading (Barcroft, Reference Barcroft2013) would be a useful follow-up to this study.
SUPPLEMENTARY MATERIAL
To view supplementary material for this article, please visit https://doi.org/10.1017/S0272263116000280.
