Background
In recent years, vocabulary has become widely recognized as a central element of second language acquisition. This has led to the development of a large number of vocabulary tests aimed at measuring L2 learners’ vocabulary knowledge (e.g., Schmitt, Nation, & Kremmel, Reference Schmitt, Nation and Kremmel2020). Keeping track of vocabulary knowledge through testing can provide teachers with a rich account of their students’ L2 performance and growth. Recent evidence suggests that tests can also be employed to enhance learning, as evidenced by retrieval practice effects on L2 vocabulary retention (e.g., Barcroft, Reference Barcroft2007; Nakata,Tada, Mclean, & Kim, Reference Nakata, Tada, Mclean and Kim2021; van den Broek, Takashima, Segers, & Verhoeven, Reference van den Broek, Takashima, Segers and Verhoeven2018). These studies have found that taking practice tests can help consolidate and increase the accessibility of knowledge.
A less commonly studied effect of testing is its potential to enhance learning of new information, even before that information has been studied (Pan & Carpenter, Reference Pan and Carpenter2023). Pretesting effects, also labelled prequestion effects, occur when learners attempt to respond to test items before a study phase. Evidence that pretesting can enhance future learning has been observed for various study materials, including word lists (e.g., Yang, Potts, & Shanks, Reference Yang, Potts and Shanks2017), paired associates (e.g., Grimaldi & Karpicke, Reference Grimaldi and Karpicke2012), and trivia facts (e.g., Richland et al., Reference Richland, Kornell and Kao2009). Although pretesting is a less common practice in the L2 classroom, recent findings suggest that testing before studying can likewise enhance L2 vocabulary gains (e.g., Seabrooke, Hollins, Kent, Wills, & Mitchell, Reference Seabrooke, Hollins, Kent, Wills and Mitchell2019).
Pretesting effects are not only pedagogically relevant but may also hold methodological implications for studies adopting a pretest-posttest design to measure the effect of an L2 learning intervention. The idea that pretesting may inflate L2 vocabulary gains is recognized by many SLA researchers (e.g., Durrant, Siyanova-Chanturia, Kremmel, & Sonbul, Reference Durrant, Siyanova-Chanturia, Kremmel and Sonbul2022), but very little research has been done to examine its impact on learning outcomes. Inadvertent pretesting effects may be particularly impactful in the context of incidental vocabulary learning, where reported learning effects from one-off interventions are typically small and highly variable (e.g., Webb, Reference Webb and Webb2020). This means that any error introduced by pretesting effects could have a strong impact on research findings. The current study explores pretesting effects on incidental vocabulary learning outcomes from L2 reading. By comparing pretested and nonpretested learning conditions in an experimental setting, our study examines the interaction between testing and subsequent word learning during meaning-focused reading.
The pretest-posttest design in incidental L2 vocabulary learning
L2 learning activities can be considered incidental when their main focus is on meaning comprehension or communication, rather than on linguistic form (e.g., Schmitt, Reference Schmitt2010). Examples include reading books, watching television for leisure, or attending academic lectures. During these activities, the main focus is on input comprehension, but vocabulary knowledge may be picked up from context, either subconsciously or by temporarily shifting focus, for example, to infer the meaning of a novel word from context (Webb, Reference Webb and Webb2020). There is now a considerable amount of evidence that L2 learners can acquire new words from activities such as TV viewing (e.g., Peters & Webb, Reference Peters and Webb2018) or listening to songs (e.g., Pavia, Webb, & Faez, Reference Pavia, Webb and Faez2019). By far the most studied type of incidental learning activity has been L2 reading (see Webb, Uchihara, & Yanagisawa, Reference Webb, Uchihara and Yanagisawa2023).
The vast majority of studies investigating the effects of reading on L2 vocabulary knowledge have employed a one-off intervention exposing learners to a relatively short reading passage before testing their knowledge of L2 vocabulary from the input. An important methodological element of such intervention studies is the control of preexisting vocabulary knowledge. Prior knowledge control ensures that observed learning gains result from learners’ engagement with the reading materials presented during the experiment, not from knowing the L2 words prior to the study (Durrant et al., Reference Durrant, Siyanova-Chanturia, Kremmel and Sonbul2022; Nation & Webb, Reference Nation and Webb2011; Schmitt, Reference Schmitt2010).
The most frequent approach to prior knowledge control in incidental L2 vocabulary learning has been the pretest-posttest design (Uchihara, Webb, & Yanagisawa, Reference Uchihara, Webb and Yanagisawa2019). As opposed to the use of pilot tests or pseudowords, measuring gains from pre- to posttest can track learning for partially known lexical items. This is particularly useful when researchers are interested in measuring the development of various word knowledge aspects or the development from partial to full word knowledge, rather than treating vocabulary knowledge as an all-or-nothing phenomenon (e.g., Schmitt, Reference Schmitt2019). Pretesting may provide a fine-grained and accurate measurement of gradual vocabulary development. A potential disadvantage of using pretests is the risk of inadvertent pretesting effects. Answering test items prior to studying can improve memory for those items in a posttest, compared with a condition in which the same materials are studied without a pretest (Pan & Carpenter, Reference Pan and Carpenter2023). In cognitive psychology, this type of testing effect has been labelled the prequestion effect, or the pretesting effect (James & Storm, Reference James and Storm2019).
Pretesting effects on L2 vocabulary learning
Pretesting effects have been reported for a wide range of study materials, including written texts (e.g., James & Storm, Reference James and Storm2019), academic lectures (e.g., St. Hilaire & Carpenter, Reference St. Hilaire and Carpenter2020), paired associates (e.g., Hays, Kornell, & Bjork, Reference Hays, Kornell and Bjork2013), and trivia facts (e.g., Kornell, Reference Kornell2014). Testing before studying tends to be more, or at least equally effective, as studying the same information multiple times (e.g., Hays et al., Reference Hays, Kornell and Bjork2013) or for a prolonged amount of time (e.g., Kornell, Hays, & Bjork, Reference Kornell, Hays and Bjork2009; Little & Bjork, Reference Little and Bjork2016). For example, Kornell et al. (Reference Kornell, Hays and Bjork2009) found that participants who made incorrect guesses on invented trivia questions (e.g., “What peace treaty ended the Calumet War?”) before studying the correct responses (e.g., “Harris treaty”) performed similarly on posttests as participants who had twice as much study time. Findings such as these suggest that attempting a response before studying could be an efficient way to enhance learning.
Little is known about pretesting effects in relation to L2 vocabulary learning, and only a few studies have examined these effects directly using an experimental design. The current evidence for pretesting effects on L2 vocabulary acquisition is situated in memory research and focuses mainly on deliberate learning of translation pairs and paired associates, not on incidental L2 vocabulary learning (e.g., Potts & Shanks, Reference Potts and Shanks2014; Seabrooke, Mitchell, Wills, Inkster, & Hollins, Reference Seabrooke, Mitchell, Wills, Inkster and Hollins2022).
In a study by Potts and Shanks (Reference Potts and Shanks2014), participants learned unknown Euskara words and their English (L1) translations in three conditions: (a) by guessing the word’s meaning followed by corrective feedback, (b) by reading each word and its corresponding meaning, or (c) by guessing the meaning from two options, followed by feedback. On a multiple-choice posttest, participants recognized a higher number of meanings for words in the guessing (i.e., pretesting) conditions, compared with words that had been preread along with their translation or definition. The effect was particularly evident for words pretested at the level of meaning recall, with items pretested at the recall level yielding higher scores than multiple-choice guessing (d = .42) and pre-reading (d = .47). The authors proposed that the generation of errors benefited learning by affecting the degree of effort or attention to corrective feedback.
In a later study by Potts, Davies and Shanks (Reference Potts, Davies and Shanks2019), participants either attempted to retrieve the translations of L2 words immediately before or after receiving corrective feedback. The aim of this manipulation was to find out whether guessing would enhance learning either through semantic preactivation (e.g., Carpenter, Reference Carpenter2009), or by increasing learners’ curiosity and learning efforts (e.g., Geller et al., Reference Geller, Carpenter, Lamm, Rahman, Armstrong and Coffman2017). The study replicated Potts and Shanks’ (Reference Potts and Shanks2014) finding that preguessing the meanings of words could enhance the effect of studying the form-meaning link by more than 4% compared with a study-only condition, whereas postguessing (i.e., retrieval) did not (Experiment 1). Furthermore, curiosity ratings for learning the correct translation of a word were higher after incorrectly guessing the meaning, indicating that generating errors can arouse learners’ curiosity to learn the correct meaning of the word.
In five experiments, Seabrooke et al. (Reference Seabrooke, Hollins, Kent, Wills and Mitchell2019) examined whether Potts and Shanks’ (Reference Potts and Shanks2014) findings would extend to other aspects of knowledge, including cued recall, two-alternative forced choice tests, and associative recognition. The study found that pretesting led to improved recognition of unfamiliar English and Euskara words, in line with Potts and Shanks’ (Reference Potts and Shanks2014) findings. The study also extended the pretesting effect to tests of multiple-choice meaning recognition and a two-alternative forced-choice test. However, the effect did not extend to cued recall, associative recognition, or a multiple-choice test containing familiar distractors. These results suggest that pretesting effects on L2 vocabulary acquisition can depend on the word knowledge aspect that is tested.
Seabrooke et al. (Reference Seabrooke, Mitchell, Wills, Inkster and Hollins2022) likewise found pretesting effects on translation recognition. Participants (English native speakers) guessed both the semantic categories and specific meanings of unknown Finnish words before receiving corrective feedback. Both correct and incorrect guesses led to higher recognition scores than pre-exposure to the correct form-meaning pairs (d = .67). The authors proposed that making small errors might trigger learners’ curiosity and motivation, leading to enhanced processing and learning of the correct word meanings.
Similar effects of guessing have been reported for learning L2 multiword expressions (e.g., Strong & Leeming, Reference Strong and Leeming2024). However, recent evidence suggests that pretesting can also be detrimental to learners’ memory of multiword combinations, which tend to be easily confused with distractors or erroneous guesses (e.g., Li & Boers, Reference Li and Boers2024). It seems, therefore, that no single mechanism can explain all pretesting effects in L2 vocabulary learning, and that item and task variables may underlie differences in vocabulary processing and learning outcomes.
Pretest format effects
Pretesting can lead to different learning outcomes depending on the pretested word knowledge component or pretest format. In addition to raising overall curiosity, the expectations raised by a particular test format could induce test-relevant processing of the study materials (Chan et al., Reference Chan, Meissner and Davis2018). For example, when learners expect a free (unprompted) recall test after studying a list of target words, they tend to score higher on both free recall of word forms and multiple-choice posttests than when they expect a multiple-choice test (e.g., Balota & Neely, Reference Balota and Neely1980; Schmidt, Reference Schmidt1988). One explanation for this is that the higher perceived difficulty of a free recall test induces stronger engagement with the tested words or their semantic associations (e.g., Leonard & Whitten, Reference Leonard and Whitten1983; Schmidt, Reference Schmidt1988).
Test format differences have also been found in the opposite direction. Compared with cued recall, multiple-choice tests may induce more efficient learning of competitive related information (e.g., Little & Bjork, Reference Little and Bjork2016). For example, in Little and Bjork’s (Reference Little and Bjork2016) study, participants completed a pretest on trivia facts before reading a text passage about Yellowstone National Park and the planet Saturn. Half of the participants completed multiple-choice questions in the pretest (e.g., What geyser is thought to be the oldest in the world? a. Old Faithful, b. Steamboat Geyser, c. Castle Geyser, d. Daisy Geyser), whereas the other half received the same questions without multiple-choice options. The results showed that the multiple-choice group outperformed the cued recall group on posttest questions related to the distractors in the multiple-choice test, suggesting that processing of the multiple-choice options influenced the learners’ engagement with relevant information in the reading passage.
Different pretest formats may also induce qualitatively different encoding strategies (i.e., transfer-appropriate processing), whereby the expectation of one format leads to increased performance in that format but decreased performance in another format (e.g., Blaxton, Reference Blaxton1989; Finley & Benjamin, Reference Finley and Benjamin2012). For example, in a study focusing on word pair associates, Finley and Benjamin (Reference Finley and Benjamin2012) manipulated learners’ posttest expectations when studying a list of associated word pairs. Participants who were led to expect a cued recall test (i.e., retrieving one word based on its paired associate) spent more time and effort focusing on cue-target associations, whereas learners expecting a free recall test (i.e., listing as many of the target words as possible) focused more strongly on individual target words as well as semantic relations across different word pairs. In both groups, learning gains tended to be higher on the expected test format. Although Finley and Benjamin’s study did not focus on the effects of pretesting, transfer-appropriate processing might also play a role when different pretest formats are compared.
Taken together, the findings presented above show that not knowing the answer to a question in a pretest can lead to both quantitative and qualitative changes in processing during a subsequent study phase. Being pre-exposed to target words in a pretest may direct learners’ attention to those words when they are later encountered in the study phase. In addition, the process of making errors can affect learning efforts and strategies, and the nature of the test format can alter learners’ engagement with unknown words.
Pretesting effects on incidental L2 vocabulary learning
Very little is known about the scale and nature of pretesting effects in the context of incidental L2 vocabulary acquisition. In a recent meta-analysis, Webb et al. (Reference Webb, Uchihara and Yanagisawa2023) examined the moderating effect of prior knowledge control on incidental L2 vocabulary acquisition. Their results showed that the pretest-posttest design was associated with larger effects compared with other methods of prior knowledge control, such as pilot testing. This suggests that pretesting effects might impact the outcomes of studies examining incidental L2 vocabulary acquisition. However, few (if any) studies on incidental L2 vocabulary learning have directly controlled pretesting effects through the inclusion of a nonpretested comparison group. As a result, current estimates of pretesting effects might be confounded by variables such as item difficulty or partial item knowledge, which are likely to covary with the method of prior knowledge control.
One way of countering pretesting effects on incidental L2 vocabulary learning is by inserting a time gap of at least one week between the pretest and the learning intervention, to reduce learners’ memory of the pretest items before the start of the treatment (e.g., Durrant et al., Reference Durrant, Siyanova-Chanturia, Kremmel and Sonbul2022; Hulstijn, Reference Hulstijn, Doughty and Long2003; Swanborn & De Glopper, Reference Swanborn and De Glopper1999). Another widely used strategy is the inclusion of distractor items in the pretest to reduce the salience of pretested words (e.g., Nation & Webb, Reference Nation and Webb2011; Schmitt, Reference Schmitt2010). Although many intervention studies on incidental L2 vocabulary acquisition have employed such strategies, there is very little empirical evidence for their effectiveness.
The current study
Pretesting effects have not yet been empirically examined in the context of incidental L2 vocabulary learning, and very little is still known about how pretesting might affect the learning outcomes of meaning-focused L2 activities such as reading. There is good reason to assume that findings for L2 word-translation pairs may not extend to the context of incidental word learning through reading. Incidental learning conditions, which may involve contextual inferencing and an overall focus on meaning comprehension, are very different from studying words and their translations in isolation. Further, because a one-week delay is often inserted between the pretest and the treatment in incidental vocabulary learning experiments, the impact of pretesting effects on learning outcomes in these studies might be less outspoken than those reported in memory research (e.g., Potts & Shanks, Reference Potts and Shanks2014).
The present study provides an initial exploration of pretesting effects on incidental L2 vocabulary acquisition from reading. The study aims to examine the effects of pretesting on the outcomes of incidental vocabulary learning through reading. More concretely, the present study aims to estimate how strongly vocabulary posttest scores are influenced by combining a pretest and an incidental learning treatment, in comparison with a non-pretested condition, while considering the potential moderating effect of the pretest format. Further, our study aims to gain initial insight into pretesting effects on learners’ perceptions of the treatment and target items. To match realistic experimental conditions, the materials, instruments, and experimental procedures were designed to reflect common practices in studies examining incidental L2 vocabulary learning. In so doing, the study aims to provide greater insight into the methodological uses of pretests in L2 vocabulary research.
Research questions
The study addresses the following research questions:
-
1. To what extent does pretesting L2 vocabulary knowledge affect incidental vocabulary learning from L2 reading?
-
2. To what extent is the pretesting effect on incidental L2 vocabulary gains moderated by the pretest format?
-
3. How does pretesting in different test formats affect learners’ perceptions of the treatment and expectations of a posttest?
Participants
One hundred and forty-three students took part in the experiment. The participants were quasirandomly assigned to four experimental groups: a form recognition pretest group (n = 36), a meaning recall pretest group (n = 35), a meaning recognition pretest group (n = 35), and a no-pretest group (n = 37). According to the background questionnaire data (see Instruments), 120 participants identified as female, 19 as male, and two participants as nonbinary (two nonresponses). Participants were between 17–59 years old (M = 21.05 years old, SD = 4.92). The students started formal English instruction at the average age of 12 (SD = 1.83). All participants were Dutch-speaking university students (L1 = Dutch) from various undergraduate programs in Flanders, including 64 English majors. Participants’ total score on the Vocabulary Levels Test (VLT, Schmitt, Schmitt, & Clapham, Reference Schmitt, Schmitt and Clapham2001) served as a proxy of their English lexical proficiency. The VLT was also used to estimate participants’ lexical coverage of the reading materials (see Materials). This test was deemed appropriate for the current study because it covers a wide range of frequency levels and can therefore better differentiate between learners from higher proficiency levels than other vocabulary level tests. Participants’ scores on the VLT are summarized in Table 1. The results of an analysis of variance (ANOVA) indicated that the four groups were matched in terms of total VLT score (F[3,139] = 0.57, p = .638). Some participants completed the experiment for partial course credit (n = 45), whereas others received a monetary reward.
Mean scores and standard deviations (in parentheses) for the Vocabulary Levels Test per experimental group
Materials
Twenty-four target words were used in the study. To minimize prior knowledge of the target items ahead of the learning phase, we selected low-frequency words that were unlikely to be familiar to the participants (e.g., nurdle, aphid, osier). The inclusion of rare words ensured minimal preexisting item knowledge, while maintaining comparability with existing pretest-posttest studies, including real words as opposed to pseudowords. The target words had an average lemma frequency of .42 per million occurrences in the Corpus of Contemporary American English (COCA, Davies, Reference Davies2008) with a maximum frequency of 1.1 per million. All target words were concrete nouns between five and seven letters long. Each of the target words appeared three times in a reading passage of 4,524 tokens. Two versions of the passage were created, each version containing a different set of 12 target words. The remaining 12 words were only included in the pre- and posttests as control items to control for practice effects and learning outside of the experiment. To control for item effects, the item sets were counterbalanced so that all words served in the experimental and control conditions. Participants were randomly assigned to one of the two item sets, so that they were pre- and post-tested on 24 target items, but encountered only 12 of those items in the reading passage.
The reading passage was a fictional story about space travel, inspired by Douglas Adams’ (Reference Adams1979) The Hitchhiker’s Guide to the Galaxy. The first version of the passage was generated by ChatGPT (OpenAI, n.d.), based on the following prompt: “Could you please write a short story about space travelers visiting a strange planet in the style of The Hitchhiker’s Guide to the Galaxy using the following words? [three target words].” The prompt was repeated until all target words had been incorporated into different story parts, each relating the characters’ visit to a different planet. The resulting text passages were then substantially edited to control the occurrence frequency of the target items and to increase the coherence and narrative flow of passages.
Each text passage was edited to ensure that 95% of running words belonged to the 3,000 most frequent English word families (Cobb, Reference Cobbn.d.). However, some low-frequency words (less than 1% of tokens in the text, e.g., voodoo, malfunction, chirp) were retained to decrease the target words’ perceptual salience, as well as to make the reading materials seem more authentic and natural. Because Flemish learners are regularly exposed to various sources of authentic English input (e.g., Peters, Noreillie, Heylen, Bulté, & Desmet, Reference Peters, Noreillie, Heylen, Bulté and Desmet2019), texts that are too heavily manipulated are likely to come across as unnatural, which may negatively impact learners’ motivation during reading (Peters, Puimège, & Szudarski, Reference Peters, Puimège and Szudarski2023). The contexts of the target words were edited so that their meanings could easily be inferred. The final story, which consisted of seven parts, was piloted with nine native speakers of EnglishFootnote 1 and 4 Dutch-English bilinguals. The text was piloted for naturalness, level of difficulty, level of enjoyment, and contextual information. The full list of target items, both versions of the final reading passage, and a description of the piloting procedure can be found in the OSF: https://osf.io/t47gq/.
Each section of the reading passage was followed by two true-or-false comprehension questions about the content of the story (14 questions in total). The questions served as an attention check throughout the treatment. They were piloted along with the reading passage to ensure that each question was unambiguous and could be answered without knowledge of the target words.
Instruments
Background questionnaire
Participants completed a background questionnaire about demographic information (e.g., gender, age, education level, first language) and contact with English at university (i.e., number of hours of English class per week) and at home (e.g., TV viewing with or without subtitles, gaming, etc.). Data pertaining to the participants’ amount of contact with English inside and outside the classroom were not analyzed as part of the present study.
Pretests
Three test formats commonly used in intervention studies on incidental L2 vocabulary acquisition were included as pretests in the present study: a meaning recall test, a meaning recognition test, and a form recognition test (see e.g., Webb et al., Reference Webb, Uchihara and Yanagisawa2023). All pretests comprised 40 items in total. Along with the 24 target and control words, 16 high-frequency fillers were included to raise the number of distractors to the 1:3 ratio recommended by Schmitt (Reference Schmitt2010). Each participant was assigned to one of the pretest formats, except for the no pretest comparison group. All pretests can be accessed in the OSF: https://osf.io/t47gq/.
The meaning recall test measured the participants’ ability to retrieve the correct meaning of the target words from memory, based on the isolated word form and without any additional cues. The test used a translation format in which participants were presented with the target word (e.g., nurdle) and were instructed to provide a Dutch translation, a definition, or a description of the word’s meaning. Participants were encouraged to write down any meaning that came to mind, even if they did not know the precise meaning of the word, and to type “I don’t know,” “/,” or “x” if they had no clue about the word’s meaning.
The meaning recognition test measured the participants’ ability to select the correct meaning of each target word from three options. Participants were instructed to select the meaning of the target word out of three options (the key and two distractors) and an “I don’t know” option. Participants were encouraged to select a response even if they were not entirely certain of the correct meaning, and to select the “I don’t know” option if they had no clue which was the correct answer. The key of each item was a Dutch translation of a high-frequency synonym of the target word, because close translations of the target words were of low frequency and therefore possibly unknown to the participants (e.g., “landengte” for isthmus). Distractors were Dutch words of similar frequency as the key (within 300 occurrences in the SUBTLEX NL word list, Keuleers, Brysbaert, & New, Reference Keuleers, Brysbaert and New2010). Care was taken to select distractors that did not occur in the story but were semantically plausible within the context of the story (e.g., words referring to animals or plants). None of the distractors were used more than once or appeared in the reading passage.
The form recognition test measured participants’ recognition of the target word form. In this test, participants were instructed to select all and only words they had seen or heard before. The pretest did not contain any pseudoword foils to match the length of the other test formats. Because prior knowledge control was not the primary aim of the pretests in the current study, the inclusion of pseudoword distractors was not deemed crucial to answer the research questions. Consequently, the pre- and posttest scores for this format might somewhat overestimate participants’ recognition of the target words.
Posttests
All participants completed three posttests, corresponding to the three formats used in the pretest. The form recognition posttest was administered first, followed by the meaning recall test and the meaning recognition test. The same 40 target words occurred in all three tests. The instructions of the form recognition test were edited slightly in the posttest: participants were instructed to indicate which words they remembered encountering in the reading passage.
Postreading questionnaire
Participants completed a short questionnaire enquiring about their perceptions of the reading task and their noticing of target words during reading. The questionnaire contained six statements with a five-point Likert scale ranging from “strongly disagree” to “strongly agree.” The statements were as follows: “I enjoyed reading the story,” “I found the content of the story easy to follow,” “I found the language of the story easy to understand,” “While reading, I mainly paid attention to the content of the story,” “While reading, I paid attention to difficult words in the text,” and “I expected to receive a vocabulary test after reading the text.” Participants in the pretested groups received one additional statement, “I noticed that words from last week’s test appeared in the story.” These participants were also asked if they had looked up any of the pretested items at home, and if so, which ones.
Procedure
The experiment was programmed in the Gorilla Experiment Builder (www.gorilla.sc, Anwyl-Irvine, Massonié, Flitton, Kirkham, & Evershed, Reference Anwyl-Irvine, Massonié, Flitton, Kirkham and Evershed2020). Participants completed the experiment individually or in small groups on computers under the supervision of the first or second author. A one-week delay was inserted between the pretest and treatment to reflect a common time course of intervention studies in incidental L2 vocabulary research (e.g., Horst, Cobb, & Meara, Reference Horst, Cobb and Meara1998; Feng & Webb, Reference Feng and Webb2020; Peters, Reference Peters2019). This time delay was chosen to avoid overestimating pretesting effects likely to affect learning outcomes in L2 vocabulary studies.
In the first session, participants first completed the background questionnaire, followed by the pretest corresponding to the assigned experimental condition. The pretest was announced as a measure of general English lexical proficiency. There were no time constraints on test completion. The pretest was followed immediately by the VLT, which was also completed without time constraints. After a one-week delay, participants returned to the lab to read the experimental text. They were instructed to carefully read the fictional story at their own pace. Participants were not forewarned of a vocabulary posttest, but were told that they would receive a comprehension test about the content of the reading passage. The reading passage was divided into 7 parts, each followed by two true-or-false comprehension questions. After reading, participants completed the postreading questionnaire and the three unannounced posttests.
The order of the two sessions was reversed for participants in the no-pretest comparison group to avoid any awareness-raising effects caused by taking the Vocabulary Levels Test. Participants in the no-pretest group thus completed the reading task and posttests in the first session, and the background questionnaire and VLT in the second session. All participants were debriefed both in spoken and written form at the end of the experiment. The full procedure is shown in Table 2.
Procedure
Scoring and analysis
Items of the pre- and posttests were scored dichotomously (1 point per correct response). In the meaning recall test, responses were coded as 1 (“correct”) when participants could provide a description, translation, or synonym in line with the contextual information provided in the reading passage. Responses referring to a vague semantic category (e.g., “an animal” or “a plant”) were coded as 0 (“incorrect”), because some target words belonged to the same category. Pre- and posttest scores of 35 participants (24%) were scored by two independent raters, with interrater agreement of 99.2%. In the meaning recognition test, correct responses were coded as 1 (“correct”), and selecting a distractor or the “I don’t know” option was coded as 0 (“incorrect”). In the form recognition test, items indicated as known or familiar received a score of 1, and new items received a score of 0.
Comparing posttest scores of target words and test-only control words enabled us to account for guessing or learning outside the experiment. Because our main independent variable was the presence or absence of a pretest (research question 1), we could not calculate gain scores or make a direct comparison between pre- and posttest scores for each of the conditions. Instead, differences in posttest scores between target and control items served as the primary source of evidence for learning from the experimental treatment. To reduce extraneous effects of look-up behavior on the learning outcomes, 39 data points (.003%) corresponding to items that participants had looked up at home (according to the postreading questionnaire) were excluded from the analysis.
To examine the effect of pretesting (RQ 1) and the pretest format (RQ 2) on the learning outcomes of the experimental treatment, we conducted generalized linear mixed models with binary item-level posttest scores as the dependent variable. The analyses were conducted using the glmer function in the lme4 package (Bates, Mächler, Bolker, & Walker, Reference Bates, Mächler, Bolker and Walker2015) in R (version 4.2.1, R Core Team, 2022). Three separate models were built, one for each posttest format. All models included fixed effects for Item type (target word or control word), Pretest condition (form recognition, meaning recognition, meaning recall, no pretest), and their interaction, as well as random intercepts for items and participants, and a by-participant random slope for Item Type. To interpret the interaction between Item type and Pretest condition, we used Bonferroni-adjusted pairwise comparisons with the emmeans function in the emmeans package (version 1.4, Lenth Reference Lenth2018). Effect sizes were calculated following the procedure outlined by Chinn (Reference Chinn2000). Model assumptions were checked using the DHARMa package (Hartig, Reference Hartig2022) and did not reveal any issues with collinearity or normality of the scaled residuals.
To examine the effects of pretesting on learners’ perceptions of the experimental treatment (RQ 3), we analyzed the results of the postreading questionnaire. The proportion of responses for different points on the Likert scale was counted for each statement and compared across pretesting conditions. We only report descriptive results for the questionnaire data, using dot plots generated in JASP (JASP Team, 2024).
Resuts
Pre- and posttests
The descriptive results of the pre- and posttests are presented in Table 3. In the meaning recall test, the average difference between pre- and posttest scores was 2.11 (SD = 2.14, range = 0–9). In the meaning recognition format, average gains were slightly higher: 2.97 (SD = 2.60, range = 0–9) new meanings were recognized in the multiple-choice test. In the form recognition test, participants could, on average, identify 6.08 out of 12 new target words (SD = 2.74, range = 1–11).
Average percentage of correct responses on pre- and posttests and standard deviations (in parentheses) by Pretest format and Item type
Note: Pretest and posttest scores that are directly comparable due to their matching format appear in boldface.
Results of the mixed models indicated a statistically significant main effect of Item type on all outcome measures. In the form recognition posttest, the odds of obtaining a correct response were 22.45 times (d = 1.72, p < .001) higher for target words seen in the experimental texts than for test-only control words. Likewise, odds of a correct response in the meaning recognition test (OR = 2.51, d = .51, p < .001) and in the meaning recall test (OR = 6.48, d = 1.03, p < .001) were significantly higher for target items than for control items.
In terms of a pretesting effect, there was no significant main effect of Pretest condition in any of the models, and no significant interactions between Item type and Pretest condition in the form recognition posttest and the meaning recall posttest. In the meaning recognition posttest, the interaction between Item type and Pretest condition reached significance (OR = 1.94, d = .37, p = .026) for the comparison between the no pretest group and the form recognition group, and approached significance for the comparison between the no pretest group and the meaning recall group (OR = 1.75, d = .31, p = .064). However, pairwise comparisons with Tukey adjustment did not reveal any significant differences between pretesting conditions for target or control items. The results are summarized in Figures 1–3, and the full model output can be found on the OSF: https://osf.io/t47gq/.
Predicted probabilities of a correct response in the form recognition posttest by group and item type. Error bars show 95% confidence intervals. Dots represent item variability jittered within groups.
Predicted probabilities of a correct response in the meaning recognition posttest by group and item type. Error bars show 95% confidence intervals. Dots represent item variability jittered within groups.
Predicted probabilities of a correct response in the meaning recall posttest by group and item type. Error bars show 95% confidence intervals. Dots represent item variability jittered within groups.
Postreading questionnaire
The results of the postreading questionnaire indicate that there was only minor variation between groups in terms of participants’ level of enjoyment of the reading passage, and in the perceived level of difficulty of the text in terms of content and language. All participants reported a high degree of attention to the text’s content as well as the difficult words in the text. Participants in all pretested groups looked up target words at home, but there were some differences between groups in terms of self-reported look-up behavior. In the form recognition and meaning recognition groups, only 11.77% and 10.71% of participants reported looking up target words at home, compared with 46.43% in the meaning recall group. Furthermore, responses to the five-point Likert items indicated that 45.71% of participants in the meaning recall group agreed (4) or strongly agreed (5) that they had recognized target words from the pretest while reading the experimental texts, compared with 30.55% in the form recognition group, and 28.56% in the meaning recognition group. Finally, 40% of participants in the meaning recall group agreed (4) or strongly agreed (5) that they had expected a vocabulary posttest, compared with 33.32% in the form recognition group, 26.70% in the meaning recognition group, and 35.12% in the no pretest group. Detailed results of the postreading questionnaire can be found in the OSF: https://osf.io/t47gq/.
Discussion
The present study set out to explore the effects of pretesting on the outcomes of incidental L2 vocabulary learning through reading. Firstly, the results show that posttest scores were significantly higher for target words encountered in the experimental reading passage than for test-only control items. This finding adds to existing evidence that significant vocabulary gains can arise from meaning-focused reading (e.g., Pellicer-Sánchez & Schmitt, Reference Pellicer-Sánchez and Schmitt2010). The descriptive results further show that gains from pre- to posttest were generally small and highly variable across individuals, as is typical of incidental vocabulary learning (Webb, Reference Webb and Webb2020). In line with previous studies (e.g., Pellicer-Sánchez & Schmitt, Reference Pellicer-Sánchez and Schmitt2010), gains were lower for meaning recall than for meaning and form recognition.
Regarding pretesting effects, the no pretest and meaning recognition groups had the smallest differences in posttest scores between target and control items, and the lowest posttest scores overall. However, looking at the target items, there were only minor differences in posttest scores across test formats. The descriptive results suggest that the inclusion of a form recognition or meaning recall pretest amounted to a difference of only one additional correct item out of twelve, two at the most. Further, no clear differences in learning outcomes were found between the different pretest conditions in the generalized linear mixed models. Although a significant interaction between pretesting condition and item type emerged in the meaning recognition posttest, pairwise comparisons did not reveal any significant differences between pretesting conditions. It is also noteworthy that the difference in posttest scores between target and control words was much more evident than the difference between pretest conditions. This shows that a treatment effect can clearly be discerned irrespective of the pretesting condition. The strong test-potentiated learning effects reported by studies focusing on word lists and translation pairs (e.g., Seabrooke et al., Reference Seabrooke, Hollins, Kent, Wills and Mitchell2019), were thus not found in the current study.
Several components of our experimental design could help explain why the results do not match those of previous studies. Firstly, the aim of the present study was to closely reflect the conditions of real-life experiments on incidental L2 vocabulary learning (e.g., Feng & Webb, Reference Feng and Webb2020; Peters, Reference Peters2019). Unlike most studies examining pretesting effects on L2 vocabulary learning (e.g., Potts & Shanks, Reference Potts and Shanks2014), forced guesses were not applied in any of the pretests. Because theoretical accounts of pretesting effects have emphasized the role of effortful guessing, or attempted retrieval (e.g., Kornell et al., Reference Kornell, Hays and Bjork2009), the absence of forced guesses in the present study design might in part explain the nonsignificant pretesting effect, as well as the large amount of individual variability in both the posttest scores and the questionnaire responses. It seems plausible that some participants took greater effort to attempt responding to unknown target words in the pretest than others, potentially leading to attempted retrieval benefits in some, but not all, learners.
Furthermore, our study procedure included a one-week interval between the pretest and the treatment. Some studies on pretest-potentiated effects have found that pretesting effects all but disappear when a delay of more than 24 hours is inserted between the pretest and study phase, depending on the nature of the studied materials (e.g., Kornell, Reference Kornell2014). It seems plausible, therefore, that pretesting effects would be more pronounced if the pretest were directly followed by the reading treatment. Another explanation for the small between-group differences could be related to the low frequency of the target items used in the current study. It has been argued that pretesting effects are mediated by the semantic quality or meaningfulness of the information in the pretest (Kornell, Reference Kornell2014). It is possible that pretesting partially known vocabulary would trigger greater semantic preactivation or retrieval efforts than pretesting entirely new lexical items. Furthermore, rare words tend to attract attention during meaning-focused reading (e.g., Godfroid et al., Reference Godfroid, Boers and Housen2013), and the presence of low-frequency items might cause learners to expect a vocabulary test regardless of prior tasks or instructions. The difficulty of the target items, in combination with a lab-based research setting, might also explain the highly variable responses in the postreading questionnaire: even in the group that did not receive a pretest, 36% of participants reported that they had expected to receive a vocabulary posttest. Finally, a potential source of noise in our data set is associated with the inclusion of real-word target items instead of pseudowords. Because of the one-week gap between the pretest and treatment, differences in learning gains may partly stem from participants looking up target words outside the experiment. Although we excluded items that participants remembered looking up from the regression analysis, undetected look-up behaviors might account for some degree of variance in the data, and could thus help to explain why no clear pretesting effects were found in the present study.
Considering the small and nonsignificant differences between different pretest formats, no clear conclusions can be drawn regarding differences between test formats or the cognitive explanations underlying pretest format effects. However, results of the postreading questionnaire suggest that pretesting may have altered learners’ perceptions of the target items and the learning intervention. For example, participants in the meaning recall and form recognition groups more often recognized pretested items in the experimental text than in the other pretesting conditions. This may have to do with the way in which items were presented in these pretests: each test item contained only the target words and fillers, unlike the multiple-choice tests, which contained a higher amount of distracting information that might have confused learners’ memory of the test items (see also Li & Boers, Reference Li and Boers2024). Previous studies have found that when distractors are interchangeable or easily confused, pretesting effects tend to be less robust (e.g., Seabrooke et al., Reference Seabrooke, Hollins, Kent, Wills and Mitchell2019; Reference Seabrooke, Mitchell, Wills, Inkster and Hollins2022). Learners who completed the meaning recall pretest were also more prone to looking up target words at home. It is possible that learners’ inability to retrieve the correct meaning of target words from memory aroused their curiosity about the words’ meanings, especially for words that were already partly known or familiar (see Seabrooke et al., Reference Seabrooke, Mitchell, Wills, Inkster and Hollins2022).
Interestingly, between-group differences in perceptions of the experimental treatment did not seem to translate into different learning outcomes. For example, even though posttest anticipation was considerably higher in the meaning recall condition (40%) than in the other conditions, this group did not obtain significantly higher posttest scores. This apparent discrepancy merits further investigation, given that vocabulary learning outcomes in incidental conditions have been widely assumed to depend on affective and cognitive variables such as motivation and attention (e.g., Laufer & Hulstijn, Reference Laufer and Hulstijn2001; Yanagisawa & Webb, Reference Yanagisawa and Webb2021). The results seem to align with the findings of recent eye-tracking studies that brief pre-exposure to L2 words can impact learners’ attention to those words during reading, but does not necessarily affect incidental learning outcomes (Elgort & Beyersmann, Reference Elgort and Beyersmann2024; Elgort et al., Reference Elgort, Wetering, Arrow and Beyersmann2024). Future studies could more closely examine how pretesting differentially affects learning processes and outcomes, and what role individual variation plays in explaining pretesting effects. For example, it is plausible that learners’ personal goals and motivations could impact how they react to the presence of pretested vocabulary in a meaning-focused task.
Finally, results of the postreading questionnaire revealed that learners in all groups, not just pretested groups, paid attention to L2 vocabulary while reading, and that many learners in the nonpretested group anticipated a vocabulary posttest. This finding calls into question the incidental nature of the present learning intervention and of incidental learning experiments more generally. Researchers have argued that pretesting can alter the nature of a learning intervention by raising learners’ awareness of the study aims or by enhancing their attention to L2 vocabulary (e.g., Swanborn & De Glopper, Reference Swanborn and De Glopper1999). However, the current findings suggest that nonpretested conditions may likewise trigger focus on L2 word forms, perhaps in particular when target words are of low frequency and encountered in an experimental setting. In line with recent calls for a more cautious approach to defining incidental vocabulary learning (e.g., Webb, Reference Webb and Webb2020; Webb et al., Reference Webb, Uchihara and Yanagisawa2023), our findings highlight that a complete lack of intention cannot be assumed when measuring incidental vocabulary learning. More research is needed to obtain a better understanding of the cognitive processes that underlie incidental vocabulary learning and how pretests and other methodological variables might affect those processes.
Methodological implications
The present study was intended as a first empirical exploration of pretesting effects in incidental L2 vocabulary acquisition through reading. The small and nonsignificant differences between pretest conditions suggest that pretesting might not substantially impact incidental vocabulary learning outcomes, in particular when target items are low-frequency words and when there is a one-week gap between the pretest and treatment. Our results also suggest that even in the nonpretested condition, participants were likely to expect a vocabulary posttest. It thus seems that administering a vocabulary pretest one week in advance may not fundamentally alter learners’ expectations or perceptions of the learning intervention.
On the one hand, these findings suggest that the pretest-posttest design does not significantly affect the outcomes of incidental L2 vocabulary learning through reading, although more evidence is needed to support this conclusion. Furthermore, a relatively short interval between pretest and treatment might suffice to minimize pretesting effects on incidental L2 vocabulary learning. In other words, a gap of four weeks, which has been recommended by some researchers (e.g., Durrant et al., Reference Durrant, Siyanova-Chanturia, Kremmel and Sonbul2022) might not be necessary. As larger intervals provide for more opportunity to learn the target vocabulary elsewhere, limiting the size of the interval might, in certain contexts, be more interesting, as well as more practically feasible. However, as the current study did not manipulate the time interval between the pretest and treatment, the effect of timing remains an empirical question for future investigation.
On the other hand, our results suggest that interventions produce similar learning outcomes regardless of whether prior knowledge is controlled through pretesting. Therefore, not using a pretest seems a legitimate approach when target words are carefully selected to be unfamiliar to participants. However, this does not mean that low-frequency target items can or should serve as a substitute for pretests in all research contexts. Considering the incremental nature of L2 vocabulary acquisition (e.g., Schmitt, Reference Schmitt2019), incidental vocabulary learning may continue long after the very first exposure to a new word, for example, when forming nuanced semantic and collocational networks based on varied contextual exposures (e.g., Webb, Reference Webb and Webb2020). Pretesting may, therefore, still be beneficial when measuring gains for already partially known vocabulary. In addition, the questionnaire results suggest that the high salience of novel word forms might in itself trigger expectations of a vocabulary posttest. It seems, therefore, that incorporating low-frequency words is not necessarily a superior approach in terms of lowering participants’ awareness of the study aims.
Limitations and future directions
The present study explored pretesting effects in a highly specific research context. Consequently, methodological and theoretical implications are limited to the setting and design examined in this experiment. There is still much to be learned about the magnitude and nature of these effects under different circumstances. Firstly, more evidence is needed to compare different time intervals between the pretest and treatment. Pretesting effects are likely to be more pronounced when the delay is shorter than one week or when a distractor activity is inserted between the pretest and the treatment. Further, the present study focused on low-frequency words, which served as an additional form of prior knowledge control to ensure a high degree of comparability in learning outcomes between the pretested and nonpretested groups. Conversely, in studies employing a pretest-treatment-posttest design, target words can belong to a variety of frequency levels, enabling researchers to track incremental gains of partially known words. Examining pretesting effects for partially known words could be an interesting avenue for future research.
An important limitation of the present study is that no a priori power analysis was conducted to estimate the required sample size. The relatively small sample size combined with the between-participants comparison implies that the null effect found for pretesting conditions needs to be interpreted with caution. Future studies could adopt a full within-participants design, which tends to have higher statistical power, to verify the validity of the current findings. However, such a study would have to be carefully designed to account for global attention-raising effects (e.g., raising learners’ overall curiosity for L2 vocabulary), which may be more difficult to detect in a within-participants design.
Further, our study focused primarily on learning outcomes. Although the results of the postreading questionnaire provide some initial insight into potential causes of pretesting effects, the cognitive mechanisms that drive these effects remain poorly understood. To obtain a clear understanding of what cognitive factors might explain pretesting effects, other experimental designs and methods are needed. For instance, think-aloud protocols, stimulated recall, and eye-tracking technology could be useful in determining learners’ attentional processes and levels of engagement both during pretesting and reading. Finally, given the highly variable designs adopted to examine incidental L2 vocabulary acquisition, replication could be a fruitful approach to provide greater insight into pretesting effects, while keeping the experimental conditions and procedures of original studies largely intact. Future studies could replicate and extend the design of existing pretest-posttest studies through the inclusion of a no-pretest control group. In the long run, this could allow for a larger-scale systematic analysis of pretesting effects on L2 vocabulary learning.
Conclusion
The present study is the first to empirically examine the effects of pretesting on incidental vocabulary learning outcomes from L2 reading. The findings suggest that giving learners a vocabulary pretest one week before they encounter the pretested words in a meaning-focused reading task does not significantly affect incidental vocabulary gains, although pretesting may enhance learners’ curiosity for learning target words, as evidenced by learners’ responses in a postreading questionnaire. Overall, the presence or absence of a pretest did not interact strongly with the treatment’s outcomes. It thus seems that, when enough time is left between the pretest and the treatment, pretesting may present only a minor source of error in incidental L2 vocabulary acquisition through reading. Nonetheless, the widespread use of pretesting in the field underscores the need for further research into its effects on incidental L2 vocabulary acquisition.
Acknowledgments
We express our sincere gratitude to the study participants for their time and effort. We are also grateful to the reviewers and editor for their invaluable and insightful comments and suggestions.
Data availability statement
The experiment in this article earned Open Material badge for transparent practices. The materials are available at https://osf.io/t47gq/.
Funding statement
This work was supported by the Research Foundation - Flanders (FWO), where the first author had a junior postdoctoral fellowship (grant number 1213023N).
