2.1 Background to the study

The aim of Schmitt's (Reference Schmitt1998) study was to examine the incremental acquisition of L2 vocabulary over the course of a year. By tracking the acquisition of a small set of words, he was able to explore how the different components of vocabulary knowledge developed over time. Vocabulary research was booming in the late 90s. In Schmitt's (Reference Schmitt1998) words there was a ‘virtual explosion of vocabulary studies’ (p. 282). However, most of these studies focused on either the words that learners needed to know in order to operate in English (e.g. Goulden, Nation & Read Reference Goulden, Nation and Read1990) or the number of words that could be learned from reading (e.g. Huckin, Haynes & Coady Reference Huckin, Haynes and Coady1993) and from different types of explicit instruction (e.g. Avila & Sadoski Reference Avila and Sadoski1996). These studies were a crucial contribution to the advancement of vocabulary research but did not really explain how words were acquired over time. Importantly, vocabulary researchers in the late 90s were advocating for the need to look at the acquisition of a range of lexical components in order to understand the interrelationships among the different components and to provide empirical evidence for the descriptive component lists available (e.g. Richards Reference Richards1976; Nation Reference Nation1990). However, judgements about words being learned in most experimental studies were based on the measurement of form and meaning. Only a couple of studies had made some initial attempts to look at the acquisition of other word knowledge aspects (e.g. Nagy et al. Reference Nagy, Diakidoy and Anderson1993; Schmitt & Meara Reference Schmitt and Meara1997). The paucity of longitudinal vocabulary research, together with the need to adopt a multi-componential approach to vocabulary learning is what instigated the present study. Schmitt's study was the first one to examine how a wider range of word knowledge components of the same words developed over time, namely written form, associations, grammatical behaviour, and meaning. After 20 years of vocabulary research, Schmitt's (Reference Schmitt1998) study continues being unique. To my knowledge, no other studies have examined how knowledge of a range of lexical aspects of the same words develop over time in a group of learners.

This exploratory study followed the lexical development of three advanced learners of English who were doing a post-graduate course at an English university. The participants were initially measured on their knowledge of a set of 11 words. An important criterion in the selection of target items was that learners would need to be exposed to them during the course of their studies. Crucially, the target items needed to fall at various points of the acquisition continuum for participants, with some words well known, others partially known, and others unknown. This was confirmed in an initial pilot with 12 other students with similar characteristics. Eleven polysemous words from the University Word List (Xue & Nation Reference Xue and Nation1984, reprinted in Nation Reference Nation1990) and from the Brown word list (Francis & Kucera Reference Francis and Kucera1982) were included in the study. Knowledge of four components (written form, associations, grammatical behaviour, and meaning) of the eleven target items was measured three times in the course of a year. Vocabulary knowledge was measured in one-on-one interviews in which participants were asked to: recall the written form of the target items presented orally (spelling test), provide three associates for each target word, recall the word class of the form and the three derivative forms, and recall the three meaning senses of the target words (first unprompted and then using prompts).

Results of the study showed that acquiring the different meaning senses of the target words was not easy, with the vast majority of meaning senses staying at that same level. However, in those cases where meaning knowledge improved, little forgetting occurred. Also, items known at the productive level were less prone to decay. Making the move from no knowledge to productive mastery was the most challenging type of growth. Knowledge of the correct form of the target words seemed to be easier. Any level of meaning knowledge almost always implied knowledge of the correct spelling. The associations provided by two of the three participants progressed and became more native-like with time, whereas for the third participant they remained at about the same level. Interestingly, meaning associations seemed to improve as knowledge of meaning senses increased, whereas they were not that closely related to grammar knowledge. Only one of the three students showed development in grammatical knowledge. Overall, the results of the study showed that different components progressed in slightly different ways, with some aspects improving more than others, and that the strength of the relationship of different word knowledge components seems to vary between different knowledge types. However, no specific developmental sequencing of word knowledge components was found.

2.2 Approaches to replication

This highly cited study deserves replication because it was the first to look at the development of the different word knowledge components of the same words over time. Although some other similar attempts followed it (e.g. Churchill Reference Churchill2008; Bell Reference Bell, Fitzpatrick and Barfield2009; Fitzpatrick Reference Fitzpatrick2012), the study by Schmitt (Reference Schmitt1998) is still the only one that looked at the vocabulary growth of a group of learners, as opposed to a single case study, and that used different tests to examine the development of a range of word knowledge components. Despite the significant contribution of this original study to the field of vocabulary research, it remains a very small-scale, exploratory investigation. A series of replications might provide more validity to the initial findings and confirm and expand the patterns found in this original study.

A first approach to close replication would be to keep the aim, research questions and methods employed in the original study but to collect data from a larger, homogeneous group of participants. As explained above, Schmitt's (Reference Schmitt1998) study reported the vocabulary development of three learners, measuring changes in the knowledge of eleven words at three points in a year's time. As Schmitt argued, participant attrition is likely to occur in this type of longitudinal studies. However, since the study only involved three sessions with each participant over the course of a year, having a larger population would still seem feasible. Adding more participants would allow researchers to perform not only descriptive (as was the case in Schmitt Reference Schmitt1998) but also inferential statistical analysis, providing more robust findings and confirming the degree to which the results might be generalized to other learners. For example, running a two-way ANOVA with time (three time points) and lexical component (written form, associations, grammatical behaviour, and meaning) as independent variables and vocabulary scores as the dependent variable (provided that the assumptions of this test are met) would allow researchers to explore the main effect of time and type of word knowledge on vocabulary scores, as well as the interaction between these two factors. This close replication of the original study (Porte & McManus Reference Porte and McManus2019) would allow researchers to look at average proportion scores and compare them to the results of the original study, while at the same time adding statistical power to the original results through inferential statistics.

Several modifications in the assessment methods used in the original study should be considered for an approximate replication. The meaning senses in Schmitt's (Reference Schmitt1998) study were assessed first unprompted, simply asking participants to recall the meanings of each target word. Keywords were then used as a prompt to elicit the meaning senses that had not been provided in the unprompted phase. As Schmitt explains, this procedure attempted to tap into the receptive–productive meaning knowledge distinction. Receptive knowledge of the different meaning senses could also be examined by means of a multiple-choice test at the end of the procedure. This approximate replication would include the same unprompted, recall phase and the prompted recall phase that Schmitt used, but also an easier, multiple-choice recognition test that might be able to tap into smaller variations in the knowledge of meaning senses. Importantly, if we have these three levels of measuring meaning, all options in the final, multiple-choice items should be related to the key words provided in the prompted stage. Otherwise, the keyword might influence response behaviour in the recognition test. This modification in the measurement instruments would allow researchers to compare the two levels of receptive knowledge (prompted recall phase and multiple-choice test) to the original prompted scores and provide a more detailed picture of receptive–productive knowledge. Another option might be to have just the unprompted recall test and the less demanding, multiple-choice meaning recognition test. A second methodological modification in this approximate replication would involve the addition of a second rater for the scoring of test results, and the consequent report of inter-rater reliability. Schmitt (Reference Schmitt1998) provides a very detailed explanation of how the different tests were scored. Although the scoring of some of the tests (e.g. grammatical knowledge) was more straightforward, as the author acknowledges, the scoring involved a certain amount of subjectivity. Adding another rater would provide reliability to the original findings.

A further suggestion for approximate replication would involve some modifications in the design to be able to provide a more systematic examination of individual variation. The three students in Schmitt's (Reference Schmitt1998) study reported different strategies and approaches to vocabulary learning. For example, one of them reported having looked up the target words in the dictionary in between sessions, which seemed to support the acquisition of components like grammatical knowledge. Interesting individual variations were found among the three participants of the original study. However, the data collected and the authors' observations did not allow for a proper discussion of those individual differences. Thus, one methodological addition in an approximate replication would be to include a more structured interview about learners’ strategies and approaches to vocabulary learning. This would allow researchers to explain some of the differences observed among participants and verify the author's observations reported in the original study. In addition to strategies for learning, the original study by Schmitt could not provide any information about the type of exposure that learners had to the target items, which could be used to explain some of the patterns found. Churchill (Reference Churchill2008) used a diary to record the learner's conscious encounters with the target word. A second modification might therefore involve adding this type of learner's diary in order to get a better picture of the type of exposure that learners have to the target items. The triangulation of the data from the vocabulary tests, the structured interview, and the learners’ diaries would allow us to get better insights into learners’ strategies for vocabulary learning and of the amount and type of exposure that they had to the vocabulary items during the study, which would help to explain patterns of variability in the quantitative results. A final, fruitful area for close replication would be to modify the type of target items included in the study so that the learning gains for single words in the original study could be compared to the acquisition of formulaic language. It is widely acknowledged that learning vocabulary involves learning items beyond the single word (e.g. Wray Reference Wray2002; Schmitt Reference Schmitt2004; Wray Reference Wray2008; Wood Reference Wood2010, Reference Wood2015). In order to achieve high levels of proficiency in an L2, learners need to be able to use appropriately the many different types of formulaic sequences that exist in language (Siyanova-Chanturia & Pellicer-Sánchez Reference Siyanova-Chanturia, Pellicer-Sánchez, Siyanova-Chanturia and Pellicer-Sánchez2019). As Nation & Webb (Reference Nation and Webb2011) argue, ‘knowledge of multiword units can be approached in the same way as knowledge of single words’ (p. 189). The depth of the knowledge of a particular set of sequences can also be explored using a multi-component approach. Nation & Webb (Reference Nation and Webb2011) provide a comprehensive list of the components involved in knowledge of multiword units (adapted from Nation Reference Nation2001, for single words), including several aspects of the form, meaning and use of the sequences, both at the receptive and productive level. Given the prominent role that the study of formulaic language has in vocabulary research, the study by Schmitt could be replicated using the same methodology but changing the focus from individual words to formulaic sequences. This replication approach would allow researchers to compare the learning of formulaic sequences with learning gains for single words in the original study, a type of comparison that is scarce in vocabulary learning research (Pellicer-Sánchez in press). In fact, Schmitt (Reference Schmitt1998) initially included collocation as one of the aspects to measure but, as he explains, the measurement instrument needed further development and test results were eliminated. Given the nature of formulaic language and the many different types of sequences that exist in language, it is unlikely that one close replication study could explore the development of more than one or two types of formulaic sequences. Researchers should choose the specific type of formulaic sequence to include in the study. Some of the currently available lists of formulaic sequences might prove useful in the selection of target items. For example, researchers might decide to explore the development of polysemous phrasal verbs, for which the use of the PHaVE List (Garnier & Schmitt Reference Garnier and Schmitt2014) would be convenient.

3.1 Background to the study

Webb & Chang's (Reference Webb and Chang2012) study aimed at shedding light on the number of words that are learned over four years of instruction. Most research studies exploring changes in learners’ vocabulary size have focused on the investigation of relatively short interventions (e.g. Cobb & Horst Reference Cobb, Horst, Flowerdew and Peacock2001). As stated in the introduction to this paper, very few studies have looked at L2 learners’ vocabulary growth over longer periods of time, and the existing studies have yielded conflicting results. The studies by Milton (Reference Milton2006, Reference Milton2008) did show improvements in vocabulary size, whereas the study by Clark & Ishida (Reference Clark and Ishida2005) failed to show a difference in learners’ vocabulary size over the course of one semester. In addition, the few available studies have employed a cross-sectional design. The study by Webb & Chang (Reference Webb and Chang2012) is the only one that looked at the vocabulary growth of the same group of learners. As the researchers argue, previous research has shown that English as a foreign language (EFL) students often fail to learn even the most frequent words in English after several years of study (e.g. Nurweni & Read Reference Nurweni and Read1999), which points to the inefficiency of vocabulary instruction in some EFL contexts. Therefore, in order to inform language pedagogy, we need to have a better understanding of the number and type of words that are learned at different points in the learning process. To this end, Webb & Chang (Reference Webb and Chang2012) examined the number of words that EFL learners in Taiwan learned over four years of instruction, as well as the frequency profile of those words.

Participants in the study were 166 EFL learners in Taiwan. Participants were initially 15–16 years old and 20–21 at the point of the last data collection session. They were from six intact classes in a vocational senior high school. Importantly, the six classes differed in the amount of exposure to English they received. The six classes were divided into three groups according to the amount of English instruction. Classes belonging to Group A were oriented towards English language and literature and received a larger amount of English instruction (10–22 hours per week across the four years), whereas classes in Groups B and C took English language learning as one of the different study requirements and therefore the amount of instruction was less (0–4 hours per week across the four years). Participants’ vocabulary size was measured using a bilingual version of the Vocabulary Levels Test (VLT) (Nation Reference Nation1990; Schmitt, Schmitt & Clapham Reference Schmitt, Schmitt and Clapham2001), including five levels: 1,000, 2,000, 3,000, 5,000, and AWL (Academic Word List). The first two levels were taken from the bilingual version of the VLT developed by Nation & Wang (available at https://www.victoria.ac.nz/lals/about/staff/paul-nation) and the remaining levels were created for the purposes of this study, translating the items from the original monolingual version. The same test was used every year but the order of items in each level changed between the years. In the analysis, Webb & Chang (Reference Webb and Chang2012) looked at the scores per level and the sum of scores across the five levels.

Results of the study revealed a gradual increase in vocabulary knowledge each year. Interesting group differences in the size of that increase emerged. The greatest increase was experienced by Group A with 12.91 points, which represents an improvement of 430 words a year (on the assumption that one item in the VLT represents knowledge of 33.3 words). The amount of instruction that the groups received played a significant role in how much learners’ vocabulary knowledge grew. The average increase in total scores for Group A, which received a considerably higher amount of instruction, was 9.0 points, whereas for Groups B and C, the increase was 4.68 and 4.83 respectively. Webb & Chang (Reference Webb and Chang2012) also looked at increases within each of the frequency levels they assessed. The increase within each level ranged from 2.79 at the 5,000 level to 5.17 points at the AWL level. The greatest increase was at the 2,000 and AWL levels for the three groups. The increase at the other three levels (1,000, 3,000, and 5,000) was similar. Importantly, results also showed that by the end of the study, 73% of the participants in Group A had mastered the 1,000 and 33% had mastered the 2,000 level (based on the argument made by Schmitt et al. Reference Schmitt, Schmitt and Clapham2001, that a score of 26 in any one level indicated mastery of the level). On the contrary, only 34% and 29% of students in Groups B and C respectively had mastered the 1,000 level. Only three participants in Groups B and C (7% and 5% respectively) mastered the 2,000 level. Overall, results of this study showed that there was considerable variation in the vocabulary growth between groups of participants who received different amounts of instruction and between years of study. The minimum increase experienced by some of the participants (18 and 76 words in a year for two groups) clearly points towards the need for a more principled approach to vocabulary instruction in this context and for an institutional vocabulary learning plan. Interestingly, even after nine years of English instruction, a considerable percentage of participants failed to master the most frequent levels, which led the authors to conclude that a greater focus on the higher frequency vocabulary should be placed.

3.2 Approaches to replication

The results of Webb & Chang (Reference Webb and Chang2012) clearly showed that the amount and type of instruction provided in certain contexts might not be sufficient to support learners in acquiring the large vocabulary learning targets that they are required to operate in English. Given the important implications that this has for curriculum design and institutional planning, it would be useful to replicate the study to substantiate the findings and assess their generalizability. As Webb & Chang (Reference Webb and Chang2012) argue, because of the many factors that affect the amount of learning over time, ‘L2 vocabulary growth in one situation may be quite different from another’ (p. 114).

Therefore, a first approach to approximate replication would involve a series of modifications in the design to explore vocabulary growth in other EFL contexts and assess the generalizability of the original findings. Educational contexts differ in the amount of exposure that learners get inside the classroom and, importantly, outside the classroom context. This certainly has an important effect on the number of words that learners know and that they learn every year. Higher vocabulary gains than those found by Webb & Chang (Reference Webb and Chang2012) could be found in other contexts with more out-of-class exposure to English language. Thus, the first modification would be to conduct the study in an EFL context where learners are usually exposed to English outside the classroom context. The study by Webb & Chang (Reference Webb and Chang2012) provided evidence for the important role that amount of in-class exposure had on learners’ vocabulary development. However, the effect of out-of-class exposure was not investigated, possibly because of the little exposure that these learners had to English language outside the classroom. Recent investigations have provided evidence for the important role that out-of-class exposure has in the development of learners’ lexical knowledge (Schmitt & Redwood Reference Schmitt, Redwood, Meunier, De Cock, Gilquin and Paquot2011; Lindgren & Muñoz Reference Lindgren and Muñoz2013; González-Fernández & Schmitt Reference González-Fernández and Schmitt2015). A recent study by Peters (Reference Peters2018), also conducted in an EFL context with secondary school students, found that both amount of instruction and out-of-class exposure to English language had an important effect on learners’ vocabulary knowledge, with out-of-class exposure having a larger effect. A second modification would involve the use of instruments to elicit information about learners’ out-of-class exposure to English. Following the methodology employed in Peters's (Reference Peters2018) study, researchers could add a questionnaire about the amount and nature of out-of-class exposure, such as the European Survey of Language Competences (European Commission 2012), where learners have to indicate how often they are exposed to English language songs, (subtitled) TV or movies, computer games, websites, books, and magazines. As Peters (Reference Peters2018) noted, the design of the study did not allow her to draw any conclusions as to the longitudinal effect of out-of-class exposure on learners’ vocabulary knowledge. The approximate replication suggested here would allow researchers to explore not only learners’ vocabulary growth over four years but also the longitudinal effect of in-class and out-of-class exposure on vocabulary development. The study by Webb & Chang (Reference Webb and Chang2012) used a bilingual version of the VLT. We should note, however, that conducting the study in other EFL contexts might mean using some other bilingual versions or the monolingual version in cases where the bilingual version is not available.

A second approach to approximate replication that also focuses on modifications to the learning context would be to replicate the study in the context of content and language integrated learning (CLIL), a prominent educational trend in many European contexts. The development of vocabulary is a key concern for CLIL researchers and teachers (e.g. Pérez-Vidal Reference Pérez-Vidal, Lorenzo, Casal, de Alba and Moore2007; Costa Reference Costa2012). In the original study by Webb & Chang (Reference Webb and Chang2012), the three groups differed in the number of hours of instruction that they received but were also attending programmes that had a very different nature. A first modification might therefore involve conducting the study in the context of CLIL, which would be a different operationalization of in-class exposure. Secondly, comparing CLIL and non-CLIL classes in the same context, which follow the same curriculum and only differ in the medium of instruction of certain subjects, would allow us to have a clearer idea of the contribution of in-class exposure to vocabulary growth. A third methodological modification might involve administering a language exposure questionnaire, along the lines of what I suggested above, in order to explore the effect of both in-class and out-of-class exposure. Another interesting context for a further approximate replication would be the SA context, as in the study by Fitzpatrick (Reference Fitzpatrick2012) reviewed in the previous section. In SA contexts the amount of out-of-class exposure is expected to be larger than in an EFL context. Moreover, the type of exposure is also expected to be different, with more opportunities for oral interaction. The comparison of results of these approximate replications conducted in different contexts to the results of the original study would shed light on the effect of context and exposure on the development of vocabulary knowledge.

A final approach to approximate replication would be to modify the vocabulary measures used in order to provide a more detailed picture of vocabulary growth. The original study used a bilingual version of the VLT, which was deemed appropriate for the low-proficiency participants in the study. It measures vocabulary knowledge at different frequency levels. Thus, it provides information about the number of words they know at each level, but does not provide an overall figure of vocabulary size (Schmitt Reference Schmitt2010). Vocabulary growth could also be measured using the Vocabulary Size Test (VST) (Nation & Gu Reference Nation and Gu2007), which is indeed intended to test overall vocabulary size and some bilingual versions are also available. The VLT and the VST could be both administered at different points in a year, allowing us to see which one is more effective in capturing changes in vocabulary knowledge over time. The VLT is a receptive measure. Thus, a further modification in the measures used would involve adding a measure of productive vocabulary knowledge such as the Productive Levels Test (Laufer & Nation Reference Laufer and Nation1999). This would allow us to look at the growth of both receptive and productive vocabulary knowledge.