The context
The English Language Institute (now the School of Linguistics and Applied Language Studies) at Victoria University of Wellington has a long history of corpus-based vocabulary research, especially after the arrival of the second director of the institute, H. V. George, and the appointment of Helen Barnard, whom George knew in India. George's successor, Graeme Kennedy, also saw corpus linguistics as a very fruitful and important area of applied language research.
This interest in corpus-based vocabulary studies led to an attempt to develop a test of receptive vocabulary size (Goulden, Nation & Read Reference Goulden, Nation and Read1990). It became clear that a standard word family unit was needed, to enable consistency between different word families and different researchers. The result was a collaboration between two researchers in the School, Laurie Bauer and Paul Nation (Reference Bauer and Nation1993). Deciding on a standard definition of what should be included in a word family is a critical step in vocabulary frequency studies and in the measurement of vocabulary size; the two major methodological weaknesses of previous vocabulary size studies had been the failure to develop a standard unit of counting and the use of spaced sampling of dictionaries (Thorndike Reference Thorndike and Bell1924; Lorge & Chall Reference Lorge and Chall1963). These weaknesses were so serious that almost all research on vocabulary size carried out during the twentieth century is virtually useless and at worst grossly misleading (Nation Reference Nation1993).
A series of steps must be followed in the creation of frequency-list based vocabulary size tests if the methodological difficulties of dictionary-based word sampling are to be avoided:
1. Develop a standard and reliable unit of word counting that suits the goals of the test.
2. Develop substantial word family lists that include virtually all the vocabulary to be tested.
3. Use corpora to arrange the word families in the lists into frequency levels.
4. Draw representative samples of word families from the lists so that no word frequency level is overrepresented or underrepresented in the samples.
5. Decide on the test format and develop items for the tests.
Thorndike's own work is a good example of this procedure. He first developed substantial word frequency lists and then sampled from these lists to create his own vocabulary size tests, thus avoiding all the difficulties that he had noted in dictionary-based sampling.
In many ways, the rate of progress through these steps has been determined by the availability of computer hardware and software. Because making word family lists is so time-consuming, for many years the 2,000 word families from the General Service List and the University Word List, the latter replaced by the Academic Word List (Coxhead Reference Coxhead2000), were the only lists available. The availability of faster computers, larger corpora, data from the British National Corpus and a specially written word family building program (AffixAppender by Chris Andreae) enabled the creation of further lists. Even with all of this support, making word families was still time-consuming, and it was only in 2012 that the twenty-fifth 1,000 word families list was finished. These lists and the programs that use them, including their source code, have always been made freely available to interested researchers.
The Vocabulary Size Test
When the fourteenth 1,000 list was completed, items were sampled from it to create the Vocabulary Size Test (VST). This test was first reported on in Nation & Beglar (Reference Nation and Beglar2007) and is now supported by a growing body of research (Beglar Reference Beglar2010; Nguyen & Nation Reference Nguyen and Nation2011; Elgort Reference Elgort2013).
Beglar's (Reference Beglar2010) examination of the 140-item monolingual VST in Japan with a mixture of native and non-native speakers showed that it had many features that typify a good test. It works well because it covers a very wide range of frequency levels, it includes a large number of items (even half of this number would work well), the items have been very carefully designed and made, and the test is designed to measure just one kind of vocabulary knowledge.
Nguyen & Nation (Reference Nguyen and Nation2011) showed that it is important for even a lower proficiency learner to sit all levels of the test because they may know some words at the lower frequency levels, perhaps because they are loan words or cognates, relate to the learner's hobbies and interests, or are technical words in fields he or she is familiar with – or because the learner just happens to have met and remembered them.
Recent research on the VST has taken two directions: the development of (a) bilingual versions of the test to use with non-native speakers of English, and (b) computerised versions of the test. Useful critical studies of the test will also soon appear.
Bilingual vocabulary size tests
The VST is designed to measure both L1 and L2 learners’ written receptive vocabulary size in English. The test measures knowledge of written word form, the form–meaning connection and, to a smaller degree, concept knowledge. The test measures largely decontextualised knowledge of each word even though the tested word appears in a single non-defining context in the test. Using Read & Chapelle's (Reference Read and Chapelle2001) framework, the VST is a discrete, selective, relatively context-independent vocabulary test presented in a multiple-choice format.
A bilingual test presents the choices in the L1 of those sitting the test:
Unpublished research by Polina Kobeleva showed that lower proficiency learners performed better on a bilingual version of the test than on the monolingual version. This is probably because a bilingual test removes the difficulty of reading definitions written in the L2, English. Definitions have difficult grammar, typically involving relative clauses or reduced relative clauses, as in the example above. L1 definitions may be a single word and are in a language that the test-takers feel most comfortable with. Kobeleva found that the low proficiency learners’ score was around 10% higher in the bilingual form of the test than in the monolingual form. Elgort (Reference Elgort2013) found similar results. The difference between monolingual and bilingual scores decreases as vocabulary knowledge increases. Elgort also found that the number of cognates in the test affected the result and recommended that the proportion of cognates in the test should mirror their proportion in the language. It is important that cognates are retained in the test, otherwise vocabulary size estimates will be severely distorted. When cognates occur in the test, they are not translated into their cognate form: instead, a definition is used so that answering the test item does not involve simply matching the English form with the L1 form.
Bilingual versions of the test now exist in Russian (prepared by Irina Elgort), Mandarin in simplified characters (Meng Fanshao), Mandarin in traditional characters (Li-Ching Chiu), Spanish (Betsy Quero), Korean (Chung Mihwa) and Japanese (Yosuke Sasao and Tatsuya Nakata). Because the bilingual VSTs get rid of a problem that is not central to vocabulary knowledge (understanding definitions in the L2, English), they can be regarded as more suitable than the monolingual test for native speakers of a particular L1. More bilingual versions are needed for teacher use.
Computerised versions of the VST
While many of the vocabulary-related resources created at the School have been freely available for download, this form of distribution has some shortcomings, chief of which are keeping teachers updated with the latest test versions and the limited application of the test results. The VocabularySize.com project was created by Myq Larson in part to address these issues. The website provides a facility for teachers to measure their students’ vocabulary size online with the latest versions of the tests while maintaining previous versions should they be needed. The tests can be customised so that teachers can collect information such as their students’ names or ID numbers to uniquely identify the results, which can then be viewed online or downloaded. Any personal information collected on behalf of the teachers is stored separately from the test responses. Every time a test is taken through VocabularySize.com, the anonymised results are permanently added to a database for future analysis. Researchers can then derive extra value from the data by using it to further refine and improve the tests or investigate trends and features which would not have been feasible with smaller data sets. As the range of learner backgrounds represented in the results database increases, reference norms can be established based on native language and age.
By the end of 2012 over 20,000 complete data sets had been collected for the VST. Initial data reveal some interesting trends that could not have been detected with traditional offline paper-and-pencil tests. For example, for correctly answered questions, the median time taken to select an answer can explain a small but significant portion of the final score on the test. These timings will be explored in future and we hope to use them to either adjust the final score or to add another dimension to it.
The order of items seems to affect the amount of effort put into each item. The paper-based item order, starting with the highest frequency words and ending with the least frequent, appears to have increased the rates of random guessing for the lowest frequency items. This is indicated by extremely short response times when longer times would be expected. However, when words are presented with a mixture of frequencies which, on average, increase in difficulty through the course of the test, the time spent answering the questions tends to remain more or less stable for all items. This suggests that more accurate scores can be obtained by altering the questions or even changing their order.
An English Word Parts Test, which tests recognition and knowledge of the most common affixes, is also available on VocabularySize.com. A new Vocabulary Levels Test (VLT) is in development, as well as VSTs for Japanese and Spanish. Other types of tests, which measure different aspects of word knowledge, as well as tests of other languages, are also planned. The VST is available online at www.lextutor.ca and at www.my.vocabularysize.com.
Testing the vocabulary size of native speakers
A sample of 14,000 words is adequate for determining the vocabulary size of non-native speakers of English, but it is by no means enough for native speakers. When the twentieth 1,000 word list was completed, a project led by Averil Coxhead created six versions of the VST, testing vocabulary up to the twentieth 1,000 level. There is a detailed description and specifications for the VST on Paul Nation's website (www.victoria.ac.nz/lals/staff/paul-nation.aspx). The 20,000 versions were created so that the vocabulary size of advanced learners of English as a second or foreign language, as well as that of young native speakers of English, could be tested. One of the major motivations for testing native speakers of English at primary and secondary school was that many teachers considered that learners’ reading problems arose from a lack of vocabulary knowledge. This belief had been largely fostered by research on young children in the United States, where large differences in vocabulary size were found between children in families at low and high socio-economic levels (Hart & Risley Reference Hart and Risley1995). This research unfortunately suffers from very severe methodological weaknesses (a critique can be found at Nation & Webb Reference Nation and Webb2011: 198–200). Our initial trialling of the test indicated that learners who were native speakers of English were highly unlikely to suffer from such a lack of vocabulary knowledge that they would be unable to read unsimplified texts written for learners of their age range. Even the few learners who were largely illiterate had a vocabulary large enough to deal with the words in their reading texts. They did not, however, have the reading skills that they needed.
During the trialling of the VST, it had also become clear that with secondary school students, particularly those who were turned off by school, the tests had to be conducted one-to-one, not with a group. When the tests were administered to a whole class, some learners did the job carefully and thoroughly; others rushed through it with minimal attention, getting it out of the way as quickly as possible (Nation Reference Nation, Daller, Milton and Treffers-Daller2007). By contrast, when the test administrator sat next to a learner and encouraged and kept him or her on task, the learner's score was much higher. This made it clear that to measure the vocabulary size of school students as accurately as possible, we may need to administer the test individually with some learners, so several versions would be required in order to avoid compromising the test. If the difference between the administrations is great, particularly for learners who score towards the lower end of the range, and if they do much better on individually administered tests, this has very important implications for a wide variety of group-administered tests used with school children. The results of group-administered tests may be unfairly misrepresenting the achievement and ability of some children, and may be leading to inappropriate and misdirected remedial teaching.
So far, using a Victoria University research grant made to Averil Coxhead, over 700 secondary school children have been tested (ages 12 to 18), 210 of them tested both with individual and group testing using different equivalent counterbalanced test versions, and controlling for order effects.
Preliminary data on vocabulary sizes generally supports the rule of thumb that to work out a young native speaker's vocabulary size, take two or three years away from their age and multiply that by 1,000. On this basis, a 13-year-old is likely to have a vocabulary size of around 13 – 2 = 11, 11 × 1,000 = 11,000 word families. This rule of thumb and the yearly increase fit well with Biemiller & Slonim's (Reference Biemiller and Slonim2001) figures for younger children. There may, however, be a substantial range of scores at any age level. The lowest score for a 13-year-old in our data is 7,000 word families and the highest is 17,000, though these are exceptional cases. However, even a 7,000 word vocabulary is enough for a learner to read texts for that age level, although reading may be a struggle.
A separate study has been carried out with adult native and non-native speakers of English. One goal of the study is to develop several parallel forms of the 20,000 version of the VST; 46 learners have sat all forms of the test. Two parallel forms have been made freely available on Paul Nation's website. Preliminary data from this study of adults shows an increase in size with age, including adults over 60. Non-native speakers of English with a European language as their L1 scored higher than non-native speakers with other L1s.
Other tests
Stuart Webb is working on a new version of the VLT (Nation Reference Nation1983; Schmitt, Schmitt & Clapham Reference Schmitt, Schmitt and Clapham2001) using the matching format and testing 30 items in each of the 1,000, 2,000, 3,000, 4,000 and 5,000 word levels, sampling from the BNC/COCA word family lists.
The most pressing need at present is for a standardised vocabulary size test that can be used with very young native speakers of English and young non-native speakers who are not yet able to read in English. Susan Ruffell's work on a picture vocabulary test (see Paul Nation's website) and Biemiller & Slonim's (Reference Biemiller and Slonim2001) research clearly show that such a test would need to go up to at least the fifth or sixth 1,000 word families.
