The availability of digitally stored texts on the Internet has opened a completely new avenue for linguists and psycholinguists to gain access to large corpora of written language. For instance, Blair, Urland, and Ma (2002) and New, Pallier, Brysbaert, and Ferrand (2004) showed that word frequency estimates obtained with Internet search engines correlate highly with those from well-established sources such as Celex for English (Baayen, Piepenbrock, & Gulikers, 1995) and Lexique for French (New, Pallier, Ferrand, & Brysbaert, 2004). This opens the possibility to obtain frequency estimates for words in languages without an existing frequency list. Similarly, Grondelaers, Deygers, Van Aken, Van Den Heede, and Speelman (2000) showed how Internet sources can be used to get access to texts from different language registers. They downloaded materials from newspapers, discussion groups, and chat channels, and showed how the presence of a particular word (“er” in Dutch, a word meaning something like “there” and in many instances facultative) varied systematically between these different language registers (see also Desmet, De Baecke, Drieghe, Brysbaert, & Vonk, 2006, for another use of this particular corpus).
A much bigger problem is to find spoken word frequencies. The method used thus far consisted of registering dialogues (e.g., from the radio or from “spontaneous” interactions) and transcribing them. Unfortunately, much of the transcription still has to be done by hand, as current programs are not good enough to yield an acceptable error rate. The estimated transcription costs amount to some 40 hr per 1 hr of spoken input. For this reason, the availability of spoken word frequencies is very limited, both in terms of the magnitude of the corpus on which they are based and in terms of the languages for which they are available. Still, it is generally accepted that spoken word frequencies are urgently needed, because there is a feeling that written word frequencies seriously underestimate the frequency with which words are encountered in everyday life (e.g., words related to eating, clothing, furniture, casual social interactions, etc.).
The ideal spoken corpus would be to record everything some people listen to and say during everyday life. However, as mentioned previously, making such a corpus would be very costly.
There is, however, one source of transcribed spoken text widely available on the Internet: subtitles of films and television programs. This type of corpus has two potentially interesting features. First, it deals with spoken interactions between people in a visible setting. Second, for many people films and television programs comprise a substantial part of their language input, given that current estimates of television watching easily reach an average of 3–4 hr per day. Below we discuss the method we used and the results we obtained for the French language. We expect very similar findings for other languages.
COLLECTING A CORPUS OF SUBTITLES
The raw materials
Digital movies allow users to watch films with and without subtitles. This is done by using two different files: one with the original movie and one with subtitles and codes to synchronize the presentation of the subtitles with the movie. Thousands of subtitle files are freely available on the Internet, and their number is constantly increasing. In French we saw the number double in 2 years. First we searched the net for Web sites providing good subtitles in French using Google. Once the Web site was found, we used a Web crawler named Wget to download subtitles for 9,474 movies and television series. The films came from four different categories1
We removed subtitles coming from Asian countries. They had an abnormally low number of word types compared to the other subcorpora. We suspect that this subcorpora has too many specific movies (e.g., mangas).
- subtitled French films for a total of 1.9 million words (e.g., Camille Claudel, C'est arrive près de chez vous),
- subtitled English and American movies for a total of 26.5 million words (e.g., Arizona Dream, Schindler's List),
- subtitled English and American television series for a total of 19.5 million words (e.g., Friends, Ally Mc Beal), and
- subtitled non-English-language European films for a total of 2.5 million words (e.g., Cria Cuervos, Good Bye Lenin!).
Most of the materials movies were from the English language, in line with the Anglo-Saxon dominance in the film industry. We made a special effort, however, to include as many French materials as we could find. Most of them were French films that had been subtitled for the hearing impaired.
Once the files had been downloaded, they needed to be cleaned for optical character recognition (OCR) mistakes. Most of those subtitles files have been scanned from DVD with an OCR system to extract the subtitles, and sometimes the OCR software confuses two letters such as “I” and “l.” We also needed to get rid of the time indications and other nonfilm-related materials (like the names of the actors and the director). This is the only part of the whole process that has to be done manually and it can be done in less than 2 min per movie. This is an example of the type of materials that remains after this cleaning process:
C'est ton ami!
Elle n'est plus aussi jolie qu'à 29 ans.
Mlle Green aimerait fixer quelques principes avant de sortir.
Veuillez ne pas employer les mots “vieux” … “sur le déclin” ou “toujours verts pour leur âge.” Ils collent bien!
Amène-toi!
Monica a préparé le petit-déj.
Des pancakes au chocolat!
On a des cadeaux!
Des bien?
Tous issus de la liste que tu nous avais filée.
Je peux garder les cadeaux et avoir encore 29 ans?
Le cap des 30 ans, c'est pas si méchant que ça.
Tu t'es dit ça, le jour où tu les as eus?
Pourquoi, Seigneur?
Pourquoi?
On avait un deal. Tu laissais les autres vieillir, pas moi! ll n'y a que moi qui le prenne aussi mal?
Le jour de mes 30 ans, je me fendais pas la poire non plus.
Et maintenant, Chandler!
On prend tous un coup de vieux!
In the end, our corpus consisted of more than 50 million words, which is considerably larger than any other source available for spoken French language.
Calculating word frequencies
On the basis of the raw materials there are two ways to calculate word frequencies. The first consists of simply calculating the frequency of all different word forms that are encountered in the corpus. This is the easiest option, but also the least informative, as the following example in English illustrates. The word “play” can be both a verb form and a noun; the same is true for “plays.” Thus, knowing the frequencies of the word forms “play” and “plays” (and “played”) does not allow us to have an idea of the frequency of the word play as a verb or the word play as a noun. Given that the processing of singular nouns is influenced by the frequency of its plural (New, Brysbaert, Segui, Ferrand, & Rastle, 2004), this is important information we are missing.
The second option is to parse the sentences, so that we know which syntactic role each word has (this is called a tagged corpus). Currently, there are many good parsers available. For our research, we opted for Cordial Analyseur 8.13, which is, to our knowledge, the best tagger for French2
Despite cordial good performances, some errors remain. We corrected some of them.
On the basis of the tagged corpus, we obtained a list of 313,656 entries, including compounds, first names, punctuations, and so forth. To clean this list, we used the spelling checker Aspell 0.50.3.3, the dictionary Le Grand Robert (Robert, 1996), the databases Morphalou 1.01 (Romary, Salmon-Alt, & Francopoulo, 2004), and Lexique 2.62 (New et al., 2004). The outcome of this filtering is available on the Internet as part of our project on French word characteristics (www.lexique.org).
On the basis of extensive testing, it seemed to us that the best frequency measure to derive from the subtitle corpus was one in which we gave equal weight to each of the four subcorpora (French films, English films, English television, and non-English films). In this way, the frequency estimates were based on the largest possible corpus, and we avoided that they were overly dependent on (American) movies. Therefore, we first calculated the frequency per million words for the French films, the English films, the English television series, and the non-English films. Then, the average was taken of these four measures.
THE VALIDITY OF THE NEW CORPUS AND THE NEW FREQUENCIES
There may be some concerns about the validity of the subtitle measure. After all, subtitles usually consist of a shortened and edited form of what is said. They lack all the hesitations and pronunciation errors common to spoken language usage. In addition, the topics covered in movies and television series are biased to certain topics. For instance, they more often deal with adultery and contacts with the police than is true for the average participant of a psycholinguistic experiment (although many participants watch a considerable number of these movies every week and hence are quite familiar with the topics).
We used two ways to test whether these are real concerns. The first is to see how the subtitle frequencies compare to those of existing sources (in test research, this is called congruent validity). The second is to see how well the new frequencies predict word processing times (called the criterion validity).
Congruent validity with another database of spoken frequencies
A first comparison we made was between the subtitle frequencies and the frequencies from a classical French spoken corpus the “Corpus de Référence du Français Parlé” (CRFP; Equipe DELIC, 2004). The CRFP consists of a series of interviews lasting between 10 and 30 min that took place in 40 French towns. Interviews have been directed and corrected by a senior researcher from the DELIC team. Their questions were mainly related to the participant's life or work. It consists of 1 million words based on 36 hr of speech. The interviews were held in real-life situations (at home, at work, in a shop, on the radio, etc.).
There were 5,206 entries common to our corpus and the CRFP. Because we only had access to the word form frequencies (i.e., play[noun + verb], plays[noun + verb]) from the CRFP, we calculated the corresponding frequencies for our corpus. All frequencies were coded as frequency per million words. The correlation between the subtitle and the CRFP frequencies (both log transformed) was .73, which is respectable.
To get a better idea of the origins of the discrepancies between the two lists, we looked at the entries that had a much higher or much lower frequency in one of the lists. We used the ratio of the subtitle frequency/CRFP frequency to select them. Table 1 presents the words for which the subtitle frequency was much higher than the CRFP frequency.
Two types of entries seem to pop out. The first category consists of words that are related to police matters (tuer [to kill], prison [jail], police [police], armes [weapons], balle [bullet]), which is in line with the fact that police-related issues figure more dominantly in movies and television series than in everyday life of most people (although many of these people watch the films and television series from our database and so do get quite a bit of exposure to these words). Finally, typical spoken expressions seem to be more frequent in the subtitle corpus than in the CRFP (dieu [god], salut [hi], désolé [sorry], laissez [let], papa [daddy], docteur [doctor], vérité [truth], con [dumb], minute [minute], devrais [should], dormir [to sleep], etc.). This is easily explained by the composition of the two corpora: the subtitle corpus is mostly made of people interacting in conversations, whereas the CRFP mainly comprises monologs from participants. Also notice that these words are words that are of a reasonable frequency in both lists.
The second question we wanted to ask was to know if our subtitle corpus would not miss some big lexical field compared to the more classical CRFP corpus. To do that we looked at Table 2, which shows the reverse situation, where the frequency in the CRFP corpus was much higher than the frequency in the subtitle one.
There seem to be five main categories of words that have a higher frequency in CRFP than in the subtitle corpus. The first category consists of words that are used in particular in some regions of France only, such as pétanque [bowls], lyonnaise [of Lyons], provençal [of Provence], Roquefort [Roquefort], calandre [a kind of Mediterranean bird], and tarot [tarot]. The second category consists of words related to French administrations, such as administrations, municipalité [municipality], collectivités [local authorities], and spécification [specification], and probably represent the questions asked to participants such as “What is your work?” The third category consists of onomatopoeias that are typical for spontaneous spoken language (euh, bé, mh, hum). The fourth category contains entries that form part of fixed expressions (parce, abord). These frequencies are an artefact because of differences in tokenization used in the two corpora. Finally, there is a subcategory of words that seem to be typically French and that do not figure in many of our films (viticole [wine producing], charcutier [butcher], viticulture [vine growing]). These would be the only words that are seriously underestimated in our list. The numbers are underrepresented because they are more represented as Arabic than Roman in the subtitle corpus. Notice, however, that many high ratios were because of very low frequencies in the subtitle corpus (e.g., omnisports [sports center] got a ratio of 800, because there were only 0.01 words per million in the subtitle corpus against 8 words per million in the CRFP).
Congruent validity with written frequencies
Another question that we can ask concerning this new corpus is to what extent it is similar to written language. To address this problem, we also compared the subtitle frequencies with written frequencies based on a corpus of 14.8 million words (New et al., 2004). These frequencies are based on 220 novels published between 1950 and 2000. Because this corpus has been tagged, we could make use of the lemma frequencies (i.e., the frequency of play[noun]), which consists of the summed frequencies of play[noun] + plays[noun]; or the frequency of play[verb], which consists of the summed frequencies of play[verb] + plays[verb] + played[verb].
We also analyzed the discrepancies for the surface frequencies but they showed essentially that the past tense is more frequent in written language than in spoken language. That's why we decided to use lemmas frequencies here.
There were 28,598 lemmas in common with a frequency larger than 0 per million. The correlation between the written and the spoken frequencies for these lemmas was .85. To get a better idea of the discrepancies, we again looked at the most extreme cases. Table 3 shows the lemmas for which the subtitle frequencies were much higher than the written frequencies.
Two types of words again seemed to be prominent. The first are words that are typical for the spoken language in everyday life (ok, désolé [sorry], super [great], info [information], petit-déjeuner [breakfast], baby-sitter, cappuccino, stress, shampooing [shampoo], etc.). The second are words related to (American) film themes (astéroïde [asteroid], capitole [capitol], missile [missile], and federal [fede ral]).
Table 4 lists the extremes at the other end, with much higher frequencies in the written corpus than in the subtitle corpus. A look at the words in the table indicates that none of them seem frequently used in everyday language.
During these four analyses, we have seen that our subtitle corpus seems to provide quite good estimates of spoken frequencies. It represents frequently heard or produced words that are not well represented in “classical” corpora. Furthermore, it does not seem to neglect very frequent lexical fields.
Criterion validity with lexical decision times
In addition to the descriptive analyses presented above, we wanted to find a more objective test to examine the psychological validity of our corpus. The lexical decision task is a very common task used in psycholinguistics to study word processing. Participants have to decide as fast as possible if a stimulus is word or a nonword. An interesting property of the lexical decision task is that the strongest predictor of the reaction times is the word frequency. We computed the correlation coefficient between several frequency measures and the lexical decision times obtained in two recent experiments. Because the CRFP does not have lemma frequencies, we limited our analyses to word surface frequencies (as has been done in English as well; see Baayen et al., 2006; Balota, Cortese, Sergent-Marshall, Spieler, & Yap, 2004).3
The variance explained by lemma frequencies is 1–5% higher. This will be covered in future work.
The first experiment examined the effects of word frequency and age of acquisition on word processing in French (Bonin, Chalard, Méot, & Fayol, 2001; Experiment 3). In this experiment, 30 participants decided for 468 letter strings whether they formed an existing French word (234 stimuli) or not (234 other stimuli). All words were nouns representing concrete things (e.g., bee, needle). Among the 234 words, only 91 were found in the CRFP.
We used four different frequency measures: the CRFP frequencies, the subtitle frequencies restricted to the French movies, the written corpus described above, and our subtitle frequencies. We added 1 to each frequency and then took log 10. In addition, because the relationship between log frequency and reaction time (RT) is not completely linear (Baayen, Feldman, & Schreuder, 2006), we added the square of the log frequency as a second predictor variable in a multiple regression analysis. The number of syllables and letters were also entered in the multiple regressions as words were varying from 3 to 12 letters and from one to four syllables. We applied the logarithmic transformation to the RT to eliminate most of the skewness of the distribution of reaction times (Baayen et al., 2006).
Table 5 lists the percentage of variance explained in the lexical decision times (adjusted R2) by each of the frequency measures. From this analysis it is clear that the CRFP did much worse than the other two corpora. This was partly because of the fact that for this corpus the log 10 frequency was 0 for nearly 150 of the stimulus words (because the word was not present in the corpus). Another reason, however, was related to the quality of the frequency measures. When the analysis was limited to the 91 words for which we had a CRFP frequency, the percentage of variance accounted for was still substantially smaller than that accounted for by the book and the subtitle frequencies and now was less than 10%, probably because the range of frequencies was too restricted. The CRFP corpus is much less diversified because the same questions were used in each interview (Tell us about you life, tell us about your work).
To find out how much the subtitle frequencies added to the book frequencies, we entered the variable log(frequency subtitles/log frequency books) as a fifth variable to the regression analyses. This extra variable gives us an idea of how much variance is explained by the relative frequency of the words in the subtitle corpus versus the book corpus (Table 6).
The second lexical decision experiment was a purpose-built experiment in which we presented a random sample of 240 two-syllable nouns with high and low frequencies from the written corpus. Seventeen participants took part. Error responses were discarded from the analysis and response times more than 2 standard deviations above or below the mean were discarded. We removed one item because of an experimental problem (bistro).
The analyses presented in Tables 6 and 7 show that the subtitle frequency measure is at least as good as the existing book frequency measure to account for differences in lexical decision times. Further large-scale studies comparable to The English Lexicon Project (Balota et al., in press), in which lexical decision data have been collected for 44,000 English words, are planned for French words. This will enable us to see whether the hint of better performance is confirmed when all French nouns are entered into the regression analyses.
CONCLUSIONS
In this article we have described a new way to obtain a corpus of social interactions in a matter of weeks, simply by making use of the availability of files with film subtitles on the Internet. Given the rate with which movies and television series are subtitled today, we foresee that the choice of materials will further increase in the coming years, which will open the possibility to make the sampled materials more representative for the language register aimed at. In the current corpus, we do have a slight bias toward American police-related matters but, as mentioned previously, these are words that people do hear quite often as they watch TV. Even so, the quality of the results surprised us. Apart from the foreseen biases (too much police matters, not enough words that refer to typical French instances), the discrepancies between the subtitle corpus and the other databases we checked intuitively turned out to be in favor of the subtitle corpus. This was confirmed when we correlated the frequencies to lexical decision times obtained in two typical experiments that addressed the word frequency issue.
In summary, the current subtitle frequency measure seems to be a useful addition to the existing spoken and written frequencies (e.g., to match stimulus materials on frequency). There is a huge advantage, in particular, related to spoken frequency measures. This kind of corpus can easily be collected without the need of manual transcription, so that it is feasible for all languages that do not yet have a spoken corpus. The corpus can also regularly be updated and further optimized as new movies are released everyday.
This research was supported by Technolangue. We thank Agnès Bontemps for the idea to use movie subtitles for making a corpus and Magali Boibeux for helping to build and run the lexical decision presented here.
