A Systematic Data-Gathering Method

The syllabi-collection procedure described in this article is based on two criteria: (1) the systematic syllabi selection and collection procedure are to be meaningfully selective; and (2) it must limit potential selection biases. Indeed, the selection process aims to gather data from syllabi from the best-ranked universities, mainly because of their high academic reputation and their unquestionable influence in the academic world.^{Footnote 1} Because we searched precisely for a specific type of document—text-analysis syllabi—it was unnecessary to randomly select among the population of syllabi at this point (George and Bennett Reference George and Bennett2005, 173–77). However, there are no selection biases in the source collection because the articles and books quoted on the syllabi are unknown a priori. The canonical source-ranking index contributes also to the source-collection process, limiting potential biases caused by the self-selection of syllabi or cases (Geddes Reference Geddes1990; King, Keohane, and Verba Reference King, Keohane and Verba1994).^{Footnote 2} Ultimately, 45 syllabi were collected. Following our main selection criteria, only syllabi for courses in or workshops covering text analysis at these universities were selected.

Source-Ranking Index

The source-ranking index introduced in this article is based on a deductive approach consistent with what is assumed to be teaching objectives of text-analysis professors in political science. A deductive approach is appropriate given a lack of existing empirical studies on how text-analysis methods are taught in major universities (George and Bennett Reference George and Bennett2005, 111–14). The ranking index overweights recently published texts assuming that they represent more current advances in text analysis. However, because the index also overweights highly cited texts, older works are not penalized if they are well cited. The index also gives greater weight to texts that appear on many syllabi so that older and less-well-cited sources that nevertheless are widely used for pedagogical reasons are not penalized.

The ranking index considers the number of times a book or an article appears on the syllabi (nSyllabi). We assume that the number of times a publication is listed in the syllabi indicates the importance that political science professors collectively accord to it. The number of times a source has been cited also is considered by the index (nCites). The index takes into account a source’s number of citations because it weights the source based on its overall contribution to science. The last variable that the ranking index considers is the text’s publication year (Year). Considering these different elements, the index-ranking formula is as follows:

$$Ran{k_{\left( {Xi} \right)}} = nSyllab{i_{\left( {Xi} \right)}} + nCite{s_{\left( {Xi} \right)}} + Yea{r_{\left( {Xi} \right)}}$$

To extract data from the syllabi, all of the articles and books listed were compiled into a BIBTEX file^{Footnote 3}; 992 sources were extracted from the syllabi collected. From these 992 sources, 110 books and articles were identified by the index as canonical.

DESCRIPTIVE ANALYSIS OF THE DATA

Figures 1 and 2 show the importance scores of the sources selected by the index. Figure 1 presents substantive texts; figure 2 presents texts about theory and software. Figure 2 shows that the most significant publication identified by the index is Grimmer and Stewart’s (Reference Grimmer and Stewart2013) Text as Data, listed in 78% of collected syllabi. Grimmer and Stewart’s article is often used as an introduction to the field and therefore can be an excellent overview at the beginning of a course semester.

Figure 1

Method-Driven Sources

Note: The importance score scale goes from 0 (not important) to 1 (highly important).

Figure 2

Theory and Software-Driven Sources

Note: The importance score scale goes from 0 (not important) to 1 (highly important).

Figure 1 shows six substantive categories of texts found across syllabi.^{Footnote 4} One striking factor is that classification and scaling are central to the teaching of text analysis. Classification methods originate from computer science and engineering. In these fields, the emphasis often is on “state-of-the-art” machine-learning models for classification and their refinement. By contrast, ideological scaling originates from and is more prevalent in political science. Since Poole and Rosenthal’s (Reference Poole and Rosenthal1987) work on spatial scaling, political scientists have been developing methods to scale the ideological positions of legislators. More recently, this has been accomplished with text data instead of roll-call vote data. In the syllabi, most scaling texts were written by political scientists. Texts about classification have more diverse origins, including natural and applied sciences.

Together, scaling and classification form the bulk of the text-analysis toolkit. In classes, they typically are introduced mid-semester and discussed for several weeks. The two classic texts for scaling are (1) the use of Wordscores by Laver, Benoit, and Garry (Reference Laver, Benoit and Garry2003) for supervised scaling based on reference texts; and (2) the work of Slapin and Proksch (Reference Slapin and Proksch2008) and Proksch and Slapin (Reference Proksch and Slapin2010) documenting the unsupervised Wordfish approach. They are “classics” in the sense that they represent two approaches central to scaling, are most prevalent across syllabi, and are used widely in research. Lauderdale and Herzog’s (Reference Lauderdale and Herzog2016) Wordshoal is an extension to Wordfish that uses a Bayesian approach to model multiple topics in a single corpus. All of the models proposed in these texts can be estimated using Benoit’s (Reference Benoit, Watanabe, Wang, Nulty, Obeng, Müller and Matsuo2018) quanteda package for R. Our syllabi analysis reveals that more than one week typically is allocated to teaching scaling. This is logical given the influence of spatial models of politics in political science, which often rely on scaling. Recently, text-based scaling using the previously described methods has been implemented using text extracted from audio and video. Other texts included in the two scaling panels of figure 1 include Lowe’s (Reference Lowe2008) “Understanding Wordscores” and the multiple implementations of scaling methods, such as Barberá’s (Reference Barberá2015) work on social media.

Regarding classification, we emphasize the difference between supervised and unsupervised approaches. Texts categorized in the unsupervised classification panel of figure 1 typically rely on one or another variant of topic modeling. We note the value of Roberts et al.’s (Reference Roberts, Stewart, Tingley, Lucas, Leder-Luis, Kushner Gadarian, Albertson and Rand2014) article that introduces the stm R package for open-ended survey responses. An advantage of this R package is that it was designed by political scientists with social-scientific applications in mind. The stm approach extends Latent Dirichlet topic modeling by allowing for the use of covariates (i.e., metadata) in topic estimation. Both approaches identify (or provide unsupervised classification of) latent topics using a statistical-mixture model. From a statistical point of view, supervised classification is simpler. Most of the work is upstream and involves the creation of a training set (Grimmer and Stewart Reference Grimmer and Stewart2013, 275). When the training set is sufficiently large and reliable, any appropriate classifier can be used, even simple logistic regression. However, this classifier inevitably misclassifies some texts. Hopkins and King (Reference Hopkins and King2010) introduced checks to optimize the population-level validity of the results rather than the accuracy of the classifier at the document level. In terms of syllabus building, it means discussing notions of accuracy, recall, precision, and imbalance in predictive accuracy among courses.

The two remaining categories in figure 1 are dictionary-based methods and word embeddings. Dictionary methods were used mostly before the emergence of supervised regression-based classification. They can be problematic to use outside of their own domains; for instance, a medical dictionary likely should not be used in political science. However, even within political science, it can be contentious to use a dictionary developed in one country to study another country. For example, words relating to “asylum seeking” tend to be very different (e.g., boats versus forest roads) in Australia and in Canada (Grimmer and Stewart Reference Grimmer and Stewart2013, 275). Nevertheless, many syllabi include the dictionary method because it can serve as a starting point to learn more sophisticated methods. Dictionary methods also are relatively easy to implement using quanteda’s “dictionary” function. The highest ranked source (0.76 on figure 1) regarding dictionary methods is Young and Soroka’s (Reference Young and Soroka2012) article, which is included in approximately one third of the syllabi collected. It introduces the Lexicoder Sentiment Dictionary, which Young and Soroka argue produces sentiment classifications closer to those made by humans.

The remaining categories analyzed group-word embeddings and artificial neural networks. Word embeddings are a promising tool to analyze large corpuses and identify patterns in speech. Given that they are mathematically dense, the articles on this topic by Mikolov et al. (Reference Mikolov, Chen, Corrado and Dean2013a; Reference Mikolov, Sutskever, Chen, Corrado and Dean2013b) are somewhat arcane for political science graduate students. The users of these methods are currently better served in Python than in R. In the near future, we can expect these methods to be implemented in the R language.^{Footnote 5}