Extending Previous Work

Our system offers three distinct advantages relative to previous work in political science using crowdsourced coding. We specifically contrast our approach with Benoit et al. (Reference Benoit, Conway, Lauderdale, Laver and Mikhaylov2016)—the most comprehensive work in political science on the subject. First, we do not rely on “gold standard” tasks, which require workers to provide judgements about precoded texts to continuously evaluate the quality of their work. These judgements, which researchers must pay for and workers must complete, are of no direct use other than to monitor worker quality and effort. Yet, Benoit et al. (Reference Benoit, Conway, Lauderdale, Laver and Mikhaylov2016, p. 286) recommend that 10% of all evaluations be focused exclusively on quality control.

In contrast, our method relies on a pairwise-comparisons framework that allows for seamless supervision of workers’ outputs in the very process of collecting the data. Indeed, as we discuss at length in our SI Appendix, our comparison framework generally works to make coding easier for workers, improve data quality, and reduce the influence of coder-specific biases. One consequence is that while Benoit et al. (Reference Benoit, Conway, Lauderdale, Laver and Mikhaylov2016, p. 286) indicate that they had to exclude “many” judgements from untrusted workers from their analyses, in our applications we were forced to remove only one worker (out of 256) and still generated highly reliable and valid scores. In all, our approach simultaneously improves the reliability of the estimates, reduces overhead costs, and makes the method more applicable to subject domains where researchers might worry about coder biases.

Second, interacting with online workforces, setting up tasks, ensuring worker quality, and providing compensation can be a time-consuming and difficult process and in some cases requires a high level of technical acumen. Our system offers a simplified approach to recruiting, training, monitoring, and aggregating online workers. The software we provide offers a suite of tools for researchers to easily set up tasks, manage and train workers, and analyze data, all within the widely used R computing environment.

Finally, the procedures outlined in Benoit et al. (Reference Benoit, Conway, Lauderdale, Laver and Mikhaylov2016) require either the use of custom-built interfaces or working through the CrowdFlower platform. The former approach is beyond the abilities of most applied researchers. For the latter option, privacy requirements or legal obligations may prevent scholars from making their text and data publicly available, transferable, and nonexclusive (a requirement for academic accounts on CrowdFlower)Footnote ¹ while the costs of commercial accounts may be prohibitive (at least $1,500/month). Our system instead provides direct access to a high-quality online workforce (Amazon’s Mechanical Turk) and includes free tools for worker management, but it allows researchers to keep private their own data and texts.Footnote ²

Workflow

The core functionality of the SentimentIt platform is a freely available cloud-based web application that interacts smoothly with AMT to post jobs, certify workers, store responses, and generally reduce the burden of interacting with AMT. Researchers access the functionality of SentimentIt via application program interfaces (APIs) that can be called from any computing environment or platform (Python, Java, etc.). However, all of the functionality described below is fully integrated into our R package, making the process for researchers accustomed to the R language especially straightforward. The complete workflow is depicted in Figure 2.

FIGURE 2.

SentimentIt Workflow

Note: See Appendix SI-10 for a detailed example.

First, we preprocess the texts ensuring they are machine-readable and, where necessary, break the document into shorter, meaningful parts, such as paragraphs. The texts are then passed to SentimentIt and randomly paired into a series of comparisons. For example, if we want 20 comparisons per document for 500 documents, we randomly create 5,000 unique comparisons (500 × 20/2 = 5,000). We then send paired document identification numbers and an associated question (e.g., “Which statement is more positive?”) via API.

At this stage, we determine the task settings, dictating how much we pay workers and whether we require certification. We then send the comparisons out to the workers. In most cases, the complete universe of micro-tasks should not be posted simultaneously. We find that posting jobs in batches of 1,000 tasks allows us to keep track of how quickly the tasks are accomplished, and, importantly, gives us the opportunity to analyze the quality of the responses. If we determine that specific workers are providing poor data, we can revoke their certification and prevent them from further contaminating the data.

Once a sufficient number of the comparisons are complete, we can download the data via API. The data simply indicate which of the two documents was selected, the unique worker ID, and the time the task was completed. We then process the data using a random utility model (closely related to standard item response models) that creates document-level estimates along the dimension of interest. Specifically, we model the probability that one document would be chosen over another, while estimating worker reliability given the choices made by that worker. Let i and j index documents in a comparison and k index the worker. The model is then:

The model is completed by specifying the following priors:

where $\mathcal {N}$ refers to the normal distribution, and $tr\mathcal {N}$ refers to the normal distribution truncated at zero to only support positive values.Footnote ³ We estimate the model using Hamiltonian Markov Chain Monte Carlo sampling using Stan (Carpenter et al. Reference Carpenter, Gelman, Hoffman, Lee, Goodrich, Betancourt, Brubaker, Guo, Li and Riddell2016). The model produces posterior estimates for the documents’ positions on the latent scale of interest (a _j ) as well as the workers’ reliability (b _k ).

We can extend this model to allow a hierarchical structure. In our second application, we deal with large documents that require simplification to create suitable micro-tasks. We construct the pairwise comparisons using paragraphs rather than entire documents. To allow for a hierarchical structure of the data, let i and j still index paragraphs in a comparison and k index the worker. We now let m index the higher-level documents. The hierarchical model is still specified as in Equation (1). However, the a estimates are now centered at a higher-level document mean for document m, denoted θ _m . Letting M be the set of paragraphs contained in document m, the priors are,

While there are many alternative ways of modeling this data, in our experience the resulting document-level estimates are largely invariant to these choices. Indeed, in our appendix, we show the correlations between the a _i estimates used in our applications below are correlated (Pearson’s r) at 0.95–0.98 with the arithmetic mean coder choice (where a document is coded as zero when it is not chosen and as one otherwise). Thus, while we feel that the model above is useful—particularly for identifying low-quality coders—the conclusions we draw below are robust to specific modeling choices and priors (see also Benoit et al. Reference Benoit, Conway, Lauderdale, Laver and Mikhaylov2016).

In both models, the parameter estimates for the workers (b _k ) give us an assessment of how well each worker performed. Intuitively, these estimates become lower for workers whose choices do not reflect how the documents are understood by other workers. If we estimate a worker as being a (low) outlier, we can ban the worker from future tasks. We find that revoking qualifications from these workers modestly improves the validity of our final estimates relative to benchmarks. After disqualifying problematic workers (if any), we can post further tasks and repeat as necessary.

Our R package can do as little or as much of this in an automated fashion as the researcher wants. If desired, researchers can execute each step specified above manually through R. Alternatively, it can be fully automated such that the software will create comparisons, post tasks, check if the tasks are completed, download the data, test for worker outliers, ban unwanted workers, and repeat the process until all of the desired data has been collected and analyzed.

Advertisement Tone

First we measure the “tone” of political ads, or their level of negativity. In the Wisconsin Advertising Project (WiscAds) dataset, ad tone is determined by expert coders who categorized ads as either promoting a single candidate, contrasting two candidates, or attacking a candidate. If the ad is contrasting, it is further categorized as either being more aimed at promoting than attacking, more attacking than promoting, or equally attacking and promoting. The result is a 5-point scale of negativity ranging from 1 (positive) to 5 (attack). We apply SentimentIt to analyze all televised ads for the U.S. Senate in 2008 (n = 942) and compare our estimates to this 5-point negativity scale.

For each ad, we created 20 pairwise comparisons for a total of 9,420 tasks. Workers were required to complete an extensive training module and were paid $0.06 per comparison. Workers were instructed to select the ad that was “most negative towards the candidate(s) mentioned, or least positive about the candidate(s) mentioned.” In all, 123 workers participated in the task and none were banned.

Figure 3 shows our estimates plotted against the 5-point negativity scale. The SentimentIt scores are correlated with the expert codings at r = 0.85. This is a very high correlation, but there are some disagreements. Table 1 shows the largest disagreements between the SenitmentIt measures and those generated by the expert coders. The first example is coded as positive by the expert coders, but the ad is negative in tone and appears to be a strongly negative contrasting ad that was miscoded. The second ad is coded as contrast (2 = more positive than attack), but SentimentIt estimates it as having almost no negative content. In fact, the ad does not mention an opposing candidate or party and is another clear miscoding. The third ad is coded as contrasting by expert coders because it mentions both candidates by name. However, the ad mostly consists of strong and negative language, causing our estimate of negativity to be quite high (1.39). Finally, the last example is coded as an attack by human coders because it does not provide positive information about a named candidate. However, the ad only mentions the target of the attack (Collins) in one sentence and otherwise conveys positive information about the Employee Free Choice Act. These last examples illustrate that the strict coding rules designed to improve the reliability of content analysis can obscure the underlying latent trait of interest.

FIGURE 3.

SentimentIt Negativity Rating Relative to 5-point Expert Codes

Note: Using the 5-point scale of human codes, we can see that as human codes increase, so do our estimates. In general, the differentiation between codes is evident. The largest disagreements are happening in the middle categories, as discussed in the text.

TABLE 1.

Example Ads Where SentimentIt Disagreed with Expert Coders

Note: When SentimentIt estimates differ from human codes, expert coding scheme’s reliance on strict coding rules mischaracterizes the overall tone within the ads.

To determine the reliability of our measure, we repeated the exercise 35 days later with 20 more comparisons on a random sample of half the ads (n = 467) using the same procedure as before. In all, 52 workers participated in this exercise. We revoked the certification from only one worker. The correlation between the two runs is 0.90.

Human Rights Reports

Next, we demonstrate how SentimentIt can be generalized to larger documents. Specifically, we turn to the sobering task of coding the section entitled “Torture and Other Cruel, Inhuman or Degrading Treatment or Punishment” from all of the U.S. State Department Human Rights Reports issued annually for nearly every country in the world. In this application, we use reports from 1999 to create a continuous scale indicating the amount of torture conveyed in the report. Hathaway (Reference Hathaway2002) codes these documents by hand on a 5-point scale, with higher values indicating more entrenched, brutal, or frequent torture. Using this measure, Hathaway (Reference Hathaway2002) argues that ratifying human rights treaties is associated with greater degrees of torture. We emphasize that our aim is not to capture the actual amount or severity of torture in each country—a task well beyond the scope of this article—but only to measure the concept of torture as it is expressed in these reports.

Because many of these reports are quite lengthy and would be difficult for even an expert to read and meaningfully compare, we divided the 182 documents into 1,652 paragraphs. We created 20 comparisons per paragraph. Since reading and understanding these documents is challenging, we paid workers $0.10 per task and required an extensive certification training. We asked workers to indicate which paragraph suggested more torture, which was defined in detail during the training. Our definition and coding rules were designed to approximate the Hathaway definitions (see the SI Appendix).

We posted the tasks in batches of 1,000. The process of gathering the data took approximately three days, and 81 workers completed tasks (we banned none). To analyze the data, we adjusted our statistical model to allow for a hierarchical structure (paragraphs within documents) as described above. The resulting document-level estimates are correlated with Hathaway’s coding at 0.69. Figure 4 plots the SentimentIt estimates of torture to the Hathaway code. In Figure 5, we show a subset of the document- and paragraph-level estimates generated by our procedure.

FIGURE 4.

SentimentIt Document-Level Estimates of Torture on Hathaway Codes

FIGURE 5.

SentimentIt Document- and Paragraph-Level Estimates of Randomly Selected Countries and Countries Estimated Differently Than Hathaway

Note: Document-level estimates are shown as red squares, and paragraph-level estimates are shown as black diamonds. There is a clear trend that as Hathaway’s codes increase, so do ours. The notable exceptions are explicitly included and discussed in the SI Appendix. Where our estimates deviate from Hathaway’s codes, there is strong reason to believe estimates from SentimentIt are outperforming expert codings.

Broadly speaking, our measure is consonant with Hathaway’s coding. For instance, all of our estimates for the countries coded as one (no torture) by Hathaway fall within the first quantile of our estimates. Yet, there are significant disagreements. One approach to arbitrating between these measures is to closely examine instances where the two codes disagree. We do this in the SI Appendix, and we believe that these examples show that the SentimentIt coding more faithfully reflects the level of torture in these documents than the original expert-coded measures.

However, within the space constraints of this letter, we do take two steps to assess the validity of these estimates. First, we compare the measures with three other well-known measures of torture drawn (in part) from this same document set. Specifically, we analyze the State Department variable of the Political Terror Scale (PTS) data on human rights violations, the Ill-Treatment and Torture (ITT) data, and the torture variable from The CIRI Human Rights Dataset.Footnote ⁴

Second, we test the degree to which Hathaway’s primary substantive claim—that signing antitorture conventions leads to more abuse (see also Neumayer Reference Neumayer2005)—is supported using our measure. Hathaway (Reference Hathaway2002) finds that signing an antitorture convention is positively but unreliably associated with torture. We argue that if our measure is valid, this relationship should hold with our measure as well.

For each variable, we conduct separate regressions on standardized measures and calculate regression coefficients. We maintain all controls of the original analysis. Our only deviation is that we look at only one year while the original analysis included 15 years of reports. The results are shown in Table 2. In almost all cases, we find that both the Hathaway and SentimentIt scores are positively related to the other measures (p ≤ 0.05). The only exception is that the relationship between level of torture and ratifying an antitorture convention is estimated unreliably using Hathaway’s coding. However, although both measures are correlated with the alternative torture measures, both the coefficients and R ² values indicate that the SentimentIt measure is more highly related to these other variables. The only exception is the CIRI measure, where the two approaches are essentially indistinguishable in terms of their predictive validity.

TABLE 2.

OLS Coefficients for Standardized SentimentIt and Hathaway Measures Regressed on Torture Measures and Treaty Ratification

Note: Bolded coefficients are significant at the p ⩽ 0.05 level. Additional controls for per capita GDP, population, population growth, trade, foreign aid, GDP growth, civil war, level of democracy, and nation durability not shown. SentimentIt and Hathaway codes are standardized for comparability. CIRI increases as the degree of torture decreases and is reverse coded in the analyses.

Finally, to test for reliability of our measures we separately estimate the (roughly) first 10 comparisons to the last 10 comparisons for each paragraph. The correlation between paragraph estimates is 0.77. The correlation between document estimates is 0.88. Considering we are only comparing the estimates between two rounds of 10 comparisons, we consider this to be strong evidence indicating the SentimentIt platform is generating highly reliable measures.Footnote ⁵