HARVESTING THE WISDOM OF CROWDS

The intuition behind crowd-sourcing can be traced to Aristotle (Lyon and Pacuit Reference Lyon, Pacuit and Michelucci2013) and later Galton (Reference Galton1907), who noticed that the average of a large number of individual judgments by fair-goers of the weight of an ox is close to the true answer and, importantly, closer to this than the typical individual judgment (for a general introduction see Surowiecki Reference Surowiecki2004). Crowd-sourcing is now understood to mean using the Internet to distribute a large package of small tasks to a large number of anonymous workers, located around the world and offered small financial rewards per task. The method is widely used for data-processing tasks such as image classification, video annotation, data entry, optical character recognition, translation, recommendation, and proofreading. Crowd-sourcing has emerged as a paradigm for applying human intelligence to problem-solving on a massive scale, especially for problems involving the nuances of language or other interpretative tasks where humans excel but machines perform poorly.

Increasingly, crowd-sourcing has also become a tool for social scientific research (Bohannon Reference Bohannon2011). In sharp contrast to our own approach, most applications use crowds as a cheap alternative to traditional subjects for experimental studies (e.g., Horton et al. Reference Horton, Rand and Zeckhauser2011; Lawson et al. Reference Lawson, Lenz, Baker and Myers2010; Mason and Suri Reference Mason and Suri2012; Paolacci et al. Reference Paolacci, Chandler and Ipeirotis2010). Using subjects in the crowd to populate experimental or survey panels raises obvious questions about external validity, addressed by studies in political science (Berinsky et al. Reference Berinsky, Huber and Lenz2012), economics (Horton et al. Reference Horton, Rand and Zeckhauser2011) and general decision theory and behavior (Chandler et al. Reference Chandler, Mueller and Paolacci2014; Goodman et al. Reference Goodman, Cryder and Cheema2013; Paolacci et al. Reference Paolacci, Chandler and Ipeirotis2010). Our method for using workers in the crowd to label external stimuli differs fundamentally from such applications. We do not care at all about whether our crowd workers represent any target population, as long as different workers, on average, make the same judgments when faced with the same information. In this sense our method, unlike online experiments and surveys, is a canonical use of crowd-sourcing as described by Galton.Footnote ⁶

All data production by humans requires expertise, and several empirical studies have found that data created by domain experts can be matched, and sometimes improved at much lower cost, by aggregating judgments of nonexperts (Alonso and Baeza-Yates Reference Alonso, Baeza-Yates, Clough, Foley, Gurrin, Jones, Kraaij, Lee and Mudoch2011; Alonso and Mizzaro Reference Alonso and Mizzaro2009; Carpenter Reference Carpenter2008; Hsueh et al. Reference Hsueh, Melville and Sindhwani2009; Ipeirotis et al. Reference Ipeirotis, Provost, Sheng and Wang2013; Snow et al. Reference Snow, O'Connor, Jurafsky and Ng2008). Provided crowd workers are not systematically biased in relation to the “true” value of the latent quantity of interest, and it is important to check for such bias, the central tendency of even erratic workers will converge on this true value as the number of workers increases. Because experts are axiomatically in short supply while members of the crowd are not, crowd-sourced solutions also offer a straightforward and scalable way to address reliability in a manner that expert solutions cannot. To improve confidence, simply employ more crowd workers. Because data production is broken down into many simple specific tasks, each performed by many different exchangeable workers, it tends to wash out biases that might affect a single worker, while also making it possible to estimate and correct for worker-specific effects using the type of scaling model we employ below.

Crowd-sourced data generation inherently requires a method for aggregating many small pieces of information into valid measures of our quantities of interest.Footnote ⁷ Complex calibration models have been used to correct for worker errors on particular difficult tasks, but the most important lesson from this work is that increasing the number of workers reduces error (Snow et al. Reference Snow, O'Connor, Jurafsky and Ng2008). Addressing statistical issues of “redundant” coding, Sheng et al. (Reference Sheng, Provost and Ipeirotis2008) and Ipeirotis et al. (Reference Ipeirotis, Provost, Sheng and Wang2014) show that repeated coding can improve the quality of data as a function of the individual qualities and number of workers, particularly when workers are imperfect and labeling categories are “noisy.” Ideally, we would benchmark crowd workers against a “gold standard,” but such benchmarks are not always available, so scholars have turned to Bayesian scaling models borrowed from item-response theory (IRT), to aggregate information while simultaneously assessing worker quality (e.g., Carpenter Reference Carpenter2008; Raykar et al. Reference Raykar, Yu, Zhao, Valadez, Florin, Bo-goni and Moy2010). Welinder and Perona (Reference Welinder and Perona2010) develop a classifier that integrates data difficulty and worker characteristics, while Welinder et al. (Reference Welinder, Branson, Belongie and Perona2010) develop a unifying model of the characteristics of both data and workers, such as competence, expertise, and bias. A similar approach is applied to rater evaluation in Cao et al. (Reference Cao, Stokes and Zhang2010) where, using a Bayesian hierarchical model, raters’ judgments are modeled as a function of a latent item trait, and rater characteristics such as bias, discrimination, and measurement error. We build on this work below, applying both a simple averaging method and a Bayesian scaling model that estimates latent policy positions while generating diagnostics on worker quality and sentence difficulty. We find that estimates generated by our more complex model match simple averaging very closely.

SCALING DOCUMENT POLICY POSITIONS FROM CODED SENTENCES

Our aim is to estimate the policy positions of entire documents: not the code value of any single sentence, but some aggregation of these values into an estimate of each document's position on some meaningful policy scale while allowing for reader, sentence, and domain effects. One option is simple averaging: identify all economic scores assigned to sentences in a document by all readers, average these, and use this as an estimate of the economic policy position of a document. Mathematical and behavioral studies on aggregations of individual judgments imply that simpler methods often perform as well as more complicated ones, and often more robustly (e.g., Ariely et al. Reference Ariely, Au, Bender, Budescu, Dietz, Gu and Zauberman2000; Clemen and Winkler Reference Clemen and Winkler1999). Simple averaging of individual judgments is the benchmark when there is no additional information on the quality of individual coders (Armstrong Reference Armstrong2001; Lyon and Pacuit Reference Lyon, Pacuit and Michelucci2013; Turner et al. Reference Turner, Steyvers, Merkle, Budescu and Wallsten2014). However, this does not permit direct estimation of misclassification tendencies by readers who for example fail to identify economic or social policy “correctly,” or of reader-specific effects in the use of positional scales.

An alternative is to model each sentence as containing information about the document, and then scale these using a measurement model. We propose a model based on item response theory (IRT), which accounts for both individual reader effects and the strong possibility that some sentences are intrinsically harder to interpret. This approach has antecedents in psychometric methods (e.g., Baker and Kim Reference Baker and Kim2004; Fox Reference Fox2010; Hambleton et al. Reference Hambleton, Swaminathan and Jane Rogers1991; Lord Reference Lord1980), and has been used to aggregate crowd ratings (e.g., Ipeirotis et al. Reference Ipeirotis, Provost, Sheng and Wang2014; Welinder et al. Reference Welinder, Branson, Belongie and Perona2010; Welinder and Perona Reference Welinder and Perona2010; Whitehill et al. Reference Whitehill, Ruvolo, Wu, Bergsma and Movellan2009).

We model each sentence, j, as a vector of parameters, θ _jd , which corresponds to sentence attributes on each of four latent dimensions, d. In our application, these dimensions are latent domain propensity of a sentence to be labeled economic (1) and social (2) versus none; latent position of the sentence on economic (3) and social (4) dimensions. Individual readers i have potential biases in each of these dimensions, manifested when classifying sentences as “economic” or “social,” and when assigning positions on economic and social policy scales. Finally, readers have four sensitivities, corresponding to their relative responsiveness to changes in the latent sentence attributes in each dimension. Thus, the latent coding of sentence j by reader i on dimension d is

$$\begin{equation*} {\mu}_{{ijd}}^{*} = {{\chi}_{{id}}}\left({{{\theta}_{{jd}}} + {{\psi}_{{\,\,id}}}} \right) \end{equation*}$$

where the χ _id indicate relative responsiveness of readers to changes in latent sentence attributes θ _jd , and the ψ _id indicate relative biases towards labeling sentences as economic or social (d = 1, 2), and rating economic and social sentences as right rather than left (d = 3, 4).

We cannot observe readers’ behavior on these dimensions directly. We therefore model their responses to the choice of label between economic, social and “neither” domains using a multinomial logit given μ* _ij1 and μ* _ij2. We model their choice of scale position as an ordinal logit depending on μ* _ij3 if they label the sentence as economic and on μ* _ij4 if they label the sentence as social.Footnote ¹² This results in the following model for the 11 possible combinations of labels and scales that a reader can give a sentence:Footnote ¹³

$$\begin{equation*} p\left({none} \right) = \left({\frac{1}{{1 + \ {\rm{exp}}(\mu _{ij1}^*) + \ {\rm{exp}}(\mu _{ij2}^*)}}} \right), \end{equation*}$$

$$\begin{eqnarray*} &&p\left({econ;scale} \right) = \left({\frac{{{\rm{exp}}(\mu _{ij1}^*)}}{{1 + \ {\rm{exp}}(\mu _{ij1}^*) + \ {\rm{exp}}(\mu _{ij2}^*)}}\} \right)\\ &&\quad\times\,\left({logi{t^{- 1}}\left({{\xi _{scale}} - \mu _{ij3}^*} \right) - logi{t^{- 1}}\left({{\xi _{scale - 1}} - \mu _{ij3}^*} \right)} \right), \end{eqnarray*}$$

$$\begin{eqnarray*} &&p\left({soc;scale} \right) = \left({\frac{{{\rm{exp}}(\mu _{ij2}^*)}}{{1 + \ {\rm{exp}}(\mu _{ij1}^*) + \ {\rm{exp}}(\mu _{ij2}^*)}}} \right)\\ &&\quad\times\,\left({logi{t^{- 1}}\left({{\xi _{scale}} - \mu _{ij4}^*} \right) - logi{t^{- 1}}\left({{\xi _{scale - 1}} - \mu _{ij4}^*} \right)} \right). \end{eqnarray*}$$

The primary quantities of interest are not sentence level attributes, θ _jd , but rather aggregates of these for entire documents, represented by the ${\bar \theta _{k, d}}$ for each document k on each dimension d. Where ε _jd are distributed normally with mean zero and standard deviation σ _d , we model these latent sentence level attributes θ _jd hierarchically in terms of corresponding latent document level attributes:

$$\begin{equation*} {\theta _{jd}} = {\bar \theta _{k\left(j \right),d}} + {\epsilon _{jd}}. \end{equation*}$$

As at the sentence level, two of these (d = 1, 2) correspond to the overall frequency (importance) of economic and social dimensions relative to other topics, and the remaining two (d = 3, 4) correspond to aggregate left-right positions of documents on economic and social dimensions.

This model enables us to generate estimates of not only our quantities of interest for the document-level policy positions, but also a variety of reader- and sentence-level diagnostics concerning reader agreement and the “difficulty” of domain and positional coding for individual sentences. Simulating from the posterior also makes it straightforward to estimate Bayesian credible intervals indicating our uncertainty over document-level policy estimates.Footnote ¹⁴

Posterior means of the document level ${\bar \theta _{kd}}$ correlate very highly with those produced by the simple averaging methods discussed earlier: 0.95 and above, as we report below. It is therefore possible to use averaging methods to summarize results in a simple and intuitive way that is also invariant to shifts in mean document scores that might be generated by adding new documents to the coded corpus. The value of our scaling model is to estimate reader and sentence fixed effects, and correct for these if necessary. While this model is adapted to our particular classification scheme, it is general in the sense that nearly all attempts to measure policy in specific documents will combine domain classification with positional coding.

BENCHMARKING A CROWD OF EXPERTS

Our core objective is to compare estimates generated by workers in the crowd with analogous estimates generated by experts. Since readers of all types will likely disagree over the meaning of particular sentences, an important benchmark for our comparison of expert and crowd-sourced text coding concerns levels of disagreement between experts. The first stage of our empirical work therefore employed multiple (four to six)Footnote ¹⁵ experts to independently code each of the 18,263 sentences in our 18-document text corpus, using the scheme described above. The entire corpus was processed twice by our experts. First, sentences were served in their natural sequence in each manifesto, to mimic classical expert content analysis. Second, about a year later, sentences were processed in random order, to mimic the system we use for serving sentences to crowd workers. Sentences were uploaded to a custom-built, web-based platform that displayed sentences in context and made it easy for experts to process a sentence with a few mouse clicks. In all, we harvested over 123,000 expert evaluations of manifesto sentences, about seven per sentence. Table 1 provides details of the 18 texts, with statistics on the overall and mean numbers of evaluations, for both stages of expert processing as well as the crowd processing we report below.

TABLE 1.

Texts and Sentences Coded: 18 British Party Manifestos

External validity of expert evaluations

Figure 2 plots two sets of estimates of positions of the 18 manifestos on economic and social policy: one generated by experts processing sentences in natural sequence (x axis); the other generated by completely independent expert surveys (y axis).Footnote ¹⁶ Linear regression lines summarizing these plots show that expert text processing predicts independent survey measures very well for economic policy (R = 0.91), somewhat less well for the noisier dimension of social policy (R = 0.81). To test whether coding sentences in their natural sequence affected results, our experts also processed the entire text corpus taking sentences in random order. Comparing estimates from sequential and random-order sentence processing, we found almost identical results, with correlations of 0.98 between scales.Footnote ¹⁷ Moving from “classical” expert content analysis to having experts process sentences served at random from anonymized texts makes no substantive difference to point estimates of manifesto positions. This reinforces our decision to use the much more scalable random sentence sequencing in the crowd-sourcing method we specify.

FIGURE 2.

British Party Positions on Economic and Social Policy 1987–2010

Notes: Sequential expert text processing (vertical axis) and independent expert surveys (horizontal). Labour red, Conservatives blue, Liberal Democrats yellow, labeled by last two digits of year.

Internal reliability of expert coding

Agreement between experts

As might be expected, agreement between our experts was far from perfect. Table 2 classifies each of the 5,444 sentences in the 1987 and 1997 manifestos, all of which were processed by the same six experts. It shows how many experts agreed the sentence referred to economic, or social, policy. If experts are in perfect agreement on the policy content of each sentence, either all six label each sentence as dealing with economic (or social) policy, or none do. The first data column of the table shows a total of 4,125 sentences which all experts agree have no social policy content. Of these, there are 1,193 sentences all experts also agree have no economic policy content, and 527 that all experts agree do have economic policy content. The experts disagree about the remaining 2,405 sentences: some but not all experts label these as having economic policy content.

TABLE 2.

Domain Classification Matrix for 1987 and 1997 Manifestos: Frequency with which Sentences were Assigned by Six Experts to Economic and Policy Domains

Note: Shaded boxes: perfect agreement between experts.

The shaded boxes show sentences for which the six experts were in unanimous agreement—on economic policy, social policy, or neither. There was unanimous expert agreement on about 35 percent of the labeled sentences. For about 65 percent of sentences, there was disagreement, even about the policy area, among trained experts of the type usually used to analyze political texts.

Scale reliability

Despite substantial disagreement among experts about individual sentences, we saw above that we can derive externally valid estimates of party policy positions if we aggregate the judgments of all experts on all sentences in a given document. This happens because, while each expert judgment on each sentence is a noisy realization of some underlying signal about policy content, the expert judgments taken as a whole scale nicely—in the sense that in aggregate they are all capturing information about the same underlying quantity. Table 3 shows this, reporting a scale and coding reliability analysis for economic policy positions of the 1987 and 1997 manifestos, derived by treating economic policy scores for each sentence allocated by each of the six expert coders as six sets of independent estimates of economic policy positions.

TABLE 3.

Interexpert Scale Reliability Analysis for the Economic Policy, Generated by Aggregating All Expert Scores for Sentences Judged to have Economic Policy Content

Despite the variance in expert coding of the policy domains as seen in Table 2, overall agreement as to the policy domain of sentences was 0.93 using Fleiss’ kappa, a very high level of inter-rater agreement (as κ ranges from 0 to 1.0).Footnote ¹⁸ A far more important benchmark of reliability, however, focuses on the construction of the scale resulting from combining the coders’ judgments, which is of more direct interest than the codes assigned to any particular fragment of text. Scale reliability, as measured by a Cronbach's alpha of 0.95, is “excellent” by any conventional standard.Footnote ¹⁹ We can therefore apply our model to aggregate the noisy information contained in the combined set of expert judgements at the sentence level to produce coherent estimates of policy positions at the document level. This is the essence of crowd-sourcing. It shows that our experts are really a small crowd.

DEPLOYING CROWD-SOURCED TEXT CODING

CROWD-SOURCED ESTIMATES OF PARTY POLICY POSITIONS

Figure 3 plots crowd-sourced estimates of the economic and social policy positions of British party manifestos against estimates generated from analogous expert text processing.Footnote ²⁷ The very high correlations of aggregate policy measures generated by crowd workers and experts suggest both are measuring the same latent quantities. Substantively, Figure 3 also shows that crowd workers identified the sharp rightwards shift of Labour between 1987 and 1997 on both economic and social policy, a shift identified by expert text processing and independent expert surveys. The standard errors of crowd-sourced estimates are higher for social than for economic policy, reflecting both the smaller number of manifesto sentences devoted to social policy and higher coder disagreement over the application of this policy domain.Footnote ²⁸ Nonetheless Figure 3 summarizes our evidence that the crowd-sourced estimates of party policy positions can be used as substitutes for the expert estimates, which is our main concern in this article.

FIGURE 3.

Expert and Crowd-sourced Estimates of Economic and Social Policy Positions

Our scaling model provides a theoretically well-grounded way to aggregate all the information in our expert or crowd data, relating the underlying position of the political text both to the “difficulty” of a particular sentence and to a reader's propensity to identify the correct policy domain, and position within domain.Footnote ²⁹ Because positions derived from the scaling model depend on parameters estimated using the full set of coders and codings, changes to the text corpus can affect the relative scaling. The simple mean of means method, however, is invariant to rescaling and always produces the same results, even for a single document. Comparing crowd-sourced estimates from the scaling model to those produced by a simple averaging of the mean of mean sentence scores, we find correlations of 0.96 for the economic and 0.97 for the social policy positions of the 18 manifestos. We present both methods as confirmation that our scaling method has not “manufactured” policy estimates. While this model does allow us to take proper account of reader and sentence fixed effects, it is also reassuring that a simple mean of means produced substantively similar estimates.

We have already seen that noisy expert judgments about sentences aggregate up to reliable and valid estimates for documents. Similarly, crowd-sourced document estimates reported in Figure 3 are derived from crowd-sourced sentence data that are full of noise. As we already argued, this is the essence of crowd-sourcing. Figure 4 plots mean expert against mean crowd-sourced scores for each sentence. The scores are highly correlated, though crowd workers are substantially less likely to use extremes of the scales than experts. The first principal component and associated confidence intervals show a strong and significant statistical relationship between crowd sourced and expert assessments of individual manifesto sentences, with no evidence of systematic bias in the crowd-coded sentence scores.Footnote ³⁰ Overall, despite the expected noise, our results show that crowd workers systematically tend to make the same judgments about individual sentences as experts.

FIGURE 4.

Expert and Crowd-sourced Estimates of Economic and Social Policy Codes of Individual Sentences, all Manifestos

Note: Fitted line is the principal components or Deming regression line.

Calibrating the number of crowd judgments per sentence

A key question for our method concerns how many noisier crowd-based judgments we need to generate reliable and valid estimates of fairly long documents such as party manifestos. To answer this, we turn to evidence from our oversampling of 1987 and 1997 manifestos. Recall we obtained a minimum of 20 crowd judgments for each sentence in each of these manifestos, allowing us to explore what our estimates of the position of each manifesto would have been, had we collected fewer judgments. Drawing random subsamples from our oversampled data, we can simulate the convergence of estimated document positions as a function of the number of crowd judgments per sentence. We did this by bootstrapping 100 sets of subsamples for each of the subsets of n = 1 to n = 20 workers, computing manifesto positions in each policy domain from aggregated sentence position means, and computing standard deviations of these manifesto positions across the 100 estimates. Figure 5 plots these for each manifesto as a function of the increasing number of crowd workers per sentence, where each point represents the empirical standard error of the estimates for a specific manifesto. For comparison, we plot the same quantities for the expert data in red.

FIGURE 5.

Standard Errors of Manifesto-level Policy Estimates as a Function of the Number of Workers, for the Oversampled 1987 and 1997 Manifestos

Note: Each point is the bootstrapped standard deviation of the mean of means aggregate manifesto scores, computed from sentence-level random n subsamples from the codes.

The findings show a clear trend: uncertainty over the crowd-based estimates collapses as we increase the number of workers per sentence. Indeed, the only difference between experts and the crowd is that expert variance is smaller, as we would expect. Our findings vary somewhat with policy area, given the noisier character of social policy estimates, but adding additional crowd-sourced sentence judgments led to convergence with our expert panel of five to six coders at around 15 crowd coders. However, the steep decline in the uncertainty of our document estimates leveled out at around five crowd judgments per sentence, at which point the absolute level of error is already low for both policy domains. While increasing the number of unbiased crowd judgments will always give better estimates, we decided on cost-benefit grounds for the second stage of our deployment to continue coding in the crowd until we had obtained five crowd judgments per sentence. This may seem a surprisingly small number, but there are a number of important factors to bear in mind in this context. First, the manifestos comprise about 1000 sentences on average; our estimates of document positions aggregate codes for these. Second, sentences were randomly assigned to workers, so each sentence score can be seen as an independent estimate of the position of the manifesto on each dimension.Footnote ³¹ With five scores per sentence and about 1000 sentences per manifesto, we have about 5000 “little” estimates of the manifesto position, each a representative sample from the larger set of scores that would result from additional worker judgments about each sentence in each document. This sample is big enough to achieve a reasonable level of precision, given the large number of sentences per manifesto. While the method we use here could be used for much shorter documents, the results we infer here for the appropriate number of judgments per sentence might well not apply, and would likely be higher. But, for large documents with many sentences, we find that the number of crowd judgments per sentence that we need is not high.

CROWD-SOURCING DATA FOR SPECIFIC PROJECTS: IMMIGRATION POLICY

A key problem for scholars using “canonical” datasets, over and above the replication issues we discuss above, is that the data often do not measure what a modern researcher wants to measure. For example the widely used MP data, using a classification scheme designed in the 1980s, do not measure immigration policy, a core concern in the party politics of the 21st century (Ruedin Reference Ruedin2013; Ruedin and Morales Reference Ruedin and Morales2012). Crowd-sourcing data frees researchers from such “legacy” problems and allows them more flexibly to collect information on their precise quantities of interest. To demonstrate this, we designed a project tailored to measure British parties’ immigration policies during the 2010 election. We analyzed the manifestos of eight parties, including smaller parties with more extreme positions on immigration, such as the British National Party (BNP) and the UK Independence Party (UKIP). Workers were asked to label each sentence as referring to immigration policy or not. If a sentence did cover immigration, they were asked to rate it as pro- or anti-immigration, or neutral. We deployed a job with 7,070 manifesto sentences plus 136 “gold” questions and screeners devised specifically for this purpose. For this job, we used an adaptive sentence sampling strategy which set a minimum of five crowd-sourced labels per sentence, unless the first three of these were unanimous in judging a sentence not to concern immigration policy. This is efficient when coding texts with only “sparse” references to the matter of interest; in this case most manifesto sentences (approximately 96%) were clearly not about immigration policy. Within just five hours, the job was completed, with 22,228 codings, for a total cost of $360.Footnote ³²

We assess the external validity of our results using independent expert surveys by Benoit (Reference Benoit2010) and the Chapel Hill Expert Survey (Bakker et al. Reference Bakker, de Vries, Edwards, Hooghe, Jolly, Marks, Polk, Rovny, Steenbergen and Vachudova2015). Figure 6 compares the crowd-sourced estimates to those from expert surveys. The correlation with the Benoit (Reference Benoit2010) estimates (shown) was 0.96, and 0.94 with independent expert survey estimates from the Chapel Hill survey.Footnote ³³ To assess whether this data production exercise was as reproducible as we claim, we repeated the entire exercise with a second deployment two months after the first, with identical settings. This new job generated another 24,551 pieces of crowd-sourced data and was completed in just over three hours. The replication generated nearly identical estimates, detailed in Table 4, correlating at the same high levels with external expert surveys, and correlating at 0.93 with party position estimates from the original crowd coding.Footnote ³⁴ With just hours from deployment to dataset, and for very little cost, crowd sourcing enabled us to generate externally valid and reproducible data related to our precise research question.

FIGURE 6.

Correlation of Combined Immigration Crowd Codings with Benoit (Reference Benoit2010) Expert Survey Position on Immigration

TABLE 4.

Comparison Results for Replication of Immigration Policy Crowd Coding

CROWD SOURCED TEXT ANALYSIS IN OTHER CONTEXTS AND LANGUAGES

As carefully designed official statements of a party's policy stances, election manifestos tend to respond well to systematic text analysis. In addition, manifestos are written for popular consumption and tend to be easily understood by nontechnical readers. Much political information, however, can be found in texts generated from hearings, committee debates, or legislative speeches on issues that often refer to technical provisions, amendments, or other rules of procedure that might prove harder to analyze. Furthermore, a majority of the world's political texts are not in English. Other widely studied political contexts, such as the European Union, are multilingual environments where researchers using automated methods designed for a single language must make hard choices. Schwarz et al. (forthcoming) applied unsupervised scaling methods to a multilingual debate in the Swiss parliament, for instance, but had to ignore a substantial number of French and Italian speeches in order to focus on the majority German texts. In this section, we demonstrate that crowd-sourced text analysis, with appropriately translated instructions, offers the means to overcome these limitations by working in any language.

Our corpus comes from a debate in the European Parliament, a multilanguage setting where the EU officially translates every document into 24 languages. To test our method in a context very different from party manifestos, we chose a fairly technical debate concerning a Commission report proposing an extension to a regulation permitting state aid to uncompetitive coal mines. This debate concerned not only the specific proposal, involving a choice of letting the subsidies expire in 2011, permitting a limited continuation until 2014, or extending them until 2018 or even indefinitely.Footnote ³⁵ It also served as debating platform for arguments supporting state aid to uncompetitive industries, versus the traditionally liberal preference for the free market over subsidies. Because a vote was taken at the end of the debate, we also have an objective measure of whether the speakers supported or objected to the continuation of state aid.

We downloaded all 36 speeches from this debate, originally delivered by speakers from 11 different countries in 10 different languages. Only one of these speakers, an MEP from the Netherlands, spoke in English, but all speeches were officially translated into each target language. After segmenting this debate into sentences, devising instructions and representative test sentences and translating these into each language, we deployed the same text analysis job in English, German, Spanish, Italian, Polish, and Greek, using crowd workers to read and label the same set of texts, but using the translation into their own language. Figure 7 plots the score for each text against the eventual vote of the speaker. It shows that our crowd-sourced scores for each speech perfectly predict the voting behavior of each speaker, regardless of the language. In Table 5, we show correlations between our crowd-sourced estimates of the positions of the six different language versions of the same set of texts. The results are striking, with interlanguage correlations ranging between 0.92 and 0.96.Footnote ³⁶ Our text measures from this technical debate produced reliable measures of the very specific dimension we sought to estimate, and the validity of these measures was demonstrated by their ability to predict the voting behavior of the speakers. Not only are these results straightforwardly reproducible, but this reproducibility is invariant to the language in which the speech was written. Crowd-sourced text analysis does not only work in English.

FIGURE 7.

Scored Speeches from a Debate over State Subsidies by Vote, from Separate Crowd-sourced Text Analysis in Six Languages

Note: Aggregate scores are standardized for direct comparison.

TABLE 5.

Summary of Results from EP Debate Coding in Six Languages