A rapidly growing area in political science has focused on perfecting techniques to treat political text as ‘data’, usually for the purposes of estimating latent traits such as left–right political policy positions.Footnote 1 More traditional approaches have applied classical content analysis to categorize sub-units of political text, such as sentences in manifestos. Prominent examples of this latter approach include the thirty-year old Comparative Manifestos Project and the Policy Agendas Project.Footnote 2 ‘Text as data’ approaches use machines to convert text to quantitative information and use statistical tools to make inferences about characteristics of the author of the text. Content analysis schemes use humans to read textual sub-units and assign these to pre-defined categories. Both methods require the prior identification of a textual unit of analysis – a highly consequential, yet often unquestioned, feature of research design.
Our objective in this Research Note is to question the dominant approach to unitizing political texts prior to human coding. This is to parse texts into quasi-sentences (QSs), where a QS is defined as part or all of a natural sentence that states a distinct policy proposition. The use of the QS rather than a natural language unit (such as a sentence defined by punctuation) is motivated by the desire to capture all relevant political information, regardless of the stylistic decisions made by the author, for example, to use long or short natural sentences. The identification of QSs by human coders, however, is highly unreliable. If, comparing codings of the same texts using quasi-sentences and natural sentences, there is no appreciable difference in measured political content, then there is a strong case for replacing ‘endogenous’ human unitization with ‘exogenous’ unitization based on natural sentences that can be identified with perfect reliability by machines using pre-specified punctuation delimiters.
We proceed as follows. First, we discuss the issues motivating the use of QSs and the implications of this for reliability. Next, we re-examine and recode, using natural sentences, a set of texts in several languages that have previously been unitized and coded using QSs, comparing results to see if this generates significant differences in measured political content. We also compare coding reliabilities that arise when using natural sentences rather than quasi-sentences. Our results provide strong evidence that unitizing text exogenously using natural sentences is systematically more reliable than endogenous unitization based on human judgement, while delivering substantively similar estimates of the positions of text authors.
The Rationale For Endogenously Defined Text Units
Expert text coding involves two data-generating steps: dividing text into units of analysis, then assigning each text unit a code.Footnote 3 While this second step (coding) typically receives the greatest scrutiny, our concern here focuses on the first step (unitization). Prior to coding, a text must be unitized by dividing it into smaller units relevant to the research question. Unitization can be specified exogenously to the research process using no human judgement, on the basis of predefined rules. This defines units of text in a manner independent of any coding decisions made as part of the analysis. Examples of such rules include using words, word sequences or n-grams, natural sentences, paragraphs, pages or even entire documents as the unit of analysis. Alternatively, text units may be defined endogenously to the coding process and involve human judgement, as part of the content analysis itself, to determine where one unit of content ends and another begins.Footnote 4 Artificial intelligence applications of natural language processing identify text units during the text processing procedure, as do the QS parsing schemes applied by human coders in the Comparative Manifestos and Policy Agendas Projects. Choosing whether to define text units exogenously or endogenously to the text coding procedure involves making a fundamental trade-off between the goals of reliability and validity.
Expert or ‘hand’ coding methods are not alone in facing the issue of how to define the unit of text analysis. Statistical scaling or classification methods, in which there have been numerous recent advances,Footnote 5 typically make the linguistic ‘bag of words’ assumption and specify the word as the fundamental unit of text analysis. All substantive decisions about text unitization are part of the research design, not the coding process. Dictionary-based methods apply a pre-defined coding dictionary, the substantive content of which is at the heart of the research design, to tag words or word stems with coding categories associated with these words by the dictionary. As with statistical methods, the goal is typically fully automated machine coding, with all substantive decisions made as part of the research design rather than the unitization or coding processes. In methods of automated natural language processing, the goal is the automated extraction of meaning from natural language. Well-known examples can be found in Google Translate, or the Watson system recently and successfully developed by IBM, part of the ‘DeepQA’ project to understand and then answer the complex natural language questions that form part of the Jeopardy television quiz programme.Footnote 6 In such applications, the unit of text analysis must be interpreted as part of the system that also processes the text, rather than exogenous to the text analysis. Thus far, this research programme has been the preserve of computer scientists and computational linguists, as opposed to political scientists.
The method deployed by research projects based on text coding by human experts, such as the Comparative Manifestos and Comparative Policy Agendas Projects, is in essence non-automated natural language processing. Crudely speaking, expert coding can be seen as using skilled humans to engage in complex pattern recognition tasks that we cannot yet programme computers to produce with valid results. As with all natural language processing, the fundamental unit of text analysis may transcend punctuation marks, may conceivably range from a short phrase to an entire text corpus, and may also require human decisions about where a relevant text unit begins and ends. Because only humans can yet be trusted to provide valid readings of complex texts for meaning, such traditional methods of content analysis inherently involve subjective judgements by humans reading, parsing and coding text. Introducing human judgement as part of the text coding process rather than just the research design, however, introduces serious concerns about reliability that are not an issue for the other methods of text analysis we have identified.
Reliability Trade-Offs With Subjective Text Unitization
The core issue when designing schemes for unitizing and then coding text units as part of an expert coding project concerns the classic trade-off between reliability and validity. In expert text coding, a procedure is reliable when ‘the reading of textual data as well as of the research results is replicable elsewhere, [so] that researchers demonstrably agree on what they are talking about’.Footnote 7 Whenever non-deterministic instruments – such as human beings – are used to unitize and code texts, then the content analysis procedure faces potential problems with reliability. No matter how carefully trained, human coders invariably disagree over qualitative decisions regarding the identification of text units and their classification into coding categories. Depending on how unreliable the procedure is, estimates constructed from the codings may also lack validity because of the level of noise or even bias introduced by the content analysis procedure. Reliability is no guarantee of validity, however, and in practice validity tends to suffer in the pursuit of maximizing reliability. Indeed, the debate over machine versus human coded content analysis largely revolves around the desire to balance the competing objectives of reliability and validity. Proponents of computerized schemes for estimating party positions from political manifestos cite perfect reliability in their favour, yet struggle to demonstrate validity.Footnote 8 Hand-coded schemes such as the CMP claim validity as a central advantage but then devote huge resources to attempts to enhance reliability.Footnote 9
Validity, at its simplest, means that the results of some content analysis can be seen to reflect the ‘true’ content of the text in a meaningful way. If researchers use expert coders to classify the content of texts then, these days, they almost surely have chosen this laborious method over machine coding because they feel that the results are more valid – that humans can currently extract more valid meaning from complex texts than machines can. Furthermore, expert codings are likely to be most valid when the unit of text analysis is endogenous, since it is unlikely that readers for whom the texts are written pay close attention to punctuation marks when they read a text for meaning. Despite the ongoing debate about whether automatic coding can produce valid representations of textual content, our focus here applies only to the validity versus reliability issue applied to unitization. If we can demonstrate that deterministic unitization rules – axiomatically meeting a perfect standard of reliability in unitization – are just as valid as subjective ones based on endogenously defined text units, then we can move the state of human-coding text research further out on the reliability–validity frontier by eliminating the unreliability of subjective unitization without suffering a trade-off in validity.
The two most widely used coding schemes in political science – the Comparative Manifestos Project and the Policy Agendas ProjectFootnote 10 – both specify the unit of textual analysis as an endogenous text fragment known as the quasi-sentence (QS): ‘an argument which is the verbal expression of one political idea or issue’.Footnote 11 This approach to unitizing is often referred to as thematic unitizing.Footnote 12 The explicit motivation for using QSs is to avoid missing separate policy statements from political texts created by more long-winded authors who tend to combine multiple policy statements into single natural sentences. More generally, the rationale for endogenous text unitization is to implement a method of natural language processing when the meaning in natural language may not respect punctuation marks.
As an example, consider this natural sentence taken from the 2001 Australian National Party manifesto. The tenth natural sentence states: ‘We know that the only way to create economic prosperity is to rely on individual enterprise / and we know that our future as a nation depends on having strong families and communities.’ The CMP coder of this document identified two QSs, indicated here by the ‘/’. The first was assigned to category 401 (Free Enterprise: Positive), the second to category 606 (Social Harmony: Positive). To see how quasi-sentences’ unitization is executed in practice on a somewhat larger scale, in Figure 1 we have reproduced a section of the Scottish National Party manifesto of 2001 – parsed into QSs by a coder from the Comparative Manifestos Project. QSs are demarcated by the pencil marks in the text, indicating that ten natural sentences have been divided into twenty-three QSs, some as short as a single word. It is clearly not self-evident that different coders would meet Krippendorff's – or indeed anyone's – definition of reliability given the large number of different, and perfectly reasonable, ways of identifying an alternative set of word strings qualifying as independent QSs from this short manifesto fragment.Footnote 13
Fig. 1Section of SNP 2001 manifesto parsed into quasi-sentences by CMP coder
Using carefully trained expert coders following well-defined instructions may mitigate problems of unitization unreliability, but we now show that serious problems remain. We obtained data from the CMP, derived from their own coder training experiments, in which many expert coders were asked to unitize and code the same training document, and the results from sixty-seven trained coders showed huge variability in the number of QSs they identified in the text. According to the CMP's master coding, applying the authoritative version of their quasi-sentence unitization, the document contains a ‘true’ number of 163 QSs. The CMP's trained expert coders, however, identified a total number of QSs ranging from about 120 to 220, with a standard deviation of 19 and an interquartile range of 148 to 173.Footnote 14 The conclusion is clear. When even well-trained human expert coders specify units of analysis endogenously – as CMP coders do when they parse a text into QSs – the results are very unreliable. Because human-coded content analyses typically combine results from different coders, furthermore, systematic differences in subjective judgements about endogenously identified text units (whereby some coders tend to see more units where others tend to see fewer) may introduce bias as well as unreliability.
One way to enhance the reliability of expert text coding is to develop an automated method for endogenously identifying QSs. The hard problems of natural language processing for complex political texts, however, mean that no automated unitization of texts into QSs is currently feasible – at least, none which we would confidently declare is valid for making the information-rich thematic distinctions motivating the use of endogenous text units in the first place. An alternative approach is to define text units exogenously to the content analysis process, following (for example) syntactical distinctions that are ‘natural’ relative to the grammar of the text.Footnote 15 Among the choices of syntactically delimited units, natural sentences are closest to the thematically defined QSs used by the CMP. Instead of endogenously defined thematic units, natural sentences are exogenously specified using predefined lists of punctuation marks. The open empirical question, addressed in the rest of this Research Note, concerns whether specifying the unit of analysis as exogenously specified natural sentences, rather than endogenously specified QSs, significantly affects inferences about the substantive content of the types of text we wish to investigate.Footnote 16 Exogenous specification of the unit of text analysis as a natural sentence is axiomatically more reliable than allowing expert coders to unitize text endogenously. If exogenous unitization does generate different results, this raises the reliability–validity trade-off for consideration. If it does not, then using perfectly reliable natural sentences as the fundamental unit of text analysis for expert coding is a dominant methodological strategy.
Data and Methods
Our comparison of the validity of expert-coded text analyses based on exogenous versus endogenous text units comes from a reanalysis of manifestos originally unitized and coded by the CMP. Ideally, we would provide a set of manifestos to a large group of coders, and ask that each be coded on the basis of natural sentences and QSs, and then compare the aggregate measures of political content. If there were no appreciable differences in the measures of aggregate political content, then we would declare both methods equally valid.Footnote 17 Of course, this comparison would not determine whether either method in itself was valid in absolute terms, but if no differences exist in the way each unitization scheme characterizes political content, then it is strong evidence that one cannot be considered less valid than the other.
Without conducting a time-consuming and expensive test such as this, however, we do have access to information that allows us to investigate whether endogenous text unitization makes a difference. This involves returning to a set of manifestos previously unitized into QSs and coded by trained CMP coders, codings that form part of the data reported in the CMP dataset. This set of fifteen documents consists of printed manifestos with unitization marks and marginal codes of the sort depicted in Figure 1. From a limited number of such texts that we were able to obtain upon request from CMP archives, we selected fifteen texts with the aim of incorporating a large range of political contexts. We maximized the number of countries covered and included a wide variety of texts in terms of language, length, party family of the authoring party and left–right orientation. The sample includes eight English texts, consisting of one manifesto from Australia, one from Ireland, one from New Zealand, three from Britain and two from the United States; three Estonian manifestos (in Estonian); two German-language manifestos from Austria; and two manifestos from Iceland (in Icelandic). A more detailed description of the sample is provided in Appendix Table A1.
Appendix Table A1Description of Manifestos Selected for Re-analysis
Using the selected texts, we proceeded in two steps. First, we recorded all QS codes indicated on the margin of the documents, and also noted whether each coded QS is identical to a natural sentence or a fragment of some natural sentence. This yields a dataset where the unit of the analysis is the natural sentence, while component QSs (one or several) are sub-units.Footnote 18 In the second step, we assigned a CMP policy code to each natural sentence. In this step, three different situations can occur:
1. A natural sentence contains one single QS. In this case, the CMP policy code assigned to the natural sentence is simply the one originally assigned to the QS.
2. A natural sentence contains more than one QS, but all were given the same policy code by the CMP coder. In this case, the natural sentence receives this code.
3. A natural sentence contains more than one QS, and these were given different policy codes. In this case, a human coding the natural sentence and faced with this choice would probably decide which of the competing policy codes best represented the natural sentence unit. Our procedure applied three different rules for making such a coding decision:
First: Assign the natural sentence the code of the first component QS.
Last: Assign the natural sentence the code of the last component QS.
Random: Assign the natural sentence the code of a randomly chosen component QS.
To be as sure as possible of the validity of our recoding exercise, the authors themselves carefully applied this method to the fifteen selected manifestos.
We first report the frequencies of the three types of relationship between natural sentences and quasi-sentences. We then aggregate text codings into widely used left–right policy scales, and compare our substantive conclusions about document content when we shift from QS unitization to natural sentences (also comparing the first, last and random rules for assigning a QS code to the natural sentence level).
Human coders faced with a natural sentence that they feel contains more than one policy statement may face a tough choice in deciding which code to assign it, if only one code may be assigned.Footnote 19 In such a case, it is possible that coding reliability – a separate issue from unitization reliability – may become an issue. As a preliminary test of this possible problem, we report the results of a coding experiment conducted using an expanded version of the CMP scheme applied to European election manifestos, designed to test whether coding unreliability increased when coders were asked to use natural rather than quasi-sentences as the basis for the CMP scheme.
Results From Recoding Manifestos Into Natural Sentences
Comparing Units of Analysis
Our revisiting and recording of the text units from the fifteen manifestos provided a dataset of a total of 6,960 natural sentences, in which were contained 8,481 QSs. These are described in Table 1.
Table 1Pattern of Natural Sentences versus Quasi-Sentences from 15 Election Manifestos
The clearest result to emerge from our analysis is that the splitting by CMP coders of natural sentences into more than one QS occurs quite infrequently: in more than eight out of ten cases (84 per cent), natural sentences contained a single QS only, meaning that the endogenous unitization problem pertains to only 16 per cent of all text units. The remaining natural sentences include mostly two (12 per cent of all natural sentences) or three QSs (2.9 per cent of all natural sentences). Natural sentences with four or more QSs are very rare, making up just 1.1 per cent of our sample.
The second strong result from our analysis is that when natural sentences are split into component QSs, these components are not necessarily coded differently. In fact, in the category of natural sentences with two QSs, fewer than half (44.1 per cent) of the natural sentences have different component codes, rising to just over half (58.4 per cent) for natural sentences split into three QSs. More of those split into four or more QSs were different, although overall, as previously mentioned, these represent a tiny fraction of all of natural sentences. Considering all natural sentences, the share of cases with varying component codes is just 7.7 per cent. In other words, before any additional comparison, we expect results that are at the very least 92 per cent identical, because there is a 92 per cent similarity between the two unitizations. Coding all QSs in the same natural sentence into the same category is at odds with the rationale for using QSs in the first place, which is that a natural sentence may have more then one component of meaning. Indeed, it amounts to an arbitrary, and as we have seen unreliable, ‘double’ or ‘triple’ counting of text units in which one natural sentence contains several identically coded QSs.
Results reported so far refer to coding differences based on the 56-category CMP scheme.Footnote 20 Published applications of CMP data typically use scales that combine many coding categories. By a country mile, the most popular application of the CMP data is the left–right index ‘Rile’, a scale that combines twenty-six of the fifty-six CMP coding categories. It is quite possible that all QSs in the same natural sentence were coded in the same direction (left, right, or neither) in relation to the Rile scale. The fifth column of Table 1 shows that, among natural sentences with two component QSs, less than three out of ten (28.7 per cent) involve QSs coded in different directions on the Rile scale. Among natural sentences with three component QSs, this figure is similar (36.6 per cent). The total share of natural sentences with component QS codes that differ in terms of left–right orientation is only 5 per cent.
In a purely descriptive sense, our analysis comparing natural sentence to QS unitization has shown that, even prior to our comparison of substantive political content, we should expect similarities of 92 and 95 per cent between codings based on perfectly reliable exogenously defined text units, natural sentences, and those based on unreliable, labour-intensive endogenously defined QSs – in practice these units of analysis are exactly the same in twelve out of thirteen cases.
Comparing Aggregate Results
Individual sentence codings are of little substantive interest to end users of political content analysis datasets, and are not even reported. Instead, the CMP dataset contains only the percentages of text units coded into each policy category – the ‘per’ codes – as well as the total number of QSs recorded in the manifesto. Our aim in this section is, therefore, to compare aggregate category percentages from each manifesto when these are reconstructed from natural sentences and QSs.Footnote 21 This involved applying our three coding rules – choose the first QS code, the last, or one at random – to code the natural sentence. We see from Table 1 that this affects just about 8 per cent of all natural sentences.
Figure 2 shows the comparison of each policy category's percentage share, in a scatterplot matrix comparing our three rules for coding natural sentences with more than one QS, to the raw QS-based results used by the CMP. Each point represents a policy percentage from one manifesto, and the dashed line shows the 45° axis of perfect agreement. To reduce skew created by low-frequency policy categories, we logged both axes (this makes no substantial difference to the results). The squares above the diagonal report Pearson's correlation coefficient, ranging from 0.98 to 0.99 – there is an almost perfect linear relationship regardless of which rule is applied. To test the overall agreement in a more numerical framework, we used a simple regression analysis of the logged QS policy category percentages on the logged natural sentence policy category percentages (see Table 2). The results confirm inferences drawn from viewing scatterplots; 98 per cent of the variance in the original QS coding is explained by the natural sentence codings, regardless which rule is applied. An F-test of whether the estimated slope coefficient differs from the 1.0 value implying perfect identity cannot reject this null hypothesis. This is strong evidence that the natural sentence and QS codings yield the same aggregate results.
Fig. 2Comparing quasi-sentence aggregate category percentages to natural sentence recodings Notes: Three rules are compared: randomly assign the code based on constituent QSs; take the first QS code for the natural sentence; and take the last QS code for the natural sentences. Total manifestos analysed: 15.
Table 2Regression of (Log) Quasi-Sentence-Based % Categories by Manifesto on (Log) Natural Sentence-Based Estimates Using Three Rules
As we noted above, the most commonly used product of the CMP dataset is the left–right ‘Rile’ index that includes twenty-six of the fifty-six CMP coding categories. Figure 3 plots Rile scores for our fifteen manifestos, using exogenous natural sentence and endogenous QS unitization, and shows a very high degree of agreement between the two. Because Figure 3 only has one aggregate data point representing each manifesto for which we recorded the text units, we re-sampled natural sentences drawing 100 samples of 100 natural sentences from each manifesto. Figure 4 reports results, plotting a total of 15 × 100 = 1,500 points representing the fifteen manifestos in our sample. The top panel of Figure 4 shows the CMP's additive, original Rile scale, while the bottom panel depicts the recently proposed aggregate logit Rile scale, a scale that has demonstrably better properties than the CMP's relative difference scale.Footnote 22 In both cases, almost perfect correspondence is observed, even given the variation to be expected in each case from the sampling procedure.
Fig. 3Actual Rile Values aggregated for each manifesto Note: Based on random assignment, which we have chosen because human coders could almost certainly do better than this rule.
Fig. 4Re-sampled Rile values Notes: Based on random assignment, we took 100 random draws of 100 natural sentences each from each manifesto, and plotted the overall distribution of scores. The bottom plot uses the log Rile introduced by Lowe, Benoit, Mikhaylov and Laver, ‘Scaling Policy Preferences from Coded Political Texts’.
Results From A Coding Reliability Experiment
The results we report above imply that using natural sentences rather than quasi-sentences as units of analysis does not affect the validity of the classification of these units following deterministic unitization. Indeed, we demonstrated that endogenous unitization so rarely results in multiple and differently coded QSs within one natural sentence unit that even random allocation of codes to the larger natural sentence units resulted in essentially the same aggregate results – suggesting that the reliability of coding has little potential to be adversely affected by the switch to natural sentence units. Our test used an exogenously specified procedure to code natural sentences, when these were split into differently coded QSs. A computer applying formal rules to do this will suffer no qualms of indecisiveness and display no favouritism towards particular policy categories or domains. It is possible, however, that a human expert coder faced with a natural sentence containing what are seen as two distinct policy statements will not use a consistent rule in coding the natural sentence text unit. While eliminating the unreliability of subjective unitization by using natural sentences, it could be that we are at the same time increasing the unreliability of coding by forcing human experts to make a Sophie's Choice when coding natural sentences that could, and perhaps should, be considered to express more than one distinct policy statement.
Our validity tests did not involve any human recoding, since the objective was to identify a lower bound of reliability by comparing three random allocation rules to code those 16 per cent of natural sentences that were split into multiple QSs. Given the extremely high convergence of results caused by all three rules we applied, we expect that human coding could only improve reliability over the computer implementation of our random rules. As an independent test where human coding of all text units permits a broader test of reliability, we assess the reliability of natural sentence versus quasi-sentence coding using results of a series of experiments applying a CMP-based coding scheme to party manifestos for the European Elections conducted by Braun, Mikhaylov and Schmitt.Footnote 23 In this experimental design, expert text coders were randomly assigned to two groups. In a setup following Mikhaylov, Laver and Benoit,Footnote 24 both groups had to code the same excerpt of the 1999 British Liberal Democratic Party Euromanifesto using an online coding platform.Footnote 25 The first group (twenty-three participants) was asked first to unitize the document into QSs, and then to code these QSs using the coding scheme. The second group (twenty-nine participants) was asked simply to treat natural sentences as text units and code these using the coding scheme. Coders were undergraduate students from the University of Mannheim, followed coding instructions as outlined by Budge et al.,Footnote 26 and used a version of the CMP-based coding scheme modified to address issues specific to European Parliament elections.Footnote 27
In order to assess the reliability and quality of the coding process and consequently of the data generation process, Braun, Mikhaylov and Schmitt calculate inter-coder agreement in each experimental group, with the aim of comparing the coding reliability of the two groups (which are defined by natural sentence versus quasi-sentence unitization).Footnote 28 Agreement was measured using Fleiss's kappa (κ).Footnote 29 The κ coefficient is by far the most widely used method of statistical analysis of agreement for categorical variables,Footnote 30 and generalizes directly to multiple coders rating the multiple items. The coefficient has a range from 0 (perfect disagreement) to 1 (perfect agreement), and takes into account the fact that some agreement may occur purely by chance.Footnote 31 While there are no universally accepted guidelines, it has been suggested that κ > 0.75 represents excellent agreement, that 0.40 < κ < 0.75 represents fair to good agreement, and that any κ < 0.40 indicates poor agreement.Footnote 32 Similar to our own results above, the QSs identified in the first stage of the Mannheim experiment were predominantly full natural sentences; furthermore, when one natural sentence consisted of more than one QS, these were typically coded into the same category. Just as with our own results, therefore, we would not expect much systematic difference in coding results between the two groups. The results relevant for our purpose are shown in Table 3.
Table 3Inter-coder Reliability Results from the Experiment Comparing Natural and Quasi-Sentence Unitizations for the Euromanifesto Coding Scheme
The key result for our purposes is that the inter-coder reliabilities of the groups coding natural versus quasi-sentences is substantively indistinguishable. Coding reliability is quite poor in this experiment overall, possibly reflecting that the main objective of Braun, Mikhaylov and Schmitt was to test the reliability of a new coding scheme for the manifestos component of the European Election Study. Overall, however, the reliability coefficients reported in Table 3 are in line with the 0.3–0.4 from tests by coders applying the standard CMP coding scheme to pre-unitized QSs reported by Mikhaylov, Laver and Benoit.Footnote 33 In sum, there is no evidence in these results that coding natural sentences rather than QSs affects the reliability of coding. The huge gain in reliability from moving to an exogenous definition of text units does not appear to come at the expense of coding reliability.
Conclusions
It is fundamental to the systematic analysis of political text that we specify the basic unit of text analysis. The ‘exogenous’ specification of the natural sentence as the text unit results in perfectly reliable unitization, but potentially pays a price in coding validity if natural sentences contain more than one message. There is also a potential price in coding reliability if human coders are asked to pick one from several messages in a single natural sentence. Addressing these issues, we draw four primary conclusions.
First, in only a small minority of cases in the manifestos we examined are natural sentences divided into separate QSs. This means that in effect there is little possible difference between a scheme requiring humans to make painstaking and unreliable decisions on parsing natural sentences into smaller units, because most QSs are also natural sentence units.
Secondly, even when the QS unitization rules call for dividing a natural sentence into multiple text units, more than half of these subdivided natural sentences contained sub-units that all have the same code. This means that no information about alternative policy emphases is lost for these units by considering only natural sentences. It also undermines the CMP rationale for using QSs, which is that one natural sentence may contain more than one distinct message, while it in effect results in double or triple counting of a subset of text units.
Thirdly, in our comparisons of the policy categories aggregated into percentages, including different left–right scales, we found no substantive differences between aggregations based on natural sentence versus quasi-sentence unitization. Our random procedure to assign a split natural sentence one of its constituent QS codes reproduced about 98 per cent of the variance in the aggregate measures based on QSs, including when subsamples were drawn to simulate the additional sampling variance that might come from having shorter manifestos. Because we think that human coders could improve on the random rules using expert judgement, furthermore, we expect our results to represent a worst-case scenario.
Finally, reporting the results from coder experiments where participants were asked to code either natural sentences or QSs, we found no evidence that inter-coder reliability differed between these two groups. The possible information loss from increasing the size of the text-coding unit from QSs to natural sentences does not on our evidence introduce additional unreliability.
The implication of these results for applying categorical coding schemes to political text is clear and simple. Natural sentences can be substituted for QSs to achieve a major gain in the reliability of text unitization without loss of validity. This implies that future text coding projects should dispense with endogenous text unitization by human experts as part of the coding process, and move to fully automated unitization based on natural sentence delimiters defined exogenously as part of the research design. Since our estimates suggest that substantive findings are unlikely to be affected by doing this, but reliability is likely to increase, the shift to natural sentence unitization could usefully be extended to the ongoing CMP and PA projects. Our analysis here implies that a substantial gain in reliability, efficiency and replicability can be achieved without sacrificing important substantive information in the texts under investigation.
%20-%20https://sketch.com%20--%3e%3ctitle%3e3ECBBF50-FD08-412A-943D-2CFC5E025A16%3c/title%3e%3cdesc%3eCreated%20with%20sketchtool.%3c/desc%3e%3cg%20id='Stylesheet'%20stroke='none'%20stroke-width='1'%20fill='none'%20fill-rule='evenodd'%3e%3cg%20transform='translate(-926.000000,%20-668.000000)'%20id='BUTTONS'%3e%3cg%20transform='translate(100.000000,%20597.000000)'%3e%3cg%20id='BTN-/-RIGHTS'%20transform='translate(816.000000,%2065.000000)'%3e%3cg%20id='ICON-/-Rights'%20transform='translate(10.000000,%206.000000)'%3e%3cpath%20d='M13.5752841,0%20L1.34943182,0%20C0.59375,0%200,0.586666667%200,1.33333333%20L0,22.6666667%20C0,23.4133333%200.620738636,24%201.34943182,24%20L17.6505682,24%20C18.40625,24%2019,23.3866667%2019,22.6666667%20L19,5.36%20L13.5752841,0%20Z%20M13.5752841,1.89333333%20L17.0838068,5.36%20L13.5752841,5.36%20L13.5752841,1.89333333%20Z%20M1.34943182,22.6666667%20L1.34943182,1.33333333%20L12.1988636,1.33333333%20L12.1988636,5.36%20C12.1988636,6.10666667%2012.7926136,6.69333333%2013.5482955,6.69333333%20L17.6235795,6.69333333%20L17.6505682,22.6666667%20L1.34943182,22.6666667%20Z'%20id='Shape'%20fill='%23595959'%20fill-rule='nonzero'%3e%3c/path%3e%3ccircle%20id='Oval'%20stroke='%23595959'%20stroke-width='1.5'%20fill='%23595959'%20cx='9.5'%20cy='13.5355339'%20r='5.5'%3e%3c/circle%3e%3cpolyline%20id='Path-2'%20stroke='%23FFFFFF'%20stroke-width='1.5'%20transform='translate(9.500000,%2013.035534)%20rotate(45.000000)%20translate(-9.500000,%20-13.035534)%20'%20points='8%2016.0355339%2011%2016.0355339%2011%2010.0355339'%3e%3c/polyline%3e%3c/g%3e%3c/g%3e%3c/g%3e%3c/g%3e%3c/g%3e%3c/svg%3e)
A rapidly growing area in political science has focused on perfecting techniques to treat political text as ‘data’, usually for the purposes of estimating latent traits such as left–right political policy positions.Footnote 1 More traditional approaches have applied classical content analysis to categorize sub-units of political text, such as sentences in manifestos. Prominent examples of this latter approach include the thirty-year old Comparative Manifestos Project and the Policy Agendas Project.Footnote 2 ‘Text as data’ approaches use machines to convert text to quantitative information and use statistical tools to make inferences about characteristics of the author of the text. Content analysis schemes use humans to read textual sub-units and assign these to pre-defined categories. Both methods require the prior identification of a textual unit of analysis – a highly consequential, yet often unquestioned, feature of research design.
Our objective in this Research Note is to question the dominant approach to unitizing political texts prior to human coding. This is to parse texts into quasi-sentences (QSs), where a QS is defined as part or all of a natural sentence that states a distinct policy proposition. The use of the QS rather than a natural language unit (such as a sentence defined by punctuation) is motivated by the desire to capture all relevant political information, regardless of the stylistic decisions made by the author, for example, to use long or short natural sentences. The identification of QSs by human coders, however, is highly unreliable. If, comparing codings of the same texts using quasi-sentences and natural sentences, there is no appreciable difference in measured political content, then there is a strong case for replacing ‘endogenous’ human unitization with ‘exogenous’ unitization based on natural sentences that can be identified with perfect reliability by machines using pre-specified punctuation delimiters.
We proceed as follows. First, we discuss the issues motivating the use of QSs and the implications of this for reliability. Next, we re-examine and recode, using natural sentences, a set of texts in several languages that have previously been unitized and coded using QSs, comparing results to see if this generates significant differences in measured political content. We also compare coding reliabilities that arise when using natural sentences rather than quasi-sentences. Our results provide strong evidence that unitizing text exogenously using natural sentences is systematically more reliable than endogenous unitization based on human judgement, while delivering substantively similar estimates of the positions of text authors.
The Rationale For Endogenously Defined Text Units
Expert text coding involves two data-generating steps: dividing text into units of analysis, then assigning each text unit a code.Footnote 3 While this second step (coding) typically receives the greatest scrutiny, our concern here focuses on the first step (unitization). Prior to coding, a text must be unitized by dividing it into smaller units relevant to the research question. Unitization can be specified exogenously to the research process using no human judgement, on the basis of predefined rules. This defines units of text in a manner independent of any coding decisions made as part of the analysis. Examples of such rules include using words, word sequences or n-grams, natural sentences, paragraphs, pages or even entire documents as the unit of analysis. Alternatively, text units may be defined endogenously to the coding process and involve human judgement, as part of the content analysis itself, to determine where one unit of content ends and another begins.Footnote 4 Artificial intelligence applications of natural language processing identify text units during the text processing procedure, as do the QS parsing schemes applied by human coders in the Comparative Manifestos and Policy Agendas Projects. Choosing whether to define text units exogenously or endogenously to the text coding procedure involves making a fundamental trade-off between the goals of reliability and validity.
Expert or ‘hand’ coding methods are not alone in facing the issue of how to define the unit of text analysis. Statistical scaling or classification methods, in which there have been numerous recent advances,Footnote 5 typically make the linguistic ‘bag of words’ assumption and specify the word as the fundamental unit of text analysis. All substantive decisions about text unitization are part of the research design, not the coding process. Dictionary-based methods apply a pre-defined coding dictionary, the substantive content of which is at the heart of the research design, to tag words or word stems with coding categories associated with these words by the dictionary. As with statistical methods, the goal is typically fully automated machine coding, with all substantive decisions made as part of the research design rather than the unitization or coding processes. In methods of automated natural language processing, the goal is the automated extraction of meaning from natural language. Well-known examples can be found in Google Translate, or the Watson system recently and successfully developed by IBM, part of the ‘DeepQA’ project to understand and then answer the complex natural language questions that form part of the Jeopardy television quiz programme.Footnote 6 In such applications, the unit of text analysis must be interpreted as part of the system that also processes the text, rather than exogenous to the text analysis. Thus far, this research programme has been the preserve of computer scientists and computational linguists, as opposed to political scientists.
The method deployed by research projects based on text coding by human experts, such as the Comparative Manifestos and Comparative Policy Agendas Projects, is in essence non-automated natural language processing. Crudely speaking, expert coding can be seen as using skilled humans to engage in complex pattern recognition tasks that we cannot yet programme computers to produce with valid results. As with all natural language processing, the fundamental unit of text analysis may transcend punctuation marks, may conceivably range from a short phrase to an entire text corpus, and may also require human decisions about where a relevant text unit begins and ends. Because only humans can yet be trusted to provide valid readings of complex texts for meaning, such traditional methods of content analysis inherently involve subjective judgements by humans reading, parsing and coding text. Introducing human judgement as part of the text coding process rather than just the research design, however, introduces serious concerns about reliability that are not an issue for the other methods of text analysis we have identified.
Reliability Trade-Offs With Subjective Text Unitization
The core issue when designing schemes for unitizing and then coding text units as part of an expert coding project concerns the classic trade-off between reliability and validity. In expert text coding, a procedure is reliable when ‘the reading of textual data as well as of the research results is replicable elsewhere, [so] that researchers demonstrably agree on what they are talking about’.Footnote 7 Whenever non-deterministic instruments – such as human beings – are used to unitize and code texts, then the content analysis procedure faces potential problems with reliability. No matter how carefully trained, human coders invariably disagree over qualitative decisions regarding the identification of text units and their classification into coding categories. Depending on how unreliable the procedure is, estimates constructed from the codings may also lack validity because of the level of noise or even bias introduced by the content analysis procedure. Reliability is no guarantee of validity, however, and in practice validity tends to suffer in the pursuit of maximizing reliability. Indeed, the debate over machine versus human coded content analysis largely revolves around the desire to balance the competing objectives of reliability and validity. Proponents of computerized schemes for estimating party positions from political manifestos cite perfect reliability in their favour, yet struggle to demonstrate validity.Footnote 8 Hand-coded schemes such as the CMP claim validity as a central advantage but then devote huge resources to attempts to enhance reliability.Footnote 9
Validity, at its simplest, means that the results of some content analysis can be seen to reflect the ‘true’ content of the text in a meaningful way. If researchers use expert coders to classify the content of texts then, these days, they almost surely have chosen this laborious method over machine coding because they feel that the results are more valid – that humans can currently extract more valid meaning from complex texts than machines can. Furthermore, expert codings are likely to be most valid when the unit of text analysis is endogenous, since it is unlikely that readers for whom the texts are written pay close attention to punctuation marks when they read a text for meaning. Despite the ongoing debate about whether automatic coding can produce valid representations of textual content, our focus here applies only to the validity versus reliability issue applied to unitization. If we can demonstrate that deterministic unitization rules – axiomatically meeting a perfect standard of reliability in unitization – are just as valid as subjective ones based on endogenously defined text units, then we can move the state of human-coding text research further out on the reliability–validity frontier by eliminating the unreliability of subjective unitization without suffering a trade-off in validity.
The two most widely used coding schemes in political science – the Comparative Manifestos Project and the Policy Agendas ProjectFootnote 10 – both specify the unit of textual analysis as an endogenous text fragment known as the quasi-sentence (QS): ‘an argument which is the verbal expression of one political idea or issue’.Footnote 11 This approach to unitizing is often referred to as thematic unitizing.Footnote 12 The explicit motivation for using QSs is to avoid missing separate policy statements from political texts created by more long-winded authors who tend to combine multiple policy statements into single natural sentences. More generally, the rationale for endogenous text unitization is to implement a method of natural language processing when the meaning in natural language may not respect punctuation marks.
As an example, consider this natural sentence taken from the 2001 Australian National Party manifesto. The tenth natural sentence states: ‘We know that the only way to create economic prosperity is to rely on individual enterprise / and we know that our future as a nation depends on having strong families and communities.’ The CMP coder of this document identified two QSs, indicated here by the ‘/’. The first was assigned to category 401 (Free Enterprise: Positive), the second to category 606 (Social Harmony: Positive). To see how quasi-sentences’ unitization is executed in practice on a somewhat larger scale, in Figure 1 we have reproduced a section of the Scottish National Party manifesto of 2001 – parsed into QSs by a coder from the Comparative Manifestos Project. QSs are demarcated by the pencil marks in the text, indicating that ten natural sentences have been divided into twenty-three QSs, some as short as a single word. It is clearly not self-evident that different coders would meet Krippendorff's – or indeed anyone's – definition of reliability given the large number of different, and perfectly reasonable, ways of identifying an alternative set of word strings qualifying as independent QSs from this short manifesto fragment.Footnote 13
Section of SNP 2001 manifesto parsed into quasi-sentences by CMP coder
Using carefully trained expert coders following well-defined instructions may mitigate problems of unitization unreliability, but we now show that serious problems remain. We obtained data from the CMP, derived from their own coder training experiments, in which many expert coders were asked to unitize and code the same training document, and the results from sixty-seven trained coders showed huge variability in the number of QSs they identified in the text. According to the CMP's master coding, applying the authoritative version of their quasi-sentence unitization, the document contains a ‘true’ number of 163 QSs. The CMP's trained expert coders, however, identified a total number of QSs ranging from about 120 to 220, with a standard deviation of 19 and an interquartile range of 148 to 173.Footnote 14 The conclusion is clear. When even well-trained human expert coders specify units of analysis endogenously – as CMP coders do when they parse a text into QSs – the results are very unreliable. Because human-coded content analyses typically combine results from different coders, furthermore, systematic differences in subjective judgements about endogenously identified text units (whereby some coders tend to see more units where others tend to see fewer) may introduce bias as well as unreliability.
One way to enhance the reliability of expert text coding is to develop an automated method for endogenously identifying QSs. The hard problems of natural language processing for complex political texts, however, mean that no automated unitization of texts into QSs is currently feasible – at least, none which we would confidently declare is valid for making the information-rich thematic distinctions motivating the use of endogenous text units in the first place. An alternative approach is to define text units exogenously to the content analysis process, following (for example) syntactical distinctions that are ‘natural’ relative to the grammar of the text.Footnote 15 Among the choices of syntactically delimited units, natural sentences are closest to the thematically defined QSs used by the CMP. Instead of endogenously defined thematic units, natural sentences are exogenously specified using predefined lists of punctuation marks. The open empirical question, addressed in the rest of this Research Note, concerns whether specifying the unit of analysis as exogenously specified natural sentences, rather than endogenously specified QSs, significantly affects inferences about the substantive content of the types of text we wish to investigate.Footnote 16 Exogenous specification of the unit of text analysis as a natural sentence is axiomatically more reliable than allowing expert coders to unitize text endogenously. If exogenous unitization does generate different results, this raises the reliability–validity trade-off for consideration. If it does not, then using perfectly reliable natural sentences as the fundamental unit of text analysis for expert coding is a dominant methodological strategy.
Data and Methods
Our comparison of the validity of expert-coded text analyses based on exogenous versus endogenous text units comes from a reanalysis of manifestos originally unitized and coded by the CMP. Ideally, we would provide a set of manifestos to a large group of coders, and ask that each be coded on the basis of natural sentences and QSs, and then compare the aggregate measures of political content. If there were no appreciable differences in the measures of aggregate political content, then we would declare both methods equally valid.Footnote 17 Of course, this comparison would not determine whether either method in itself was valid in absolute terms, but if no differences exist in the way each unitization scheme characterizes political content, then it is strong evidence that one cannot be considered less valid than the other.
Without conducting a time-consuming and expensive test such as this, however, we do have access to information that allows us to investigate whether endogenous text unitization makes a difference. This involves returning to a set of manifestos previously unitized into QSs and coded by trained CMP coders, codings that form part of the data reported in the CMP dataset. This set of fifteen documents consists of printed manifestos with unitization marks and marginal codes of the sort depicted in Figure 1. From a limited number of such texts that we were able to obtain upon request from CMP archives, we selected fifteen texts with the aim of incorporating a large range of political contexts. We maximized the number of countries covered and included a wide variety of texts in terms of language, length, party family of the authoring party and left–right orientation. The sample includes eight English texts, consisting of one manifesto from Australia, one from Ireland, one from New Zealand, three from Britain and two from the United States; three Estonian manifestos (in Estonian); two German-language manifestos from Austria; and two manifestos from Iceland (in Icelandic). A more detailed description of the sample is provided in Appendix Table A1.
Description of Manifestos Selected for Re-analysis
Party family abbreviations: AGR = Agrarian, CHR = Christian Democratic, COM = Communist, CON = Conservative, ETH = Ethnic-regionalist, LIB = Liberal, SOC = Social Democratic.
* The number of QSs recorded by the CMP in the coding of this document, e.g. the Austrian 1971 manifesto was determined by the CMP coder to contain 213 QSs.
Using the selected texts, we proceeded in two steps. First, we recorded all QS codes indicated on the margin of the documents, and also noted whether each coded QS is identical to a natural sentence or a fragment of some natural sentence. This yields a dataset where the unit of the analysis is the natural sentence, while component QSs (one or several) are sub-units.Footnote 18 In the second step, we assigned a CMP policy code to each natural sentence. In this step, three different situations can occur:
1. A natural sentence contains one single QS. In this case, the CMP policy code assigned to the natural sentence is simply the one originally assigned to the QS.
2. A natural sentence contains more than one QS, but all were given the same policy code by the CMP coder. In this case, the natural sentence receives this code.
3. A natural sentence contains more than one QS, and these were given different policy codes. In this case, a human coding the natural sentence and faced with this choice would probably decide which of the competing policy codes best represented the natural sentence unit. Our procedure applied three different rules for making such a coding decision:
First: Assign the natural sentence the code of the first component QS.
Last: Assign the natural sentence the code of the last component QS.
Random: Assign the natural sentence the code of a randomly chosen component QS.
To be as sure as possible of the validity of our recoding exercise, the authors themselves carefully applied this method to the fifteen selected manifestos.
We first report the frequencies of the three types of relationship between natural sentences and quasi-sentences. We then aggregate text codings into widely used left–right policy scales, and compare our substantive conclusions about document content when we shift from QS unitization to natural sentences (also comparing the first, last and random rules for assigning a QS code to the natural sentence level).
Human coders faced with a natural sentence that they feel contains more than one policy statement may face a tough choice in deciding which code to assign it, if only one code may be assigned.Footnote 19 In such a case, it is possible that coding reliability – a separate issue from unitization reliability – may become an issue. As a preliminary test of this possible problem, we report the results of a coding experiment conducted using an expanded version of the CMP scheme applied to European election manifestos, designed to test whether coding unreliability increased when coders were asked to use natural rather than quasi-sentences as the basis for the CMP scheme.
Results From Recoding Manifestos Into Natural Sentences
Comparing Units of Analysis
Our revisiting and recording of the text units from the fifteen manifestos provided a dataset of a total of 6,960 natural sentences, in which were contained 8,481 QSs. These are described in Table 1.
Pattern of Natural Sentences versus Quasi-Sentences from 15 Election Manifestos
The clearest result to emerge from our analysis is that the splitting by CMP coders of natural sentences into more than one QS occurs quite infrequently: in more than eight out of ten cases (84 per cent), natural sentences contained a single QS only, meaning that the endogenous unitization problem pertains to only 16 per cent of all text units. The remaining natural sentences include mostly two (12 per cent of all natural sentences) or three QSs (2.9 per cent of all natural sentences). Natural sentences with four or more QSs are very rare, making up just 1.1 per cent of our sample.
The second strong result from our analysis is that when natural sentences are split into component QSs, these components are not necessarily coded differently. In fact, in the category of natural sentences with two QSs, fewer than half (44.1 per cent) of the natural sentences have different component codes, rising to just over half (58.4 per cent) for natural sentences split into three QSs. More of those split into four or more QSs were different, although overall, as previously mentioned, these represent a tiny fraction of all of natural sentences. Considering all natural sentences, the share of cases with varying component codes is just 7.7 per cent. In other words, before any additional comparison, we expect results that are at the very least 92 per cent identical, because there is a 92 per cent similarity between the two unitizations. Coding all QSs in the same natural sentence into the same category is at odds with the rationale for using QSs in the first place, which is that a natural sentence may have more then one component of meaning. Indeed, it amounts to an arbitrary, and as we have seen unreliable, ‘double’ or ‘triple’ counting of text units in which one natural sentence contains several identically coded QSs.
Results reported so far refer to coding differences based on the 56-category CMP scheme.Footnote 20 Published applications of CMP data typically use scales that combine many coding categories. By a country mile, the most popular application of the CMP data is the left–right index ‘Rile’, a scale that combines twenty-six of the fifty-six CMP coding categories. It is quite possible that all QSs in the same natural sentence were coded in the same direction (left, right, or neither) in relation to the Rile scale. The fifth column of Table 1 shows that, among natural sentences with two component QSs, less than three out of ten (28.7 per cent) involve QSs coded in different directions on the Rile scale. Among natural sentences with three component QSs, this figure is similar (36.6 per cent). The total share of natural sentences with component QS codes that differ in terms of left–right orientation is only 5 per cent.
In a purely descriptive sense, our analysis comparing natural sentence to QS unitization has shown that, even prior to our comparison of substantive political content, we should expect similarities of 92 and 95 per cent between codings based on perfectly reliable exogenously defined text units, natural sentences, and those based on unreliable, labour-intensive endogenously defined QSs – in practice these units of analysis are exactly the same in twelve out of thirteen cases.
Comparing Aggregate Results
Individual sentence codings are of little substantive interest to end users of political content analysis datasets, and are not even reported. Instead, the CMP dataset contains only the percentages of text units coded into each policy category – the ‘per’ codes – as well as the total number of QSs recorded in the manifesto. Our aim in this section is, therefore, to compare aggregate category percentages from each manifesto when these are reconstructed from natural sentences and QSs.Footnote 21 This involved applying our three coding rules – choose the first QS code, the last, or one at random – to code the natural sentence. We see from Table 1 that this affects just about 8 per cent of all natural sentences.
Figure 2 shows the comparison of each policy category's percentage share, in a scatterplot matrix comparing our three rules for coding natural sentences with more than one QS, to the raw QS-based results used by the CMP. Each point represents a policy percentage from one manifesto, and the dashed line shows the 45° axis of perfect agreement. To reduce skew created by low-frequency policy categories, we logged both axes (this makes no substantial difference to the results). The squares above the diagonal report Pearson's correlation coefficient, ranging from 0.98 to 0.99 – there is an almost perfect linear relationship regardless of which rule is applied. To test the overall agreement in a more numerical framework, we used a simple regression analysis of the logged QS policy category percentages on the logged natural sentence policy category percentages (see Table 2). The results confirm inferences drawn from viewing scatterplots; 98 per cent of the variance in the original QS coding is explained by the natural sentence codings, regardless which rule is applied. An F-test of whether the estimated slope coefficient differs from the 1.0 value implying perfect identity cannot reject this null hypothesis. This is strong evidence that the natural sentence and QS codings yield the same aggregate results.
Comparing quasi-sentence aggregate category percentages to natural sentence recodings Notes: Three rules are compared: randomly assign the code based on constituent QSs; take the first QS code for the natural sentence; and take the last QS code for the natural sentences. Total manifestos analysed: 15.
Regression of (Log) Quasi-Sentence-Based % Categories by Manifesto on (Log) Natural Sentence-Based Estimates Using Three Rules
Notes: The constant was constrained to be zero. The F-test reported in the last line is a test of the null hypothesis that the slope coefficient is the identity value of 1.0.
As we noted above, the most commonly used product of the CMP dataset is the left–right ‘Rile’ index that includes twenty-six of the fifty-six CMP coding categories. Figure 3 plots Rile scores for our fifteen manifestos, using exogenous natural sentence and endogenous QS unitization, and shows a very high degree of agreement between the two. Because Figure 3 only has one aggregate data point representing each manifesto for which we recorded the text units, we re-sampled natural sentences drawing 100 samples of 100 natural sentences from each manifesto. Figure 4 reports results, plotting a total of 15 × 100 = 1,500 points representing the fifteen manifestos in our sample. The top panel of Figure 4 shows the CMP's additive, original Rile scale, while the bottom panel depicts the recently proposed aggregate logit Rile scale, a scale that has demonstrably better properties than the CMP's relative difference scale.Footnote 22 In both cases, almost perfect correspondence is observed, even given the variation to be expected in each case from the sampling procedure.
Actual Rile Values aggregated for each manifesto Note: Based on random assignment, which we have chosen because human coders could almost certainly do better than this rule.
Re-sampled Rile values Notes: Based on random assignment, we took 100 random draws of 100 natural sentences each from each manifesto, and plotted the overall distribution of scores. The bottom plot uses the log Rile introduced by Lowe, Benoit, Mikhaylov and Laver, ‘Scaling Policy Preferences from Coded Political Texts’.
Results From A Coding Reliability Experiment
The results we report above imply that using natural sentences rather than quasi-sentences as units of analysis does not affect the validity of the classification of these units following deterministic unitization. Indeed, we demonstrated that endogenous unitization so rarely results in multiple and differently coded QSs within one natural sentence unit that even random allocation of codes to the larger natural sentence units resulted in essentially the same aggregate results – suggesting that the reliability of coding has little potential to be adversely affected by the switch to natural sentence units. Our test used an exogenously specified procedure to code natural sentences, when these were split into differently coded QSs. A computer applying formal rules to do this will suffer no qualms of indecisiveness and display no favouritism towards particular policy categories or domains. It is possible, however, that a human expert coder faced with a natural sentence containing what are seen as two distinct policy statements will not use a consistent rule in coding the natural sentence text unit. While eliminating the unreliability of subjective unitization by using natural sentences, it could be that we are at the same time increasing the unreliability of coding by forcing human experts to make a Sophie's Choice when coding natural sentences that could, and perhaps should, be considered to express more than one distinct policy statement.
Our validity tests did not involve any human recoding, since the objective was to identify a lower bound of reliability by comparing three random allocation rules to code those 16 per cent of natural sentences that were split into multiple QSs. Given the extremely high convergence of results caused by all three rules we applied, we expect that human coding could only improve reliability over the computer implementation of our random rules. As an independent test where human coding of all text units permits a broader test of reliability, we assess the reliability of natural sentence versus quasi-sentence coding using results of a series of experiments applying a CMP-based coding scheme to party manifestos for the European Elections conducted by Braun, Mikhaylov and Schmitt.Footnote 23 In this experimental design, expert text coders were randomly assigned to two groups. In a setup following Mikhaylov, Laver and Benoit,Footnote 24 both groups had to code the same excerpt of the 1999 British Liberal Democratic Party Euromanifesto using an online coding platform.Footnote 25 The first group (twenty-three participants) was asked first to unitize the document into QSs, and then to code these QSs using the coding scheme. The second group (twenty-nine participants) was asked simply to treat natural sentences as text units and code these using the coding scheme. Coders were undergraduate students from the University of Mannheim, followed coding instructions as outlined by Budge et al.,Footnote 26 and used a version of the CMP-based coding scheme modified to address issues specific to European Parliament elections.Footnote 27
In order to assess the reliability and quality of the coding process and consequently of the data generation process, Braun, Mikhaylov and Schmitt calculate inter-coder agreement in each experimental group, with the aim of comparing the coding reliability of the two groups (which are defined by natural sentence versus quasi-sentence unitization).Footnote 28 Agreement was measured using Fleiss's kappa (κ).Footnote 29 The κ coefficient is by far the most widely used method of statistical analysis of agreement for categorical variables,Footnote 30 and generalizes directly to multiple coders rating the multiple items. The coefficient has a range from 0 (perfect disagreement) to 1 (perfect agreement), and takes into account the fact that some agreement may occur purely by chance.Footnote 31 While there are no universally accepted guidelines, it has been suggested that κ > 0.75 represents excellent agreement, that 0.40 < κ < 0.75 represents fair to good agreement, and that any κ < 0.40 indicates poor agreement.Footnote 32 Similar to our own results above, the QSs identified in the first stage of the Mannheim experiment were predominantly full natural sentences; furthermore, when one natural sentence consisted of more than one QS, these were typically coded into the same category. Just as with our own results, therefore, we would not expect much systematic difference in coding results between the two groups. The results relevant for our purpose are shown in Table 3.
Inter-coder Reliability Results from the Experiment Comparing Natural and Quasi-Sentence Unitizations for the Euromanifesto Coding Scheme
Notes: Bootstrapped bias-corrected 95% confidence intervals from 1,000 replications. Bias correction gives better coverage probability for a possibly biased statistic, and produces the same results as the percentile method for an unbiased statistic. See Braun, Mikhaylov and Schmitt, ‘Computer-Assisted Human Coding: Experimental Tests of Reliability’; and Anthony C. Davison and David V. Hinkley, Bootstrap Methods and Their Application (Cambridge: Cambridge University Press, 1997).
The key result for our purposes is that the inter-coder reliabilities of the groups coding natural versus quasi-sentences is substantively indistinguishable. Coding reliability is quite poor in this experiment overall, possibly reflecting that the main objective of Braun, Mikhaylov and Schmitt was to test the reliability of a new coding scheme for the manifestos component of the European Election Study. Overall, however, the reliability coefficients reported in Table 3 are in line with the 0.3–0.4 from tests by coders applying the standard CMP coding scheme to pre-unitized QSs reported by Mikhaylov, Laver and Benoit.Footnote 33 In sum, there is no evidence in these results that coding natural sentences rather than QSs affects the reliability of coding. The huge gain in reliability from moving to an exogenous definition of text units does not appear to come at the expense of coding reliability.
Conclusions
It is fundamental to the systematic analysis of political text that we specify the basic unit of text analysis. The ‘exogenous’ specification of the natural sentence as the text unit results in perfectly reliable unitization, but potentially pays a price in coding validity if natural sentences contain more than one message. There is also a potential price in coding reliability if human coders are asked to pick one from several messages in a single natural sentence. Addressing these issues, we draw four primary conclusions.
First, in only a small minority of cases in the manifestos we examined are natural sentences divided into separate QSs. This means that in effect there is little possible difference between a scheme requiring humans to make painstaking and unreliable decisions on parsing natural sentences into smaller units, because most QSs are also natural sentence units.
Secondly, even when the QS unitization rules call for dividing a natural sentence into multiple text units, more than half of these subdivided natural sentences contained sub-units that all have the same code. This means that no information about alternative policy emphases is lost for these units by considering only natural sentences. It also undermines the CMP rationale for using QSs, which is that one natural sentence may contain more than one distinct message, while it in effect results in double or triple counting of a subset of text units.
Thirdly, in our comparisons of the policy categories aggregated into percentages, including different left–right scales, we found no substantive differences between aggregations based on natural sentence versus quasi-sentence unitization. Our random procedure to assign a split natural sentence one of its constituent QS codes reproduced about 98 per cent of the variance in the aggregate measures based on QSs, including when subsamples were drawn to simulate the additional sampling variance that might come from having shorter manifestos. Because we think that human coders could improve on the random rules using expert judgement, furthermore, we expect our results to represent a worst-case scenario.
Finally, reporting the results from coder experiments where participants were asked to code either natural sentences or QSs, we found no evidence that inter-coder reliability differed between these two groups. The possible information loss from increasing the size of the text-coding unit from QSs to natural sentences does not on our evidence introduce additional unreliability.
The implication of these results for applying categorical coding schemes to political text is clear and simple. Natural sentences can be substituted for QSs to achieve a major gain in the reliability of text unitization without loss of validity. This implies that future text coding projects should dispense with endogenous text unitization by human experts as part of the coding process, and move to fully automated unitization based on natural sentence delimiters defined exogenously as part of the research design. Since our estimates suggest that substantive findings are unlikely to be affected by doing this, but reliability is likely to increase, the shift to natural sentence unitization could usefully be extended to the ongoing CMP and PA projects. Our analysis here implies that a substantial gain in reliability, efficiency and replicability can be achieved without sacrificing important substantive information in the texts under investigation.