In this research note we present a new method of extracting party position data from election manifestos and other types of political text. The core of our procedure is a hand-coded relational, kernel sentence-based method of content analysis as originally developed by Kleinnijenhuis and colleagues for the analysis of media content (e.g., Kleinnijenhuis and Pennings Reference Kleinnijenhuis and Pennings2001). Based on the adaptations made by Kriesi et al. (Reference Kriesi, Grande, Lachat, Dolezal, Bornschier and Frey2008), we have further developed this approach, which now has three core features: (1) a unitization process based on syntax leading to semantically complete units, which are typically smaller than natural sentences; (2) a relational structure that uses a “predicate” to link a “subject” (an actor) with an “object” (either another actor or a policy issue); and (3) a fine-grained hierarchical issue coding scheme that allows for flexibility in analyzing the data.
Our approach is a new attempt to meet the classic challenges of extracting party positions from political text. The first challenge is unitizing where the goal is to arrive at homogeneous and unambiguous units. Here a controversy exists concerning the usage of “quasi-sentences” (as traditionally used by the Comparative Manifestos Project (CMP), e.g., Klingemann et al. Reference Klingemann, Volkens, Bara, Budge and McDonald2006) and natural sentences (Däubler et al. Reference Däubler, Benoit, Mikhaylov and Laver2012). Quasi-sentences divide natural sentences primarily based on content criteria. Our method splits natural sentences strictly according to grammatical rules. The resulting units are more reliable than quasi-sentences and typically shorter (and hence easier to code) than natural sentences. The second challenge is producing a valid coding scheme that contains enough detail to capture the relevant policy positions without becoming too fine-grained for coders to make reliable distinctions between the categories. Our scheme takes benefit from a three-level hierarchical organization that allows for extracting the full content of manifestos. It permits cost-efficient adaptations to new developments and produces highly reliable results. In addition, it goes far beyond extracting policy saliencies and party positions allowing, in particular, for the analysis of inter-actor relations.
We have developed the method of coding manifestos presented here in the context of the Austrian National Election Study (AUTNES). The method is designed to meet the needs of an “integrated” election study (i.e., one paying equal attention to parties, voters, and the mass media). This requires that information extracted from manifestos can be precisely linked to information from other party or media textual sources and to mass and elite surveys. Our method provides such “connectivity” and suggests answers to the problems of existing manifesto research the vibrant scholarly debate has identified. As with most methods, there is nothing country specific to its basic principles. Rather, it can be used for extracting information from manifestos in all languages that share the basic syntax with the German language (all Indo-Germanic languages and beyond). Our coding scheme is designed to cover party issue competition in contemporary Europe. Although the specifics of issue coding may need adaptation over space and time, our method is designed to allow for such flexibility. Nevertheless, as our empirical study relates to Austria, all examples and applications relate to this case and the most recent elections (already coded) in particular.
In the next section, we describe the unitizing and coding procedures. Here we also present reliability measures. In the subsequent section, we discuss the face validity of our results and engage in cross-validation. Specifically, we compare our results for the 2002, 2006, and 2008 Austrian national elections with those yielded by established measurement strategies. Here we also show how to link our new manifesto data with mass and elite (party candidate) surveys and thus put them into an electoral context. Throughout this report we refer to an online appendix where we present detailed information on several aspects mentioned in the main text.
The Method
Starting with unitization, we have developed a procedure that takes the texts’ syntactic structure into account but is not sensitive to writing style (short versus long sentences), nor is unitization based on the units’ thematic contents. Thus, our approach differs from employing a purely numeric criterion (i.e., a fixed number of words) as proposed by Laver and Garry (Reference Laver and Garry2000), but also from the CMP’s “quasi-sentences,” which result from dividing natural sentences into thematic units. Our system draws on Chomsky’s (Reference Chomsky1957, Reference Chomsky1965) phrase structure model. According to Chomsky, texts can be unitized into the smallest possible but nevertheless grammatically complete sentences (“kernel sentences”). A natural sentence may consist of one or more kernel sentences. As sentences in manifestos often include various issues, it is necessary to unitize them into these smaller units to extract the full information. For example, in 2008 the Austrian People’s Party (ÖVP) proposed, “We will finance the care fund with privatization revenues”. The ÖVP thus positioned itself positively toward the introduction of a care fund as well as to privatization in general. Based on the kernel sentence approach, we extract two units from this sentence (online Appendix 1).
Compared with the CMP, we extract on average more than twice as many observations (factor 2.16). However, the observation counts per manifesto correlate strongly (r=0.90), even though, as demonstrated in online Appendix 1, especially long natural sentences lead to multiple kernel sentences.
Däubler et al. (Reference Däubler, Benoit, Mikhaylov and Laver2012) expressed some concerns regarding the reliability of unitizing by human coders. Yet, our results (Table 1) display excellent overall agreement, with only one of 16 manifestos closely missing the conventional benchmark of 0.8 (Krippendorff’s α). This performance is a result of high-quality coder selection and training: unitizing and coding is carried out by outstanding BA and MA students, of which a total of 17 have been recruited over the course of five years. Equipped with an extensive codebook, they undergo an intense training process of several weeks, including group sessions as well as individual meetings and feedback by AUTNES researchers. Each manifesto is unitized twice, with pairs of coders rotating between manifestos. Between four and nine different coders are involved in the unitizing of each set of manifestos for the years 2002–2008. In order to produce a single set of units that represents the starting point for further coding, inter-coder disagreements are resolved authoritatively by AUTNES researchers.
Inter-Coder Reliability of the Unitizing Procedure
Note: figures are Krippendorff’s αs for the number of kernel sentences extracted from each natural sentence by rotating pairs of coders. Each cell entry represents agreement between two coders.
a Data for 2006 are based on random samples of 10 percent of the manifesto sentences.
b The BZÖ was founded in 2005.
c The LIF did not run in 2006.
Not only does the unitization procedure generate highly reliable coding units, it also allows us to capture considerably more policy detail than other approaches. Yet, the overall issue emphasis in a manifesto does not change if we compare our results with sentence-based saliencies (correlations are >0.97), indicating that the unitizing procedure does not bias the saliencies toward one issue area or the other (online Appendix 5).
In a second step, the units are transformed into statements following a subject–predicate–object structure, e.g., “ÖVP // for // privatization.” We differentiate three types of statements as follows: (1) an actors’ position toward an issue (actor–issue statements), (2) the relationship between actors (actor–actor statements), and (3) statements describing external circumstances such as economic developments (reality statements). In all three types the “subject” (usually the party) positions itself with respect to an “object” that can either be a policy issue or another actor. Subject and object are linked by a “predicate” indicating a positive (+1), neutral (0), or negative (−1) relationship.
Distinguishing these types of statements offers a number of advantages when it comes to data analysis: scholars interested in parties’ positions will use actor–issue statements. Here, the numerical value of the predicate is the basis for calculating policy positions. However, our scheme also provides comprehensive information about saliencies, as issues are recorded for all three types of statements. In addition, actor–actor statements are a rich source for the study of interactions of parties or politicians (e.g., negative campaigning).
In a third step, the issues (and actors, see online Appendix 3) recorded in the statements are categorized. Any such scheme is required to be comprehensive and capture specific as well as general topics; it should also be open to new issues. Our coding scheme covers >650 categories and is structured into three hierarchically nested levels as follows: 15 issue categories (level 1), 88 issue sub-categories (level 2), and over 650 issues (level 3). Coders are trained to allocate each statement on level 3. Table 2 shows the 15 issue categories on level 1; all others are displayed in online Appendix 2.
Issue Categories (Level 1)
The wording of the individual issues on level 3 is close to the common use of language, e.g., “unemployment” or “tuition fees” rather than “labor market” or “education.” Coders spotting a keyword in the statement can carry out a targeted search within the issue list. This often leads to the appropriate category. Our method relieves coders from assigning specific issues to broader and more abstract categories and thus enhances efficiency and data validity (online Appendix 3).
Coders are allowed to come up with new issues by adding them to the list when coding the manifestos.Footnote 1 Later, AUTNES researchers collectively decide on the new issues’ inclusion. As a consequence, the coding scheme is flexible enough to adequately represent future contents of party competition or, when applied to old documents, contents that had long before withered away. It is thus suitable for longitudinal analyses and for being applied to other types of text. To be sure, the benefit of flexibility certainly comes at a cost in terms of consistency over time. Yet, as the alternative would be to stick with a given scheme even when new and highly salient issues arise, this trade-off is worth making.
Taking problems of reliability of human coding seriously, we double code all statements by rotating pairs of coders. Coders are assigned to manifestos that they had not worked on in the unitizing stage. Table 3 presents the average reliability scores across parties per year. Reliability increases as we move from the very specific issues (level 3) to the broad policy areas on level 1. In other words, coders may disagree as to whether a statement should be coded, for instance, under “trade unions” or “collective bargaining,” but agree about the superordinate categories (“employees” on level 2, “economy” on level 1). Applying Krippendorff’s benchmark of 0.8, it can be concluded that our initial coding results (i.e., before checking and resolving inter-coder differences) are highly reliable already at the intermediate level of our issue scheme (for more on uncertainty, see online Appendix 4). In order to produce the final data set, the two sets of codes are merged into an authoritative version with AUTNES researchers making the final decision on cases where coders disagree.
Inter-Coder Reliability at Three Levels of the Issue Coding Scheme
Note: figures are Krippendorff’s αs for manifestos double coded by rotating pairs of coders. Each reported mean is the average of agreements for all pairs of coders across all manifestos per election. The levels refer to our system of categories. The actual coding is done at level 3 (see the main text).
This fine-grained coding scheme in combination with the innovative unitizing procedure allows us to pick up much more policy information—as well as information on actors—than other approaches. When averaging the results for the manifestos in all three elections, we find that 33.1 percent of the sentences contain issues from different level-3 categories, 24.8 percent from different level-2 categories, and 16.6 percent from different level-1 categories (online Appendix 5). Combining a very detailed coding scheme with our unitization method thus considerably increases the validity of our measurement.
Empirical Results: Cross-Validation and Comparisons
Although the aim of our manifesto analysis approach is much more ambitioned than simply reproducing the position estimates that could be generated with less time- and resource-consuming methods, cross-validation is still necessary to assess the plausibility of our results. We draw these external data from the four most common approaches to the measurement of party policy positions: the analysis of manifesto content by human coders and computer algorithms as well as expert and party elite surveys. We conduct comparisons on different levels of aggregation such as a general left–right scale, a two-dimensional configuration, and an example of a specific issue position that allows for easy comparison with other data sources.
Deriving policy positions from the coding of a myriad of individual statements requires some method of aggregation. For the construction of a general left–right scale, we used the literature on left versus right in both “old” and “new politics” (e.g., Fuchs and Klingemann Reference Fuchs and Klingemann1990; Corbetta, Cavazza and Roccato Reference Corbetta, Cavazza and Roccato2009). Our aim is to accommodate a broad range of policies; on average, 55 percent of all statements are included in the calculation of the left–right score (online Appendix 6). The CMP’s RILE scale covers 50 percent in the same sample of manifestos. Nevertheless, we abstain from grouping issue areas such as “Europe” into this ideological dimension. Online Appendix 2 informs on all issues included.
Once the decision of which issues to use for the left–right scale has been settled, means are calculated for the predicates’ values across all relevant actor–issue statements. The result is a value between −1 and 1, which represent the left and right extreme ends of the scale, respectively (for the reliability of this scale, see online Appendix 7). This mode of aggregation, mathematically equivalent to (R−L)/(R+L), differs from the standard CMP procedure that applies the formula (R−L)/N. We thus follow the suggestions made by Kim and Fording (Reference Kim and Fording1998) because their formula yields results that are independent of the manifesto’s share of units that are substantively irrelevant for this scale (online Appendix 8).
Figure 1 depicts the party placements along the general left–right scale from a variety of sources: the CMP,Footnote 2 the Chapel Hill Expert Survey (CHES), the AUTNES candidate survey, and the two automated text analysis algorithms Wordscores and Wordfish.Footnote 3
General left–right scale Source: Austrian National Election Study (AUTNES) (Müller et al. Reference Müller, Jenny, Steininger, Dolezal, Philipp and Preisl-Westphal2012), CMP (manifesto-project.wzb.eu), Chapel Hill Expert Survey (CHES) (Hooghe et al. Reference Hooghe, Bakker, Brigevich, de Vries, Edwards, Marks, Rovny, Steenbergen and Vachudova2010), Candidates/MPs (Müller et al. Reference Müller, Eder and Jenny2001; Müller, Eder and Jenny Reference Müller, Dolezal, Ennser-Jedenastik and Winkler2012). Party abbreviations: B=BZÖ; F=FPÖ; G=Greens; L=LIF; S=SPÖ; V=ÖVP. See Table 1 for the parties’ full names.
Footnote The AUTNES manifesto data show a relatively stable party landscape with an ordinal ranking of Greens–SPÖ–ÖVP–BZÖ–FPÖ from left to right—a result that appears quite plausible, not only for those acquainted with Austrian politics but also when compared with other sources of party positional data.
In comparison with the CMP, AUTNES party placements seem superior. To anyone familiar with Austrian politics, it appears implausible that the right-wing populist FPÖ is reported as the most left-wing party in 2002 (as the CMP data suggest). Similarly, the CMP data awkwardly position the Greens and the BZÖ (an FPÖ split-off) at almost the same location in 2008. There is only one party placement in the AUTNES data that may cause concern at first sight: the Liberals assuming the most rightist position in 2002. This is due to their manifesto being devoted mostly to economic issues, whereas remaining relatively silent on cultural ones. In contrast, the thematically more balanced 2008 manifesto moves the Liberals to the left, close to the SPÖ.
Comparing our results with those derived by other methods, we find overwhelming agreement for most parties on the left–right dimension in most elections (Table 4; online Appendix 10). The correlations are particularly high with data from expert surveys (CHES), the Wordscores approach, and the AUTNES candidate survey. Interestingly, the AUTNES manifesto data show the least degree of agreement with the Wordfish and the CMP scores, even though all three are derived from the same documents. However, the comparison across data sources suggests that the AUTNES manifesto estimates provide a more accurate representation of the party landscape than the CMP data that share the property of exclusive reliance on hand-coded manifestos.
Correlations Between Different Left–Right Measures
Source: see notes below Figure 1.
Note: the two-dimensional CMP scores are derived according to the scaling procedure suggested by Linhart and Shikano (Reference Linhart and Shikano2009).4 CHES variables “lreco” and “galtan” are used for expert survey data. Elite survey data are party averages on questions referring to socio-economic and cultural issues.
Although the left–right continuum may be a useful tool for voters and parties in their everyday communication, in many cases it clearly represents an over-simplification of the space in which parties compete. For many West European party systems, there is indeed good evidence of at least two dimensions that structure the political space: one pertaining to economic, the other to cultural questions (Kriesi et al. Reference Kriesi, Grande, Lachat, Dolezal, Bornschier and Frey2008; Dolezal et al. Reference Dolezal, Eder, Kritzinger and Zeglovits2013). The AUTNES economic scale correlates highly with expert and elite survey data and the Wordscores estimates, but only at 0.50 with the CMP scores. The AUTNES cultural scale displays very high agreement with most other sources.
Although there is thus some disagreement as to the “correct” party placements, we can still conclude that also the two-dimensional AUTNES estimates appear valid, albeit that there is greater consensus as to the placement of parties along the cultural dimension than there is agreement regarding the economic one. The lower correlations for the economic dimension may also be owing to the fact that the variation along all economic scales is consistently lower across data sources than on the cultural scales, whereas measurement error should be similar.
For all three scales, the AUTNES estimates outperform the CMP scores in terms of correlations with elite and expert surveys. Neither elite data, which were recommended especially by de Swaan (Reference de Swaan1973), nor expert survey data are perfect (see Budge Reference Budge2000; Mair Reference Mair2001). Yet, compared with manifestos both are less prone to be biased by cheap talk and the use of policy rhetoric as a strategic device. Finding the AUTNES manifesto estimates in high agreement with both kinds of survey data thus gives particular credibility to the estimates produced.
Spatial representations of policy positions are theoretically possible at almost any level of abstraction. Typically, however, party placements are presented as aggregate measurements, comprising observations from a large number of lower-level categories. Although such practice may serve some research purposes well, it constitutes a very crude approach in other circumstances. Assume, for instance, that a researcher wants to compare party positions on very specific issues to corresponding question items in a voter or elite survey. The fine-grained nature of the AUTNES coding procedures easily allows for the construction of policy scales also at such a low level of abstraction.
To provide an example, Figure 2 presents data from several AUTNES sources on attitudes toward criminal punishment, where the manifesto estimates refer to all statements about issues such as prison sentences, preventive custody, or extrajudicial settlements (online Appendix 2). The voter and candidate surveys reflect agreement or disagreement with the statement that criminal offenders should face harsher punishment. The manifesto data display a divide between the major parties (SPÖ, ÖVP) taking a moderate to soft position and the populist right (FPÖ, BZÖ) advocating tough measures against criminal offenders. The voters, by contrast, are largely united in holding rather harsh attitudes.
Criminal punishment (soft versus harsh), Austrian National Election Study (AUTNES) data Source: voters 2009 (AUTNES 2009), all others see Figure 1. Note: the Greens made no statement about criminal punishment in their 2008 manifesto. Party abbreviations: B=BZÖ; F=FPÖ; G=Greens; L=LIF; S=SPÖ; V=ÖVP. See Table 1 for the parties’ full names.
It is obvious that party and voter placements are not identical across sources. There are real differences between, for instance, the masses and elites. Yet, with the exceptions noted above, the ordinal rankings of the parties are comparable. Moreover, one needs to take into account that the uncertainty associated with position estimates rises as the number of issue-related statements in a manifesto decreases (Benoit, Laver and Mikhaylov Reference Benoit, Laver and Mikhaylov2009). The trade-off between higher specificity and lower uncertainty must, of course, always be acknowledged. The advantage of the AUTNES scheme is to allow researchers to choose virtually any point along this continuum.
Conclusion
In this research note, we have presented our new approach to the analysis of parties’ manifestos in their electoral context. In so doing, we have aimed at realizing two goals: first, our aim was to conceptualize manifesto analysis in a way that advances the empirical study of elections by locating party positions at almost any level of specificity. Second, our task was to offer resolutions to the problems in extant manifesto research that the contemporary debate on political text analysis has identified.
Our cross-validations of party placements derived from AUTNES manifesto data with the main alternatives have revealed enough commonalities to claim the overall validity of our data. The remaining differences can partly be attributed to differences in the real world phenomena studied and thus are not particularly relevant here. With regard to variation among the different manifesto-based party placements, our results appear superior if checked for plausibility before the background of qualitative accounts of party competition.
Of course, our content analysis method requires considerable resources compared with other methods. Therefore, skeptics may question whether gains in precision justify the effort. We think it does. On the one hand, better measured variables should lead to better results when included in statistical analyses. On the other, our approach provides greatly enhanced opportunities to investigate novel research questions on many core topics of political science including elections, parties, coalitions, and government policies. This flexibility in analysis should provide multiple and long-term returns on investment and thus justify the resource input.
Supplementary Material
To view supplementary material for this article, please visit http://dx.doi.org/10.1017/psrm.2015.38
