Does the choice of democracy measure matter? Earlier research has shown that different democracy measures are highly correlated with each other (see, for example, Arat Reference Arat1991; Bollen Reference Bollen1980; Coppedge and Reinicke Reference Coppedge and Reinicke1990). In some studies high correlations between measures of democracies have also been used as evidence of validity (for example, Bollen Reference Bollen1980). If indices are highly correlated it might be easy to assume that they will generate similar results when the indices are used in empirical research. However, Pemstein et al. (Reference Pemstein, Meserve and Melton2010) found that there are discrepancies between measures of democracy and their ratings of different countries’ levels of democracy, even if the correlations between the measures are high. They stated that research that uses only one measure of democracy will have potentially serious problems with the robustness of its results. In this article I will investigate if different measures of democracy generate the same empirical results. I will apply two of the most widely used measurements of democracy, Freedom House and Polity IV indices, as dependent variables and apply the same set of independent variables and see if the empirical results are similar in statistical significance and explanatory power. In total four different independent variables will be applied, and these variables have all been used in previous research to explain variation in the level of democracy.
Measures of Democracy: An Overview
There are many ways to define democracy, and there are also many ways to measure democracy (political regimes) (for different definitions of democracy, see Dahl Reference Dahl1971; Przeworski et al. Reference Przeworski, Alvarez, Cheibub and Limongi2000; Sartori Reference Sartori1987; Saward Reference Saward1994; Schumpeter Reference Schumpeter1992; Vanhanen Reference Vanhanen1997). A widely discussed question in the political science field is how democracy should be measured, and many measures (indices) of democracy have been constructed over the years.Footnote 1 In the late 1950s Seymour Martin Lipset's classic and often-cited article ‘Some Social Requisites of Democracy: Economic Development and Political Legitimacy’ (1959) was published, in which Lipset classified a total of 48 countries from the two different groups of countries: European and English-speaking countries, and Latin American countries by their degree of stable democracy. He classified the European and English-speaking countries into two categories: stable democracies, and unstable democracies and dictatorships. The Latin American countries were also classified into two categories: democracies and unstable dictatorships, and stable dictatorships. In the early 1960s, Phillips Cutright (Reference Cutright1963) developed a democracy measure that he named the Index of Political Development. Cutright (Reference Cutright1963: 254) was critical of the fact that Lipset had not scaled the indicator of political regime, and so he developed his index on a continuous scale. The index was based on the legislative and executive branches of government, covering 77 countries (African countries were excluded) and a time period of 21 years (1940–60). In the following decade, Robert Jackman (Reference Jackman1973) developed a measure of democracy that covered 60 countries in the year 1960 (Communist countries were excluded). Jackman included four indicators in his measure of democracy: level of voter turnout, competitiveness of the party system, electoral irregularity and freedom of the press. Six years later, Kenneth Bollen (Reference Bollen1979, Reference Bollen1980) presented a measurement for political democracy (POLDEM) which was based on two dimensions: political liberties and popular sovereignty. In his index of political democracy, Bollen included three indicators of political liberties (press freedom, freedom of group opposition and government sanctions) and three indicators of popular sovereignty (fairness of elections, executive selection and legislative selection). Bollen's index originally covered over 100 countries for the two years 1960 and 1965, which he later enlarged to cover more years. More recently, Tatu Vanhanen (see, for example, Vanhanen Reference Vanhanen1990) constructed a democracy index that he named the Index of Democratization (ID), and used this index in several studies (Vanhanen Reference Vanhanen1990, Reference Vanhanen1997, Reference Vanhanen2000, Reference Vanhanen2003). Vanhanen's aim was to provide a theoretical explanation for democratization and to test that explanation through empirical evidence. In Vanhanen's opinion (1990: 12) it was better to use quantitative indicators instead of using subjective evaluations when measuring democracy. He included two dimensions in the index – competition and participation – and he covered different numbers of countries in his index in different studies; for example, he included 170 countries (of population 200,000 or higher) in one study (Vanhanen Reference Vanhanen2003) and used data for 187 countries in another (Vanhanen Reference Vanhanen2000).
In the 1990s several measures of democracy were developed. Michael Coppedge and Wolfgang Reinicke (1990) constructed a measure based on Dahl's concept (1971) of polyarchy, and named it the Polyarchy Scale. Coppedge and Reinicke included five indicators in the measure: freedom of expression, freedom of organization, media pluralism, the extent of the suffrage and the holding of fair elections. Zehra Arat's measure of democracy (1991) consists of four dimensions of popular sovereignty – participation, inclusiveness, competitiveness and civil liberties – and covers a 35-year period, with the numbers of countries included in the index varying between 29 and 109. Axel Hadenius (Reference Hadenius1992) constructed an index of democracy that originally covered 132 less-developed countries in 1988. The index consisted of two dimensions: elections and political liberties. The Alvarez, Cheibub, Limongi and Przeworski (ACLP) index used a nominal (dichotomous) measurement of democracy where countries are classified as autocracies or democracies,Footnote 2 and the index has been used in different studies (Alvarez et al. Reference Alvarez, Cheibub, Limongi and Przeworski1996; Przeworski et al. Reference Przeworski, Alvarez, Cheibub and Limongi1996, Reference Przeworski, Alvarez, Cheibub and Limongi2000). The ACLP index that was used in Przeworski et al. (Reference Przeworski, Alvarez, Cheibub and Limongi2000) covers the time period between 1950 and 1990, and in total 141 countries are included; the index has now been expanded by Cheibub et al. (Reference Cheibub, Gandhi and Vreeland2010) to include data from 1800 to 2008. The ACLP index has its focus on contestation, and countries are classified as democracies if: (1) the chief executive is elected; (2) the legislative is elected; (3) there is more than one political party; and (4) there is some alternation in power (an incumbent regime has lost power). The Economist Intelligence Unit's (EIU) Democracy Index is a newly developed index that to date has been calculated for three years (2006, 2008 and 2010 (EIU 2010)). The EIU index consists of five dimensions: electoral process and pluralism, civil liberties, the functioning of government, political participation and political culture. It is based on the ratings for a total of 60 indicators for the five dimensions and in 2008 covered 165 independent states and two territories.
Currently, the two most widely used measurements of democracy are Freedom House and Polity IV. The underlying concept for Freedom House is freedom and not democracy, but the Freedom House data (political rights, civil liberties and freedom status) are often used by researchers as measures of democracy. Freedom House has been rating the freedom status in each country of the world every year since 1972,Footnote 3 and the dataset that covered 2009 (Freedom House 2010a) consisted of a total of 194 countries and 14 select territories. The Freedom House index includes two main categories: political rights and civil liberties. Political rights are based on three subcategories – electoral process, political pluralism and participation, and functioning of government – and these subcategories consist of a total of 10 indicators (checklist questions: Freedom House 2010b). Civil liberties are based on four subcategories – freedom of expression and belief, associational and organizational rights, rule of law, and personal autonomy and individual rights – which consist of a total of 15 indicators (checklist questions). The subcategories are measured on an ordinal five-point scale (0–4; where 0 represents the lowest degree of freedom and 4 represents the highest degree of freedom).Footnote 4 The scores for the subcategories are then separately added for the two main categories, political rights and civil liberties. The highest score for political rights is 40 (10 × 4), and for civil liberties the highest score is 60 (15×4). Freedom House uses a scale from 1 to 7 when measuring political rights and civil liberties, where 1 indicates the highest degree of freedom and 7 indicates the lowest degree of freedom. For example, a total score between 36 and 40 in political rights gives the value 1 on the rating score for political rights, and a total score between 53 and 60 in civil liberties gives the value 1 on the rating score for civil liberties, which thus represents the highest degree of political rights and civil liberties. The mean average of political rights and civil liberties, determines the freedom status. The freedom status is divided into three different categories: free, partly free and not free. Those countries that have the combined mean average between 1.0 and 2.5 on political rights and civil liberties are rated as free, countries with a mean average of 3.0 to 5.0 are rated as partly free and countries with a mean average of between 5.5 and 7.0 are rated as not free. The two measurements, political rights and civil liberties, can be used separately, but often they are used together. The Freedom House index can also be used as a scale of 2–14 by adding the scores for political rights and civil liberties. The Freedom House freedom status variable that consists of the three categories, free, partly free and not free, is also an alternative that can be used as a measure of democracy.
Next, let us turn to Polity IV (2011), which covers all independent countries with a total population of 500,000 or more and provides data since 1800. The index that covers 2009 includes 163 countries. Polity IV provides two aggregate indices, democracy (DEMOC) and autocracy (AUTOC). Both the democracy index and the autocracy index are measured on an ordinal scale and the index consists of five expert coded categorical indicators. The Polity IV democracy index and autocracy index include the following dimensions: competitiveness of executive recruitment, openness of executive recruitment, constraints on the chief executive, regulation of participation and competitiveness of participation. In the Polity IV indices the five dimensions are weighted differently.Footnote 5 The two indices are normally used together by subtracting autocracy (AUTOC) from democracy (DEMOC), which provides the Polity variable, which ranges from −10 (high autocracy) to +10 (high democracy). Accordingly, the Polity variable consists of a 21-point scale that does not divide countries into subgroups such as democracies and non-democracies.Footnote 6 The Polity dataset also offers a modified version of the Polity variable that is named the Polity 2 variable. The Polity 2 variable has converted ‘standardized authority scores’ to conventional policy scores and therefore the Polity 2 variable can be used in time-series analyses (see Polity IV 2009).
Methodological Considerations
Selection of Measures of Democracy
As the overview above shows, there are a variety of ways to measure democracy, and the questions about measurements are many. One relevant question is whether democracy is a dichotomous or a continuous (graded) phenomenon. Researchers such as Huntington (Reference Huntington1991), Alvarez et al. (Reference Alvarez, Cheibub, Limongi and Przeworski1996), Przeworski et al. (Reference Przeworski, Alvarez, Cheibub and Limongi1996, Reference Przeworski, Alvarez, Cheibub and Limongi2000) and Cheibub et al. (Reference Cheibub, Gandhi and Vreeland2010) see democracy as a dichotomous phenomenon (democracy/autocracy). To quote Przeworski et al. (Reference Przeworski, Alvarez, Cheibub and Limongi2000: 57), ‘democracy can be more or less advanced, one cannot be half-democratic: There is a natural zero point’. Cheibub et al. (Reference Cheibub, Gandhi and Vreeland2010) also reject the idea that there is some middle ground between democracy and dictatorship, but they agree that there are different types of democracies and dictatorships. Huntington (Reference Huntington1991: 11) stated that using a continuous measure of democracy poses many problems, for example, the weighting of indicators. On the other hand, researchers such as Arat (Reference Arat1991), Bollen (Reference Bollen1990), Hadenius (Reference Hadenius1992) and Vanhanen (Reference Vanhanen1990) see democracy as a continuum. Some of them, such as Hadenius and Teorell (Reference Hadenius and Teorell2005) and Elkins (Reference Elkins2000), argue that information will be lost if a dichotomous measure of democracy is used, and Bollen (Reference Bollen1990: 13) stated that countries that are near a cutoff point are difficult to place in a dichotomous measure. Collier and Adcock (Reference Collier and Adcock1999) discuss justifications for the use of a dichotomous or graded approach of democracy in comparative research, and they conclude that ‘decisions about gradations versus dichotomies are often built into the framing of research questions’ (1999: 561). However, Collier and Adcock are doubtful about the justification given by Przeworski and collaborators for why they see democracy as a dichotomous phenomenon, and they write that ‘in our view it remains unclear why a regime that has competitive elections for the presidency, rotation in the presidential office, and more than one party – but lacks competitive elections for legislative office – is not at least partially democratic’ (Collier and Adcock Reference Collier and Adcock1999: 549). It should be added that some indices use a trichotomous measure of democracy (Freedom House, freedom status variable) and a four-grade measure of democracy (EIU).
I find it relevant to view democracy as both a dichotomous and a continuous phenomenon. Sometimes, in the selection of cases or when dealing with transitions from autocracy to democracy and the reverse, it is necessary to know which countries are democracies and which are not. However, when a dichotomous measure is used, it is not possible to measure the variation in democracy (per se). At other times it is relevant to see democracy as a continuous phenomenon; for example, when the level of democracy is used as a dependent variable. Therefore, a strength of measures of democracy such as Freedom House and EIU is that they offer both a polychtomous measurement and a graded scale of democracy.
In this article, democracy is seen as a continuum, and so the ACLP index of democracy is not included as a dependent variable. Many of the other indices mentioned earlier have only been calculated for one or a few years and several of them are no longer updated. Therefore, I have decided to concentrate the empirical test in this article on the two measurements of democracy that are currently most widely used: Freedom House and Polity IV. The comparisons made in this study will be based on these two measures of democracy.
Time Period Covered
Another important question is which time period(s) the article should cover, and a decision has to be taken whether a cross-sectional or a longitudinal approach will be employed. Until then, it is relevant to examine if the variation in democracy is mainly across space or across time. If the variation is mostly across space, a cross-sectional approach can be justified, and if most of the variation in democracy is across time a longitudinal approach can be justified. To examine how much of the variation in democracy is across space and across time I run two different Anova tests with measures of association, each for Freedom House and Polity IV (1972–2009). The first test shows the relationship between space (countries) and democracy and the other test show the relationship between time (years) and democracy.
As shown in Table 1, the Eta squared for democracy (Freedom House) and space indicates that almost 80 per cent of the variation in democracy is between countries. The test for democracy and time shows that 3 per cent of the variation in democracy is between years. The tests with Polity IV show that 69 per cent of the variation in democracy is between countries and 10 per cent of the variation is between years. Thus, the results of the four tests show that most of the variation in democracy (measured with Freedom House and Polity IV) is across space and not over time. Consequently, I will apply a cross-sectional approach in this study. I also consider it relevant to examine a time period close to the current time period, and therefore I will use 2009 as the year to measure the dependent variable. The independent variables are lagged by one year and are therefore measured for 2008.
ANOVA Test with Eta Freedom House and Polity IV, 1972–2009
Selection of Cases
Freedom House and Polity IV have not got exactly the same selection of countries in their surveys. In the index from Freedom House (2010a), which covers 2009, 194 countries are included. The Polity IV index that covers 2009 consists of 163 countries. In some of the empirical comparisons it is important that the two indices cover the same countries and therefore it is necessary to make a selection of countries so that the same countries are used for both indices. Polity IV has a lower number of countries in its index and I will start with the Polity IV selection.
As regards which countries are included in the two indices, the main difference is that Polity IV excludes microstates (countries with fewer than 500,000 people) in its research. However, Polity IV codes countries with transition, interregnum and interruption differently from its −10 to +10 scale. The codes are named standardized authority codes (−66, −77, −88); countries with foreign interruption are coded as −66 and are treated as system missing; countries of interregnum or anarchy are coded as −77 but they are converted to a neutral score of 0; countries of ongoing transition are coded as −88 and converted to system missing. Because of that, Polity IV has no values on the +10 to −10 scale for 2009 for Afghanistan, Bosnia-Herzegovina, Iraq (all coded −66, foreign interruption), Côte d'Ivoire (coded as −88, ongoing transitions) or Somalia (coded as −77, country of interregnum or anarchy). In this study I follow Treier and Jackman (Reference Treier and Jackman2008: 203) and exclude countries which Polity IV has coded −66, −77 and −88, and accordingly Afghanistan Bosnia-Herzegovina, Iraq, Côte d'Ivoire and Somalia will be excluded from the following research section. Thus, the Polity IV index that will be used in this study consists of 158 countries; to enable a fair comparison of the indices I will include Freedom House index in this study with the same 158 countries. However, I will also use the Freedom House full set of countries (194) in the analysis of the study. I will use the Freedom House index by adding the scores of political rights to the scores of civil liberties. In the Freedom House index, low values indicate high levels of democracy, but in this study I will invert the scale so that high values indicate high levels of democracy, and the scale ranges between 2 and 14. The Polity IV measure (Polity 2 variable) ranges from −10 to +10, where high values indicate high levels of democracy. The correlation (Pearson's r) between the two indices for 2009 is 0.875.
Statistical Method
It is necessary to discuss which statistical technique is best to use when Freedom House and Polity IV are used as dependent variables. Often OLS regressions are applied when Freedom House and Polity IV are used in this way. However, when using OLS regression the dependent variable should be an interval variable, and, strictly speaking, Freedom House and Polity IV indices are ordinal variables; so, by using an ordinal variable as dependent variable in an OLS regression the assumptions of OLS are violated. I will employ OLS regression in this article, though, and to ensure that the use of OLS does not seriously distort the findings it is necessary also to employ ordinal regressions. To control if an ordinal variable is suitable to be applied as a dependent variable in an OLS regression it is possible to compare the model with an ordinal regression, and to control if the multivariate models in this study fit with OLS regressions I employ ordinal regressions (PLUM, Polytomous Universal Model). In this way I run the same multivariate models with OLS and PLUM (ordinal regression) and compare the unstandardized coefficients (size, direction and statistical significance) in OLS with the estimated coefficients in the PLUM models. The results of the comparison show that the differences in the unstandardized coefficients in OLS and the estimates in PLUM are small as far as size, direction and statistical significance as concerned. Thus I see it as justified to use OLS regressions in the study,Footnote 7 not only due to the fact that OLS regressions are more commonly employed, but also because they are easier to interpret and understand.
Independent Variables
I will use four different independent variables, and have chosen independent variables where several previous studies claim to have established a relationship between these independent variables and the level of democracy.
Modernization and Democracy
The first two independent variables are drawn from the modernization theory – the relationship between socioeconomic development and democracy, which has been studied frequently over more than 50 years (see, for example, Arat Reference Arat1988; Barro Reference Barro1999; Cutright Reference Cutright1963; Epstein et al. Reference Epstein, Bates, Goldstone, Kristensen and O'Halloran2006; Hadenius Reference Hadenius1994; Jackman Reference Jackman1973; Lipset Reference Lipset1959; Neubauer Reference Neubauer1967; Przeworski et al. Reference Przeworski, Alvarez, Cheibub and Limongi2000). Lipset (Reference Lipset1959) found that economic development measured by different indicators such as wealth, industrialization, urbanization and education was much higher in stable democracies than in unstable democracies and dictatorships. To cite Lipset (Reference Lipset1959: 75): ‘the more well-to-do a nation, the greater the chances that it will sustain democracy’. After Lipset's study many others have confirmed that there is a positive relationship between democracy and economic development. Therefore, I will apply two independent variables that have often been used as independent variables in studies of the relationship between socioeconomic development and democracy. The variables are GDP per capita and the enrolment ratio in education as measured by UNESCO. It is expected that GDP per capita and the enrolment ratio in education will be positively associated with democracy. GDP per capita is measured in thousands of US dollars and UNESCO (Institute for Statistics) defines total enrolment ratio in education as ‘Total enrolment in a given level of education, regardless of age, expressed as a percentage of the official school-age population for the same level of education’.Footnote 8
Size and Democracy
The third and fourth independent variables are physical variables and are drawn from the literature that examines the relationship between size of countries and level of democracy. In the book Size and Democracy, Robert Dahl and Edward Tufte (Reference Dahl and Tufte1973: 1) raised the question, is ‘democracy’ related in any way to ‘size’? In general, Dahl and Tufte found no empirical evidence to support the idea that democracy was related to size, but, after Dahl and Tufte's study (Reference Dahl and Tufte1973), many subsequent studies have investigated the relationship between democracy and size and many researchers have found that small countries are more likely to be democracies than larger countries are (see C. Anckar Reference Anckar2008; D. Anckar Reference Anckar2004, Reference Anckar2010; Anckar and Anckar Reference Anckar and Anckar1995; Diamond Reference Diamond1999; Hadenius Reference Hadenius1992; Teorell Reference Teorell2010). Thus, I will apply two independent variables which have been used as independent variables in studies concerning the relationship between size and democracy – area and population size. The size of population is measured in millions of people, and countries’ territory is measured in square kilometres (km2). It is expected that smaller countries are associated with a higher level of democracy.
Research Strategy
As mentioned earlier, OLS regressions will be used in this study and I will start with a linear version of the independent variables. However, to reduce the skewness, I will also use a logarithmic version of three of the independent variables (GDP per capita, population and area). I will include three different variants of the dependent variable: (1) Freedom House with all 194 countries included; (2) Freedom House with 158 countries included; and (3) Polity IV with 158 countries included. I will start by running bivariate regressions with the three variants of the dependent variable and the four independent variables. For the independent variables that have been transformed I will run two different types of bivariate regression. In the first regression the independent variables will be untransformed, and in the second the independent variables will be logarithmically transformed. Finally, I will run a multivariate regression with each of the three variants of the dependent variable and with all four independent variables included in each model.
Result
Tables 2, 3 and 4 show the bivariate regressions.
Bivariate Regressions, Freedom House Index as Dependent Variable, 194 Countries
Note: In each row, the independent variable is listed first, followed by the explanatory power (explained variance), the unstandardized regression coefficient, the standard error, statistical significance and number of countries included in the regression.
Bivariate Regressions, Freedom House Index as Dependent Variable, 158 Countries
Note: In each row, the independent variable is listed first, followed by the explanatory power (explained variance), the unstandardized regression coefficient, the standard error, statistical significance and number of countries included in the regression.
Bivariate Regressions, Polity IV Index as Dependent Variable, 158 Countries
Note: In each row, the independent variable is listed first, followed by the explanatory power (explained variance), the unstandardized regression coefficient, the standard error, statistical significance and number of countries included in the regression.
GDP per Capita
The results from the bivariate regressions show that GDP per capita (untransformed) explains 12.5 per cent of the variation in the degree of democracy when Freedom House index (194 countries) is used as the dependent variable. When GDP per capita is transformed (logarithmic) it explains 27.2 per cent of the variation. When Freedom House with 158 countries is used as the dependent variable, GDP per capita (untransformed) explains 19.0 per cent, and the logarithmic version of GDP per capita explains 26.9 per cent of the variation in the dependent variable. All four regressions are highly statistically significant. When Polity IV (158 countries) is used as the dependent variable, GDP per capita (untransformed) explains 3.5 per cent of the variation in democracy; when GDP per capita is transformed, the Adj.R2 increases to 7.4 per cent. Both regressions with Polity IV are highly statistically significant. The findings show that GDP per capita has a much higher explanatory power as independent variable when Freedom House is used as the dependent variable than when Polity IV is used as the dependent variable. With the same 158 countries included, GDP per capita (transformed) explains 19.5 percentage units more of the variation in democracy when Freedom House is used as the dependent variable compared to when Polity IV is used as the dependent variable.
Enrolment Ratio in Education
The enrolment ratio in education has about the same explanatory power for democracy with Freedom House containing 194 countries and Freedom House containing 158 countries as the dependent variable. The explanatory power is 24.5 per cent and 25.4 per cent, respectively. When Polity IV is used as the dependent variable, education's explanatory power is 11.1 per cent. All three regressions are highly statistically significant. The results indicate that the enrolment ratio in education has 14.3 percentage units less explanatory power when Polity IV is used as dependent variable than when Freedom House is used as dependent variable (same 158 countries). Thus we find that the difference in explanatory power is also huge when enrolment ratio in education is used as an independent variable.
Area
Area (untransformed) as independent variable is statistically insignificant with all three variants of the dependent variable. When area is logarithmically transformed, the variable is statistically significant when Freedom House with 194 countries is used as the dependent variable and it explains 11.6 per cent of the variation in democracy; as expected, the coefficient is negative. However, the logarithmic version of area is statistically insignificant when both Freedom House (158 countries) and Polity IV (158 countries) are used as the dependent variable. Based on these estimations it can be concluded that it matters whether the countries with a population of fewer than 500,000 people are included or not if area is to be statistically related to democracy with Freedom House as the dependent variable.
Population Size
Concerning the independent variable population size, the regressions show that when population size is untransformed it is not statistically related to democracy for any of the three variants of the dependent variable. However, when population size is logarithmically transformed it is highly statistically related to democracy when Freedom House with 194 countries is used as the dependent variable, and population size explains 8.8 per cent of the variation in the dependent variable. The coefficient is, as expected, negative, indicating that countries with a smaller population are positively related to democracy. When Freedom House (158 countries) and Polity IV (158 countries) are used as the dependent variable the independent variable population size is highly insignificant, even when the variable is transformed. Again, we may note that estimations are sensitive to the set of countries included in the analysis; if population is statistically related to democracy with Freedom House as the dependent variable, the countries with populations of less than 500,000 must be included.
The next step is to run multivariate regressions with the three variants of the dependent variable and the four independent variables. The logarithmic versions of GDP per capita, area and population size are used in the multivariate models. Three different multivariate models are tested: Freedom House with 194 countries, Freedom House with 158 countries and Polity IV with 158 countries as the dependent variable. Table 5 shows the results.
Multivariate Regressions on Freedom House and Polity IV
Note: In each row, the unstandardized regression coefficient is listed first, followed by standard error in brackets. *** = p < 0.01 level; ** = p < 0.05 level.
In the first multivariate regression where Freedom House with 194 countries is used as the dependent variable three independent variables – GDP per capita, education and area – are statistically related to the dependent variable. The sign of the coefficients are also in the expected directions. The model explains 34.1 per cent of the variation in the dependent variable. In the second regression, where Freedom House with 158 countries is used as the dependent variable the variables GDP/capita and education achieve statistical significance. The model explains 28.2 per cent of the variation in the level of democracy. In the third and last model, where Polity IV (158 countries) is used as the dependent variable, only the variable education is statistically related to the dependent variable.Footnote 9 The model explains 10.4 per cent of the variation in the dependent variable. The results show that the multivariate regressions differ to a great extent in explanatory power and statistical significance. For example, with the same 158 countries included, the explanatory power is 17.8 percentage points higher when Freedom House is used as the dependent variable than when Polity IV is used as the dependent variable. Regarding statistical significance, when Freedom House with the full set of countries is used as the dependent variable, three independent variables (GDP per capita, education and area) are statistically related to democracy. When Freedom House with 158 countries is used as the dependent variable, two independent variables (GDP per capita and education) achieve statistical significance with the dependent variable. Finally, when Polity IV is used as the dependent variable, only one independent variable (education) is statistically related to democracy.
The results from the bivariate and multivariate regressions show that the level of statistical significance and explanatory power for the independent variables differ greatly depending on which democracy index is used as the dependent variable. The correlation between the two indices is high at 0.875, so the result can be seen as unexpected. Can these different results be because the Polity IV and Freedom House rate some countries very differently? I will control for that being the case, and therefore I will identify the countries that differ most in rating in the two indices. To do so I convert each index so that they vary between 0 and 100, where high values indicate high levels of democracy and low values indicate low levels of democracy. Figure 1 shows a scatterplot with Freedom House and Polity IV indices with 158 countries’ levels of democracy (converted to a scale 0 to 100).
Level of Democracy 2009, Freedom House and Polity IV
When comparing which countries differ most in the two indices’ rating, the results displayed in Figure 1 show that Congo Kinshasa is the country that differs most. In the scale of 0 to 100 the Polity IV rating for Congo Kinshasa is 75.00 where the Freedom House rating is 16.67, which means that the Polity IV rating for Congo Kinshasa is 58.33 units higher in democracy level than the Freedom House rating; this is a substantial difference. There are four other countries that Polity IV rates as more than 40.00 units higher in democracy level compared with Freedom House: Kosovo (48.33), Nicaragua (45.00), Russia (45.00) and Armenia (41.67). Overall, Polity IV in total rates 29 countries between 30.00 and 75.00 democracy units higher than Freedom House does. There are also countries that Freedom House rates higher than Polity IV. Three countries are rated more than 20.00 units higher by Freedom House compared with Polity IV: Kuwait (35.00), Qatar (25.00) and Morocco (21.67). (The eight countries mentioned above are marked in Figure 1.) When there is evidence that the two indices rate many countries so differently it is easy to understand why the indices generate such different empirical results. Thus, one conclusion is that the results from the study show that the choice of democracy measures matters in empirical research.
Conclusions
The correlation between the two democracy indices, Freedom House and Polity IV, in the 2009 rating is high at 0.875 (for 158 countries). When indices are so highly correlated it is easy to expect that they will generate similar results when the indices are used in empirical research. However, the findings of this article show that there are significant discrepancies in the results concerning statistical significance and the explanatory power of cross-national statistical analyses, according to which of the two most widely used measures of democracy, Freedom House or Polity IV, is used as the dependent variable. The four independent variables that are included in this study in general seem to be more statistically related and have a higher explanatory power with Freedom House as the dependent variable compared to Polity IV as the dependent variable. The results also indicate that Freedom House and Polity IV rate many countries differently concerning their level of democracy. The countries that differ most between the two indices are Congo Kinshasa, Kosovo, Nicaragua, Russia and Armenia, and each of these five countries is rated with a much higher level of democracy by Polity IV than by Freedom House. However, some countries – for example, Kuwait, Qatar and Morocco – are rated with a higher level of democracy by Freedom House than by Polity IV.
Finally, the findings from this study show that a strong correlation between measures of democracy cannot be taken as evidence that the measures will generate the same results in empirical research. On the basis of the result of this study, it can be seen as worrying and problematic for comparative studies of democracy that empirical results differ so much according to which measure of democracy is used. It can also be seen as problematic and surprising that Freedom House and Polity IV come to such different conclusions about the level of democracy in several countries in the world.
