Democracy is a central concept in political science. Scholars have linked it to patterns of war and peace, alliance formation, conflict escalation, success in war, political stability, economic development, and improvements in a wide array of social indicators. These claims are subject to challenge, but the existence of a substantial literature on each of them underlines the importance of the questions. What, however, is the mechanism by which democracy generates these benefits? Is it a matter of norms, behavior, or institutions? Although democracy is a key concept, there is no consensus regarding its theoretical underpinnings.

It is for these reasons that the appearance of The Logic of Political Survival (Bueno de Mesquita et al. Reference Bueno de Mesquita, Smith, Siverson and Morrow2003) was hailed as a breakthrough. The book provides a simple explanation for the effects of democratic institutions: democratic leaders face different incentives than autocrats because democracies have different reselection institutions. The selectorate theory holds that political power depends upon satisfying a winning coalition (W) of the relevant selectorate (S). As the size of the winning coalition increases, leaders shift public policy from providing private benefits to supporters to providing public goods to the masses. Coalitions are most loyal when W is small relative to S. As the ratio W/S increases, defection becomes less risky, and leaders must spend more to keep their coalitions satisfied.

If correct, the selectorate theory has important implications for the long-standing debate over whether democracy should be understood primarily in terms of attitudes and behavior (e.g., Almond and Verba Reference Almond and Verba1963: Dahl Reference Dahl1971: Putnam Reference Putnam1994: Przeworski et al. Reference Przeworski, Alvarez, Cheibub and Limongi2000) or in terms of institutions (e.g., Shugart and Carey Reference Shugart and Carey1992: Cox Reference Cox1997: Lijphart Reference Lijphart1999: Powell Reference Powell2000). According to the theory, the differences between democratic and authoritarian politics and the differences in politics among democracies are reducible to institutional incentives; thus, institutions and not behaviors are the crux of the matter.

Bueno de Mesquita et al. test the predictions of their model by regressing a wide range of dependent variables against winning coalition size (W) and its ratio to the size of the selectorate (W/S). W and S are derived from commonly used measures of democracy, notably the Polity IV score, and W and W/S are highly correlated with democracy (r=0.8). Consequently, the authors are sensitive to the criticism that these variables are no more than proxies for democracy. They attempt to show that the effects of these variables are robust to models that include democracy as a control. In fact, one of their key findings is that W and W/S outperform democracy when included in equations containing democracy.

We demonstrate that these conclusions are incorrect. When controlling for democracy, the size of a polity's selectorate and winning coalition are often not associated with the decisions made by leaders or are associated in ways that contradict the model's predictions. We show that Bueno de Mesquita et al. are misled because they misuse a technique known as residualization in a way that induces omitted variable bias. The regression results they report are equivalent to the results obtained when democracy is omitted from their specifications. Their use of residualization exaggerates the importance of their variables of interest by increasing the size of the estimated coefficients and decreasing the estimated standard errors. When we reestimate the main results from Chapters 4 and 5 of their book using the correct estimator, their most important findings do not survive. We show, for example, that the apparent strength of winning coalition size is an artifact of the biased estimator and the correlation between W and democracy.

THE PROBLEM

To give a flavor for Bueno de Mesquita et al.'s results and what is incorrect about their analysis, we initially focus on a single example, public expenditures. The selectorate model implies that institutions that call for large winning coalitions should be associated with more effort to produce public goods, and the authors expect this to be reflected in the level of public expenditures. They write, “when the coalition is small, government expenditures are expected to be relatively small, and when the coalition is larger, so are government expenditures. Small-coalition leaders can survive spending a smaller quantity of what they collect than can large-coalition leaders” (Bueno de Mesquita et al. Reference Bueno de Mesquita, Smith, Siverson and Morrow2003, 161–162).

The authors test this proposition by regressing expenditures as a proportion of GDP on W, the size of the minimum winning coalition, and S, the size of the selectorate.Footnote ¹ To control for alternative explanations, the authors include democracy and the natural logarithm of GDP in their equations as control variables.Footnote ² (Democracies, for example, might spend more than autocracies for reasons other than the logic of the selectorate model. At the same time, governments in wealthier countries are able to spend a greater percentage of GDP.) The regression results confirm their expectations. Unfortunately, these results are flawed. Instead of actually using democracy and ln(GDP) as control variables, they use the residuals from two auxiliary regressions in their place. The first of these auxiliary equations regresses democracy on W and S, and the second regresses ln(GDP) on W and S.Footnote ³ This technique, substituting residuals for control variables, serves to exaggerate the size of the estimated coefficient on W because both democracy and ln(GDP) are positively correlated with W.

Consider Table 1. In the first column, we replicate Model 2 in Table 4.6 of Bueno de Mesquita et al. (Reference Bueno de Mesquita, Smith, Siverson and Morrow2003, 163). The coefficient for W is in the expected direction—larger coalitions exhibit higher levels of expenditures—and it is statistically significant (p=0.02). About this result, the authors write “. . .societies with high per capita incomes spend more. However, even after controlling for these effects (or the independent, positive, and significant impact of democracy), large coalition size and a weak loyalty norm are still powerful indicators of higher government spending” (163).

TABLE 1.

Expenditures, n = 2373

^aEstimates obtained by Bueno de Mesquita et al. using their residualization technique.

In the second column, we report results from a corrected regression in which we replaced the residuals of democracy and ln(GDP) used by Bueno de Mesquita et al. (Reference Bueno de Mesquita, Smith, Siverson and Morrow2003) with the actual variables. The results no longer support the selectorate theory. The coefficient on W changes sign—larger coalitions are associated with lower levels of expenditures—and it is statistically significant (p=0.01). Our replications of the evidence in Chapters 4 and 5 of The Logic of Political Survival show that the broad pattern of evidence that Bueno de Mesquita et al. find is an artifact of their biased estimation procedure, and the strong effects that they attribute to W or W/S are in fact due to their correlation with democracy and ln(GDP). When democracy and ln(GDP) are included correctly in the specifications, the effects of W and W/S disappear, change signs, or are substantially attenuated.

The technique Bueno de Mesquita et al. (Reference Bueno de Mesquita, Smith, Siverson and Morrow2003) use induces omitted variable bias. The procedure, known as “residualization,” comes from a suggestion by Goldberger (Reference Goldberger1964, 196) and is well-known in certain areas of American politics, although recent uses, such as Hicken, Satyanath, and Sergenti Reference Hicken, Satyanath and Sergenti2005, cite Bueno de Mesquita et al. directly.

The problem with residualization is easily explained on an intuitive level. W and democracy are correlated, and when included in the same regression, the shared variation between them is not used to calculate their individual effects. When democracy is replaced with the residuals from the auxiliary regression of democracy on W, W and democracy residuals are uncorrelated. Thus, the shared variation between W and democracy is now assumed to belong solely to W. The shared variation between W and the dependent variable, therefore, is larger than it should be. The effect is to bias the estimated coefficient on W and lower the standard error (due to the increased information). The true danger of using residualization becomes clear when we realize that the effect on W's estimated coefficient is precisely the same as if we had omitted democracy from the regression altogether.

For the sake of notational simplicity, let us assume a model with only two independent variables: y=β₀ + wβ_w + dβ_d + ε, where w is winning coalition size, and d is democracy. As Bueno de Mesquita et al. are concerned with the correlation between winning coalition size and democracy, they replace democracy in this equation with the residuals from a regression of democracy on winning coalition size: , where are the residuals from the auxiliary regression, d = γ₀ + wγ_w + ν. (We have left the coefficient on as β_d to emphasize that the residualization technique does not change this coefficient.) The expected value of this estimator, under the usual assumption that the expectation of the error term is zero, E[ε] = 0, is . The bias on the estimated coefficient on W is therefore

(1)

Equation (1) is precisely the bias that would result if democracy were left out of the specification completely (Greene Reference Greene2003, 148–49). The residualization procedure amounts to generating omitted variable bias on the coefficient of interest while still getting an unbiased estimate on the residualized variable. (Note that the authors could have achieved an unbiased estimate of W by residualizing W instead of democracy.) The formal derivation of equivalence between the residualization estimator and omitted variable bias is in Appendix A.

This equivalence cannot be observed directly in the results Bueno de Mesquita et al. report. When they move from models that include only W to models that include residualized variables, they include additional covariates, such as ln(population). The inclusion of these additional variables, which are correlated with W and each other, obscures the relationship between residualization and omitted variable bias in the results. We can observe the equivalence, however, if we compare a regression of expenditures on W and democracy residuals to a regression of expenditures on just W. Table 2 shows that the estimated coefficients on W are precisely the same in both cases.

TABLE 2.

Expenditures, n = 2500

^aEstimates obtained by Bueno de Mesquita et al. (Reference Bueno de Mesquita, Smith, Siverson and Morrow2003) using their residualization technique.

Democracy is not the only variable for which Bueno de Mesquita et al. Reference Bueno de Mesquita, Smith, Siverson and Morrow2003 substitute residuals. They use, in various specifications, the “residual impact” of GDP, civil liberties, political rights, investment, and savings. How these biases interact with one another is demonstrated easily with an example. Consider the results in Table 3, in which we replicate and reestimate the results of a regression of savings on W and various control variables (the replication is of model 2 in Table 4.1 of Bueno de Mesquita et al. Reference Bueno de Mesquita, Smith, Siverson and Morrow2003, 144).

TABLE 3.

Savings, n = 3942

^aEstimates obtained by Bueno de Mesquita et al. using their residualization technique.

When we substitute the actual variables for the residuals used by the authors, the estimated coefficient on W changes sign. Equation (1) tells us why. Consider first the effect of democracy residuals. The estimated coefficient in the regression of democracy on W, (w′w)⁻¹w′d, is 1.01. , reading off Table 3, is −7.74. The bias is therefore . Now consider the effect of ln(GDP) residuals. The estimated coefficient in the regression of ln(GDP) on W is 3.69. The effect of ln(GDP), reading from the table, is 6.24. The bias in this case is 3.69*6.24 = 23.04. The total bias is therefore −7.82 + 23.04 = 15.22, and the corrected coefficient on W is 12.64 − 15.22 = −2.58, which is what we find in column three of Table 3. (The difference in the constants is a function of the different s in the two equations and the “demeaning” of democracy residuals and ln(GDP) residuals.)

This result explains a number of our other findings. First, replacing democracy residuals with democracy has a large effect on the estimated coefficient of W when the effect of democracy on the dependent variable is large and has a concomitantly small effect when the effect of democracy is small. (These effects are clearly seen in this case because the first part of the bias, (w′w)⁻¹w′d, the estimated coefficient of W on democracy, is approximately 1.) Consequently, if democracy does not have a large effect on the dependent variable, using the procedure will create only a small bias, which may not affect our inferences from hypothesis tests.Footnote ⁴ Second, when the coefficient on democracy is negative, the bias decreases, rather than increases, the estimated coefficient on W. Third, when the specification includes more than one set of residuals, the effect on W's estimated coefficient is a combination of biases, and in the case of OLS, the effects are additively separable. In the example we gave in the previous paragraph, the negative bias that including democracy residuals has on W's estimated coefficient is more than offset by the positive bias that including ln(GDP) residuals has on W's estimated coefficient.

These results cut through the arguments Bueno de Mesquita et al. advance in favor of residualization. The fundamental question is quite simple: does some measure of democracy belong in their specifications? If it is necessary to control for democracy or some function of democracy, then residualization is incorrect because it is equivalent to leaving democracy out altogether. On the other hand, if it is not necessary to control for democracy or some function of democracy, there is no reason to use residualization. The authors make it clear, however, that democracy does belong in their regressions; they repeatedly emphasize the independent effect of W after controlling for democracy.

To be specific about their reasoning, Bueno de Mesquita et al., in a private communication, argue that democracy is a function of W, and residualization was used to deal with the resulting endogeneity. Residualization was used by Soper (Reference Soper1976) for the same reason. He was criticized by Swan (Reference Swan1978) and Kennedy (Reference Kennedy1982), and the same criticisms apply here. Briefly, if democracy is a function of W, and the various dependent variables are functions of W, democracy, and other variables, then the system of equations is recursive (all variables in the democracy equation plus some additional variables appear in the main equation). Recursive systems are problematic for linear regression only if the error terms are correlated. If the error terms of the two equations are uncorrelated, linear regression is unbiased, and it is the appropriate technique to employ. If the error terms are correlated, it makes no sense to include an estimate of the first error term (the democracy residuals) in the main equation. The error terms would remain correlated.

Bueno de Mesquita et al.'s (Reference Bueno de Mesquita, Smith, Siverson and Morrow2003) concerns about the interdependence of democracy and W can be thought of in terms of multicollinearity, which was also a concern of Soper's (42). In fact, the residualization technique has traditionally been thought of as a “fix” for multicollinearity (see Goldberger Reference Goldberger1964). We ran multicollinearity diagnostics for all of our replications, and the results demonstrate that multicollinearity is not a problem in these data. (The full diagnostics are available from the authors upon request.) Thus, whether we think of the interrelatedness of winning coalition size and democracy as endogeneity or multicollinearity, multiple regression is the correct way to proceed.

Replication and Reestimation Results

Bueno de Mesquita et al.'s (Reference Bueno de Mesquita, Smith, Siverson and Morrow2003) model is attractive because it generates a wide range of observable implications, which the authors use to conduct a series of ambitious statistical tests. We focus on chapters 4 and 5 as they comprise the core chapters of the book, and they rely heavily on residualization. In chapter 4, Bueno de Mesquita et al. test the implications of their model regarding taxation, labor, leisure, and total spending and “assess the implications of the theory regarding personal income, economic growth, and kleptocracy . . .” [129]. In chapter 5, Bueno de Mesquita et al. test the implications of their model for the provision of “core public goods” (civil liberties, political rights, transparency, and peace), “general public goods” (public health, education, social security, and international trade), and private goods (black markets, construction, and corruption). We replicate and reestimate the results in these two chapters, and we present them briefly. Few of their important results survive reestimation when the residuals are replaced with the original variables.

Table 4 summarizes our results, which are categorized by dependent variable. For instance, when we reestimate the effect of W on savings, the resulting coefficient is no longer statistically significant and has, in fact, changed signs (last row, column 2). The results show that of the 70 coefficients reported by Bueno de Mesquita et al. (Reference Bueno de Mesquita, Smith, Siverson and Morrow2003), 33 (47%) are either no longer significant (28 cases) or have changed signs and are significant (5 cases). In 64 of the 70 regressions (91%), use of the residualization technique exaggerated the substantive effect of coalition size. The authors incorrectly conclude that W has strong effects on cross-national variation in economic growth, civil liberties, political rights, literacy, and public health. For example, the authors claim, “A shift from the smallest to the largest coalition structure produces a 20% to 40% reduction in the illiteracy rate among those fifteen years or older” (189). These estimates are approximately three times larger than they should be. Similarly, they conclude, “living in a polity that depends on the largest-size coalition contributes nearly 13.5 more years of life [expectancy] than does living in a society that relies on the smallest-size coalitions” Bueno de Mesquita et al., 194). These estimates are more than four times too large. The broad pattern of evidence that Bueno de Mesquita et al. marshal in support of their theory does not exist.

TABLE 4.

Reestimation Results for Chapters 4 and 5

^aSuperscripts refer to the particular model when there is a question.

^bWe treat 0.05 as the threshold for significance.

There are a few cases (investment, trade, the number of doctors, and immunity to measles) where the biased procedure makes the authors' results appear to be weaker than they should be. However, the results that the authors themselves indicate are the most important are overturned. These key findings include the results on government expenditures and economic growth in chapter 4 and the results on core public goods in chapter 5. These important results are overturned because these tests involve dependent variables that are correlated with democracy and thus are most strongly affected by omitted variable bias. Bueno de Mesquita et al. (Reference Bueno de Mesquita, Smith, Siverson and Morrow2003) obtain their misleading results because W is correlated with democracy, and the biased procedure they use exploits this fact to inflate the apparent effect of W and decrease its standard error. Even in cases where the coefficient on W remains statistically significant in our reestimations, its effects turn out to be quite modest. The combination of W's apparent invincibility—it seemed to never fail a hypothesis test—and its inflated substantive effects lead the authors to conclude that coalition size is the core of the matter where democracy is concerned. Results that indicated that W outperformed democracy were artifacts of a biased estimation technique.

Thus, we find little empirical support for the selectorate theory. It remains possible that the theory is correct, and the data are to blame. Bueno de Mesquita et al. (Reference Bueno de Mesquita, Smith, Siverson and Morrow2003) acknowledge the “crude and primitive” nature of their measures—W takes five possible values, and S takes three—although they argue that their approximations “should, however, be adequate to evaluate whether the central tendencies of politics are aligned with the expectations that follow from the selectorate model” (133). As a robustness check, we reestimated their results using a more appropriate measure of W/S drawn from Powell Reference Powell2000. We found 10 statistically significant results that rejected their hypotheses, 17 insignificant results, and only 2 results (for income and trade openness) that supported them. Although space considerations prevent from us presenting the full suite of results, all of our findings and replication files can be found on the Web site listed in the acknowledgements.