The Issue

The reason that the relationships between the individual Xs and Y can be manipulated apparently with such ease is simple: the results of the test are not being interpreted correctly. β₁ captures the impact of X₁ on Y when X₂ = 0 (and vice-versa),⁴

not the impact of X₁ on Y in general. Because interactive relationships imply that the impact of X₁ on Y varies depending on the level of X₂, the idea of “the impact of X₁ on Y in general” is in fact a meaningless one. Nevertheless, even highly respected scholars continue to interpret lower-order interaction-term coefficients as if they were ordinary coefficients in a strictly additive model.

Such an interpretation is erroneous. If β₁ is statistically significant, it is only reasonable to conclude that H₁ : β₁ ≠ 0 cannot be rejected when X₂ = 0. The hypothesis may or may not be supported at other levels of X₂. Unless the hypothesis makes some claim of the (highly unusual) form

this information is of little immediate use in hypothesis testing.

Moreover, the conclusion that H₁ : β₁ ≠ 0 cannot be rejected when X₂ = 0 is especially unhelpful if the range of the data, or of the relevant cases, does not include zero. If X₂ were gross national product (GNP), for example, β₁ would describe the estimated impact of X₁ on Y when GNP = 0. In short, it would tell one, literally, nothing. Concluding that a statistically significant relationship between X₁ and Y exists based on such information is simply incorrect.

For these reasons, statements about the statistical significance of β₁ and β₂, rather than being statements about the nature of the political world, at best represent statements about reality that only apply to a subset of the cases. That subset is typically quite small and there is no reason, a priori, to believe that it is representative of the rest—in fact, the presence of an interactive relationship guarantees that it will not be. These statements surely cannot help a researcher to evaluate hypotheses relating X₁ and X₂ to Y in general.⁵

Finally, it is worth noting that the interpretation of coefficients in models with higher-order interactions is more convoluted still. Given the equation

the meaning of the coefficients is even more restricted, and their applicability to hypothesis tests is even more tenuous, than in the two-variable case:

• β₁₂₃ describes the impact of a joint increase of X₁, X₂, and X₃ on Y.
• All other coefficients reflect the singular or joint impact of the independent variables to which their subscripts correspond on Y when all other independent variables are equal to zero.

So, for example, β₃ describes the impact of an increase in X₃ on Y when X₁ = X₂ = 0, and β₁₃ describes the impact of a joint increase in X₁ and X₃ on Y when X₂ = 0.

Figure 1 illustrates these effects. In terms of the graph, which depicts Y at two different values of X₃, β₃ permits the surface to vary at the leftmost point in the graph, β₁₃ permits variation of the slope along the edge at which X₂ = 0, and β₁₂₃ permits a change in the curvature of the surface. In this case, β₃ and β₁₃ are positive, while β₁₂₃ is negative: an increase in X₃, therefore, raises the point at X₁ = X₂ = 0 and increases the slope along X₂ = 0 while making the surface less convex. Accordingly, the consequences of misinterpretation of coefficients becomes more severe as the number of variables increases: far from saying something universal about the relationship between X₃ and Y, for example, a significant and positive β₃ says nothing when either X₁ ≠ 0 or X₂ ≠ 0. One might reasonably wonder whether theories about social phenomena really make predictions about the sign and magnitude of most of these coefficients.

Moreover, the illustration highlights the perils of omitting some or all of the lower-order terms. In any interaction of k independent variables, a full set of

coefficients must be estimated to avoid forcing the estimated hyperplane to assume a shape that may not conform to the general tendency of the pointcloud that it is intended to describe. If β₁₃ x₁ x₃ were omitted from the equation, for example, the slope of the line at X₂ = 0 would be held constant across all levels of X₃, by assumption, and the remaining coefficients most likely biased as a result.

⁶

The outcome is analogous to omitting the constant term from a simple bivariate regression, thereby forcing the regression line through the origin regardless of the pattern of the data: the consequences for inference may be negligible or severe.

Why Does It Matter?

Political science is a discipline in which an inordinate amount of importance is placed on t- or z-statistics.⁷

Even those reviewers willing to overlook a significance level of 0.051 pay attention to the ratio of the coefficient to the standard error. Judging by the contents of even the top journals, few scholars realize that that ratio is both arbitrary and not representative of any general trend for the coefficients on the lower-order terms of a set of interactive variables. At the same time, a single article or book with a significant result on a prominent topic can be immensely persuasive, especially if no critics point to flaws in the data or methodology. In combination, these two facts imply that large numbers of scholars can be misled for long periods of time by the simple misinterpretation of a coefficient.

Without pointing fingers—the discipline, not any individual, is really culpable—let me illustrate this point with a reexamination of three articles, all of which were written by prominent and respected scholars and have played substantial roles in ongoing academic debates.⁸

Schultz's (1999) article on whether democratic institutions lead to peace by constraining leaders or informing other states is an exemplary piece in many ways: a clean formal model leads to opposite predictions based on the two theories, and an empirical test favors the informational perspective. Similarly, Mansfield and Snyder's (2002) comprehensive test of the relationship between various regime transitions and war suggests quite strongly that incomplete democratization is hazardous: in twenty-nine of the thirty-five variants of the statistical model that are tested, there is a significant and positive relationship between incomplete democratic transition and war. Finally, Adserà and Boix's (2002) examination of government size (that is, the size of the public sector) makes a strong case for the importance of politics: whereas previous studies suggested that trade and the size of the public sector would be related for purely economic reasons, the authors derive a model in which both are the result of political decisions based ultimately on the distribution of domestic interests. As a result, the authors argue, government should be large in free-trading democracies but small elsewhere,⁹ and the tendency toward large government should be exacerbated when state exports become less diversified, especially in democracies.¹⁰ In all cases, an interaction term was included as part of the analysis. Schultz included the democracy of the initiator, the democracy of the target, and an interaction term, noting only that “[t]he hypotheses do not speak to the expected sign and significance of the coefficients on demtarg and demdem, but they are nevertheless included as controls.”¹¹ In Mansfield and Snyder's case, incomplete democratic transition was multiplied by a variable measuring the concentration of domestic authority in twenty-five of the thirty-five models; and in all twenty-five, the variable capturing incomplete democratic transition was statistically significant. In Adserà and Boix, the interaction of trade openness, democratic institutions, and export concentration was examined in the search for the sources of variation in government size.

Moreover, in all cases the main conclusions depended on the interpretation of coefficients from lower-order terms, and those coefficients are interpreted as if they applied across all cases. Schultz notes that

Mansfield and Snyder, who find significant results for coefficients on both incomplete democratic transitions and the interaction of same with the concentration of domestic authority, discuss the implications of both but emphasize the former:

Adserà and Boix multiply three variables together (export concentration, trade openness, and democratic institutions), making the task of drawing inferences much more complex; moreover, they omit one term (export concentration × democratic institutions), forcing that coefficient to zero. Their conclusions based on lower-order terms are unconditional ones:

Similar examples pervade the field, but more would simply belabor the point.

Does the general critique above imply that the authors' conclusions are wrong? No such simple assertion is possible, because—and this is the key point—the tests were never designed in such a way that the conclusions reached were meaningful ones for more than a subset of the data. Schultz's significant coefficient describes what happens to militarized interstate disputes when the initiators are democratic and the targets are not (demtarg= 0), not what happens to militarized interstate disputes when the initiators are democratic in general. Mansfield and Snyder's coefficient describes what happens when there is an incomplete democratic transition and domestic concentration is at its lowest. Adserà and Boix's conclusion about the effects of export concentration apply, but only when trade openness and democratic institutions equal zero. Each conclusion is therefore correct, but only for a subset of the cases. That said, in the latter two instances that subset is actually empty: Mansfield and Snyder's data set contains no instances of incomplete democratic transitions when domestic concentration is at its lowest, and Adserà and Boix's contains no instances in which trade openness equals zero.

Reanalysis and Reinterpretation

How can one obtain more generalizable answers about the relationships of these key independent variables to the dependent variable of interest?

In the case of Schultz, the interaction term was added to the equation without much theoretical justification, and the coefficient suggests strongly that it adds little to the results. The simple remedy is to drop it. In Table 2, therefore, I have reanalyzed the main model from Schultz's article, dropping first the interaction term and then the demtarg variable. As intuition might suggest, the omission of the relatively insignificant interaction term makes little difference for the effects of democratic initiation. The coefficient decreases in magnitude by roughly 25 percent but remains statistically significant (p = 0.037, versus p = 0.015 in the replication). Happily, the main result stands, though its substantive and statistical significance are somewhat attenuated.

In the Mansfield and Snyder piece, on the other hand, omitting the interaction term is not a legitimate option: both theory and empirical results suggest that it belongs in the equation. At the same time, one would like to know more about the relationship between incomplete democratic transitions and war in cases in which domestic concentration is greater than zero.

One simple solution¹⁶

is to take advantage of the fact that β₁ measures the impact of X₁ on Y when X₂ = 0 by recoding X₂ in steps and describing how β₁ changes. In this case, that means subtracting n (n = 1, 2,…, 9) from domestic concentration and reestimating the logit equation to see how the coefficient and standard errors on incomplete democratic transition vary as a function of domestic concentration.

Figure 2 illustrates the results of such an analysis. The figure demonstrates that the positive relationship between incomplete democratization and war, far from being a general result, is limited only to cases in which democratic concentration is relatively low—say, from zero to 4 on the 10-point concentration scale. Only about 27 percent of the cases of incomplete democratization fall into this range. In the bulk of the cases—those in which concentration ranges from 5 to 7, which constitute 67 percent of all cases of incomplete transition—there is enough uncertainty about the relationship that it cannot reliably be distinguished from zero and would fail conventional tests of statistical significance. Interestingly, in cases of high concentration (an 8 or a 9, which constitute about 6 percent of the transition cases), the coefficients are negative and significant, suggesting that incomplete democratization produces peace, not war.

Rather than concluding, then, that incomplete democratic transitions “are a potent impetus to war, especially when the level of domestic concentration is low,”¹⁷

the authors could more reasonably have concluded that such transitions are an impetus to war only in those few cases in which the level of domestic concentration is low, and that they might even be conducive to peace if levels of domestic concentration are high. In more than 70 percent of the cases, an unconditional assertion that incomplete democratic transitions increase the probability of war cannot be supported.

Finally, the Adserà and Boix analysis provides an illustration of some of the intricacies involved in analyzing interactions with more than two variables. Although the interaction of product concentration of exports and democratic institutions is not thought to play a prominent theoretical role in their story, the interaction term must nevertheless be included because the product of those two variables is multiplied by trade openness. Omitting concentration × institutions (or ec × di) implies that the impact of a joint increase in those two variables when trade openness is zero must equal zero, by assumption. In terms of Figure 1, the slope of the surface along the X₂ (democracy) axis where X₁ (trade openness) equals zero cannot vary.

The results of a reanalysis including (concentration × institutions), summarized in Table 3, demonstrate this point. When ec × di is included in the model, the estimated coefficient is far from zero. Accordingly, while the coefficients on the variables not included in the interaction remain roughly the same, those associated with variables in the interaction term (and the constant) vary wildly. The lower-order coefficients indicate some changes that appear more alarming—and more meaningful—than they really are, given the correct interpretation of the coefficients: trade openness appears to have a negative impact on size of government (when institutions are nondemocratic and concentration, which never reaches zero, is zero); democratic institutions have a much larger and more significant negative impact on size of government (when trade openness and concentration, which never reach zero, are zero). The most striking change is the finding that joint increases in trade openness, concentration, and democratic institutions, which were previously thought to have no impact on size of government, are now shown to have a negative and significant impact. Figure 3 illustrates the impact of changes in product concentration on the relationship between trade, democratic institutions, and size of government.¹⁸

Substantively, the results tell a rather complex story, one that only partly agrees with the original conclusions. The authors argued that a large public sector is the product of the combination of high trade openness and political democracy. When product concentration is at its lowest (top illustration in figure), that is, when the country exports a diverse array of products, the generalization holds—but the positive interaction term and the negative marginal terms produce a saddle effect. The results suggest that size of government is nearly as large in autarkic autocracies as it is in free-trading democracies, a finding unanticipated by the theory.

When product concentration is at its highest (bottom illustration in figure), meaning that the country only exports a single product, the marginal relationships are reversed, and the joint effect of trade and democracy becomes negligible—in fact, ever so slightly negative. The effects of decreasing diversification, therefore, have more nuanced effects than the authors suggest: it drives the pressure for domestic compensation (as measured by size of government) down rather than up for a wide range of states, and it actually increases that pressure more in free-trading autocracies than it does in free-trading democracies.