Current Advice to Applied Researchers

Before the publication of Berry, DeMeritt and Esarey (Reference Berry, DeMeritt and Esarey2010), the literature suggested that when analysts faced a dichotomous outcome and suspected interaction, there was only one path forward: include a product term and conclude that there is interaction between X and Z if and only if the coefficient for the product term was statistically significant (Nagler Reference Nagler1991; Brambor, Clark and Golder Reference Brambor, Clark and Golder2006, though see Berry and Berry Reference Berry and Berry1991). However, Berry, DeMeritt and Esarey (Reference Berry, DeMeritt and Esarey2010, Reference Berry, DeMeritt and Esarey2015) have added much needed nuance to this advice. In particular, Berry, DeMeritt and Esarey (Reference Berry, DeMeritt and Esarey2010) lay out an initial framework describing the contexts in which researchers should hypothesize in terms of the latent variable Y* or Pr(Y), describe the implications for whether one performs hypothesis tests on the coefficient of the product term or a second-difference, and make suggestions about when to include a product term. Table 1 synthesizes the advice from Nagler (Reference Nagler1991), Berry, DeMeritt and Esarey (Reference Berry, DeMeritt and Esarey2010), and Berry, DeMeritt and Esarey (Reference Berry, DeMeritt and Esarey2015), summarizing the current advice to applied researchers.

Table 1

Current Advice for Testing Interactive Theories Using Logistic Regression

Note: this table summarizes three situations in which analysts might theorize interaction in the context of logistic regression. Notice that the quantity of interest varies across these situations, as well as whether the literature recommends including a product term. In this paper, I address the situation in which an analyst suspects interaction due to compression alone. In particular, I agree with Berry, DeMeritt and Esarey (Reference Berry, DeMeritt and Esarey2010) characterization of the theoretical value of compression and the implied quantity of interest, but argue that analysts must still include a product term.

I find the advice in Table 1 sound, useful, and large improvement over the coarse practice before Berry, DeMeritt and Esarey (Reference Berry, DeMeritt and Esarey2010). However, I disagree with the recommendation to exclude a product term when the analyst puts forward a strong theory predicting interaction on the basis of compression alone (the bold “No” in Table 1). In this paper, I explain why I find Berry, DeMeritt and Esarey’s (Reference Berry, DeMeritt and Esarey2010) argument for excluding a product term incomplete.

The Argument for Excluding a Product Term

Berry, DeMeritt and Esarey’s (Reference Berry, DeMeritt and Esarey2010) (hereafter, BDE) overarching claim, which I wholeheartedly agree with, is that “a statistically significant product term is neither necessary nor sufficient for concluding that there is substantively meaningful interaction among independent variables in their influence on [the probability of an event]” (Reference Berry, DeMeritt and Esarey2010, 261). Given this point though, BDE ask whether it is always necessary to include a product term in the model. To decide whether to include a product term, they suggest that analysts draw a sharp distinction between theories that focus on the unbounded latent variable, denoted as Y* (perhaps conceptualized as “utility”), and theories that focus on the probability of an event, denoted as Pr(Y).Footnote ⁴

We agree on Part (a). As both BDE and Nagler (Reference Nagler1991) note, this situation is entirely analogous to linear regression (e.g., Y*=β ₀+β ₁ X ₁+β ₂ X ₂+ … +β _k X _k ), making a product term essential.

BDE explain:

They continue by suggesting:

Although I find BDE’s suggestion to exclude a product term problematic, I take two other points from their paper as a starting point.

1. 1. Compression can be substantively interesting and theoretically meaningful. Consider the example from Wolfinger and Rosenstone (Reference Wolfinger and Rosenstone1980) discussed above. Recall that their model does not have a product term, the only interaction is due to compression. Yet if their model is correct, then restrictive voting rules not only reduce turnout, but also increase the inequality of political participation because restrictive rules have a larger negative effect on the probability of voting among the least educated. This relationship has enormous normative consequence and policy relevance.

2. 2. If analysts are interested in interaction due to compression, then the quantity of interest is not the product term, but the second-difference (or second derivative). Several authors show that the sign and significance of the second-difference does not depend on the sign and significance of the product term (e.g., Ai and Norton Reference Ai and Norton2003; Tsai and Gill Reference Tsai and Gill2013).

Although, I find BDE’s argument for these two points compelling, I disagree with the advice to exclude a product term when theorizing interaction on the basis of compression alone.

The Argument for Including a Product Term

Researchers must employ a statistical model that can represent (1) relationships that are consistent with the theory and (2) plausible relationships that are inconsistent with the theory. It is obvious that models should be able to represent relationships that are consistent with theory. But if researchers use models that cannot also represent situations inconsistent with the theory, then they have assumed their theories are correct. Though the theoretical rationale may be strong, it has not been made vulnerable to the data.

Thus, BDE’s analysis is incomplete when they write that “no product term is required if the analyst’s only reason for positing interaction between X and Y in influencing Pr(Y) is an expectation that when the value of X is extreme, Pr(Y) is near its limit of 0 or 1, and thus there is little room for Pr(Y) to change as Z changes.” Although they correctly note that the theorized relationship can be represented by a model with no product term, they fail to discuss whether this model can also represent relationships that are inconsistent with the theory.

Without a product term, a logistic regression model cannot represent many reasonable situations that are inconsistent with an interactive theory. In particular, it cannot represent a situation in which two explanatory variables have non-zero effects and the probability of an event is greater or less than one half, but there is no interaction. Specifically, for a logistic regression model including X and Z, but excluding their product, if all of the following conditions hold, then X and Z must interact in influencing Pr(Y).Footnote ⁵

1. 1. X has a non-zero effect.

2. 2. Z has a non-zero effect.

3. 3. Pr(Y)<0.5 or Pr(Y)>0.5.

This can easily be seen by inspecting the second derivative of the logistical regression model with no product term, given by

(1) $${{\partial ^{2} {\rm Pr(}Y{\rm )}} \over {\partial X\partial Z}}={\rm Pr}(Y)[1{\minus}{\rm Pr}(Y)][1{\minus}2{\rm Pr}(Y)]\beta _{x} \beta _{z} .$$

If any of Conditions 1–3 hold, then the second derivative is exactly 0. Otherwise, the second derivative is non-zero, which implies an interactive relationship model. Consequentially, if the researcher incorrectly theorizes compression, but 1–3 hold, then a logistic regression model without a product term will overstate the evidence for interaction.Footnote ⁶

With a product term, however, a logistic regression model can represent some plausible situations inconsistent with an interactive theory. For a logistic regression model including X, Z, and XZ, if all of the conditions (1–3) identified above hold, then X and Z may or may not interact in influencing Pr(Y). The intuition of this claim can be seen by simply inspecting the second derivative of a logistic regression model including a product term, given by

(2) $$\eqalignno{ {{\partial ^{2} {\rm Pr(}Y{\rm )}} \over {\partial X\partial Z}} &#x0026; ={\rm Pr}(Y)[1{\minus}{\rm Pr}(Y)]\beta _{{xz}} \cr &#x0026; {\plus}{\rm Pr}(Y)[1{\minus}{\rm Pr}(Y)][1{\minus}2{\rm Pr}(Y)](\beta _{x} {\plus}\beta _{{xz}} Z)(\beta _{z} {\plus}\beta _{{xz}} X). $$

The presence of the product term creates another coefficient that allows the second derivative to be zero, even if Conditions 1–3 hold. Thus, this model is capable of representing situations in which 1–3 hold, but no interaction is present. This means that a product term “uncouples” hypotheses about the effects of X and Z from their interaction.

Although including a product term does not entirely remove the bias toward confirming interactive hypotheses, it does greatly reduce the bias. Thus, including a product term makes the empirical argument more compelling by making the theory more vulnerable to the data.Footnote ⁷

Simulations

To illustrate the potential consequences of excluding a product term, I report two simulation studies. The first examines a single relationship in detail. The second looks at a wide range of potential relationships. Because I am particularly interested in what happens if the theory (or researcher’s hypothesis) is wrong, I imagine that a researcher incorrectly theorizes interaction due to compression and follows BDE’s suggestion to exclude the product term. In almost all cases, excluding the product term biases the researcher toward finding support for her theory.

But why focus specifically on what happens if the researcher is wrong? Why not look at what happens if the researcher correctly theorizes interaction on the basis of compression alone? Careful readers recognize the tradeoff between the size of a hypothesis test (i.e., the probability of finding interaction when there is none) and power (i.e., the probability of finding interaction when it is present). Although analysts usually face a tradeoff between these two, I focus my attention squarely on the size of tests for interaction.

Hypothesis test form compelling statistical arguments because, by design, they rarely lead a researcher to reject the null hypothesis when the null hypothesis is true. This is because hypothesis tests are constructed by fixing α=Pr(reject null | null is true) at an acceptably low value, typically α=0.05. This is important because researchers and readers interpret a decision to reject the null hypothesis as strong evidence against the null hypothesis (or for the research hypothesis).

Social scientists typically use hypothesis tests because they want to conclude that their research hypothesis is correct only when the data offer strong evidence in favor of their research hypothesis (or equivalently, against their null hypothesis). If a test is improperly sized (e.g., α=0.75), then the decision to reject the null hypothesis is no longer compelling. Making matters worse, readers and reviewers are still likely to interpret the results of this over-sized test as compelling. I focus squarely on the size of tests for interaction because the logic of hypothesis testing requires that the size be fixed at a suitably small value, usually 0.05 in the social sciences, whereas allowing the power of a test to fluctuate with other variables, such as the number of observations. My primary focus on size, or Type I or false-positive errors, should not be taken as meaning that power is unimportant. To address the secondary issue of power, I include a discussion of the statistical power of models that exclude and include a product term in the online Appendix. These simulations show that the ability of a model with a product term to “detect” an interactive relationship (i.e., statistical power) is nearly equal if the amount of interaction is substantial or the sample size is large. However, it is important to point out that when the sample size is small (e.g., <1000) and the amount of interaction is small (e.g., second-difference of 0.04−0.09=−0.05), then including a product term might lead to a substantial drop in statistical power. However, it is important to note that the gain in statistical power comes from assuming an interactive relationship, not detecting one.

One might also worry about the impact of including a potentially unnecessary product term XZ in the model (i.e., β _XZ =0) on the estimated effects of other variables in the model. In the online Appendix, I show that including an unnecessary product term does not lead to bias in the estimated effects of other variables in the model.

Study No. 1: A Fixed Relationship

I first consider the non-interactive relationship (i.e., data-generating process) Pr(Y)=0.3+0.2X−0.3Z shown in the left panel of Figure 1. I imagine that the analyst correctly suspects that X has a positive effect on Pr(Y) and Z has a negative effect on Pr(Y). She also theorizes that because the event is relatively rare and Z pushes Pr(Y) even closer to 0, there is less room for X to affect Pr(Y). Therefore, she expects that X should have a smaller effect when Z=1. But this hypothesis is not correct—she incorrectly expects that X and Z interact in influencing Pr(Y) due to compression. Still, she notices that a logistic regression model can represent the kind of interaction suggested by her theory and decides that no product term is necessary (Berry, DeMeritt and Esarey Reference Berry, DeMeritt and Esarey2010).

Fig. 1

Logistic regression fits to a non-interactive relationship Note: this figure shows the true relationship in the first simulation study (dotted lines) as well as how logistic regression models with and without a product term fit the process (solid, bold lines). Notice that X and Z have large effects on Pr(Y) and Pr(Y) is always <0.5. In this case, excluding a product term forces the model to point toward interaction, despite the fact that the relationship is non-interactive. Including a product term allows the model to fit this non-interactive process surprisingly well.

Because X is approximately uniformly distributed from 0 to 1 and Z is dichotomous with half of the observations equal to 1 and the other half equal to 0, she decides to test for interaction by estimating a logistic regression model with no product term, calculating the second-difference as X changes from 0.25 to 0.75 and Z changes from 0 to 1. Her hypothesis suggests that this second-difference should be negative, as X should have a smaller effect when Z=1.

Notice the middle panel of Figure 1, which shows how a logistic regression model without a product term might fit the true relationship. Although there is no interaction in the actual data-generating process, the logistic regression model with no product term suggests that there is substantial interaction between X and Z in influencing Pr(Y). When Z=1, the estimated effect of moving X from 0.25 to 0.75 is 0.05. When Z=0, however, the effect of X triples to 0.15 (the second-difference is 0.05−0.15=−0.1). Although the hypothesis of interaction is wrong, the logistic regression model with no product term suggests there is strong evidence in favor of it.

Now notice the right panel of Figure 1, which shows the estimates when the model contains a product term. Although this model does not exactly capture the true data-generating process (i.e., notice the non-linearity when Z=1), it estimates the amount of interaction very well. The estimated effect of X when Z=0 is 0.1. When Z=1, the estimate is identical. Therefore, the estimated second-difference is 0 (0.1−0.1=0) and the model correctly suggests that there is no interaction.

Rather than examining how models fit the data-generating process, it is perhaps more informative to evaluate each approach in repeated trials. To do this, I simulate 2,000 data sets containing 1,000 observations each. I use each data set to estimate a second-difference and its confidence interval using a model with and without a product term. Finally, I estimate α=Pr (Find Interaction), the probability of finding interaction when none is present (i.e., a Type I or false-positive error), by calculating the proportion of data sets that pointed toward interaction with each approach (i.e., the confidence interval does not contain 0). By convention, this probability should be close to 0.05. When the model does not include a product term, the probability of finding interaction is 0.98. Thus, finding a statistically significant second-difference is not at all a compelling evidence against the null when the model does not include a product term, as the researcher will likely find this even when there is no interaction. However, the probability of finding interaction drops to 0.06 when the researcher includes a product term in the model. This suggests that finding a statistically significant second-difference is compelling when the model includes a product term, as the researcher would rarely find this when the null is true.

To help understand why excluding a product term biases the researcher toward finding interaction, Figure 2 shows the estimates and confidence intervals from only 50 simulated data sets, sorted by the size of the estimated second-difference. Notice the left panel of Figure 2, which shows that the models without a product term always estimate a substantial amount of interaction in the expected direction and that the confidence intervals rarely overlap 0. This happens because compression serves as the theoretical motivation for the hypothesis and is assumed by the model. But notice the right panel of Figure 2, which shows that most of this bias disappears when a product term is included in the model.

Fig. 2

Fifty estimated second differences and confidence intervals Note: this figure shows 50 simulations of estimated second-differences and confidence intervals using models with and without a product term. Although second-difference is actually 0 (a non-interactive relationship), the model with no product term consistently finds interaction. Including a product term removes almost all of this bias.

To get a sense of the robustness of this conclusion, I systematically vary the sample size, the distribution of X, and the change in X used to calculate the second-difference while the true relationship remains the same. For each combination of the simulation parameters, I simulate 2000 data sets. For each data set, I estimate a logistic regression model with and without a product term and compute the confidence interval around the second-difference, as BDE recommend. I then calculate the proportion of simulations in which each model points incorrectly toward interaction to estimate the probability of finding interaction when none is present. The results are presented in Figure 3.

Fig. 3

The size of tests for interaction Note: this figure shows the probability of concluding interaction when the true relationship is given by Pr(Y)=0.3+0.2X−0.3Z (i.e., there is no interaction) using logistic regression models with and without a product term as the sample size, the distribution of X, and the change in X considered in computing the second-difference varies. Notice that the model including a product term performs remarkably well, although the improvement diminishes as the simple size gets large, especially when the distribution of X is skewed and the researcher considers the effect of changing X nears its extremes.

Across all values of the simulation parameters, the models without a product term bias the researcher toward finding interaction and sometimes strongly. The bias is reduced, usually dramatically, when the researcher includes a product term. Indeed, for symmetric distributions of X and centered second-differences, adding a product term to the model eliminates nearly all of the bias, lowering the probability of finding interaction from nearly 1.00 (the worst situation) to about 0.05 (the ideal situation).Footnote ⁸

Study No. 2: A Diverse Set of Relationships

To demonstrate that the patterns seen in Figure 3 do not depend on any particular relationship, I now examine a diverse collection of 1000 simulated analyses. Each analysis is generated through a random process (described in the online Appendix) that varies in the following ways:

1. 1. The relationship between Pr(Y), X, and Z varies, but follows the form of Pr(Y)=β ₀+β ₁ X+β ₂ Z+β ₃ X ²+β ₄ Z ² and is always monotonic across the range of X and Z. Importantly, X and Z never interact in changing Pr(Y).Footnote ⁹

2. 2. The distributions of X and Z vary and might be binary, flat, unimodal, or skewed to the right.

3. 3. The values of x _lo , x _hi , z _lo , and z _hi used to construct the second-difference vary from 0 to 1. All combinations are possible, though if a variable is binary, its values are set to 0 and 1.

4. 4. The sample size varies, ranging from 500 to 100,000, though smaller samples are more likely.

Each “analysis” consists of features the researcher totally controls (e.g., values of x _lo , x _hi , z _lo , and z _hi ), somewhat controls (e.g., sample size), and does not control (e.g., distributions of explanatory variables and the true data-generating process). For each simulated analysis, I use the same procedure as before to estimate the probability of finding interaction using models with and without a product term. Because none of the true relationships are interactive, this probability should be near 0.05.

The left panel of Figure 4 shows that when the researcher uses a logistic regression model without a product term, the probability of finding interaction when none exists is less than 0.1 for only about 40 percent of the simulated analyses. It is >0.9 for about 20 percent of the analyses. This suggests that excluding a product term strongly biases the researcher toward finding interaction for many plausible relationships. On the other hand, the right panel of Figure 4 shows that, when using a model with a product term, the probability of finding interaction is less than 0.1 for almost all of the simulated analyses (over 98 percent). This suggests that including a product term can eliminate most of the bias toward finding interaction for many plausible relationships and analyses.

Fig. 4

The distribution of the size of tests for interaction across a diverse set of analyses Note: the histogram on the left shows that logistic regression models without a product term tend to find interaction when none exists. The histogram on the right shows researchers can eliminate most of this bias by simply including a product term.

Figure 5 directly compares the two approaches. Except for a single outlier, notice that every point falls below the diagonal line, except when both approaches fall below 0.1 (i.e., both perform well). This means that in almost every situation in which excluding a product term led to substantial bias, the model with a product term reduced this bias. In only one case did the model without a product term perform substantially better than the model with a product term. Regardless of the absolute performance of each, the model with a product term performs at least as well as the model without a product term in almost all the analysis generated in my simulation, except when both perform well. In cases in which the model with a product term performs somewhat poorly (e.g., the probability of finding interaction is about 0.3, for example), the model without a product term performs terribly (e.g., the probability of finding interaction is nearly 1.00).

Fig. 5

The relationship between the size of the test for interaction with and without a product term across a diverse set of analyses Note: this figure shows the probability of concluding interaction when none exists from logistic regression models with and without a product term for many different analyses. Points that fall in the vertical band indicate an analysis in which the model without a product term performs well. Points that fall in the horizontal band indicate an analysis in which the model with an product term performs well. For many analyses, the model without the product term performs quite poorly, whereas the model with a product term performs well. Finally, notice that almost all points fall below the diagonal line indicating that models without a product term are more likely to mislead researchers in almost every simulated analysis.

Recommendations For Applied Researchers

I suggest that applied researchers take several steps to ensure their claims of interaction are empirically meaningful.

1. 1. Clearly explicate your interactive theory (Clark, Gilligan and Golder Reference Clark, Gilligan and Golder2006; Berry, Golder and Milton Reference Berry, Golder and Milton2012) and carefully consider which relationships are consistent and inconsistent with the theory. Only use empirical models that can represent both relationships.

2. 2. As Berry, DeMeritt and Esarey (Reference Berry, DeMeritt and Esarey2010) suggest, specify whether the relevant outcome variable is the latent variable Y* or the expected value Pr(Y). If you use the concept Y*, then test for interaction by testing the statistical significance of the product term. If you rely on Pr(Y), then test for interaction by simulating confidence intervals around a carefully chosen second-difference or second derivative. See Berry, DeMeritt and Esarey (Reference Berry, DeMeritt and Esarey2010) for the details of calculating a second derivative or second-difference and King, Tomz and Wittenberg (Reference King, Tomz and Wittenberg2000) for the details of the simulation procedure. Note that Hanmer and Kalkan (Reference Hanmer and Kalkan2013) offer an alternative approach in which researchers average the quantity of interest across the observed data rather than choosing a specific comparison of interest.

3. 3. Regardless of whether the theoretically relevant outcome is Y* or Pr(Y), and even if you theorize interaction on the basis of compression alone, you must include product terms in order to draw compelling substantive conclusions about interaction from logistic regression models. Otherwise, you are simply pulling assumptions through data and treating them as conclusions.

4. 4. As Nagler (Reference Nagler1991) and Frant (Reference Frant1991) note, if you do not include a product term, then avoid making claims about interaction. The interaction is assumed and you should understand and clearly identify it as such.Footnote ¹⁰

Democracy, Distance, and the Probability of Conflict

To illustrate how to conduct a compelling test for interaction due to compression, I replicate analyses by Oneal and Russet (Reference Oneal and Russet2001).Footnote ¹¹ Berry, DeMeritt and Esarey (Reference Berry, DeMeritt and Esarey2010) explain that the probability of war between two democracies is nearly 0, but much higher if one or both countries is non-democratic. Thus, on the basis of compression alone, Berry, DeMeritt and Esarey (Reference Berry, DeMeritt and Esarey2010) argue that one can meaningfully hypothesize that explanatory variables should have smaller effects on the probability of conflict when both countries are democratic, simply because this probability is so close to 0. Using the example of geographic distance, the authors note that we should expect distance to have a much smaller effect on the probability of conflict when both countries are democracies than when at least one country is not democratic. Importantly, we have no reason to expect that democracy and distance interact in influencing the latent outcome (e.g., the utility of conflict). Thus, we have no reason to theorize about the sign of the product term. Further, the only reason to include a product term is to allow the model to represent a situation in which we are wrong (i.e., there is no interaction between democracy and distance in influencing the probability of conflict).

This leads to three hypotheses about the effects of distance and democracy on the probability of conflict. First, as has been commonly suggested (e.g., Ray Reference Ray1998; Tomz and Weeks Reference Tomz and Weeks2013), democracies are much less likely to fight each other.

Second, other literature in international relations points out that states that are located geographically further from one another are less likely to engage in conflict (e.g., Diehl Reference Diehl1991; Bremer Reference Bremer1992).

As Berry, DeMeritt and Esarey (Reference Berry, DeMeritt and Esarey2010) point out, these two hypotheses imply a third hypothesis. Because democratic dyads are much less likely to engage in conflict, we should expect the negative effect of distance to be much smaller in these dyads simply due to compression.

Is Parsimony Preferable?

When models with and without a product term provide similar (or nearly identical) conclusions, applied researchers might be tempted to discard the model with a product term for three reasons.

1. 1. The logistic regression model with no product term captures the relationship (democracy and distance have negative effects and interact due to compression) suggested by the strong theory. Berry, DeMeritt and Esarey (Reference Berry, DeMeritt and Esarey2010) recommend excluding the product term in this situation.Footnote ¹⁴

2. 2. The AIC suggests the two models fit equally well (AIC=13,520 for both models) and the BIC points toward the simpler with no product term (BIC=13,580 for the model without the product term and BIC=13,589 for the model with the product term).

3. 3. The product term is not statistically significant (p=0.23).

Researchers might be tempted to drop the product term for any of these three reasons. In spite of these temptations, parsimony is not preferable. Researchers must include the product term XZ when making arguments about X and Z interacting in influencing Pr(Y). Although the models might suggest similar conclusions, they do not offer similar evidence for those conclusions. A model without a product term does not offer compelling evidence for interaction because it suggests interaction in spite of the data, whereas a model with a product term suggests interaction because of the data.

In the applied example, it is important to include the product term because it “uncouples” the Interaction Hypothesis from the Distance and Democracy Hypotheses. If I do not include a product term in the model, then the large negative effect of democracy and distance guarantee that the two interact in influencing Pr(Y) (see the left panel of Figure 2 and Equation 1). With no product term in the model, the Distance and Democracy Hypotheses imply the Interaction Hypothesis and nothing is learned from examining the second-difference. By including the product term, however, I am able to uncouple the Democracy and Distance Hypotheses from the Interaction Hypotheses (see the right panel of Figure 2 and Equation 2). When the model includes a product term, the Distance and Democracy Hypotheses no longer imply the Interaction Hypothesis and something is learned from examining the second-difference. Although the two estimates of the second-difference presented in Figure 9 are similar, they represent different amounts of evidence for the Interaction Hypothesis. The estimate from the model with no product term is assumed by the structure of the empirical model. The estimate from the model with a product term, on the other hand, represents a meaningful relationship based on the data.