Hostname: page-component-6898cfcb55-zkb6p Total loading time: 0 Render date: 2026-11-11T23:56:15.805Z Has data issue: false hasContentIssue false
  • English
  • Français

The Possibility Principle: Choosing Negative Cases in Comparative Research

Published online by Cambridge University Press:  01 November 2004

JAMES MAHONEY and
GARY GOERTZ
JAMES MAHONEY
Affiliation:
Brown University
GARY GOERTZ
Affiliation:
University of Arizona
Rights & Permissions [Opens in a new window]

Abstract

A central challenge in qualitative research is selecting the “negative” cases (e.g., nonrevolutions, nonwars) to be included in analyses that seek to explain positive outcomes of interest (e.g., revolutions, wars). Although it is widely recognized that the selection of negative cases is consequential for theory testing, methodologists have yet to formulate specific rules to inform this selection process. In this paper, we propose a principle—the Possibility Principle—that provides explicit, rigorous, and theoretically informed guidelines for choosing a set of negative cases. The Possibility Principle advises researchers to select only negative cases where the outcome of interest is possible. Our discussion elaborates this principle and its implications for current debates about case selection and strategies of theory testing. Major points are illustrated with substantive examples from studies of revolution, economic growth, welfare states, and war.

Information

Type
ARTICLES
Information
American Political Science Review , Volume 98 , Issue 4 , November 2004, pp. 653 - 669
Copyright
© 2004 by the American Political Science Association

“I see nobody on the road,” said Alice. “I only wish I had such eyes,” the King remarked, in a fretful tone. “To be able to see Nobody! And at that distance, too!”

Lewis Carroll

(Through the Looking Glass)

Where and when do “nonsocial revolutions” occur? Certainly the United States in 1900 qualifies, but Skocpol (1979) never considered this case in her famous study of social revolutions. Nor did she choose to analyze Canada in 1890, Australia in 1950, or most of the millions of nonsocial revolutions that have occurred in world history. Instead, she selected a sample of “negative cases”1

One can think of “negative” cases as “control” cases. We prefer the term negative because the contrast group is constituted by the observations that are “positive” on the dependent variable. Here we assume that cases are coded dichotomously on the dependent variable, an assumption that we relax below. It bears emphasis, however, that case selection is largely a dichotomous affair in research: Either an observation is included in the analysis or it is not.

that she regarded as relevant and appropriate for testing her theory of social revolution. In qualitative research, most analysts must—like Skocpol—select a set of negative cases to test their theories. However, the rules for choosing and justifying a set of cases defined by the occurrence of a nonevent are far from straightforward.

Intuitively, most qualitative analysts would claim that the United States in 1900 is not relevant or informative for testing theories of social revolution. Does this therefore mean that the case can be legitimately ignored when testing a theory of social revolution? Philosophers have puzzled over this question for half a century in the form of the “ravens paradox” (Hempel 1945). In this example, the hypothesis under consideration is the proposition that “all ravens are black.” The positive cases that clearly support the hypothesis are black things that are ravens and ravens that are black. The paradox arises from the logical fact that all nonblack, nonraven things also support the hypothesis. We intuitively feel that most—though probably not quite all—nonblack, nonraven things are not very useful in testing this hypothesis, just as the United States in 1900 is not an informative case for testing theories about the causes of social revolution. However, without any clear guidelines for differentiating relevant from irrelevant cases, it is hard to justify excluding these cases.

In this article, we propose a principle—the Possibility Principle—that provides explicit, rigorous, and theoretically informed guidelines for choosing a set of negative cases. The Possibility Principle holds that only cases where the outcome of interest is possible should be included in the set of negative cases; cases where the outcome is impossible should be relegated to a set of uninformative and hence irrelevant observations. We show that the application of this principle can help scholars avoid errors and maximize leverage for making valid causal inferences.

The Possibility Principle implicitly informs much experimental research. For example, when testing new varieties of crops, researchers do not usually put test plots in the desert. Not only would the use of these test plots be a waste of resources, but their inclusion could distort inferences about the efficacy of crop strands in settings where the outcomes of interest are possible. Or suppose scientists seek to test a drug to prevent breast cancer. Should they include men and children in the test population? Although men and children can develop breast cancer, it is quite rare. One might therefore argue that men and children are irrelevant when testing a drug to prevent breast cancer, given that the outcome of interest is such a low-probability event for them. The Possibility Principle states that the negative cases should be those where the outcome has a real possibility of occurring—not just those where the outcome has a nonzero probability.

It is useful to contrast the problem of selecting negative cases with the problem of selecting on the dependent variable. As is well known, selecting cases based on their value on the dependent variable can lead to the overrepresentation of positive cases in the sample, which can bias results in regression studies. The inclusion of irrelevant observations has the opposite effect: one introduces too many negative cases into the population. In short, selecting on the dependent variable normally means too many positive cases, whereas including irrelevant observations normally means too many negative cases. Just as the solution for selecting on the dependent variable is to include more negative cases, so too the solution to the negative case problem is to exclude irrelevant cases.

In developing this argument, we focus on qualitative, small-N research in the fields of comparative politics and international relations. We are particularly concerned with studies that seek to test theory about the causes of outcomes of exceptional interest such as revolution, war, genocide, welfare state development, and sustained economic growth. To explain these kinds of outcomes, nearly all research designs require the examination of negative cases. This is true both of research designs in large-N, quantitative work (see Hewitt and Goertz 2004) and of small-N research methods such as Mill's method of difference (Skocpol 1984), typological theory (George and Bennett 2005), Boolean algebra (Ragin 1987), and fuzzy-set analysis (Ragin 2000).2

Research designs focused on necessary causes are perhaps the only partial exception to this claim. As Braumoeller and Goertz (2000), Dion (1998), and Ragin (2000) have shown, one can test necessary cause hypotheses by selecting only cases with positive outcomes. However, Braumoeller and Goertz (2000) have argued that negative cases are required to test whether or not a necessary cause is trivial.

We introduce the Possibility Principle as a means of identifying the universe of these negative cases.

THE TOPOLOGY OF CASE SELECTION

The problem of case selection entails at least two central challenges. One challenge is selecting an appropriate sample of cases from a larger population about which one wishes to generalize. The literature on selection bias in comparative research focuses on this problem, attempting to offer insights for choosing samples in ways that do not bias inferences (e.g., Collier and Mahoney 1996; Geddes 2003, chap. 3; King, Keohane, and Verba 1994, 124–39). However, a second and more basic challenge involves drawing the boundaries between different kinds of cases. Most scholars have discussed this boundary challenge in terms of distinguishing positive and negative cases. By contrast, we focus attention on the rarely discussed boundary issues involving negative and irrelevant cases. We suggest that these negative/irrelevant boundary issues must be resolved before scholars can implement procedures for choosing a representative sample of cases.

Positive–Negative Boundary

The most often discussed boundary divides positive and negative cases. In the small-N research that interests us, the analyst seeks to explain the positive cases that possess the outcome of interest by contrasting them with negative cases that lack the outcome. Typically, when working with exceptional outcomes it is relatively easy to distinguish positive from negative cases, because the vast majority of observations will lack the outcome of interest and thus be negative cases. For example, most observations clearly are not social revolutions or wars or sustained high-growth economies, and thus they are negative cases. Even so, some cases will be difficult to classify as positive or negative, representing partial instances of the outcome of interest (e.g., partial revolutions or partial wars). Because of these borderline cases, one can think of the intersection between positive and negative cases as a nonempty space.

We use the expression gray zone to refer to this nonempty intersection point of the positive and negative sets where the outcome is partially present, the classically half-empty/half-full cases (see Figure 1). Techniques such as fuzzy-set analysis (Ragin 2000) are explicitly designed to help qualitative researchers conceptualize borderline cases in the gray zone.

Case Selection: Positive, Negative, and Irrelevant Cases

The issue of drawing the boundary between the positive and the negative cases is an important problem; likewise, once this boundary is established, the selection of a representative sample of positive and negative cases is a key issue. However, these are not the concerns of our argument. Rather, we are considering a prior issue involving the construction of a relevant population of cases in the first place.

Negative–Irrelevant Boundary

The problem of negative case selection involves the difficulties of distinguishing nonpositive cases that are relevant (i.e., negative cases) from nonpositive cases that are irrelevant (i.e., irrelevant cases). In Figure 1, the zone of irrelevant cases next to the negative cases highlights the structure of this boundary problem. The question raised here is, How should scholars draw the line between the negative and the irrelevant cases?

To this point, methodologists have offered only very general answers. They do not explicitly declare certain nonpositive cases to be irrelevant but, rather, advise that some nonpositive cases are more analytically useful than others. In particular, nonpositive cases that closely resemble positive cases, including on key hypothesized causal factors, are seen as highly useful. For example, in her discussion of the method of difference, Skocpol (1984) suggests that negative cases should be “as similar as possible to the ‘positive’ cases” in all respects except for their value on the dependent variable (378). Przeworski and Tuene's (1970) “most similar system design,” which examines positive and negative cases, is also grounded in the belief that cases “as similar as possible with respect to as many features as possible constitute the optimal samples for comparative inquiry” (32). Ragin (2000, 60) frames the issue of negative case selection in similar terms: “Negative cases should resemble positive cases in as many ways as possible, especially with respect to the commonalities exhibited by the positive cases.” Indeed, in order to maximize similarities between positive and negative cases, many scholars use time periods within a given spatial unit as their cases (Haydu 1998).

We consider these suggestions to be good advice for selecting a sample of negative cases in small-N research.3

By virtue of focusing on negative cases that resemble positive cases on certain potential causal variables, the advice is consistent with the Rule of Inclusion that we develop below.

By encouraging a focus on negative cases that resemble positive cases, they help small-N researchers control for background features and thereby achieve greater leverage for causal analysis. At the same time, however, the advice still assumes that all nonpositive cases are theoretically relevant or at least theoretically neutral, failing to note that serious problems may arise if irrelevant nonpositive cases are included in the analysis.

Positive–Irrelevant Boundary

It would seem unlikely that a boundary exists between positive observations and irrelevant ones. Indeed, as we shall see, the Possibility Principle holds that irrelevant observations are those where the positive outcome is impossible. However, the impossible can happen if an observation is mistakenly put into the irrelevant category but it in fact has a positive outcome. Thus, at the irrelevant–positive boundary we have a situation where the “impossible happens.” As we explore below, the impossible is much more likely to happen in research designs where the analyst selects cases without prior knowledge of their value on the dependent variable.

Scope Boundary

A well-known boundary involves the scope of a theory. In Figure 1, this boundary is represented by the box itself; all observations within the box are assumed to meet the scope conditions of the theory. Typically, scope conditions define cases as irrelevant when causal processes are not homogeneous due to the lack of certain specified characteristics. For example, Skocpol argues that the basic causal processes of social revolutions in states with colonial histories differ from those in noncolonial states, and her scope includes only the latter kind of cases. There might well be social revolutions outside this scope (i.e., in the area outside the box in Figure 1), but these cases are irrelevant to testing her theory.

IRRELEVANT CASES: WHY ARE THEY A PROBLEM?

A common reflex in statistical analysis is to consider all cases as relevant for testing theory. This reflex is grounded in the belief that excluding cases as irrelevant entails the loss of potentially helpful information. It finds philosophical support in the advice of Hempel (1945), who resolved the ravens paradox by arguing that all nonblack, nonraven things are relevant to confirming the proposition that all ravens are black. Likewise, it is consistent with an “all-cases” design in qualitative analysis, which advises researchers to sample from the entire population when testing hypotheses about necessary or sufficient causation (Seawright 2002).

What is wrong with the statistical reflex to consider all cases as relevant when testing theories? We argue that there are three fundamental problems. First, the assumption that all cases are relevant leads the researcher to waste time and resources by analyzing a huge number of cases that do not teach us anything because the outcome of interest was obviously impossible. For example, it is pointless for an investigator studying the causes of industrialization to spend energy on cases such as the precolonial Americas or contemporary Antarctica. Because industrialization is not possible in these cases, they do not help us test theories of industrialization. Consider research on the emergence of social democracy. Lipset's (1977) famous query—“Why no socialism in the United States?”— made sense because social democracy was possible during earlier periods of U.S. history. However, one could scarcely believe that we can also learn about the causes of social democracy by asking questions such as “Why no social democracy in contemporary Sierra Leone?” and “Why no social democracy during the Roman Empire?” In medical research, where research findings have obvious real-world implications and analysts are highly conscious of using resources in the most productive ways possible, it is common for scholars to focus on cases where the outcome of interest is possible and treat others as irrelevant. Social scientists could benefit by following this example.

Second, the inclusion of all cases will artificially inflate the number of observations that confirm a theory. In effect, this practice can make a false or weak theory appear much stronger than it really is. For example, consider the theory that most ravens are white. Although this theory will not be supported by black ravens, it will be confirmed by all nonraven, nonwhite things. Insofar as the number of confirming observations is orders of magnitude larger than the number of disconfirming observations, one could conclude that the theory is almost always supported by the data.4

Hempel (1965, 48) recognized this problem, and he suggested that some confirming observations may carry less weight than others when testing a theory (see also Earman 1992). This problem also motivated Popper (1968) to focus on disconfirming observations rather than confirming observations.

This issue underlies a recent debate between Seawright (2002) and his critics (Braumoeller and Goertz 2002; Clarke 2002). Seawright suggests that all cases in an “appropriately defined universe” are relevant to testing a proposition about causal sufficiency, even negative cases that lack the hypothesized sufficient cause.5

The debate applies equally to necessary and sufficient causes. We focus here on sufficient causes because of their close connection to theories that require negative cases.

He shows that the inclusion of all cases can substantially enhance statistical significance by increasing the number of confirming observations. By contrast, Clarke argues that including all cases will lead one to confirm a proposition through irrelevant observations, in much the same way that “most ravens are white” might be confirmed by observing yellow pencils and blue books. Braumoeller and Goertz's argument likewise suggests that, when testing a hypothesis about a sufficient cause, cases that lack both the cause and the outcome are irrelevant, since the hypothesis does not imply anything about the number or proportion of these cases that should be present.

We follow Seawright's critics in arguing that some cases that lack both the causes and the outcome of interest must be deemed irrelevant for tests of causal sufficiency. At the same time, however, we recognize that much of this debate depends on Seawright's stipulation of an appropriately defined universe. For example, if this universe is defined as only cases that are analytically useful, then the so-called “all-cases” design would exclude as irrelevant any case that lacks both the causes and the outcome of interest (i.e., potentially the majority of cases).

A third problem concerns the error that can be generated when irrelevant cases are treated as relevant. As noted above, selecting on the dependent variable in regression studies can bias results by overrepresenting positive cases in the sample (e.g., King, Keohane, and Verba 1994). By contrast, selecting irrelevant cases can lead one to include too many negative cases in the sample. A sample with too many negative cases can produce erroneous causal inferences, just as can a sample with too many positive cases (see Clark and Nordstrom 2003).

To illustrate this problem, it is useful to draw on a concrete example from the international relations literature. A central issue in this literature concerns the impact of power parity versus power preponderance on militarized dispute. Many argue that power parity leads to more conflict and war because both sides believe they have a chance to prevail. By contrast, power preponderance leads to less conflict because the weaker side knows it is weaker, allowing the two states to peacefully negotiate outcomes that roughly reflect their relative power.

The unit of analysis in this literature is the state dyad-year and the dependent variable is militarized dispute. For some scholars, there are no irrelevant cases: all dyads fall into the negative or positive sets. In contrast to this “all-dyad” approach, however, other scholars propose the use of only “politically relevant dyads,” which in practice are defined as (1) dyads consisting of one or two major powers or (2) any contiguous pair of states. These scholars argue that some dyads, e.g., Belgium–Burma, should not be included in the research design because militarized conflict is impossible; only states with opportunity (Most and Starr 1989) for conflict can be considered legitimate negative cases. Hence, the use of politically relevant dyads is an informal application of the Possibility Principle.

Table 1 shows what happens for a simple test of this hypothesis with the two different sets of negative cases. When the criterion of politically relevant dyad is applied, the number of dyad-years decreases dramatically, from over 650,000 dyads to only about 90,000 politically relevant dyads. With this different and smaller set of cases, the probability of disputes arising from situations of preponderance and parity also changes dramatically. In particular, when only politically relevant dyads are selected, the hypothesis that preponderance reduces the likelihood of militarized dispute relative to parity is strongly supported (i.e., an odds ratio of 2.7). By contrast, when all dyad-years are selected, there is little difference between preponderance and parity (i.e., an odds ratio of 1.25).

In sum, our inferences regarding the effects of power superiority on war depend quite significantly on how we define the population of negative cases. We can see why when we recognize that the irrelevant cases excluded through the Possibility Principle are usually not a random sample but rather will tend to have distinctive values on key causal variables. In this example, many politically irrelevant dyads are composed of noncontiguous minor powers (like Belize–Bolivia) that are more equal in power than is true of all dyads. These are also cases where militarized dispute is understood to be impossible. Hence, when the Possibility Principle is applied, many cases that exhibit both power parity and the absence of militarized are excluded as irrelevant. Concomitantly, this selection process increases the relative proportion of nondispute dyads with power preponderance. Since the proportion of cases marked by both power preponderance and no militarized dispute increases, power preponderance becomes more strongly associated with nondispute behavior.

For all of these reasons, the definition of the full population of relevant cases has large implications for theory testing and research findings. Yet the literature on sampling techniques often makes it appear as if the definition of the population can be treated as unproblematic and given. Consider case–control sampling methods when studying rare events. Here the analyst strictly differentiates between positive and negative observations and then selects all positive observations and a random sample of (perhaps matched) negative observations (King and Zeng 2001, 142; see also Goldstone et al. 2000). This approach simply assumes that the analyst has a good understanding of the full population of negative cases. In their discussion of militarized conflict among dyads of states, for instance, King and Zeng (2001) assume that determining the fraction of positive cases is “straightforward” because “the denominator, the population of countries or dyads, is easy to count” (144). By contrast, we think that determining the population size is quite problematic: it depends on how one defines a relevant dyad.

Likewise, scholars who have sounded alarm bells about the dangers of selection bias assume that the scholar is working with a well-defined larger population of relevant cases. Yet we believe that—unless the Possibility Principle is applied—the full population of cases may include many irrelevant observations. These irrelevant cases may be systematically different from the relevant cases on many independent variables. In addition, the inclusion of irrelevant cases will produce an explosive increase in the number of cases with zero values on the dependent variable, much as selecting on the dependent variable often leads to an overrepresentation of positive cases. Because samples selected from populations that include irrelevant cases have too many cases of zero on the dependent variable, one can say that failure to apply the Possibility Principle is a potential source of selection bias.6

The consequences of selection bias for qualitative research are sharply debated. For different views, compare Geddes 2003 and King, Keohane, and Verba 1994 with Collier and Mahoney 1996. See also Brady and Collier 2004.

THE POSSIBILITY PRINCIPLE

In this section, we more formally introduce and elaborate the Possibility Principle. Many qualitative researchers already have implicitly applied the principle in making and justifying their case selection decisions, and thus we are in many ways only formalizing a widely held intuition. Nevertheless, we argue that greater explicitness and rigor in applying the principle can improve the quality of research and help resolve debates about case selection in the social sciences.

Basic Rules

The Possibility Principle of negative case selection has the basic form:

Possibility Principle: Choose as negative cases those where the outcome of interest is possible.

Obviously, much depends on how we interpret the key concept of “possible,” which is used to draw the boundary between the negative and the irrelevant observations. We propose two rules for implementing this principle in qualitative analysis: a Rule of Inclusion and a Rule of Exclusion.

The Rule of Inclusion holds that an outcome should be seen as possible if at least one independent variable of the theory under investigation predicts its occurrence. This is true even if other independent variables predict its absence. Thus, the basic rule is:

Rule of Inclusion: Cases are relevant if their value on at least one independent variable is positively related to the outcome of interest.

We call this the Rule of Inclusion because it serves as a means of selecting observations into the population of relevant cases.

The Rule of Inclusion is applied in conjunction with the theory under investigation. In qualitative research, investigators usually develop parsimonious theories in which the number of independent variables is relatively limited. For example, five or fewer independent variables often constitute the core of the theory, whereas it is rare for more than seven or eight independent variables to be included. In this sense, in the context of qualitative research, a case that exhibits even one core independent variable that is hypothesized to be positively related to the outcome of interest should be considered within the domain of observations where the outcome is possible. In other kinds of research, theories may contain many more independent variables, and these variables may be seen as only weakly related to the outcome of interest. For these studies, the Rule of Inclusion could be adjusted such that the presence of more than one positively related independent variable is needed for a case to be included in the relevant category.

Many qualitative analysts have implicitly applied the Rule of Inclusion in their research. For example, Bennett (1999) uses this logic when selecting cases to test a theory of the causes of Soviet and Russian military intervention. He identifies five factors that provide “opportunities” for intervention, such as the presence of a pro-Soviet/Russian faction or a low level of U.S. threat. He then considers as relevant only those countries or territories where one or more of these factors provides the opportunity for intervention; he does not consider the vast majority of countries or territories, because these cases lack all opportunities and thus Soviet or Russian intervention is simply not possible.

In contrast to the Rule of Inclusion, the Rule of Exclusion provides a means of declaring an observation to be irrelevant and thus excluding it from analysis. Under this rule, a case is considered irrelevant if it possesses a value on a variable that is known from previous research to make the outcome of interest impossible. For example, in her study of the causes of genocide, Harff (2003) notes that almost all genocides (i.e., 36 of 37) occur during or immediately after political upheavals. Accordingly, she excludes cases like France and Canada that lack political upheaval when testing her theory of genocide. These politically stable cases have such a low probability of experiencing genocide that their inclusion would distort inferences about other cases where the outcome of interest is possible.

The Rule of Exclusion depends on the analyst's having good knowledge about one or more “eliminatory variables” that are important enough to remove a case from the domain of relevant observations all by themselves. These eliminatory variables may be necessary causes of the positive outcome of interest, or they may be sufficient causes of the negative outcome. It is not uncommon for multiple eliminatory variables to be present in a given case and, thus, for the zero value on the dependent variable to be overdetermined. For example, one can come up with many reasons why social revolution in United States in 1900 was theoretically impossible. Given that nonsocial revolution was overdetermined, it makes little sense to use the United States when testing theories of social revolution.

The Rule of Exclusion takes precedent over the Rule of Inclusion: eliminatory variables can lead an analyst to declare a case as irrelevant even if the case is considered relevant via the Rule of Inclusion. For example, one may have a theory of genocide that highlights ethnic divisions as a key independent variable. Under the Rule of Inclusion, contemporary Canada could therefore be considered a relevant case. However, under Harff's (2003) exclusion criterion, Canada is irrelevant because its value on the political upheaval variable eliminates it from the analysis. In short, then, the Rule of Exclusion has the following basic form:

Rule of Exclusion: Cases are irrelevant if their value on any eliminatory independent variable predicts the nonoccurrence of the outcome of interest. This rule takes precedent over the Rule of Inclusion.

As we explore below, the Rule of Exclusion is closely related to the use of scope conditions in comparative research.

Finally, it is crucial to call attention to the different ways in which the Rule of Inclusion and Rule of Exclusion draw on theory. The Rule of Inclusion is used in conjunction with the core independent variables of the theory under investigation. Hence, the application of this rule draws on a theory that has not yet been fully tested. By contrast, the Rule of Exclusion is used in conjunction with one or more independent variables that have already been tested and established as robust eliminatory variables. Hence, this rule does not use the variables of the theory under investigation to exclude cases. For example, when employing the Rule of Exclusion, Harff draws on established knowledge about the key causal role of political upheavals; by contrast, when employing the Rule of Inclusion, Bennett draws on several variables that are part of the theory to be tested.

Uses with Boolean Theories

To further investigate the Possibility Principle, it is helpful to consider some standard “Boolean” theories from qualitative research. We define these as theories that use logical ANDs and/or ORs to specify hypotheses.7

The logical OR is used in conjunction with Boolean addition. If any of the additive terms are present, then the outcome is also present. Thus, the logical OR is a means of specifying different paths to the same outcome or what is sometimes called equifinality (George and Bennett 2005) and multiple causation (Ragin 1987). By contrast, the logical AND is used in conjunction with Boolean multiplication. A product refers to the combination of causal conditions. Analyzing a Boolean product with the logical AND allows researchers to specify a combination of conditions that are jointly for sufficient for an outcome, or what is sometimes called conjunctural causation (see Ragin 1987).

Boolean theories can use dichotomous variables (Ragin 1987) or continuous ones (Braumoeller 2003) or fuzzy-set ones (Ragin 2000). Likewise, they can adopt either a probabilistic or a veristic understanding of causation. Furthermore, one can translate these theories into other mathematical frameworks; for example, the logical OR can be translated into the arithmetic “+” and the logical AND into the arithmetic “*.”

For illustrative purposes, Skocpol's (1979) States and Social Revolutions is a good example. The core theory is relatively straightforward: State breakdown and peasant revolt are individually necessary and jointly sufficient for social revolution (see Goertz and Mahoney 2005). Thus, Skocpol argues that

She claims that if state breakdown occurs at the same time as peasant revolt, then social revolution will occur (given her scope conditions; see below).

Here we have a very simple Boolean theory that uses the logical AND with two positively related causal variables. The Rule of Inclusion states that we should choose as negative cases those where either causal variable is present. Hence the set of negative cases consists of

Notice that we have replaced the AND of the theory of the positive cases with an OR to capture the full relevant population. Here we see a key rule for linking Boolean theories with the Possibility Principle:

Change the logical AND in Boolean theories of the positive outcome to the logical OR when selecting the population of relevant cases.

This procedure is a version of the Rule of Inclusion that we call the AND-to-OR Replacement Rule.

Many qualitative theories posit conjunctural causation in which there are multiple paths to a given outcome (Ragin 1987, 2000). Conjunction implies the use of the logical AND to connect independent variables. Again, because the logical AND makes reference to the positive outcome, a useful general rule when applying the Possibility Principle to conjunctural causation is to replace all ANDs in the theory with ORs. For example, a typical result from a preliminary Boolean analysis might look like

This theory could then be tested using other techniques (e.g., process tracing) and in light of new cases.8

Boolean algebra is a method of both theory formulation and theory testing. In theory formulation, the technique is used with an initial set of cases to arrive at a set of hypotheses. These hypotheses may then be evaluated with a broader array of cases during a subsequent phase of more explicit theory testing (Ragin 1987). Much the same is true of large-N, statistical research in practice: analysts conduct early tests to explore relationships among variables before arriving at a final theory that is formally tested.

At that point, one has enough information to apply the AND-to-OR Replacement Rule to arrive at

In this example, the researcher should sample all cases where at least one of the independent variables is present. While the researcher might not be confident about which combinations are sufficient for the positive outcome (i.e., equation [1]), knowledge about the basic causal factors is enough to select the negative cases.

Boolean results often include both the presence of some factors (indicated by capital letters) and the absence of others (signaled by lower-case letters). The question then arises about how the absence of a certain variable should be used to select cases with the Possibility Principle. The answer depends on what is meant by the “absence” of the variable. In some cases, the absence of a variable actually refers to the presence of a clear causal condition. For example, a Boolean analyst might code a variable for religion using two values: Protestant (i.e., P) and Catholic (i.e., p). In this case, one can argue that the absence of being Protestant (i.e., being Catholic) is a positive cause of the outcome. However, if the variable values correspond to simply Protestant and non-Protestant, there is no clear causal condition associated with the absence of the variable. In this case, where the absence of a variable is undertheorized and does not correspond to a clear positive category, the Possibility Principle cannot be easily applied.

To this point, we have examined Boolean theories that employ dichotomous independent variables. To consider how the Possibility Principle works with continuous independent variables, let us imagine a theory in which four independent variables are jointly sufficient for the positive outcome of interest. Further, let us assume that these variables are coded from zero to one, where values close to zero mean that a positive factor is absent.

How would the analyst differentiate negative cases from irrelevant cases in this kind of design? Drawing on Ragin's (2000) work on fuzzy-set analysis, we can formulate a general rule in two steps. First, if one is testing to determine whether variables coded from zero to one are jointly sufficient for an outcome, then one should apply the AND-to-OR Replacement Rule. In fuzzy-set analysis, the logical OR is implemented by taking the maximum value of the independent variables. For example, if the variable scores for a given case are .17, .33, .33, and .67, then the case receives an overall score of .67, since this is the highest value (maximum) of the independent variables. In short, there is no problem in applying the AND-to-OR rule with continuous variables: the OR is defined as the maximum.

As a second step, the analyst must decide and justify the exact threshold or cutoff point at which the outcome is considered possible. In practice, one often sets this threshold at a fairly high level (e.g., >.50) to ensure that at least one independent variable is clearly present in all cases. Under some circumstances, however, the analyst may be better served by intentionally setting the threshold at a lower level. This is especially true if the analyst has good reason to believe that the higher threshold will exclude too many cases as irrelevant. For example, if the theory used to select cases is known to be missing key variables for which data cannot be collected, the analyst might legitimately be concerned that some cases will be prematurely excluded as irrelevant. By adopting the lower threshold, however, one increases the number of negative cases and decreases the number of irrelevant cases, thereby guarding against this error. Overall, the rule for continuous independent variables can be stated as follows:

All cases whose maximum of the positively related independent variables is equal to or above the selection threshold should be included in the set of negative cases. Cases whose maximum does not meet the threshold are irrelevant.

We can again use Skocpol's (1979) States and Social Revolutions as a concrete example of this rule. To develop and test her theory, Skocpol considers three positive cases of social revolution (France 1787–1800, Russia 1917–1921, and China 1911–1949) and five negative cases (England 1640–1689, Prussia 1807–1814, Germany 1848–1850, Japan 1868–1873, and Russia 1905–1907). Elsewhere, we have summarized and evaluated her argument by coding the two main variables using fuzzy sets (see Table 2). In Table 2, columns 2 and 3 report the fuzzy-set values for the two independent variables—state breakdown and peasant revolt. Since Skocpol is interested in whether the combination of these two variables is sufficient for social revolution, we adopt the rule listed above and focus on the maximum value of the two variables to determine whether her cases are indeed relevant. This maximum value is reported in the fourth column; the final column states whether the case is positive (i.e., social revolution is present) or negative (i.e., social revolution is absent).

We believe that Skocpol implicitly used the Possibility Principle in identifying her negative cases. Again, the AND-to-OR Replacement Rule gives us: possible Social Revolution = state breakdown OR peasant revolt. With respect to Skocpol's work, this proposition means that the negative cases should include all observations where either a state breakdown or a peasant revolt is present (or both are present). As the fourth column (“maximum value”) in Table 2 suggests, at least one of the two major variables is significantly present in all five of the negative cases. If we assume a threshold of at least .50 as a basis for retaining cases, then all five of the negative cases are relevant following the rule introduced above. More generally, this interpretation means that relevant negative cases include all those country-periods when a causal factor is as much present as absent.9

Skocpol's (1979) description of her case selection is also consistent with the Possibility Principle: “I shall invoke negative cases for the purpose of validating various particular parts of the causal argument. In doing so, I shall always construct contrasts that maximize the similarities of the negative case(s) to the positive case(s) in every apparently relevant respect except the causal sequence that the contrast is supposed to validate” (37). This passage suggests that Skocpol selected negative cases that resembled positive cases in terms of certain causal factors but not others, which is congruent with the guidelines above.

SCOPE CONDITIONS AND THE POSSIBILITY PRINCIPLE

In this section, we consider scope conditions as an alternative method through which researchers may exclude cases as irrelevant. Whereas the Possibility Principle excludes cases in which the outcome is not theoretically possible, scope conditions exclude cases where theory suggests that causal patterns are not homogeneous. Here we spell out the implications of these different modes of case selection. We also consider several examples in which researchers purport to exclude cases through scope conditions but, in fact, appear to be implicitly using the Possibility Principle.

What Are Scope Conditions?

Scope conditions refer to the parameters within which a given theory is expected to be valid (Cohen 1989; Walker and Cohen 1985). The need for scope conditions grows out of the fact that social scientists rarely formulate universal propositions that hold across all times and places; rather, they formulate conditional propositions that apply to specific contexts.10

Ideally, researchers use scope conditions to identify general parameters that could exist in multiple times and places, not scope conditions that identity specific times and places themselves (Kiser 1996, 257; Walker and Cohen 1985, 291).

Cases that do not meet the scope conditions of a given theory are routinely considered irrelevant and are not used to evaluate that theory.

Typically, the methodological justification for imposing scope conditions involves the need to meet the standard of unit homogeneity (e.g., Bartels 1996; George and Bennett 2005; Collier and Mahoney 1996; Ragin 2000, 61–62; Zelditch 1971, 272–88).11

This concern is implicit in Kiser 1996 and Walker and Cohen 1985. These analysts mostly justify scope conditions on practical grounds, in particular, the failure of theories to apply to all times and places. They do not link the need for scope conditions with possibility ideas.

Units are homogeneous when a given change on an independent variable is expected to have the same net effect on the dependent variable across these units (cf. King, Keohane, and, Verba 1994, 91–93). Cases that fall outside scope conditions do not meet the demands of unit homogeneity and, in many kinds of research, are not considered relevant for testing the theory at hand.

Unit homogeneity is almost always a theoretical assumption, and thus scope conditions—like the Possibility Principle—are theory-laden. Although one may have good reasons for believing that particular scope conditions specify a domain of causal homogeneity, it is difficult to know for certain without actually examining cases outside this domain. If the theory underlying the scope conditions is weak, the researcher may inappropriately exclude certain homogeneous cases or inappropriately include certain cases that introduce unrecognized heterogeneity into the population. In turn, these failures can seriously jeopardize one's findings.12

One might argue that the Possibility Principle offers a less theory-laden basis for excluding cases than scope conditions. The theory underpinning the Possibility Principle is evaluated against the positive and negative cases that are selected. In this sense, there is some “check” on the validity of the theory underlying the Possibility Principle, even if this check is based on cases that were selected in light of the theory itself. By contrast, a theory of unit homogeneity usually is not tested; rather, it is an untested assumption that analysts accept on theoretical grounds alone.

Relationship to the Possibility Principle

The kinds of cases that are excluded using scope conditions and the Possibility Principle are not symmetrical. Scope conditions are designed to exclude any case—positive or negative—that does not meet the standard of unit homogeneity. By contrast, the Possibility Principle is designed to exclude nonpositive cases that fall within scope conditions but that, nevertheless, provide little useful information for causal inference.

The relationship between scope conditions and the Possibility Principle can be more formally specified with Boolean notation. Let us assume that an analyst has a theory in which three independent variables (A, B, C) are understood to be jointly sufficient for an outcome (one could assume any Boolean model here). To select cases to test this theory, the analyst applies the AND-to-OR Replacement Rule of the Possibility Principle and adds a separate term Z to represent scope conditions as follows:

The scope conditions (term Z) act as an eliminatory variable in the same way as discussed above for the Rule of Exclusion. That is, the absence of Z is sufficient to declare an observation to be irrelevant. To specify this idea, the logical AND is used to link the eliminatory variable with the core Boolean model. In this sense, the Rule of Exclusion and scope conditions are built around the logical AND, whereas the Rule of Inclusion draws on the logical OR.

In practice, researchers are not explicit about whether they exclude cases using scope conditions or the Possibility Principle. However, because these two techniques approach positive and negative cases differently, we can formulate a simple diagnostic rule of thumb:

If only nonpositive cases are excluded, then it is likely that the Possibility Principle is being used. If positive and nonpositive cases are excluded, then it is likely that scope conditions are being used.

For example, in her study of social revolutions, we know that Skocpol uses scope conditions because she excludes positive cases of social revolution like Cuba 1959. If she were exclusively using the Possibility Principle, she would have no basis for declaring positive cases where social revolution is obviously possible as irrelevant to her theory.

Scope Conditions or the Possibility Principle? Examples from the Literature

The extent to which cases are excluded as irrelevant through scope conditions versus the Possibility Principle will vary. However, because scope conditions are widely accepted as legitimate in social science research, whereas the Possibility Principle has not been formally discussed, analysts may state that they are excluding cases through scope conditions even if they are in fact applying the Possibility Principle.

A good example of this tendency comes from theories of welfare state development. This research has shown that the chances of having a welfare state among poor countries are approximately zero. For example, Hicks (1999) finds that poverty is sufficient for the absence of a welfare state (see also Huber and Stephens 2001, 370–71). This empirical finding is important in its own right. It also has clear implications for scholars who seek to explain welfare state development: the less-developed countries are not useful. Their inclusion in the population hinders our ability to understand why some wealthier countries develop welfare states but others do not. For example, whereas left-leaning governments are related to welfare state development among wealthy countries, the relationship is much weaker or nonexistent among all countries. Inclusion of the poor countries distorts results in ways that inhibit substantive understanding of welfare state development.

To avoid these problems, many analysts of welfare states include only OECD countries (see Amenta 2003 and Pierson 2000 for recent reviews). Typically, they justify the exclusion of poorer countries through the use of scope conditions. However, they exclude only negative cases, and we believe that they are really employing the Possibility Principle, not scope conditions. In particular, they use the Rule of Exclusion to eliminate countries that possess a condition sufficient for the absence of welfare state development—namely, poverty. Indeed, the finding that economic wealth is related to welfare state development among all countries but not among rich countries is what we might expect if all cases are homogeneous (i.e., if scope conditions do not apply).13

Why is this true? Because wealth is correlated with the dependent variable of interest (welfare state development), and a selection strategy that chooses only wealthy countries excludes many negative cases without welfare states. In this context, independent variables other than economic prosperity are likely to appear as especially important despite the existence of causal homogeneity (see Collier and Mahoney 1996).

The failure of analysts to be explicit about their use of the Possibility Principle also introduces confusion into case selection debates surrounding the literature that seeks to explain the spectacular growth rates of certain East Asian countries since the 1960s. In this field, scholars almost always focus on Korea and Taiwan as positive cases, and sometimes Hong Kong and Singapore as well. These successful cases are often contrasted with less successful developers in Latin America, especially Brazil and Mexico. Overall, the negative cases are not representative of all countries in the world but, rather, tend to be wealthier nations. One might therefore argue that case selection is systematically biased and that different results would appear if a more representative sample of cases is used. For example, Geddes (2003, 93–105) argues that studies of the NICs that select only cases with higher levels of economic development mischaracterize the effects of labor repression on growth.

Here we use the Possibility Principle to explore the argument that the literature on the NICs inappropriately restricts the domain to only economically successful cases. Although the theories that animate this literature are varied, several prominent analysts argue that the ability of countries to move from import-substitution industrialization (ISI) policies to export-oriented industrialization (EOI) policies before heavy industry was established produced the high growth rates (e.g., Gereffi 1991 and Haggard 1990). In this theory, the formula for success is the combination of early ISI policies (normally before the 1960s) to achieve light industrialization and subsequently the adoption of EOI policies to move toward heavy industrialization. Sequence and timing are important, since EOI policies without the early ISI policies are not believed to produce the economic development of interest.

According to the Possibility Principle, only cases in which exceptional growth is possible should be included when testing this theory. When the Rule of Inclusion is formally applied, the analyst selects those cases that adopt ISI policies during the light phase of industrialization as candidates for exceptional growth. Exceptional growth is considered impossible in countries that lack this condition as of the 1960s. Usually, countries without ISI by this time are characterized by nonindustrial forms of commodity exportation.

The more developed nations of Latin America such as Brazil and Mexico are appropriate negative cases, given that they engaged in ISI policies beginning in the 1930s and 1940s. However, most other countries of Latin America are irrelevant cases for the theory, since they were still oriented toward basic commodity exportation well into the 1960s. In fact, Argentina, Chile, and possibly one or two others are the only countries within Latin America that made clear-cut early moves toward ISI and could therefore be considered definitely relevant. Outside of Latin America, there are few countries that were characterized by ISI before the 1960s. For example, nearly all of sub-Saharan Africa would be excluded, as would most of South Asia. On the other hand, some countries—perhaps several in the Middle East such as Turkey, Syria, and Iraq—might be argued to have engaged in ISI during this period and, thus, could be included as negative cases (see Waldner 1999).

In short, the small number of cases evaluated in this literature appears to come close to the full population of cases for which the theory is relevant. Hence, we believe that Geddes (2003) is mistaken to characterize this literature as inappropriately restricting the scope of analysis. Given the actual theory under investigation in much of this literature, exceptional growth is impossible in most countries, and hence the majority of potential negative cases can legitimately be excluded.

WHEN THE IMPOSSIBLE HAPPENS

The impossible happens when an observation is put into the irrelevant category but it in fact has a positive outcome. Although the impossible is unlikely to occur in small-N research, it is very likely to happen in large-N research. Here we briefly consider this problem and the lessons it raises for case selection procedures more generally.

Impossible-but-Happens Cases in Quantitative Research

A good example of the impossible happening in quantitative research is provided by the literature on militarized international disputes at the dyadic level (Jones, Bremer, and Singer 1996; Lemke and Reed 2001). As noted above, many scholars have argued that the negative cases should not consist of all possible dyads but, rather, should include only “politically relevant dyads.”

Unfortunately, militarized disputes do occur between politically irrelevant dyads and, in fact, constitute about 10% to 20% of all the positive cases. These “impossible-but-happens” cases are selected in part because of the weak theory: only two variables— contiguity and power—determine whether a given dyad qualifies as relevant. The full range of independent variables that affect international disputes is actually much larger, and this larger range of variables should be used for case selection purposes. Even with a stronger theory, however, it is still likely that some impossible-but-happens cases will be present.

Scholars in the quantitative tradition often seek to allow as many observations as possible to be relevant for theory testing. This approach helps guard against impossible-but-happens cases by reducing the number of cases that are considered impossible. Yet the approach can also inflate the pool of irrelevant cases that are mistakenly considered relevant, much like reducing Type I error can increase Type II error. The Possibility Principle suggests that analysts must avoid blindly maximizing the number of cases included in analysis and instead carefully weigh the costs and benefits of inclusive versus exclusive approaches to case selection.

Alternative Applications of the Possibility Principle

In our discussion of qualitative research, we have suggested that case selection proceed according to the following steps: (1) choose positive cases, (2) use the Possibility Principle to choose the negative cases from the remaining population, and (3) treat the union of these two sets as the population of interest. The quantitative literature on international conflict suggests a different procedure: (1) use the Possibility Principle to distinguish relevant and irrelevant cases, (2) treat the relevant cases as the whole population of interest, and (3) select the positive cases as a subset of this population.

The key difference between these two procedures is how they deal with the impossible cases. In the first procedure, positive cases are not put into the irrelevant category, since the Possibility Principle is used to classify only nonpositive cases as irrelevant. Hence, assuming the outcome is not incorrectly coded, the first procedure does not allow the impossible to happen. In the second procedure, however, the Possibility Principle classifies cases as relevant or irrelevant without regard for their value on the dependent variable. As a result, positive cases can be coded as irrelevant, such that the impossible happens.

We do not feel that one procedure is conclusively better than the other. The first procedure reflects the realities of small-N research, where the positive instances are well-defined, few in number, and thus will not likely be excluded from the analysis. The second procedure reflects the realities of large-N research, where it is a good practice to declare cases as relevant or irrelevant without reference to their value on the dependent variable. Our more general point is simply to take note of the fact that there are two ways to use the Possibility Principle: (1) use it just to select the negative cases, or (2) use it to select all cases (positive and negative) that are included in the analysis.

A GEOMETRIC INTERPRETATION OF THE POSSIBILITY PRINCIPLE

Many of the fundamental case selection issues raised by the Possibility Principle can be illustrated through a geometric interpretation that graphically situates negative cases in relationship to other types of cases. Figure 2 offers this geometric interpretation. The cube in the figure is constituted by three dimensions: X1, X2, and Y. The representation assumes that the theory under investigation takes the form of Skocpol's argument, that is, two independent variables that are individually necessary and jointly sufficient for the dependent variable. Furthermore, it assumes that no eliminatory variables are applied in conjunction with the Rule of Exclusion. To add substantive content to the representation, the two X-axes are labeled state breakdown and peasant revolt, while the vertical Y-axis is social revolution. All axes are standardized into a [0,1] interval. The variables can thus be interpreted as fuzzy-set membership scores or can be considered regular regression variables that have undergone scale transformations to range from zero to one.

A Geometric Interpretation of the Possibility Principle: shape States and Social Revolutions (Skocpol 1979)

The cube is divided into eight zones that dichotomize variables at the .5 value. This dichotomous approach is adopted here for illustrative purposes; in real research, the decision about where to separate one zone from another must be driven by substantive and theoretical considerations. Furthermore, whereas the cube draws a sharp and clear separation line between the zones, the actual cutoff points from one zone to another will rarely be so stark. Rather, there typically will be a gray zone at the boundary between any two zones.

For our purposes, we are interested in five kinds of cases that occupy these eight zones: (1) positive, (2) negative, (3) irrelevant, (4) impossible-but-happens, and (5) disconfirming. Exemplary cases of these types can be found near the corners of the cube. By contrast, as cases approach the center of the cube on one or more dimensions, they may enter the gray zone, and their membership in a given category may become ambiguous.

The prototypical positive cases are those where the relevant causes (the two X dimensions in Figure 2) and the outcome of interest (the Y dimension) are present. These observations are located in the right-back-top zone near the (1,1,1) corner, where all the variables—independent and dependent—have values greater than .50. In the case of Skocpol's theory, the successful revolution cases of France, Russia, and China would be found in this zone.

According to the Possibility Principle, one should select as negative cases only those observations where at least one independent variable predicts the outcome. Accordingly, prototypical negative cases are found in the right-front-bottom corner and the left-back-bottom corner (the latter is not visible in Figure 2), the two zones where one independent variable has a value greater than .5, while the other independent variable and the dependent variable have values less than .5. In Skocpol's theory, the negative cases of Japan and England would occupy the right-front-bottom corner (where state breakdown is present, but peasant revolt and social revolution are absent), while Russia 1905 and Germany would fall more into the left-back-bottom corner (where peasant revolt is present, but state breakdown and social revolution are absent).

The Possibility Principle states that the irrelevant cases are those with low values on all the independent variables, that is, where the max(Xi) is near zero. If the theory is correct, then values for Y should also be near zero for these observations. In practice, this means that all irrelevant cases will lie near the origin (i.e., the (0,0,0) point). Accordingly, we can define the threshold that separates the negative from the irrelevant cases as those that lie within Euclidean distance X from the origin.14

The Euclidean distance rule actually generates a sphere around the origin. By contrast, the implementation of the maximum in conjunction with the logical OR produces a cube. However, the spirit of the Euclidean distance and maximum rules is the same.

All other observations beyond this distance should be included in the analysis as relevant. We thus have an important geometric version of the Possibility Principle:

The irrelevant cases are those near the origin in the N-dimensional space of the positively related independent variables.

In Figure 2, the impossible-but-happens cases are those where Y has a value greater than .5 (i.e., Y occurs), but both X1 and X2 have values less than .5 (i.e., are absent). These cases appear in the left-front-top zone near the (0,0,1) corner. In the Skocpol example, these would be cases where both state breakdown and peasant revolt are absent, but social revolution nevertheless takes place.

There are two types of disconfirming cases for the kind of theory we are considering here. One type includes any observations in which Y does not occur (i.e., is closer to zero than to one) when it should (i.e., when X1 and X2 are both closer to one). These cases are found in the right-back-bottom zone near the (1,1,0) corner. These disconfirming observations would initially be selected as negative cases because they have a positive value on at least one independent variable but lack the outcome of interest. They would then be classified as disconfirming once it became apparent that their values across both independent variables predict the outcome, even though it does not occur. In Skocpol's theory, for example, disconfirming observations would be cases where both state breakdown and peasant revolt are present but social revolution is absent. In effect, the cases in this region disconfirm the hypothesis of (joint) sufficiency.

The other kind of disconfirming cases contradicts the hypothesis that each independent variable is individually necessary for the outcome. These cases have a positive value on the outcome, but only one of the two independent variables is present. We find these observations in the right-front-top zone near the (1,0,1) corner and the left-back-top zone near the (0,1,1) corner. In qualitative research, it may be common to discover these cases during the initial stages of theory formulation, but they are far less common by the later stages. For example, Skocpol is aware of all cases of social revolution within her scope conditions, and it would be unlikely for her to present a final theory that is clearly contradicted by even one of these cases.

The cube thus provides a nice visual summary of disconfirming observations for both necessary and sufficient condition hypotheses. Necessary condition hypotheses are disconfirmed by cases where the outcome takes place, which in the cube are located in the top half (Y>.5). Sufficient condition hypotheses are disconfirmed by cases where the outcome does not happen, and thus these cases are found in the bottom half of the cube (Y<.5).

The geometric interpretation is instructive for thinking about the distribution of cases in qualitative research. In terms of Figure 2, few cases will be situated in the top of the cube, because analysts tend to study outcomes that are only rarely present. Furthermore, cases in the top will generally fall into the positive case zone, because analysts formulate theories explicitly designed to account for the positive cases. Within the bottom half of the cube, where most cases are located, the irrelevant space may be the single most populated area. This is true insofar as qualitative researchers develop theories in which causal variables are only rarely present, such that many or most cases cannot be considered relevant via the Rule of Inclusion. Likewise, assuming that one is working with a good theory, very few or no cases will fall into the disconfirming zone, leaving the remaining cases in the bottom half within the negative observation zone.

This geometric interpretation of the Possibility Principle offers a solution to an important problem that arises in fuzzy-set analysis. The problem involves what to do with cases that are near the origin (i.e., cases near the (0,0,0) corner). As Ragin (2000, 250–51) notes, when testing whether variables are causally sufficient for an outcome, observations with a zero for all the independent variables will always satisfy causal sufficiency and thus artificially inflate the number of cases where the theory works (this dilemma is the ravens paradox mentioned above).15

Smithson (1987) discusses other more technical problems of fuzzy-set analyses when membership scores are zero or near zero.

The Possibility Principle solves this problem by eliminating all of these cases with the exception of any impossible-but-happens cases. Ragin similarly advises that one should not include cases with zero values across all independent variables when testing theories of causal sufficiency. However, his argument is based on practicality. By contrast, the Possibility Principle provides a coherent methodological rationale for excluding these problematic cases.

Figure 2 also allows us to think systematically about the trade-offs that arise in qualitative research. We can do this most easily by considering the implications of expanding or shrinking the cutoff points between zones. As presented, Figure 2 separates all zones at the .5 point, but this decision was made for illustrative purposes, and researchers could have good theoretical reasons for expanding one zone at the expense of another. If the threshold for the presence of the outcome was lowered from .5 to .25 (e.g., from social revolution to political revolution), the size of the positive cases would increase, while the size of the negative cases would decrease. With a theory like Skocpol's, this move would help the analyst avoid disconfirming observations (because this zone would be reduced in size), but it would increase the likelihood of encountering an impossible-but-happens case (because this zone is enlarged). Inevitably, to increase the size of one zone is to reduce one risk but increase another.

These trade-offs underscore the importance of making substantively and theoretically informed choices about where to draw the line when including or excluding cases vis-à-vis a given zone. The issue is not just where to draw the line for the dependent variable. Rather, different thresholds on independent variables can shrink or expand the size of any given zone, with major implications for theory testing.

TESTING THEORY WITH THE POSSIBILITY PRINCIPLE

To illustrate concretely the value-added of explicitly applying the Possibility Principle, we consider in this section how the principle could be used to retest Skocpol's (1979) States and Social Revolutions. We begin by noting that previous tests of this argument have lacked a clear rationale for choosing negative cases. We then apply the Possibility Principle within Skocpol's scope conditions to identify what may be the full set of observations relevant to testing the theory. Finally, we assess Skocpol's argument in light of this set.

Previous Tests of Skocpol's Theory

Many scholars have used new cases, especially Third World countries, to explore Skocpol's theory. In some instances, they draw on the evidence from these cases to directly test her theory. For example, Geddes (2003, 106–14) draws on evidence from nine Latin American countries to show that Skocpol's specific arguments about international warfare and state breakdown are not supported.16

Geddes (2003) uses correlational analysis to test Skocpol's argument about international pressure and revolution. She also briefly considers necessary causation. However, we have argued that Skocpol's claim involves equifinality in the context of a two-level model, for which these tests are not appropriate (see Goertz and Mahoney 2005).

Other scholars use Skocpol's work to build their own theories of social revolution, perhaps then testing these theories in conjunction with implicit applications of the Possibility Principle. For example, Foran's (1997) Boolean analysis of social revolution selects as negative cases only state-periods that have a positive value on at least one of his five major independent variables. Goodwin (2001) likewise selects as negative cases only state-periods where at least one key independent variable is present. More generally, scholars of social revolution rarely focus on negative cases where revolution appears to be impossible, such as modern Costa Rica or Korea.

Analysts often recognize that Skocpol's theory cannot be directly tested in light of Third World countries because these cases clearly violate her scope conditions. Indeed, Skocpol's scope is limited to politically ambitious agrarian states that have not experienced colonial domination (Skocpol 1979, 33–42, 287–90). She explicitly excludes cases in which the possibilities for revolution have been shaped by the legacies of colonialism, dependence in the international economy, and the rise of modern militaries differentiated from dominant classes. Hence, nearly all modern Third World countries are excluded by Skocpol's scope statement.

Selecting the Negative Cases

The first step in retesting Skocpol's argument involves identifying cases that fall within her scope. In addition to the cases analyzed in States and Social Revolutions, we believe that the following nine states meet Skocpol's scope conditions: the Austrian Empire (1804–66) and Austria–Hungary (1867–1918), the Dutch Republic (1579–1795), Mughal India (1556–1857), Spain (1492–1823), Portugal (1641–1822), Sweden (1523–1814), the Polish–Lithuanian Commonwealth (1569–1795), and the Ottoman Empire (1520–1922). Although we cannot claim that these nine cases represent all states relevant to Skocpol's theory, we can say with some confidence that these cases cannot be excluded as irrelevant on the grounds of falling outside of Skocpol's scope conditions.

Here we pose the following question: What specific periods in the histories of these nine new cases are relevant for testing her theory? To answer, we apply the Possibility Principle by considering a case as relevant if it has a positive value on at least one of Skocpol's two main causes (i.e., state breakdown and peasant revolt). Although we examine nine states, our actual unit of analysis is the state-period, in that we are looking for specific periods of time in the histories of these states that are relevant for testing Skocpol's theory. In terms of the outcome of social revolution, all of these cases are non-positive—i.e., none experienced an event transformative enough to meet Skocpol's definition of social revolution. Hence, our efforts focus on differentiating the negative state-periods from the irrelevant state-periods.

For the state breakdown variable, we consider periods relevant when international wars, state–elite conflict, or agrarian backwardness fostered large-scale political instability and the collapse of reigning governmental and bureaucratic structures. For the peasant revolt variable, we include cases where peasant rebellions against landlords and state agents encompassed broad regions of the state.17

Because we are not experts on most of these cases, it is possible that we have overlooked specific time periods when a causal variable was present, especially given the sparse data on peasant revolts. Thus, we emphasize that this exercise does not constitute the final word concerning the set of cases relevant to testing Skocpol's theory. Furthermore, it is likely that additional relevant negative cases could be generated by evaluating new time periods for the original cases analyzed in States and Social Revolutions. For example, given that peasant revolts were common in late eighteenth-century Russia, a broader array of country-years than 1905–7 and 1917–21 in Russia is almost certainly relevant.

As Table 3 suggests, we conclude that the following state-periods are relevant negative cases for testing Skocpol's theory: mid-nineteenth century Austria, early twentieth-century Austria–Hungary, late seventeenth- and early eighteenth-century India, early nineteenth-century Ottoman Empire, eighteenth-century Poland–Lithuania, early nineteenth-century Portugal, mid- to late seventeenth- and early nineteenth-century Spain, and early eighteenth-century Sweden. Although none of the nine original states are eliminated as completely irrelevant, the Possibility Principle greatly reduces the range of cases that are considered relevant—most periods in the histories of these states are irrelevant to Skocpol's theory. When a period is relevant, it corresponds to a situation of political instability, given that it was selected precisely because of the presence of state crisis or peasant revolt. Said differently, all periods of political stability are irrelevant.

Because only situations of political instability are selected, one might argue that the Possibility Principle leads to truncation on the dependent variable by restricting its range of variation. However, the alternative would be to much more severely jeopardize valid inferences by including a nearly infinite number of negative case observations. For example, every year of nonrevolution in Spain from the late fifteenth century to the early nineteenth century would become a negative case. This huge number of negative cases would make it inevitable that Skocpol finds a strong association between her causal factors and revolution. By contrast, the Possibility Principle focuses attention only on cases where social revolution is possible, avoiding all negative cases that are bound to confirm the theory.

Testing the Theory

Skocpol's full theory of social revolution has a complex two-level structure, and a complete test of the theory would consider causal claims at both levels (see Goertz and Mahoney 2005). Here we evaluate only her core argument that state breakdown and peasant revolt are individually necessary and jointly sufficient for a social revolution.

The evidence from the additional relevant cases in Table 3 is consistent with Skocpol's theory, with the exception of the Polish–Lithuanian Commonwealth in the mid-eighteenth century. Outside of this case, no territory simultaneously features state breakdown and peasant revolt. All of the territories had at least one major state breakdown, but many of them never witnessed large-scale peasant revolts. Hence, we find substantial support for Skocpol's theory from a consideration of several cases not originally analyzed by Skocpol.

The seemingly disconfirming case of mid-eighteenth-century Poland–Lithuania corresponds with the partition and eventually obliteration of this commonwealth by Russia, Prussia, and Austria. After the first partition was initiated by Russia in 1772, radical reformers in Poland moved to adopt a progressive constitution, and events unfolded in a manner that suggested a social revolution could be on the horizon. However, the threat of revolution caused the Polish nobility to call in occupation forces, and Russia, Prussia, and Austria dissolved the commonwealth by 1795. Thus, instead of social revolution, Poland–Lithuania experienced a loss of sovereignty. The extent to which this outcome should be seen as a disconfirmation of Skocpol's theory could be debated, though we believe it is difficult to hold Skocpol too accountable for the absence of social revolution in a political entity that ceased to exist.

Negative Case Selection and Causal Inference

One might legitimately raise the “so what” question: even though Skocpol did not include the whole population of negative cases, the addition of these cases really does not change how we view her theory since there are no unambiguously disconfirming observations. Yet it is only because of this kind of analysis that we can say that a survey of all the negative cases produces no clearly disconfirming cases. Certainly others, such as Geddes (1990, 2003), have proposed that the inclusion of a broader range of cases casts doubt on Skocpol's theory. By contrast, our survey finds no evidence that Skocpol selected only negative cases where her theory works; in fact, the inclusion of additional relevant cases may strengthen her argument.

However, our survey suggests at least one important respect in which Skocpol's theory should be viewed in a new light. Of Skocpol's two main causal factors, it is fair to say that most readers have focused attention on the state breakdown variable. For example, students of Skocpol who have developed their own major theories of social revolution have zeroed in on this variable (Goldstone 1990; Goodwin 2001). Yet our results suggest that, empirically speaking, the peasant revolt variable is the causally more important one.

Of the negative cases, the state breakdown variable is significantly more common than the peasant revolt variable. In fact, state breakdown appears in all of the negative cases that we analyze here as well as all of Skocpol's original negative cases, except Germany in 1848. By contrast, peasant revolt is absent from most of our new negative cases as well as from Japan and England in Skocpol's original analysis. Precisely because it is more difficult for peasants to stage large-scale revolts than for states to experience major crises, it is appropriate to view peasant revolts as the more important cause. While this might seem counterintuitive, it does make intuitive sense. For example, if a gas leak results in an explosion in the house of a smoker, one is inclined to think that the thousands of cigarettes that have been smoked are less important than the rare gas leak (see Honoré and Hart 1985 for an extensive analysis of this point). Alternatively, one can make the same point by thinking in terms of correlations. The correlation between state crisis and social revolution is much lower than that for peasant revolt and social revolution. If we include Skocpol's original cases with our new cases, there are 13 times when state breakdown occurs without social revolution but only six peasant revolts that do not lead to social revolutions.

In addition to providing a check for disconfirming observations, then, the selection of negative cases can significantly influence the relative importance we attribute to different causal factors, even in qualitative studies that employ Boolean theories. This observation serves to reinforce our point about the risks of including irrelevant cases. If we had included irrelevant cases from a region such as modern Latin America, it is possible that the peasant revolt variable would have been more common than the state breakdown variable, leading us to erroneously conclude that state breakdown is more important to Skocpol's theory than peasant revolt.

CONCLUSION

Qualitative researchers who study events such as revolutions, welfare state development, genocide, and sustained economic growth generally do not analyze negative cases where the outcome of interest is impossible. Rather, they tend to focus on negative cases where the outcome has a real possibility of occurring. In this paper, we have made explicit and formalized this common research practice. In doing so, we have created a new set of rules to guide case selection practices in qualitative research.

We have shown how the Possibility Principle addresses the fundamental but rarely discussed issues entailed in defining a relevant population of cases. The definition of the relevant population can affect findings about the significance, strength, and even direction of hypothesized causal relationships. The relevant population therefore should be carefully defined before one turns to specific techniques of case sampling. Regardless of the sampling procedure, when irrelevant cases are included in one's understanding of the population, one will overrepresent observations with a zero value on the dependent variable. Thus, our discussion has called attention to ways of avoiding the error that arises when the definition of the population generates too many negative cases, whereas most of the literature on case selection has focused on procedures for overcoming the bias generated from an insufficient number of negative cases.

More generally, the Possibility Principle offers a powerful tool for thinking about case selection in qualitative research. An analysis of this principle sharpens our understanding of how scholars use scope statements, and it provides a basis for formalizing the alternative logic that scholars follow when they exclude only negative cases from tests of their theories. Likewise, by representing the Possibility Principle visually, one can see the overall geography of case selection in qualitative research, including the distinction among positive, negative, irrelevant, impossible-but-happens, and disconfirming cases.

Just as Lewis Carroll's King had trouble seeing Nobody on the road, scholars have struggled to identify nonwar, nonrevolution, and the like. The Possibility Principle together with a theory of the positive outcome of interest allows analysts to identify the full range of negative cases relevant to testing their causal theories.

Both authors contributed equally to this article. We would like to thank the three anonymous reviewers for many useful suggestions. In addition, we thank Bear Braumoeller, John Gerring, Joe Hewitt, Jason Seawright, Geoffrey Surrette, and the students at the Annual Training Institute on Qualitative Research Methods 2003 for feedback on earlier versions. James Mahoney's research on this project is supported by the National Science Foundation under Grant 0093754. A “teaching” version of this article is available from the authors. It is a lightly edited version of the current article with exercises and answer key.

References

Amenta Edwin. 2003. “What We Know about the Development of Social Policy: Comparative and Historical Research in Comparative and Historical Perspective.” In Comparative Historical Analysis in the Social Sciences, ed. James Mahoney and Dietrich Rueschemeyer. New York and Cambridge: Cambridge University Press, 91– 130.
Bartels Larry M. 1996. “Pooling Disparate Observations.” American Journal of Political Science 40 (August): 905– 42.Google Scholar
Bennett Andrew. 1999. Condemned to Repetition? The Rise, Fall, and Reprise of Soviet-Russian Military Interventionism, 1973–1996. Cambridge, MA: MIT Press.
Brady Henry E., and David Collier, eds. 2004. Rethinking Social Inquiry: Diverse Tools, Shared Standards. Lanham, MD: Rowman and Littlefield.
Braumoeller Bear. 2003. “Causal Complexity and the Study of Politics.” Political Analysis 11 (Summer): 209– 33.Google Scholar
Braumoeller Bear, and Gary Goertz. 2000. “The Methodology of Necessary Conditions.” American Journal of Political Science 44 (October): 844– 58.Google Scholar
Braumoeller Bear, and Gary Goertz. 2002. “Watching Your Posterior: Comment on Seawright.” Political Analysis 10 (Spring): 198– 203.Google Scholar
Clark David H., and Timothy Nordstrom. 2003. “Risky Inference: Unobserved Treatment Effects in Conflict Studies.” International Studies Quarterly 47 (September): 417– 29.Google Scholar
Clarke Kevin A. 2002. “The Reverend and the Ravens: Comment on Seawright.” Political Analysis 10 (Spring): 194– 97.Google Scholar
Cohen Bernard P. 1989. Developing Sociological Knowledge: Theory and Method. Chicago: Nelson–Hall.
Collier David, and James Mahoney. 1996. “Insights and Pitfalls: Selection Bias in Qualitative Research.” World Politics 49 (October): 56– 91.Google Scholar
Dion Douglas. 1998. “Evidence and Inference in the Comparative Case Study.” Comparative Politics 30 (January): 127– 46.Google Scholar
Earman John. 1992. Bayes or Bust? A Critical Examination of Bayesian Confirmation Theory. Cambridge, MA: MIT Press.
Foran John. 1997. “The Comparative-Historical Sociology of Third World Social Revolutions: Why a Few Succeed, Why Most Fail.” In Theorizing Revolutions, ed. John Foran. London: Routledge, 227– 67.
Geddes Barbara. 1990. “How the Cases You Choose Affect the Answers You Get: Selection Bias in Comparative Politics.” In Political Analysis, Vol. 2, ed. James A. Stimson. Ann Arbor: University of Michigan Press, 131– 50.Google Scholar
Geddes Barbara. 2003. Paradigms and Sandcastles: Theory Building and Research Design in Comparative Politics. Ann Arbor: University of Michigan Press.
George Alexander L., and Andrew Bennett. 2005. Case Studies and Theory Development. Cambridge, MA: MIT Press, Forthcoming.
Gereffi Gary, ed. 1991. Manufacturing Miracles: Paths of Industrialization in Latin America and East Asia. Princeton, NJ: Princeton University Press.
Ghosn Faten, and Scott Bennett. 2003. Codebook for the Dyadic Militarized Interstate Data, Version 3.02. online: http://cow2.la. psu.ed.
Goertz Gary, and James Mahoney. 2005. “Two-Level Theories and Fuzzy-Set Analysis.” Sociological Methods and Research. Forthcoming.Google Scholar
Goldstone Jack A. 1991. Revolution and Rebellion in the Early Modern World. Berkeley: University of California Press.
Goldstone Jack A., et al. 2000. State Failure Task Force Report: Phase III Findings. University of Maryland.
Goodwin Jeff. 2001. No Other Way Out: States and Revolutionary Movements, 1945–1991. New York: Cambridge University Press.
Haggard Stephan. 1990. Pathways from the Periphery: The Politics of Growth in Newly Industrializing Countries. Ithaca, NY: Cornell University Press.
Harff Barbara. 2003. “No Lessons Learned from the Holocaust? Assessing Risks of Genocide and Political Mass Murder Since 1955.” American Political Science Review 97 (February): 57– 74.Google Scholar
Haydu Jeffrey. 1998. “Making Use of the Past: Time Periods as Cases to Compare and as Sequences of Problem Solving.” American Journal of Sociology 104 (September): 339– 71.Google Scholar
Hempel Carl G. 1945. “Studies in the Logic of Confirmation.” Mind 54 (213–14): 1–26, 97– 121.Google Scholar
Hempel Carl G. 1965. “Postscript (1964) on Confirmation.” In Aspects of Scientific Explanation and Other Essays in the Philosophy of Science, ed. Carl G. Hempel. New York: Free Press, 47– 51.
Hewitt J. Joseph, and Gary Goertz. 2004. “Choosing Populations: Opportunity, Willingness, Political Relevance and the Possibility Principle.” Manuscript, University of Missouri.
Hicks Alexander. 1999. Social Democracy and Welfare Capitalism: A Century of Income Security Policies. Ithaca, NY: Cornell University Press.
Honoré T., and H. L. A. Hart. 1985. Causation in the Law. 2nd ed. Oxford: Oxford University Press.
Huber Evelyne, and John D. Stephens. 2001. Development and Crisis of the Welfare State. Chicago: University of Chicago Press.
Jones Daniel, Stuart Bremer, and J. David Singer. 1996. “Militarized Interstate Disputes, 1816–1922: Rationale, Coding Rules, and Empirical Patterns.” Conflict Management and Peace Science 15 (Spring): 163– 213.Google Scholar
King Gary, Robert O. Keohane, and Sidney Verba. 1994. Designing Social Inquiry: Scientific Inference in Qualitative Research. Princeton, NJ: Princeton University Press.
King Gary, and Langche Zeng. 2001. “Logistic Regression in Rare Events Data.” Political Analysis 9 (Spring): 137– 63.Google Scholar
Kiser Edgar. 1996. “The Revival of Narrative in Historical Sociology: What Rational Choice Theory Can Contribute.” Politics and Society 24 (September): 249– 71.Google Scholar
Lemke Douglas and William Reed. 2001. “The Relevance of Politically Relevant Dyads.” Journal of Conflict Resolution 45 (February): 126– 44.Google Scholar
Lipset Seymour Martin. 1977. “Why No Socialism in the United States?” In Radicalism in the Contemporary Age, ed. Seweryn Bialer and Sophia Sluzar. Boulder, CO: Westview Press, 31– 149.
Most Benjamin A., and Harvey Starr. 1989. Inquiry, Logic, and International Politics. Columbia: University of South Carolina Press.
Pierson Paul. 2000. “Three Worlds of Welfare State Research.” Comparative Political Studies 33 (August): 791– 821.Google Scholar
Popper Karl R. 1968. The Logic of Scientific Discovery. London: Hutchinson.
Przeworski Adam and Henry Tuene. 1970. The Logic of Comparative Social Inquiry. New York: John Wiley.
Ragin Charles C. 1987. The Comparative Method: Moving Beyond Qualitative and Quantitative Strategies. Berkeley: University of California Press.
Ragin Charles C. 2000. Fuzzy-Set Social Science. Chicago: University of Chicago Press.
Seawright Jason. 2002. “Testing for Necessary and/or Sufficient Causation: Which Cases Are Relevant?” Political Analysis 10 (Spring): 178– 93.Google Scholar
Skocpol Theda. 1979. States and Social Revolutions: A Comparative Analysis of France, Russia, and China. Cambridge: Cambridge University Press.
Skocpol Theda. 1984. “Emerging Agendas and Recurrent Strategies in Historical Sociology.” In Vision and Method in Historical Sociology, ed. Theda Skocpol. Cambridge: Cambridge University Press, 356– 91.
Smithson Michael. 1987. Fuzzy Set Analysis for Behavioral and Social Sciences. New York: Springer Verlag.
Waldner David. 1999. State Building and Late Development. Ithaca, NY: Cornell University Press.
Walker Henry A. and Bernard P. Cohen. 1985. “Scope Statements: Imperatives for Evaluating Theories.” American Sociological Review 50 (June): 288– 301.Google Scholar
Zelditch Morris Jr. 1971. “Intelligible Comparisons.” In Comparative Methods in Sociology: Essays on Trends and Applications, ed. Ivan Vallier. Berkeley: University of California, 267– 307.
Figure 0

Case Selection: Positive, Negative, and Irrelevant Cases

Figure 1

Impact of Negative Cases on Causal Inference: Politically Relevant Dyads and Dispute Initiation

Figure 2

Fuzzy-Set Codes for Skocpol's Variables

Figure 3

A Geometric Interpretation of the Possibility Principle: shape States and Social Revolutions (Skocpol 1979)

Figure 4

Negative Cases for Skocpol's Theory

Submit a response

Comments

No Comments have been published for this article.