The Problem of Causal Inference

Consider, to begin, a situation in which some individuals in a diseased population are observed to have received a treatment while others have not (X). Assume that, subsequently, a researcher observes which individuals become healthy and which do not (Y). Let us further assume that each individual belongs to one of four unobserved “types,” defined by the potential effect of treatment on the individual:Footnote ²

• • adverse: Those who would get better if and only if they do not receive the treatment

• • beneficial: Those who would get better if and only if they do receive the treatment

• • chronic: Those who will remain sick whether or not they receive treatment

• • destined: Those who will get better whether or not they receive treatment

Throughout, we will use the letters a, b, c, and d to denote these causal types and the terms, λ _a , λ _b , λ _c , λ _d , to denote the relative share of these types in a population of interest.

These types differ in their “potential outcomes”—that is on what outcomes, Y, they would take on under alternative treatment conditions, X (Rubin Reference Rubin1974). More formally, we let Y(x) denote a case or type’s potential outcome when X = x. Thus, the potential outcomes are Y(0) = 1, Y(1) = 0 for type a; Y(0) = 0, Y(1) = 1 for type b; Y(0) = 0, Y(1) = 0 for type c; and Y(0) = 1, Y(1) = 1 for type d. These potential outcomes are illustrated in Table 1.

TABLE 1.

Potential Outcomes and Causal Types with a Binary Causal and Binary Outcome Variable

Note: What would happen to each of four possible types of cases if they were or were not treated?

Shifting to the social world, consider the effect of economic crisis on the collapse of an authoritarian regime, where “collapse” is understood as the positive outcome (Y = 1). An a case is one in which crisis, if it occurs, prevents an authoritarian regime from collapsing; in a b case, economic crisis, if it occurs, generates authoritarian collapse in a country that would otherwise have remained authoritarian; a c case is a regime that will not collapse with or without economic crisis; and a d case is one that will collapse with or without crisis.

The treatment effect for a case is defined as the difference in potential outcomes for that case between the treatment and control conditions, Y(1) − Y(0). The well-known fundamental problem of causal inference is that, for any given case, it is only possible to observe Y(1) or Y(0). Thus, for no case is it possible to observe the difference between these two quantities and, hence, the treatment effect. Put differently, it is impossible to directly observe the type of an individual case.

Table 2 displays the ambiguity that we face about the type of a case given an observation of X and Y for that case.

TABLE 2.

The Fundamental Problem of Type Ambiguity

Note: Conditional on X and Y values, each unit is one of two possible causal types.

Importantly, as seen in the table, the ambiguity always involves one element of a, b and one element of c, d. To return to our regime-change example, if we observe an authoritarian country that has experienced crisis and has subsequently democratized, we do not know whether the regime collapsed because of the economic crisis (and is, thus, of type b) or would have collapsed in any case (type d). Similarly, if we observe a country that does not experience economic crisis and does not collapse, we do not know whether the regime would have collapsed in case of crisis (and is, thus, of type b) or would have survived intact in any case (and is, thus, of type c). The table also indicates, however, that we can rule out either a or b and either c or d based solely on the observation of values on X and Y. For instance, a regime that collapsed after economic crisis is with certainty neither an a nor a c type.

Identifying the type of a given case is to make a case-level causal claim for the case in question. Population-level causal claims, on the other hand, are claims about the distribution of types in the population—that, is the quantities λ _a , λ _b , λ _c , and λ _d . A quantity of particular interest is thus the value λ _b − λ _a , which is the average causal effect for the population.

We turn next to describe, and formalize in Bayesian terms, strategies used in qualitative and quantitative analyses to support case- and population-level causal claims, respectively.

The Process Tracing Approach

While process tracing can be put to different purposes, we focus here on process tracing as an approach that draws causal inferences—about whether X caused Y or how X caused Y—for a given case, by examining within-case data that is believed to shed light on whether a given causal relationship exists or causal process is in operation.

Translated into our typological setup, we can summarize the standard view of process tracing as a method that inspects a case for evidence of that case’s type: that is, to determine whether or not the outcome in that case was generated by the case’s treatment status on a given X. We refer to the within-case evidence gathered during process tracing as clues in order to underline their probabilistic relationship to the causal relationship of interest. Readers familiar with Collier, Brady, and Seawright’s (Reference Collier, Brady, Seawright, Collier and Brady2010)) framework can usefully think of our “clues” as akin to causal process observations, although we highlight that there is no requirement that the clues be generated by the causal process per se. Process tracing can be understood as a search for clues that will be observed with some probability if the case is of a given causal type and that will be observed with some differing probability if the case is of a different causal type.

It is relatively straightforward to express the logic of process tracing in Bayesian terms, a step that will aid the integration of qualitative with quantitative causal inferences. As noted by others (e.g., Beach and Pedersen (Reference Beach and Pedersen2013), Bennett (Reference Bennett, Box-Steffensmeier, Brady and Collier2008), Rohlfing (Reference Rohlfing2012)), there is an evident connection between the use of evidence in process tracing and Bayesian inference. In the article’s Supplementary Materials (Sec. A), we provide a survey of recent accounts and applications of process tracing that follow a logic akin to that underlying our formalization.

In a Bayesian setting, we begin with a prior belief about the probability that a hypothesis is true. New data then allow us to form a posterior belief about the probability of the hypothesis.

Formally, we express Bayes’ rule as

H represents our hypothesis, which may consist of beliefs about one or more parameters of interest. $\mathcal {D}$ represents a particular realization of new data (e.g., a particular piece of evidence that we might observe). Thus, our posterior belief derives from three considerations. First, the “likelihood”: how likely are we to have observed these data if the hypothesis were true, $\Pr (\mathcal {D}|H$ )? Second, how likely were we to have observed these data regardless of whether the hypothesis is true or false, $\Pr (\mathcal {D})$ ? Our posterior belief is further conditioned by the strength of our prior level of confidence in the hypothesis, Pr(H). The greater the prior likelihood that our hypothesis is true, the greater the chance that new data consistent with the hypothesis have in fact been generated by a state of the world implied by the hypothesis.

In formalizing Bayesian process tracing, we start with a very simple setup, which we then elaborate. To return to our running example, suppose that we already have X, Y data on one authoritarian regime: we know that it suffered economic crisis (X = 1) and collapsed (Y = 1). We want to know if X caused Y. We answer the question by collecting one or more clues that we believe are related to the case-level causal effect of X on Y. We use the variable K to register the outcome of the search for a clue (or collection of clues), with K = 1 indicating that a specific clue (or collection of clues) is searched for and found, and K = 0 indicating that the clue is searched for and not found.

Bayesian inference involves five steps: (a) defining our parameters, which are the key quantities of interest, (b) stating prior beliefs about the parameters of interest, (c) defining a likelihood function, (d) assessing the probability of the data, and (e) the application of Bayes’ rule. We discuss each of these steps, in the context of process tracing, in turn.

Parameters. For an X = Y = 1 case, the inferential challenge is to determine whether the regime collapsed because of the crisis (the case is a b type) or whether it would have collapsed even without it (d type). The parameter of interest is thus the causal type. Let j ∈ {a, b, c, d} refer to the type of an individual case. Our hypothesis, in this initial setup, consists of a belief about j for the case under examination: specifically whether the case is a b type (j = b).Footnote ³

Prior. We then assign a prior degree of confidence to the hypothesis (Pr(H)). This is, here, our prior belief that an authoritarian regime that has experienced economic crisis is a b. For now, we express this belief as a prior point probability.

Likelihood. We next indicate the likelihood, Pr(K = 1|H). This is the probability of observing the clue, when we look for it in our case, if the hypothesis is true—i.e., here, if the case is a b type. We thus require beliefs relating clues to causal types.

The key feature of a clue is that the probability of observing the clue is believed by the researcher to be a function of the case’s causal type. For the present example, we will need two such probabilities: we let ϕ _b denote the probability of observing the clue for a case of b type (Pr(K = 1|j = b)), and ϕ _d the probability of observing the clue for a case of d type (Pr(K = 1|j = d)). We note that, for the hypothesis that the case is a b, ϕ _b corresponds to Van Evera (Reference Van Evera1997)’s concept of “certainty.”Footnote ⁴ The key idea in process tracing is that the difference between the probability of the clue for a b type (ϕ _b ) and for a d type (ϕ _d ) provides the clue with “probative value,”—that is, the ability to generate learning about causal types.Footnote ⁵

In process tracing, analysts’ beliefs about the probabilities of observing clues for cases with different causal effects typically derive from theories of, or evidence about, the causal process connecting X and Y. Suppose we theorize that the mechanism through which economic crisis generates collapse runs via the regime’s diminished capacity to reward its supporters. A possible clue to the operation of a causal effect, then, might be the observation of diminishing rents flowing to regime supporters shortly after the crisis. Given our theory, this is a clue that we might believe to be highly probable for cases of type b that have experienced economic crisis (where the crisis in fact caused the collapse) but of moderate probability for cases of type d that have experienced crisis (where the collapse occurred for other reasons). This would imply a high value for ϕ _b and moderate value for ϕ _d .

Here the likelihood, Pr(K = 1|H), is simply ϕ _b .

Note that the likelihood takes account of known features of the data-gathering process. The likelihood given here can be thought of as following from an implicit assumption that the case is randomly sampled from a population of X = Y = 1 cases for which share ϕ _b of the b cases have clue K = 1 and share ϕ _d of the d cases have clue K = 1.

Probability of the data. This is the probability of observing the clue when we look for it in a case, regardless of its type, (Pr(K = 1)). More specifically, it is the probability of the clue in a treated case with a positive outcome. As such a case can only be a b or a d type, this probability can be calculated simply from ϕ _b and ϕ _d , together with our beliefs about how likely an X = 1, Y = 1 case is to be a b or a d type. This probability aligns (inversely) with Van Evera’s concept of “uniqueness.”

Inference. We can now apply Bayes’ rule to describe the learning that results from process tracing. If we observe the clue when we look for it in the case, then our posterior belief in the hypothesis that the case is of type b is

Suppose, in our running example, that we believe the probability of observing the clue for a treated b case is ϕ _b = 0.9 and for a treated d case is ϕ _d = 0.6, and that we have prior confidence of 0.5 that an X = 1, Y = 1 case is a b. We then get

Analogous reasoning follows for process tracing in cases with other X, Y values. For an X = 0, Y = 1 case, for instance, we need prior beliefs about whether the case is an a or a d type and beliefs about the probabilities ϕ _a and ϕ _d for the clue being sought.

The inferential leverage in process tracing thus comes from differences in the probability of observing K = 1 for different causal types. As should also be clear, the logic described here generalizes Van Evera’s familiar typology of tests by conceiving of the certainty and uniqueness of clues as lying along a continuum.

Van Evera’s four tests (“smoking gun,” “hoop,” “straw in the wind,” and “doubly decisive”) represent, in this sense, special cases—particular regions near the boundaries of a “probative-value space.” To illustrate, we represent the range of combinations of possible probabilities for ϕ _b and ϕ _d as a square in Figure 1 and mark the spaces inhabited by Van Evera’s tests. As can be seen, the type of test involved depends on both the relative and absolute magnitudes of ϕ _b and ϕ _d . Thus, a clue acts as a “smoking gun” for proposition “b” (the proposition that the case is a b type) if it is highly unlikely to be observed if proposition “b” is false, and more likely to be observed if the proposition is true (bottom left, above diagonal). A clue acts as a “hoop” test if it is highly likely to be found if “b” is true, but less likely to be found if it is false. Doubly decisive tests arise when a clue is very likely if “b” and very unlikely if not. It is also easy to imagine clues with probative qualities lying in the large space amidst these extremes.

FIGURE 1.

Probative-Value Space

Note: A mapping from the probability of observing a clue if a proposition “b” (i.e., that a case is a b type) is true (ϕ b ) or false (ϕ d ) to a generalization of the tests described in Van Evera (1997).

At the same time, the probative value of a test does not fully describe the learning that takes place upon observing evidence. Following Bayes’ rule, inferences also depend on our prior confidence in the hypothesis being tested. At very high or very low levels of prior confidence in a hypothesis, for instance, even highly probative evidence has minimal effect on posteriors; the greatest updating generally occurs when we start with moderate prior probabilities. Figure 5 in the Supplementary Materials (Sec. B) graphically illustrates the effect of prior confidence on learning.

We have so far described a very simple application of Bayesian logic. A further elaboration, however, can place process tracing in a more fully Bayesian setting, allowing for considerable gains in learning. Instead of treating clue probabilities (ϕ values) as fixed, we can treat them as parameters to be estimated from the data. In doing so, we allow the search for clues to provide leverage not only on a case’s type but also, given a belief about type, on the likelihood that a case of this type generates the clue. We can define our hypothesis as a vector, θ, that includes both the causal type of the case and the relevant ϕ values, e.g., ϕ _b , ϕ _d . We can then define our prior as a prior probability distribution p(θ) over θ.Footnote ⁶ We can thus express any prior uncertainty about the relationship between causal effects and clues. Our likelihood is then a function that maps each possible combination of type and the relevant ϕ values to the probability of observing the clue when we search for it, given those parameter values.

In this multiparameter approach, updating produces a joint posterior distribution over type and our ϕ values. Observing the clue will shift our posterior in favor of type and ϕ-value combinations that are more likely to produce the clue. In sum, and critical to what follows, we can simultaneously update beliefs about both the case’s type and the probabilities linking types to clues—learning both about causal effects and empirical assumptions. We provide further intuition on, and an illustration of, this elaboration in the Supplementary Materials (Sec. B).

The Correlational Approach

The correlational solution to the fundamental problem of causal inference is to focus on population-level effects. Rather than seeking to identify the types of particular cases, researchers exploit covariation across cases between the treatment and the outcome variables—i.e., dataset observations—in order to assess the average effect of treatment on outcomes for a population or sample of cases.

In the simplest, frequentist approach, under conditions described by Rubin (Reference Rubin1974), the average effect of a treatment may be estimated as the difference between the average outcome for those cases that received treatment and the average outcome for those cases that did not receive treatment.

Although this frequentist approach to estimating causal effects from correlational data is more familiar, recasting the strategy in Bayesian terms will facilitate the integration of within-case and between-case data that we undertake below. The general utility of the Bayesian framework for cross-case data analysis is already well appreciated, and so we simply review Bayesian correlational inference as applied to the binary setup.Footnote ⁷

Suppose, returning to our running example, that we are interested in determining the distribution of causal types in a population of authoritarian regimes. We again need to specify our parameters, priors, likelihood, and the probability of the data, and then draw our inference via the application of Bayes’ rule:

Parameters. One set of parameters to be estimated are our λ values: i.e., the proportion of the population of authoritarian regimes for which economic crisis would prevent collapse (λ _a ), the proportion for which it would cause collapse (λ _b ), and so on.

As in our multiparameter process-tracing setup, we also include a set of parameters capturing analytic assumptions. In correlational inference, these parameters relate to the process of assignment of types to treatment. Let π _j denote the (possibly unknown) probability that a case of type j is assigned to treatment (X = 1).Footnote ⁸ Thus, for instance, π _b indicates the likelihood that a country of type b (one susceptible to a regime-collapsing effect of crisis) has been “assigned” to experiencing economic crisis. Critically, the X, Y data pattern consistent with any given belief about the distribution of types depends on beliefs about these assignment probabilities.

We can now define our hypothesis as a vector, θ = (λ _a , λ _b , λ _c , λ _d , π _a , π _b , π _c , π _d ), that registers a possible set of values for the parameters over which we will update: type proportions in the population and assignment propensities by type.

Prior. We next need to assign a prior probability to θ. In the general case, we will do so by defining a prior probability distribution, p(θ), over possible values of the elements of θ.

Likelihood. Our data, $\mathcal {D}$ , consist of X and Y observations for a sample of cases. With a binary X and Y, there are four possible data realizations (combinations of X and Y values) for a given case. For a single case, it is straightforward to calculate an event probability w_xy —that is, the likelihood of observing the particular combination of X and Y given the type shares and assignment probabilities in θ. For instance,

More generally, let w_XY denote the vector of these event probabilities for each combination of X and Y values, conditional on θ. Further, let n_XY denote a vector containing the number of cases observed with each X, Y combination and n the total number of observed units. Under an assumption of independence (data are independently and identically distributed), the full likelihood is then given by the multinomial distribution:

We again assume here that cases are randomly drawn from the population, though more general functions can allow for more complex data gathering processes.

Probability of the data. We calculate the unconditional probability of the data, $\text{Pr}(\mathcal {D})$ , by integrating the likelihood function above over all parameter values, weighted by their prior probabilities.

Inference. After observing our data, $\mathcal {D}$ , we then form posterior beliefs over θ by direct application of Bayes’ rule, above:

This posterior distribution reflects our updated beliefs about which sets of parameter values are most likely, given the data. Critically, upon observing X and Y data, we simultaneously update beliefs about all parameters in θ: beliefs about causal effects (type shares) in the population and beliefs about the assignment propensities for cases of each type. We provide further detail and a simple illustration in the Supplementary Materials (Sec. C).

Baseline Model

Turning now to the formalization of the baseline model, we describe a complete BIQQ model with respect to parameters, priors, likelihood, and inference.

Extensions

The baseline model simplifies certain features of real-world research situations: treatments and outcomes are binary; neither measurement error nor spillovers occur; there is only one treatment of interest and one clue to examine; and our focus is on a single causal estimand (population-level treatment effects).

Importantly, however, the basic logic of integration underlying the BIQQ framework does not depend on any of these assumptions. In what follows, we suggestively indicate how BIQQ can be extended to capture a wider range of research situations and objectives.

Application 1: The Origin of Electoral Systems

We now apply the BIQQ framework to an issue that has received both quantitative and qualitative treatment in comparative politics: explaining variation in electoral systems across democracies. We use this application in part to illustrate the substantive effects that integration can have on causal conclusions. Equally important, the application demonstrates the dependence of conclusions on how many and which cases are selected for qualitative analysis.

Boix (Reference Boix1999) advances one influential theory of electoral-system choice as well as a quantitative test of the theory. In brief, Boix theorizes that the preferences of governing parties between plurality rules and proportional representation (PR) depend on the threat posed by challenger parties under the current rules. In particular, the presence of a strong opposition party together with coordination failure among the ruling parties will create strong incentives for governments to shift from plurality to PR. Boix’s central test of the theory focuses on a set of 22 interwar European cases. In this context—the extension of universal suffrage, enabling the rise of socialist challengers—the theory implies that ruling parties should have been most likely to shift to PR where the left was electorally strong and the right was fragmented. In his main model, Boix regresses the effective electoral threshold on the electoral strength of the left, the effective number of right parties, the interaction of the two, and a number of controls, and finds (consistent with the theory) a strong negative interaction between left strength and right division.

Kreuzer (Reference Kreuzer2010) then undertakes a qualitative analysis of Boix’s cases. He does so by collecting within-case information relating to three implications of Boix’s causal logic. For each case, Kreuzer asks (1) whether any proposal for PR followed a major suffrage expansion, (2) whether it was the ruling parties who initiated the move, and (3) whether the ruling parties were united in their support for PR. Kreuzer reports that the process tracing yields full support for the theory in 36 percent of the cases, partial support in 21 percent, and no support in the remaining cases.

What conclusion can be drawn from the adjoining of these two analyses? Kreuzer draws the reasonable inference that Boix’s theory offers an incomplete explanation of PR adoption. Yet it is not at all obvious how much this particular mix of confirmatory and disconfirmatory qualitative evidence should unsettle the quantitative findings. That is, it is not clear what these two separate analyses—one quantitative, the other qualitative—ought to imply for our beliefs about the effect of left threat on PR adoption in interwar Europe. The separated analyses also do not allow us to learn about other parameters of interest, such as the distribution of case types, the probative value of the clues, or the propensities of different types of cases to be assigned to treatment.

We illustrate how BIQQ can be applied to a somewhat simplified version of the problem, focusing on the average causal effect of left threat as our estimand of interest. Consistent with the setup introduced above, we treat all variables as binary, employing the dichotomized codings of Boix’s independent and dependent variables as provided in Table 5 of Kreuzer (Reference Kreuzer2010). Since the move to PR is theorized to depend both on the left being strong and the right being divided, we code a single independent variable, X, as 1 if and only if Kreuzer codes both conditions as present and 0 otherwise. We include in the analysis only those cases that started with single-member districts, thus dropping 2 of Boix’s cases. And we code Y as 1 if the country moved to a form of PR and 0 otherwise.Footnote ⁹ Further, we maintain the bivariate setup for the correlational analysis, setting aside the covariates in Boix’s analysis. We note that, even with these simplifications, the basic bivariate relation between X and Y remains consistent with Boix’s theory and strong (correlation = 0.47, p = 0.036).

Finally, we collapse Kreuzer’s three process-tracing tests into a single clue. We code the clue K = 1 for a given case if all three clues are present and K = 0 if one or more of the indicators is absent.

Table 4 summarizes the X, Y, and clue (K) data that enter into the analysis, indicating the number of cases with each combination of X, Y, and K values.

TABLE 4.

Mixed Data for Electoral Systems Analysis

Source: Based on Kreuzer (Reference Kreuzer2010).

Alongside the data given in Table 4, we need to supply priors or known values on the parameters of interest: the distribution of types (a, b, c, and d) in the population; the probability of each type being assigned to treatment; and the probative value of clues. In this exercise, we fix assignment propensities to 0.5 for a and b types but use a flat prior for the assignment of c and d types. We employ a flat Dirichlet distribution for the proportion of each type in the population.

Finally, while Kreuzer does not indicate the probative value he assumes for his clues, we assign probabilities of observing the clues for each type and treatment status by reasoning with Boix’s theoretical framework. The idea here is that these are values that a reader sympathetic to the theory might take as plausible expectations about the clue generation process. For this analysis we assume no uncertainty over the probative value of clues, fixing the eight ϕ parameter values. The specific ϕ values that we employ and our detailed reasoning can be found in the Supplementary Materials (Sec. G.1). Among the more important ϕ probabilities here are those associated with b and d types for cases with X = 1 values. For cases with high left threat and a shift to PR, the inferential task is to determine whether they would have (d) or would not have (b) shifted to PR without left threat. The ϕ _b1 and ϕ _d1 values that we have chosen are such that the clue operates as a hoop test for the proposition that such a case is a b type. On the other hand, for cases with no left threat and no shift to PR (which could be b or c types), ϕ values are such that the search for Kreuzer’s clue will be minimally informative. This is because, regardless of type, the theory would not expect ruling parties to unanimously push for PR in the absence of left threat. Off the diagonal, in low-threat cases that shifted to PR, the ϕ probabilities make the clue a smoking gun for designation as a d type; and for high-threat cases that did not shift, the clue serves as a hoop test for designation as an a type.

We present the results of the analysis in Figure 2. The figure displays the estimated causal effects for a range of possible research strategies. All strategies examined use the correlational data for all 20 cases, but they differ in the number of cases (k) for which clues are sought. We imagine for the purpose of this analysis that researchers randomly select the set of cases for which they gather within-case data, giving rise to a distribution of posterior distributions for each design. The X axis indicates the number of cases for which clues are sought, while the Y axis reports the resulting distribution of the posterior expected value of the average causal effect. On the far left, where we employ only Boix’s correlational data and no within-case data, we estimate a posterior mean just over 0.3. On the far right we see the analysis where all cases are examined qualitatively, which yields a posterior mean just below 0.15. Between these extremes we see the distribution of results for each research design, representing the causal-effect estimates for 500 random samples of cases for each level of k.

FIGURE 2.

Distribution of Posterior Expected Values for Average Causal Effect under Alternative Research Designs: Electoral Systems Analysis

Note: Each point represents the mean of the posterior distribution for the average treatment effect from a research design in which a random k of 20 available cases is studied qualitatively. The black circle marks the mean estimate for each k; for each k, 95% of the simulations lie along the black line.

Three features of the graph are of particular interest. The results indicate, first, that the impact of Kreuzer’s full analysis, under our stipulated priors, is large: although it does not eliminate the effect, it cuts the estimated impact of left threat on electoral-system reform roughly in half. Second, we see that in designs in which clue data are not collected for all cases, the results depend substantially on which cases are selected for process tracing. Even with as many as 12 cases, there are samples that result in a higher estimate of the causal effect than is found in the pure correlational analysis. Third, we note how slowly the effect of process-tracing data cumulates as the qualitative sample size increases. If we collect clues on a sample of the size typical of much qualitative work—say, between one and four cases—the expected result moves us no more than about a quarter of the way toward the full 20-case finding. Moreover, with random sampling at least, the variance in estimates resulting from qualitative samples of this size is large. These findings are all the more striking given that the clue data are on the whole in substantial tension with Boix’s claims, that the clues have been assumed to have fairly high probative value, and that the correlational sample is itself quite small.

Application 2: Natural Resources and Conflict

Our second substantive application focuses on the relationship between natural resources and civil conflict. This application illustrates the use of multiple clues, shows how conclusions can be reported conditional on beliefs about priors, and highlights an important feature of integration: that even strongly probative within-case evidence collected for a small number of cases may make only a small contribution to inference when combined with a substantial amount of correlational data.

We base our analysis on Ross (Reference Ross2004), who undertakes qualitative analyses of 13 cases that feature both high levels of natural resources and the emergence of civil conflict. Ross draws his population of cases from a frame used in a quantitative study by Collier and Hoeffler (Reference Collier and Hoeffler2004). Ross’s study identifies all cases of civil wars that started or were ongoing between 1990 and 2000 for which “scholars, nongovernmental organizations, or UN agencies suggested that natural resource wealth, or natural resource dependence, influenced the war’s onset, duration, or casualty rate.”Footnote ¹⁰ In his analysis he seeks evidence of whether natural resources were associated with looting, grievance, or separatism; on examination of the cases, he also concludes that foreign intervention and evidence of resource-based rebel financing are also clues for a causal link between resources and conflict onset. Ultimately, Ross’s determination of whether natural resources have a causal effect is based on whether any of the above clues were observed in these cases. In the event, at least one such clue is found in 5 cases out of the 13 in which clues are sought.

Ross’s overall conclusion is that “in these thirteen conflicts, there is strong evidence that resource wealth has made conflict more likely to occur.” This conclusion is consistent with the claim in Collier and Hoeffler (Reference Collier and Hoeffler2004) that resources are causally linked to conflict, though Ross’s analysis does not support the interpretation favored by Collier and Hoeffler. For this analysis, we do not call into question any of Ross’s conclusions about these cases. We focus instead on assessing the inferences that one can make from these cases, in combination with the correlational data, about average causal effects in the population.

Note that Ross’s analysis shares many of the characteristics of the type of qualitative analysis described above. He focuses on a binary outcome variable—war onset. Although the underlying treatment variable—natural-resource dependence—is continuous, he dichotomizes the variable, considering natural resources to be either present or absent in each case. Most importantly, Ross’s analysis does not focus on variation in X and Y to make inferences on causal effects; in fact, only cases with X = Y = 1 are examined. Inferential leverage is based instead on additional within-case information—clues—and indeed ultimate conclusions are based on a single clue, albeit a complex one, for each case: the presence of one of a number of “subclues,” each of which is indicative of a possible mechanism connecting resources to war onset.

Qualitative analysts frequently seek to use findings from a sample of this kind to shed light on broader causal theories. How, then, do Ross’s findings in these 13 case studies—together with the correlational evidence from Collier and Hoeffler—add up to a set of population-level causal inferences?

To carry out this integration in the BIQQ framework, we need first to be able to identify the quantitative sample from which Ross draws his cases. For reasons that we explain in the Supplementary Materials (Sec. G.2), we identify this sample as all cases in which there was not a conflict ongoing in the 1990s. Surprisingly, in this subset of cases the positive relation between X and Y, identified in Collier and Hoeffler’s study, is not present; indeed there is a small negative correlation.

Second, we need to situate the cases in Ross’s qualitative analysis within this quantitative sample. An important feature of Ross’s analysis is that selection into the process-tracing sample depends not simply on values of X and Y but also on expert assessments of whether X and Y are causally linked.Footnote ¹¹ This implies nonrandom selection of cases into the process-tracing sample. In our framework, we can take account of this nonrandom selection by treating expert statements that natural resources caused a conflict as a first clue, K ₁: these statements can be conceptualized as evidence suggesting that a case is a b type rather than a d type. We can then conceive of the evidence uncovered in Ross’s own analysis as a second clue, K ₂, which is gathered only conditional upon observation of the first clue (and which may or may not reflect the same underlying information used to generate the first clue).

Employing this interpretation, Table 5 shows the distribution over X, Y, K ₁, K ₂ values used in this analysis. Note that there are no K ₂ data for cases with K ₁ = 0 (given Ross’s selection rule), and there are only K ₁ data for cases with X = Y = 1 (since Ross uses expert causal assessments only of those cases with resources that experienced wars). This nonrectangularity presents no special problem for the analysis, however.

TABLE 5.

Mixed Data for Civil Conflict Analysis

Note: Left panel shows X, Y data based on Collier and Hoeffler (Reference Collier and Hoeffler2004); right panel shows K ₁, K ₂ data for cases with X = 1, Y = 1, drawn from Ross (Reference Ross2004).

Alongside the data given in Table 5, we need to form priors about the probative value of clues and about the assignment process. For the purposes of this analysis, we adopt the positions taken implicitly by Collier and Hoeffler for the X, Y data: we assume that assignment probabilities are similar for all types—though we allow for uncertainty over these assignment probabilities. For the probative value of clues, we carry out the analysis under multiple sets of assumptions. First, we use optimistic assumptions most consistent with Ross’s discussion, treating K ₁ as a hoop test (providing entry to the sample) and K ₂ as doubly decisive. However, we also examine the sensitivity of the results to the possibility that either K ₁ or K ₂, or both, is uninformative (of low probative value).Footnote ¹² We note that the situation in which neither clue has probative value is equivalent to an analysis of X, Y data only. With respect to the distribution of types, we assume a flat prior (Dirichlet) over the proportions λ _a , λ _b , λ _c , and λ _d .

The first panel of Table 6 provides statistics from the prior distribution on λ _b − λ _a and π _c . With the flat prior over type shares, the expected value of λ _b − λ _a is 0 with a wide credibility interval. The second panel shows the posteriors from the mixed-method analysis under the assumption that the experts’ determinations were uninformative (low probative value). On this assumption, it is as if Ross had sampled at random from the set of X = Y = 1 cases. We see here that, if K ₂ is also uninformative, our mean estimate of λ _b − λ _a is negative, consistent with the negative correlation in the X, Y data. Though not reported in the table, our posterior also contains a positive association between beliefs about causal effects and beliefs about π _c (see Supplementary Materials (Sec. G.2)). This association reflects the fact that a weak correlation in the X, Y data is consistent with either of two states of the world: (1) no true effect and no confounding (equal treatment propensities across types) or (2) a true positive effect that is masked by confounding (c types being more likely to be assigned to treatment).

TABLE 6.

Priors and Posteriors: Civil Conflict Analysis

Note: Table provides statistics of the prior and posterior distribution over λ _b − λ _a and π _c for the civil conflict analysis. Top panel provides statistics on priors; middle panel provides statistics on posteriors assuming that expert assessments (K ₁) are uninformative; lower panel provides statistics on posteriors assuming that expert assessments are informative.

What happens if we treat the qualitative clues examined by Ross (K ₂) as highly informative about effects in the individual cases? We see the result in the second row of the second panel. Here our mean estimate of λ _b − λ _a hardly moves, rising to approximately zero, with a very wide credibility interval. Likewise, the posterior on π _c increases just marginally.

In the third panel, we now assume that expert assessments—the clue used to determine process-tracing sample selection—are highly informative. Our willingness to make such an assumption would, of course, depend on our beliefs about how such judgments were formed. Focusing on the second row, we see that, even if we assume both clues to be highly informative, the process-tracing yields virtually no change in posterior means or uncertainty relative to the estimates derived from purely correlational evidence.

This analysis illustrates some general features of the integration of qualitative and quantitative causal inferences. First, it highlights some of the informational demands that are required to perform formal inference from mixed data. Of particular importance here is the need to understand, and explicitly model, the criteria used to select cases for process tracing from the quantitative sample. Second, the analysis—especially in comparison to Application 1—highlights an important limit to the capacity of small-n work to shape population-level inferences. Recall that Ross finds causal-process evidence that natural resources were a driver of conflict in four of five cases on which process tracing was carried out. This finding stands in apparently sharp contrast to the X, Y pattern of a small negative correlation. However, even when we take Ross’s clues to be highly probative—treating causal conclusions about these five cases to be nearly certain—our posterior hardly shifts from the correlational finding. The result suggests that, at least for estimating population-level quantities, relative numbers matter: that in some circumstances even strong process-tracing evidence, gathered on a small number of cases, will have only a very modest impact on conclusions drawn from the quantitative analysis of a much larger sample. We turn now to a more general examination of the consequences of mixing quantitative and qualitative observations in differing proportions.

Varieties of mixing

What are the marginal gains from additional pieces of correlational and process-tracing evidence for the accuracy of causal estimates? Figure 3 displays the results, plotting the errors associated with different mixes of correlational and process data. Each dot represents a single possible research design, with the x axis charting the total the number of cases examined. For all cases, X and Y data are collected. The shading of the dots in each column then represents the proportion of cases for which process tracing is also carried out. An unshaded dot is a design in which only correlational data have been collected for all cases; a black dot is a design in which the process-tracing clue is sought in all cases; and shades of grey, as they darken, indicate process tracing for increasing shares of cases. For n ⩽ 4 we report results for all designs; for n > 4 we report only results when within case information is sought for all, half, or none of the cases.

FIGURE 3.

Expected Errors with Varying Mixes of Qualitative and Quantitative Data

Note: The figure displays the expected errors in the estimation of average treatment effects for designs in which X, Y data are sought in n studies (horizontal axis) and clue data are sought within k of these. The shading of dots indicates the proportion of cases for which within-case data are sought (white = none; black = all). For small sample sizes (n ∈ {1,2,3,4}), we show results for all designs (m ∈ {1,2,. . .,n}). For larger sample sizes, we show only designs with clues sought in zero, half, and all cases.

We see first from the graph that, as one would expect, moving from lower-n to higher-n designs reduces the expected error of estimates. Further, both adding a correlational case and doing process tracing on an additional case improve accuracy. The figure also suggests that there are diminishing marginal returns to both types of data: in particular, the grey point reflecting 50% process tracing is generally well below the midpoint of the white and black dots, and converges toward the black dot (100% process tracing) as sample size increases. Other, less obvious results include the following:

• • Qualitative and quantitative data can act as partial substitutes for assessing causal effects. We see, in the smaller sample sizes, that the marginal gains from adding an extra correlational case are lower when there is more within-case information on existing cases. Similarly, the marginal gains from gathering more within-case information are lower when there are more correlational cases (for example adding one case study when n = 1 has about the same effect as adding eight cases studies when n = 16).

• • The relative marginal gains from going wider and going deeper vary with the study design. Suppose that the costs of gathering X, Y data and gathering clue data were the same per case. Suppose further that we have an n of 2 and only correlational data. Then, for the case illustrated in Figure 3, if we have additional resources to invest, we do about as well adding a third case as we would do from gathering information within one of the two existing cases. However, at this point the tradeoff shifts. From this new situation we are better off gathering information within one of the three cases than we are seeking correlational data on a fourth case.

• • Optimal strategies might involve going deep in a subsample of cases only. Suppose again that the costs for gathering X, Y data and gathering K-type data were the same and, now, that researchers can gather four pieces of data of any type. The results in Figure 3 suggest that, for the priors chosen here, gathering X, Y data on three cases and K-type data on one produces more accurate estimates than either going maximally wide (gathering X, Y data on four cases) and at least as accurate an estimate as going maximally deep (gathering X, Y, K data on two cases).

Designs in Context

More generally, the optimal level of mixing ought to depend on context—on features of the research situation that affect the problem, and the available tools, of inference. In the next subsections, we report results from experiments in which we vary the researcher’s priors about (a) the probative value of clues, (b) heterogeneity of treatment effects, (c) uncertainty regarding assignment processes, and (d) uncertainty regarding the probative value of clues. In all cases we report the expected loss for the design in question, as given in Equation (4).