3.1 Hypotheses and priors

The first step in explicit Bayesian process tracing entails specifying the set of hypotheses we wish to consider, $\{H_{k}\}$ . Whereas most treatments compare a single working hypothesis $H$ against its logical negation, ${\sim}H$ , we advocate identifying one or more concrete rival hypotheses. This approach is critical in social science, because ${\sim}H$ generally will not be a well-defined proposition— $H$ could fail to hold in an essentially infinite number of ways. The more specific the hypothesis is, the more possibilities are embodied in ${\sim}H$ . Directly assessing likelihoods of the form $P(E\mid \sim HI)$ will be practically impossible if we have not first contemplated what concrete possibilities ${\sim}H$ might actually entail. Suppose $H_{A}=$  Suspect A killed the victim, and we have a clue $E=$  A’s glove was found at the crime scene. If the only plausible alternative hypothesis is $H_{B}=\sim H_{A}=$  Suspect B, A’s bitter ex-spouse, killed the victim and framed A, then $P(E\mid \sim H_{A}I)$ could be fairly high, assuming the ex-spouse still had some access to $A$ ’s personal effects. However, if the only plausible alternative hypothesis is $H_{C}=\sim H_{A}=$  Suspect C, a total stranger to A, killed the victim, then $P(E\mid \sim H_{A}I)$ might be quite low. If both $H_{B}$ and $H_{C}$ are considered plausible alternatives, the only sensible way to proceed is to evaluate the likelihood conditional on each hypothesis separately, $P(E\mid H_{i}I)$ . We can then calculate posterior odds ratios for $H_{A}$ vs. $H_{B}$ and $H_{A}$ vs. $H_{C}$ , or we can use Bayes’ rule to obtain the posterior probability on each of the three hypotheses and then calculate $P({\sim}H_{A}\mid EI)=1-P(H_{A}\mid EI)$ , keeping in mind that our background information includes the assumption that ${\sim}H_{A}=H_{B}+H_{C}$ .

Whenever possible, mutually exclusive hypotheses are preferable. In principal, we can always take a set of nonrival hypotheses and construct a set of mutually exclusive rivals. For example, consider two factors that could motivate presidents to violate protectionist policy mandates: a desire to represent voters’ best interests, and a desire to seek rents associated with neoliberalism (Stokes Reference Stokes2001). If we think both factors may contribute, we could delineate five rivals: $H_{1}=$  predominantly representation, $H_{2}=$  both but mostly representation, $H_{3}=$  both in relatively equal measure, $H_{4}=$  both but mostly rent-seeking, $H_{5}=$  predominantly rent-seeking. Strictly speaking, ensuring that these possibilities are mutually exclusive requires greater precision—what exactly do we mean by “predominantly” vs. “mostly” vs. “relatively equal?” This specification issue can pose challenges in explicit Bayesian process tracing.Footnote ³ In practice, it is important to specify hypotheses as carefully as possible and to explicitly include the assumption that they are mutually exclusive and exhaustive as part of the background information. If new evidence suggests that a more complex hypothesis would provide the best explanation, we should incorporate it in $\{H_{k}\}$ and redo the analysis.

Once we have specified rival hypotheses, we must assign prior probabilities. In principle, we advocate starting from an initial state of maximal ignorance, $I_{0}$ , and placing equal prior probabilities on each hypothesis,Footnote ⁴ in accord with the “principle of indifference” or Laplace’s principle of “insufficient reason,” (Gregory Reference Gregory2005, pp. 37–38; Jaynes Reference Jaynes2003, pp. 40–41). We then build up via Bayes’ rule from $I_{0}$ to the actual prior state of knowledge $I$ . In practice, especially for social science, systematically incorporating all of our background information is infeasible. There is simply too much such information to exhaustively list, setting aside the challenges of formally applying Bayes’ rule to each such piece of information. Given this reality, there are several reasonable options for proceeding. First, we can specify priors that aim to reflect our background information as best as possible, recognizing that the logical Bayesian approach is aspirational and some subjectivity will inevitably enter in practice. Second, we can simply use equal priors, which avoids biasing the initial assessment in favor of any of the hypotheses. Third, we can report likelihood ratios instead of posterior probabilities and allow readers to supply their own priors. If the evidence is strong, scholars may converge on a single hypothesis even if they start from different priors. If the evidence does not strongly favor a single hypothesis, scholars may at least be able to agree on the direction in which their beliefs should be shifted. Another option entails carrying out the analysis starting from several different prior probability distributions, which allows us to assess how sensitive our conclusions are to these initial choices.

Leading literature on Bayesian process tracing does not always follow these guidelines. Consider Bennett’s (Reference Bennett, Bennett and Checkel2015) discussion of Tannenwald’s (Reference Tannenwald2007) research on the nonuse of nuclear weapons in the postwar period. He focuses on Tannenwald’s three principle alternative hypotheses, which we denote $H_{D}=$  deterrence, $H_{M}=$  lack of military utility, and $H_{T}=$  norms, in the form of a “nuclear taboo.” Bennett (Reference Bennett, Bennett and Checkel2015, p. 277) observes that these hypotheses “at first glance seem equally plausible.” In accord with this assessment, which corresponds to the indifference principle, we should use equal prior probabilities of 1/3 for each hypothesis. However, Bennett (Reference Bennett, Bennett and Checkel2015, p. 278) instead chooses a prior of 40 percent for $H_{T}$ and 60 percent for ${\sim}H_{T}(=H_{D}+H_{M})$ . Rohlfing (Reference Rohlfing2013, pp. 13–16) in contrast constructs priors through a process that entails identifying one working hypothesis, assigning a preliminary prior probability of 50 percent, discovering a rival hypotheses by exploring the literature, and then reducing the prior probability on the working hypothesis by what appears to be an arbitrary amount. After two iterations corresponding to the discovery of two alternative hypotheses, he ultimately gives the working hypothesis a prior of 30 percent. Instead, best practice mandates that we should state each hypothesis from the outset, before assigning priors. When comparing three mutually exclusive hypotheses, we could then assign equal prior probabilities of 1/3, or we could choose values that aim to reflect our background information, with an explanation of why we favor some hypotheses over others.

Reiterating the critical points, before assigning priors, we must elaborate a clearly articulated set of hypotheses that we assume to be mutually exclusive and exhaustive. We should then assign a probability to each hypothesis, rather than considering only the working hypothesis and its logical negation, which implicitly contains all of the rivals. If we discover or devise a new hypothesis later on, we must start the problem over and reassign priors.

3.2 Evidence

We take a broad, common-sense view of what constitutes evidence in process tracing—any relevant observation or information (beyond our background knowledge) that bears on the truth of our hypotheses. Evidence often contains information about timing and sequencing, actors’ goals and intentions, and other aspects of causal mechanisms, as obtained from a wide range of sources including interviews, archives, media records, and secondary literature. Bayesianism does not face any of the restrictions that limit frequentist statistics to analyzing only replicable observations or data generated from some stochastic process, such as random sampling. The type of evidence—regardless of how distinctions are delimited—does not matter for the fundamental logic of Bayesian analysis, because we are allowed to evaluate the probability of any proposition.Footnote ⁵ For example, the evidence we evaluate in our empirical example includes not only “within-case” observations about the causal process driving Chile’s 2005 tax reform, but also “cross-case” evidence from previous tax reform episodes that bears on the hypotheses for explaining the 2005 reform (see $E_{1}$ , Section 4). We view classification of evidence as superfluous unless it helps us evaluate likelihoods, because the evidence enters Bayesian calculations only through likelihoods, regardless of its origin or form.

In a Bayesian framework, attempting to provide a definition of what constitutes “good” evidence is largely beside the point.Footnote ⁶ Evidence can be more or less informative, as determined by the likelihood ratio, but probative value—or “weight of evidence”Footnote ⁷ —falls on a continuum. Moreover, what matters is how strongly the total body of evidence discriminates between rival hypotheses. For example, multiple pieces of information that each yield a very small weight of evidence together with one piece of information that produces a large weight of evidence may give an overall inference of essentially the same strength as a single highly probative piece of information, or two pieces of information with intermediate weights of evidence.  While we would ideally like every piece of evidence we gather to be highly probative, in social science we must often make inferences based on the accumulated weight of evidence from many clues, none of which is strongly decisive.  On the other hand, we sometimes do encounter a highly decisive piece of evidence, in which case we need not spend time systematically assessing other marginally probative pieces of evidence, because the latter will not affect the overall inference.

Likewise, there is no general prescription for designating what constitutes a distinct “piece” of evidence. We are free to disaggregate the overall body of evidence $E$ into components $E_{1}$ – $E_{N}$ as finely or as coarsely as we see fit, with the goal of parsing information at a level that facilitates reasoning about likelihoods (Section 3.3). As a broad rule of thumb, observations that intuitively appear to favor different hypotheses, or information derived from distinct types of sources (e.g. a right-wing politician vs. a labor-union leader) are usually best treated as separate pieces of evidence, whereas similar information arising from similar sources (e.g. two government informants tell a similar story about a policy process) might usefully be treated as a single piece of evidence. A second rule of thumb would be to avoid disaggregating the evidence too finely; if we seek to make too many analytical steps explicit, we risk becoming lost in minutia. On the other hand, if we aggregate too much evidence, it becomes difficult to assess the overall likelihood of what is actually a conjunction of many distinct propositions.

3.3 Likelihoods and likelihood ratios

Assessing likelihoods and likelihood ratios is the key inferential step in Bayesian process tracing. The likelihood of the evidence tells us how we should update our prior degree of belief in a given hypothesis, and, more importantly, likelihood ratios allow us to adjudicate between rival hypotheses and thereby identify the best explanation in light of the evidence. This section explains how best to interpret and assess likelihoods, with guidelines for handling information from sources that may not be reliable and accounting for logical dependence among multiple pieces of evidence.

3.3.3 Condition on previously incorporated evidence

When the total body of evidence $E$ consists of multiple observations $E_{1}$ – $E_{N}$ , we can decompose the likelihood into a product as follows:

(2) $$\begin{eqnarray}\displaystyle P(E\mid H_{k}I) & = & \displaystyle P(E_{1}E_{2}\cdots E_{N}\mid H_{k}I)=P(E_{N}\mid E_{1}E_{2}\cdots E_{(N-1)}H_{k}I)~P(E_{1}E_{2}\cdots E_{(N-1)}\mid H_{k}I)=\cdots \nonumber\\ \displaystyle & = & \displaystyle P(E_{N}\mid E_{1}\cdots E_{(N-1)}H_{k}I)\cdots P(E_{2}\mid E_{1}H_{k}I)~P(E_{1}\mid H_{k}I),\end{eqnarray}$$

because the product rule allows us to write the joint probability of $A$ and $B$ as $P(AB)~=$ $P(A\mid B)^{\ast }P(B)$ . The right-hand side of the Equation (2) is essentially a product of likelihoods for each observation $E_{x}$ , with the nuance that previously analyzed evidence must be incorporated along with $I$ as conditioning information. In other words, the likelihood for $E_{x}$ must be assessed conditional not only on a hypothesis and the background information, but also on all evidence from the current problem that we have previously incorporated, $E_{\text{prev}}$ . We must therefore ask if $E_{x}$ is any more or less likely given that we already know $E_{\text{prev}}$ , beyond whatever $H_{k}$ and $I$ imply.

Conditioning on previously incorporated evidence requires careful thought about logical dependence between $E_{x}$ and $E_{\text{prev}}$ under each hypothesis. Logical dependence is distinct from causal (or physical) dependence, in that $E_{\text{prev}}$ need not exert any causal influence on $E_{x}$ for it to affect our degree of belief in $E_{x}$ . Consider a textbook example of drawing without replacement from an urn containing two red marbles and one black marble. Suppose we close our eyes, draw one marble from the urn, and set it aside without looking. We then repeat the process and draw a second marble from the urn. We are interested in the probability that the first draw produced a red marble, $P(E_{1R}\mid I)$ . Before looking at either marble taken from the urn, $P(E_{1R}\mid I)=2/3$ . If we observe the second marble drawn from the urn and discover that it is black, we gain information that is highly relevant for inferences about the color of the first marble—in light of this information, we become certain that the first marble is red: $P(E_{1R}\,\mid \,E_{2B}I)=1$ . The outcome of the second draw can in no way exert a causal influence backward in time on the outcome of the first draw; the dependence is entirely logical in nature. Causal interactions between systems or causal influences in their common past can of course lead to logical dependence, but in a Bayesian framework, it is essential to remember that conditional probabilities reflect logical dependencies between propositions, not physical connections.

Returning to social science, suppose an informant interviewed in December 2005 tells a story $X$ , which we will denote as evidence $E_{\text{Inf}(X)}$ , and a news article from May 2005 reports a similar story, denoted as evidence $E_{\text{News}(X)}$ . Suppose we conduct and analyze the interview first, and then subsequently discover the news article, such that $E_{1}=E_{\text{Inf}(X)}$ and $E_{2}=E_{\text{News}(X)}$ . In this case, $E_{1}$ cannot have any causal effect on $E_{2}$ —what the informant has told us cannot possibly influence a news article written seven months earlier. Nevertheless, $P(E_{2}\mid E_{1}HI)$ will not equal $P(E_{2}\mid HI)$ . Whether $P(E_{2}\mid E_{1}HI)$ is higher or lower than $P(E_{2}\mid HI)$ may depend on the hypothesis $H$ . The point is that $E_{\text{News}(X)}$ and $E_{\text{Inf}(X)}$ are logically dependent given possible causal connections that might have occurred in the past. For instance, our informant may have learned $X$ from reading the news article, so under many hypotheses, we would be less surprised to encounter the article after talking to the informant, in which case $P(E_{2}\mid E_{1}HI)>P(E_{2}\mid HI)$ .

It is also important to note that dependence (or independence) is not a physical property of our sources—e.g. the newspaper vs. the informant in the example above—notwithstanding what might be suggested by common advice to seek “independent sources of evidence.”Footnote ¹⁰ Instead, it is a logical relationship between pieces of evidence given a specific hypothesis. For any two pieces of evidence—which should include information about the source—it is almost always possible to concoct some hypothesis under which they are dependent. Drawing on different sources and distinct types of information is certainly good practice, but it does not absolve us from thinking carefully about potential logical dependence among the data. The degree to which one source corroborates another depends on the hypothesis under consideration.

Conditioning on previously incorporated evidence is critical for inference—the imperative emerges directly from the mathematics of probability theory. However, reasoning about logical dependence in qualitative research can be extremely challenging. Evidence can be connected in arbitrarily complex ways; $E_{x}$ and $E_{y}$ may even be dependent in multiple ways, some of which might lead us to raise $P(E_{x}\,\mid \,H_{j}E_{y}I)$ relative to $P(E_{x}\,\mid \,H_{j}I)$ , whereas others might lead us to lower $P(E_{x}\,\mid \,H_{j}E_{y}I)$ relative to $P(E_{x}\,\mid \,H_{j}I)$ . Moreover, evidence may be dependent under some hypotheses but independent under others. In practice, we should aim to do the best we can while recognizing that it may not be feasible to carefully handle multiple nuanced dependencies in the data.

3.3.4 Sequence the evidence as convenient

The rules of conditional probability imply that the order of the evidence does not affect the final posterior probabilities. Using the product rule of probability and commutativity, the joint likelihood of two pieces of evidence can be written in any of the following equivalent ways:

(3) $$\begin{eqnarray}P(E_{A}E_{B}\mid HI)=P(E_{B}E_{A}\mid HI)=P(E_{A}\mid E_{B}HI)~P(E_{B}\mid HI)=P(E_{B}\mid E_{A}HI)~P(E_{A}\mid HI).\end{eqnarray}$$

Some literature in the psychological Bayesian tradition introduces nonstandard rules for updating probabilities, implying that the order in which evidence is analyzed does matter (e.g. Jeffrey Reference Jeffrey1983). However, such approaches necessarily violate the fundamental rules of probability (Equation (3)) and conflict with the notion of rationality that lies at the heart of logical Bayesianism—information incorporated in logically equivalent ways should lead to the same conclusions. If in practice we do not arrive at the same posterior probabilities upon analyzing the evidence in a different order, we have made a mistake somewhere in our reasoning that needs to be corrected.

Because we are free to consider the evidence in any order, we can look for sequences that facilitate conditioning on previously incorporated evidence when assessing likelihoods. Placing strongly discriminating evidence last can preclude having to condition other pieces of evidence on the conjunction of a hypothesis and an observation that is extremely implausible under that hypothesis—a difficult mental exercise that requires imagining very surprising flukes or coincidences. Incorporating highly decisive evidence last can also obviate careful conditioning on previous evidence, because the likelihood ratios will be extremely large regardless. On the other hand, if the evidence is decisive enough, we could incorporate it first and be done, since additional evidence will contribute only marginally to our posteriors.Footnote ¹¹

3.4 Logarithmic scales for probabilities

Whereas most research on Bayesian process tracing assigns values to probabilities on essentially linear, often course-grained scales (e.g. Rohlfing Reference Rohlfing2013, p. 19), we advocate a logarithmic scale for odds and likelihood ratios, which is standard practice in the natural sciences and information sciences. Using a logarithmic scale, in conjunction with an analogy to sound, better leverages intuition, enhances consistency when working with qualitative information, and may facilitate intersubjective agreement.

This recommendation is grounded in psychophysics, which shows that sensory perception tends to be a logarithmic function of stimulus strength. Stated in differential terms, a just-noticeable difference in the loudness of sound, brightness of light, or pressure on the skin is proportional to the magnitude of the stimulus. This relationship (the Weber–Fechner Law), which works well for a wide variety of stimuli and a large range of magnitudes, is sensible given that humans experience stimuli of highly varied intensity. By building in a logarithmic scale, evolution has increased the nervous system’s dynamic range. Given this characteristic feature of the nervous system, a logarithmic scale is more natural than a linear scale for measuring and analyzing sensory inputs. Sound, for example, is measured in decibels (dB), such that increasing the intensity of the sound wave by a factor of ten corresponds to an additive increment of 10 dB.

For similar reasons, logarithmic scales were introduced to assess perceptions of uncertainty in probabilistic inference. Good’s (Reference Good, Bernardo, de Groot, Lindley and Smith1985) weight of evidence in favor of one hypothesis compared to a rival, measured in decibels, is proportional to the logarithm of the likelihood ratio:

(4) $$\begin{eqnarray}WOE(H_{i}:H_{j})=10\log _{10}[P(E\mid H_{i}I)/P(E\mid H_{j}I)].\end{eqnarray}$$

The weight of evidence describes the probative value of the evidence—how strongly it discriminates between two rival hypotheses. Good (Reference Good, Bernardo, de Groot, Lindley and Smith1985) contends that a change in weight of evidence of one decibel, for example from even odds of 1:1 to odds of around 5:4, is as fine-grained as we can reliably quantify our degree of belief in competing hypotheses. A change in probability from 75 percent to 90 percent corresponds to an increase in log-odds of about 5 dB, which is salient, but in the natural sciences, cogent evidence might regularly lead to swings of several tens of decibels. Notice that in Bennett’s (Reference Bennett, Bennett and Checkel2015, p. 281) illustration of a smoking-gun test, where $P(E\mid H)=0.2$ and $P(E\mid \sim H)=0.05$ , the weight of evidence is only 6 dB—salient, but not decisive enough by Good’s standards to serve as a smoking gun for $H$ .

Measuring log-odds in decibels lets us leverage everyday experience with sound, while providing a quantitative underpinning for Gull’s metaphor of Bayesian inference as a “dialogue with the data”—in essence we can ask whether the evidence is whispering or shouting in favor of a given hypothesis. In acoustics, the minimal noticeable change is roughly 3 dB. A change of 5 dB is clearly noticeable, while an increase of 10 dB is perceived as about twice as loud; 20 dB is roughly four times louder. Table 1 provides typical reference sounds in decibels.

In qualitative research, we suggest regarding decisive evidence that strongly favors one hypothesis over a rival as roughly 30 dB, the difference between a quiet bedroom and a conversation—in other words, the data are “talking clearly.” Likewise, a very low prior log-odds against a hypothesis relative to a more plausible rival could reasonably be set at $-$ 50 dB (Jaynes Reference Jaynes2003, pp. 99–100), the difference between a pin drop and a normal conversation.

Table 1.

Typical Sound Levels (dB). Reference scales vary across sources. See for example: www.osha.gov/dts/osta/otm/new_noise/.

3.5 Inference via Bayes’ rule

The last step in explicit Bayesian process tracing entails applying Bayes’ rule to draw an inference. When working with multiple pieces of evidence, we can successively apply Bayes’ rule (Equation (1)), or we can multiply likelihood ratios to conduct inference in a single step:

(5) $$\begin{eqnarray}\frac{P(H_{i}\mid EI)}{P(H_{j}\mid EI)}=\frac{P(H_{i}\mid I)}{P(H_{j}\mid I)}\times \frac{P(E_{1}\mid H_{i}I)}{P(E_{1}\mid H_{j}I)}\times \frac{P(E_{2}\mid E_{1}H_{i}I)}{P(E_{2}\mid E_{1}H_{j}I)}\times \cdots \times \frac{P(E_{N}\mid E_{1}\cdots E_{N-1}H_{i}I)}{P(E_{N}\mid E_{1}\cdots E_{N-1}H_{j}I)},\end{eqnarray}$$

which follows from Equations (1) and (2).

Taking the logarithm of the Equation (5) gives a particularly simple, additive form of Bayes’ rule—the posterior log-odds equals the prior log-odds plus the weight of evidence:

(6) $$\begin{eqnarray}10\log _{10}\left[\frac{P(H_{k}\mid EI)}{P(H_{l}\mid EI)}\right]=10\log _{10}\left[\frac{P(H_{k}\mid I)}{P(H_{l}\mid I)}\right]+WOE(H_{k}:H_{l}).\end{eqnarray}$$

This formulation offers the computation advantage that weights of evidence are also additive:

(7) $$\begin{eqnarray}WOE(H_{k}:H_{l})=WOE_{1}(H_{k}:H_{l})+WOE_{2}(H_{k}:H_{l},E_{1})\cdots +WOE_{N}(H_{k}:H_{l},E_{1}\cdots E_{N-1}),\end{eqnarray}$$

where we must remember to condition on $E_{\text{prev}}$ as appropriate.

5.1 Caveats and limitations

The foremost challenge of explicit Bayesian process tracing entails assigning numerical values to all probabilities (priors and likelihoods). In the natural sciences, a strong underlying theory and a description of the measurement apparatus (along with various simplifying assumptions) lead to a specific error model and thereby a likelihood function that unambiguously specifies probabilities for the various possible measurements that may be observed. In the social sciences, there is no clear procedure for translating complex, narrative-based, nonreproducible, often qualitative information into precise probability statements. Efforts to explicitly apply Bayesian learning in qualitative social science will inevitably run up against the difficulty of quantifying likelihoods without precision or objectivity. Specifying a range of probabilities rather than a precise value (Humphreys and Jacobs Reference Humphreys and Jacobs2015) may be adequate for some purposes, but ultimately, it simply relocates the arbitrariness of assigning numerical values.Footnote ¹² Even from a purely pragmatic perspective, this sort of approach has limited value, because propagating interval probabilities through Bayes’ rule is nontrivial when working with multiple hypotheses. While words used to express probability in common parlance are certainly ambiguous, quantification may simply disguise that ambiguity with false precision.

A second caveat is that substantial training may be necessary before explicit Bayesian analysis can improve upon intuition. Our teaching experience thus far indicates that a day or two of intensive workshops is not adequate to successfully apply this approach. In fact, as part of the learning process, reasoning may get worse before getting better. While the metaphor of weighing evidence is in some ways intuitive, assessing the weight of evidence changes the way that we use our intuition. Practice will be needed to master this fundamental aspect of Bayesian thinking.

Third, explicit Bayesian analysis becomes intractable beyond fairly simple causal models, which are rarely adequate in social science. Recall that Bayesian analysis entails specifying mutually exclusive hypotheses, which is nontrivial and may require oversimplification. Some of the hypotheses we assess against Fairfield’s (Reference Fairfield2013) explanation involve causal mechanisms that—in the real world—could potentially operate simultaneously or in interaction. Assessing such possibilities requires elaborating a more complex but still mutually exclusive hypothesis space, which would aggravate the challenges of assigning numerical values to likelihoods. By contrast, in the natural sciences, Bayesian analysis is usually applied to very simple hypothesis spaces (even if the underlying theory and experiments are highly complex); for example: $H_{1}=$  the Higgs boson mass is $124{-}126~\text{GeV}/\text{c}^{2}$ , $H_{2}=$  the mass is $126{-}128~\text{GeV}/\text{c}^{2}$ , etc.

Fourth, practical considerations may restrict the use of explicit Bayesian analysis. Appendix A, which includes various cross checks to ensure that likelihoods are sensible and reasonably consistent across the problem, exceeds the length of Fairfield’s (Reference Fairfield2013) article, which included three additional case studies; Fairfield’s (Reference Fairfield2015) book includes over 28 case studies. Assessing weights of evidence (likelihood ratios) instead of absolute likelihoods (Section 4) is a more feasible if somewhat less careful approach. But either way, explicit Bayesian analysis cannot replace case narratives, which are an effective and efficient way to holistically communicate context, evidence, and analysis. As such, requiring scholars to provide explicit Bayesian analysis for all of their cases, above and beyond their narrative work, would create heavy disincentives for process tracing.

Finally, explicit Bayesian analysis should not be directly equated with transparency. On the one hand, this approach can obscure rather than clarify inference, especially if we disaggregate the evidence too finely and unpack our analysis into too many steps—we may become lost in minutiae. Moreover, making too many steps explicit may lull readers into uncritically accepting or glossing over the author’s reasoning, rather than assessing whether they can arrive at the conclusions through their own independent logical pathways, thereby undermining the scholarly scrutiny of inferences that analytical transparency is intended to promote. Even mathematicians routinely skip steps in published proofs; readers must fill in and verify themselves, which provides an important cross-check. On the other hand, transparency does not require assigning numerical values to probabilities and applying Bayes’ rule to derive an inference. Scholars can make the assumptions and logic behind their inferences explicit without resorting to numbers. While these considerations do not necessarily constitute an argument against explicit Bayesian process tracing, they clarify that transparency is not necessarily an argument for adopting this approach.

5.2 Applications

Given the caveats, when might explicit Bayesian analysis prove most useful? We begin with situations where we expect this approach to be of limited value and proceed toward those that proffer higher value. If all observations strongly favor a particular hypothesis over the rivals, explicit Bayesian analysis is unlikely to improve on intuition. Scholars can explain why the evidence is decisive without quantifying probabilities, and if the evidence is indeed compelling, readers should recognize it as such on its face. Likewise, if the evidence has weak probative value, explicit Bayesian analysis may simply confirm the realization we would have obtained intuitively—the evidence is insufficient to strongly support any particular hypothesis (unless we already had strong priors or cannot think of reasonable alternatives).

Moderate gains for inference may arise when the evidence is complex and does not clearly favor a single hypothesis. On the one hand, when some observations favor one hypothesis whereas others favor a rival, it may be difficult to intuitively ascertain which provides the best explanation. Explicit Bayesian analysis helps us keep track of nuances, consistently assess the weight of evidence for each observation, and systematically aggregate inferences across observations. On the other hand, there is a danger when evidence is ambivalent that conclusions derived via explicit Bayesian analysis may be driven by arbitrariness inherent in assigning numerical values to probabilities. Natural scientists would only believe that noisy data accumulates into a significant signal if the error model is well understood; in qualitative social science, analogous situations may rarely arise. Almost by definition, if the evidence pulls in different directions, small changes in probabilities may swing the inference in favor of one hypothesis or another.

Nevertheless, explicit Bayesian process tracing in such cases may be merited for the sake of analytical transparency and informing future research. Regarding transparency, if we must make inferential claims on the basis of ambivalent or weak evidence—if important questions are at stakeFootnote ¹³ and obtaining better evidence is infeasible—explicit Bayesian analysis can at least clarify the basis on which those claims rest and facilitate debate among scholars. Looking forward to future data-gathering opportunities, explicit Bayesian analysis could also elucidate what kinds of additional evidence would be most useful for strengthening the inference.

We envision a more valuable role for explicit Bayesian analysis in identifying the locus of contention when scholars disagree on inferences. As Hunter (Reference Hunter1984, p. 88) argues, through Bayesian analysis, “the sources of the disagreement can be determined much more easily than in normal verbal analysis.” Explicit Bayesian analysis provides a clear framework for pinpointing disagreements: Do they arise from different background information and assumptions (e.g. a source’s motives or sincerity), different priors, or different assessments of likelihoods? If the problem lies with the probative value of evidence, which observations are most contested, and why? For these purposes, numbers serve primarily to stimulate discussion about inferential logic, assumptions, and judgments, and the ad hoc component of quantification will be less problematic. Our Bayesian sensitivity analysis in Appendix A illustrates how this clarification and adjudication process might work; we show that to remain unconvinced, a skeptical reader would need to have extremely strong priors against the equity-appeal explanation and/or contend that the evidence is far less discriminating than we have argued.

We also foresee a valuable pedagogical role for explicit Bayesian analysis. Reading examples and conducting exercises can train intuition to follow this inferential logic more systematically, thereby improving traditional narrative-based process tracing. For example, one of the most salient lessons from our empirical case application is that the weight of evidence depends by definition on which hypotheses we compare; we cannot judge how decisive the evidence is with respect to our working hypothesis alone, without considering concrete alternatives. Thinking in terms of the weight of evidence, even without assigning numbers, may help scholars identify their most discriminating observations. More generally, many of our guidelines for explicit Bayesian analysis (summarized in Table 3) have direct analogs for more heuristic Bayesian reasoning in narrative-based process tracing—from comparing clearly specified mutually exclusive hypotheses and explaining why our background information suggests some explanations and justifies disregarding others from the outset, to considering how a source’s potential biases and instrumental incentives might change under rival hypotheses, and thinking about logical dependencies among the evidence when drawing inferences.

Table 3.

Guidelines for explicit Bayesian process tracing.

Relatedly, elaborating an explicit Bayesian appendix for an illustrative case from one’s own research might help establish process-tracing “credentials.” As much as we try to make our analysis transparent, multiple analytical steps will inevitably remain implicit. Qualitative research draws on vast amounts of data, often accumulated over multiple years of fieldwork. There is simply too much evidence and too much background information that informs how evidence is evaluated to fully articulate or catalog. Qualitative research is not replicable as per a controlled laboratory experiment; at some level, we must trust that scholars have made sound judgments. To that end, scholars might use an explicit Bayesian illustration to demonstrate their care in reasoning about the evidence and the inferences it permits.