Bayesian Foundations

Frequentism conceptualizes probability as a limiting proportion in an infinite series of random trials or repeated experiments. For example, the probability that a coin lands “heads” on a given toss is equated with the fraction of times it turns up heads in an infinite sequence of throws. In this view, probability reflects a state of nature—e.g., a property of the coin (fair or weighted) and the flipping process (random or rigged). In contrast, Bayesianism understands probability as a degree of belief based on a state of knowledge. The probability an individual assigns to the next toss of a coin represents her strength of confidence about the outcome after taking into account all relevant information she knows. Two observers watching the same coin flip would rationally assign different probabilities to the proposition “the next toss will produce heads” if they have different information about the coin or tossing procedure. For example, an observer who has had the opportunity to examine the coin in advance and discerns that it is weighted in favor of heads would rationally place a higher probability on that outcome than an observer who is not privy to such information.

The Bayesian notion of probability offers multiple advantages—most centrally: it fits better with how people intuitively reason under uncertainty; it can be applied to any proposition, including causal hypotheses, which would be nonsensical from a frequentist perspective; it is well suited for explaining unique events or working with a small number of cases, without need to sample from a larger population; and inferences can be made from limited amounts of information, using any relevant evidence (e.g., open-ended interviews, historical records), above and beyond data generated from stochastic processes. These features make Bayesianism especially appropriate for qualitative research, which evaluates competing explanations for complex sociopolitical phenomena using evidence that cannot naturally be conceived as random samples (e.g., information from expert informants, legislative records, archival sources). Strictly speaking, “frequentist inference is inapplicable to the nonstochastic setting,” quoting Jackman and Western.^{Footnote 23}

The school of Bayesianism we advocate as the foundation for scientific inference—logical Bayesianism—seeks to represent the rational degree of belief we should hold in propositions given the information we possess, independently of hopes, subjective opinion, or personal predilections. In ordinary logic, truth-values of all propositions are known with certainty. But in most real-world contexts, we have limited information, and we are always at least somewhat unsure about whether a proposition is true or false. Bayesian probability is an “extension of logic”^{Footnote 24} in that it provides a prescription for how to reason when we have incomplete knowledge and are thus uncertain about the truth of propositions. When degrees of belief assume limiting values of zero (impossibility) or one (certainty), Bayesian probability automatically reduces to ordinary logic.

A central tenet of logical Bayesianism is that probabilities should encode knowledge in a unique, consistent manner. Incorporating information in different but logically equivalent ways (e.g., learning the same pieces of information in different orders) must produce identical probabilities, and individuals possessing the same information must assign the same probabilities. Cox, Jaynes, and subsequent scholars proved mathematically that if we represent degrees of confidence in the truth of propositions with real numbers and impose these consistency requirements, we are led directly to the standard sum and product rules of probability, which in turn give rise to all other operations for manipulating and updating probabilities.^{Footnote 25}

The consistency requirements of logical Bayesianism are more demanding than requirements imposed in approaches that draw on the “psychological” or “subjective” school of Bayesianism common in the philosophy of science literature and many Bayesian statistics textbooks for social science. In this latter personalistic approach, rationality requires probabilities to be coherent, which means that utility-maximizing agents must decline “Dutch Book” bets, where loss is certain. Coherence in turn implies that probabilities satisfy the sum and product rules. But as long as probabilities satisfy these rules, they can be based on pure opinion or whim—whatever happens to motivate an individual to hold some particular subjective degree of belief. Accordingly, within psychological Bayesianism, individuals possessing the same information need not assign identical probabilities.

We will show that the consistency requirements are the key to understanding the powerful methodological foundation that logical Bayesianism provides for iterative research. First, however, we review the mechanics of Bayesian inference.

Bayesian Inference

Intuitively speaking, Bayesian reasoning is simply a process of updating our views about which hypothesis best explains the phenomena or outcomes of interest as we learn additional information. We begin by identifying two or more alternative hypotheses. The literature we have read along with our own previous experiences and observations give us an initial sense, or prior view, about how plausible each hypothesis is—e.g., before heading into the field or the archives, do we believe the median-voter theory is a much stronger contender for explaining levels of redistribution in democracies than approaches focusing instead on the power of organized actors including business associations and social movements? Or are we highly dubious that the median-voter hypothesis provides an accurate explanation for the politics of inequality? As our research proceeds, we ask whether the evidence we gather fits better with one hypothesis as opposed to another. When we have finished collecting data, we arrive at a posterior view regarding which hypothesis is most plausible. Bayes’ rule provides a mathematical framework for how we should revise our confidence in a given hypothesis, considering both our previous knowledge and the information we discovered during our research. If we remain too uncertain about which hypothesis performs best after analyzing the data in hand, we may continue our research and collect additional evidence.

Stated more formally, Bayesian inference generally proceeds by assigning prior probabilities to salient rival hypotheses. These prior probabilities represent our rational degree of belief (or confidence) in the truth of each hypothesis taking into account all relevant initial knowledge, or background information (I), that we possess. Symbolically, we represent the prior probability for hypothesis H as P(H|I). This follows the conventional notation whereby a conditional probability P(A|B) represents the rational degree of belief that we should hold in proposition A given proposition B—that is, how likely is A if we take proposition B to be true. We then consider evidence E obtained during the investigation at hand. The evidence includes all observations (beyond our background information) that bear on the plausibility of the hypotheses. Finally, we employ Bayes’ rule to update our degree of confidence in hypothesis H in light of evidence E:

(1)$$P\left( {H{\rm{|}}EI} \right) = P\left( {H{\rm{|}}I} \right) \times P\left( {E{\rm{|}}HI} \right)/P\left( {E{\rm{|}}I} \right).$$

On the left-hand side of Bayes’ rule, P(H|EI) is the posterior probability of H given evidence E and background information I. On the right-hand side, P(H|I) is the prior probability discussed earlier, and P(E|HI) is the likelihood of the evidence—the probability of observing evidence E if the hypothesis is actually true. In the denominator, P(E|I) acts as a normalization factor. Bayes’ rule is nothing more than a rearrangement of the product rule of probability:

(2)$$P\left( {AB} \right) = P\left( {BA} \right) = P\left( {A{\rm{|}}B} \right) \times P\left( B \right) = P\left( {B{\rm{|}}A} \right) \times P\left( A \right),$$

where we substitute H and E for propositions A and B, while also explicitly conditioning all probabilities on our background information I. ^{Footnote 26}

Because inference always involves comparing hypotheses, it is easier to work with the odds-ratio form of Bayes’ rule:

(3)$${{P\left( {{H_i}{\rm{|}}EI} \right)} \over {P\left( {{H_j}{\rm{|}}EI} \right)}} = {{P{\rm{(}}{H_i}{\rm{|}}I)} \over {P\left( {{H_j}|I} \right)}} \times {{P\left( {E{\rm{|}}{H_i}I} \right)} \over {P\left( {E{\rm{|}}{H_j}I} \right)}}$$

where the factor P(E|I) conveniently cancels out. The factor on the left-hand side represents the posterior odds on hypothesis H _i relative to H _j—namely, how much more plausible one hypothesis is versus a rival hypothesis in light of the evidence learned as well as the background information we initially brought to the problem. The first factor on the right-hand side is the prior odds—the plausibility of one hypothesis compared to the other based only on our background information. For posterior odds and prior odds, we can think in terms of how willing we would be to bet in favor of one hypothesis versus the other. The second factor on the right-hand side is the likelihood ratio—how plausible the evidence is under one hypothesis relative to the other, or in other words, how likely the evidence would be if we assume H _i is true, compared to how likely the evidence would be if we instead assume H _j is true. The odds-ratio form of Bayes’ rule states that the posterior odds equal the prior odds multiplied by the likelihood ratio. Bayes’ rule tells us that evidence fits better with a hypothesis H _i than an alternative H _j to the extent that H _i makes the evidence more plausible. How much we end up favoring one hypothesis over another depends on both our prior views and the extent to which the evidence weighs in favor of one hypothesis over another.

Assessing likelihood ratios P(E|H _iI)⁄P(E|H _jI) is therefore the critical inferential step that tells us whether evidence E should make us more or less confident than we were initially in one hypothesis relative to a rival. The likelihood ratio can be thought of as the probability of observing E in a hypothetical world where H _i is true, relative to the probability of observing E in an alternative world where H _j is true. When evaluating likelihoods of the form P(E|H _iI), we must in effect (a) suppress our awareness that E is a known fact, and (b) suppose that H _i is correct, even though the actual status of the hypothesis is uncertain. Recall that in the notation of conditional probability, everything that appears to the right of the vertical bar is either known or assumed as a matter of conjecture when reasoning about the probability of the proposition to the left of the bar. In qualitative research, to use Hunter’s phrase, we need to “mentally inhabit the world” of each hypothesis and ask how surprising (low probability) or expected (high probability) the evidence E would be in each respective world.^{Footnote 27} If E seems less surprising in the “H _i world” relative to the “H _j world,” then that evidence increases our odds on H _i vs. H _j.^{Footnote 28} Again, we gain confidence in a given hypothesis to the extent that it makes the evidence we observe more plausible compared to rivals.

Assessing likelihoods entails thinking about how consistent the evidence is with the world of the hypothesis in question. Would the events, decisions, and statements that E represents follow plausibly and naturally in the world of a particular hypothesis? Or would they seem unusual and unexpected, perhaps requiring additional flukes, coincidences, or complex chains of intervening events to occur? Here it may be helpful to consider Coleridge’s notion of “willing suspension of disbelief.” When reading a story involving supernatural forces or time travel, for example, the conjured world might strike us as highly implausible—meaning that P(H|I) is very low. But if we nevertheless provisionally accept this hypothesized world, the story should feel compelling and self-consistent, obeying the internal logic of the imagined world—meaning that P(E|HI) for the events that unfold should not be too low compared to other scenarios we can envision.

Elsewhere, we elaborate guidelines for explicit Bayesian analysis in qualitative research, which entails quantifying all probabilities.^{Footnote 29} To illustrate how Bayesian logic can be applied heuristically—i.e., without specifying numerical values for probabilities—consider an example drawing on Kurtz’s state-building research.^{Footnote 30} We wish to ascertain whether the resource-curse hypothesis, or the warfare hypothesis (assumed mutually exclusive), better explains institutional development in Peru:

For simplicity, suppose we have no relevant background knowledge about state-building in Peru. Since both hypotheses find substantial support in literature on other countries, we might reasonably assign even prior odds. We now learn the following:

Intuitively, E ₁ strongly favors the resource-curse hypothesis. Applying Bayesian reasoning, we must evaluate the likelihood ratio P(E ₁|H _RI)⁄P(E ₁|H _WI). Imagining a world where H _R is the correct hypothesis, mineral dependence in conjunction with weak state capacity is exactly what we would expect, and external threats are not surprising given that a weak state with mineral resources could be an easy and attractive target for invasion. In the alternative world of H _W, E ₁ would be quite surprising; something very unusual, and hence improbable, must have happened for Peru to end up with a weak state if the warfare hypothesis is nevertheless correct, because weak state capacity despite military threats contradicts the expectations of the theory. Because E ₁ is much more probable under H _R relative to H _W —that is, P(E ₁|H _RI) is much greater than P(E ₁|H _WI)—the likelihood ratio is large, and it significantly boosts our confidence in the resource-curse hypothesis.

Prior versus Posterior Probabilities and Old versus New Evidence

While testing hypotheses with new evidence is pervasively espoused, distinctions between old and new evidence, and hence exploratory and confirmatory research, are far less consequential within logical Bayesianism compared to frequentism or psychological/subjective Bayesianism. To be clear, “new evidence” refers to information unknown to the scholar before devising the hypothesis—regardless of the historical timing of when that information was generated. For example, in figure 1, E ₁ is old evidence relative to H, whereas E ₂ is new evidence, even though E ₂ existed before E ₁.

Figure 1

“New” versus “old” evidence

The key to unraveling the false dichotomies lies in understanding that prior and posterior are not temporal notions—they are logical notions. In astro-statistician Tom Loredo’s words:

In other words, prior and posterior refer to degrees of belief before and after a piece of evidence is incorporated into our analysis—not to the timing of when we happened to learn or obtain that evidence. Prior/posterior describe idealized states of knowledge without/with specific pieces of evidence included. Hypotheses can contain temporal structuring, and evidence can contain temporal information. However, probabilities themselves carry no intrinsic time stamps.

These points merit expounding. Recall that within logical Bayesianism, only the data at hand and the background knowledge are relevant for assessing the degree of belief that a hypothesis merits. Nothing else about our state of mind should influence our probabilities. The relative timing of when we stated the hypothesis, worked out its implications, and gathered data falls into this later category of logical irrelevance.

To further stress the logical irrelevance of keeping track of what we knew when, the rules of conditional probability mandate that we can incorporate evidence into our analysis in any order without affecting the posterior probabilities. Using the product rule (2) and commutativity, the joint likelihood of two pieces of evidence can be written in any of the following equivalent ways:

(4)$$\eqalign{P({E_1}{E_2}|HI){\rm{\ }} = {\rm{\ }}P({E_2}{E_1}|HI) = \cr{P({E_1}|{E_2}HI) \times P({E_2}|HI) = P({E_2}|{E_1}HI) \times P({E_1}|HI).$$

Evidence learned at time one (E ₁) may thus be treated as logically posterior to evidence learned at time two (E ₂). If in practice conclusions differ depending on the order in which evidence was incorporated, there is an error in our reasoning that should be corrected. Otherwise we have violated the fundamental notion of rationality that lies at the heart of logical Bayesianism (refer to Bayesian Foundations)—information incorporated in equivalent ways should lead to the same conclusions.

Once we recognize that timing is irrelevant in probability theory, it follows that each step below is logically distinct:

• • drawing on evidence E to inspire hypotheses;^{Footnote 32}

• • assigning prior probabilities to those hypotheses given background information I that does not include E;

• • assessing the likelihood of E under alternative hypotheses to derive posterior probabilities.

Information is neither “exhausted” nor “double-counted” in this process (online appendix A expounds). All relevant knowledge can be sorted as convenient into background information on which all probabilities are conditioned and into evidence that we use to update probabilities.

Psychological/subjective approaches to Bayesianism often diverge from logical Bayesianism on these points, because the former focus on individuals’ personal degrees of belief and how their psychological states evolve over time. Jeffrey’s “probability kinematics” is a prominent example;^{Footnote 33} his approach introduces non-standard rules for updating that violate the laws of probability and imply that the order in which evidence is analyzed can matter.^{Footnote 34}

In sum, probability theory requires keeping track of what information has been incorporated into our analysis, not when that information was acquired.^{Footnote 35} Time-stamps indicating when hypotheses were composed or when evidence was observed or incorporated are not relevant to scientific inference.^{Footnote 36}

Curtailing Confirmation Bias and Ad-Hoc Theorizing

Careful application of Bayesian logic helps guard against confirmation bias and ad-hoc hypothesizing in iterative research. We consider these dual pitfalls in turn.

Two common variants of confirmation bias entail overfocusing on data that fit a particular hypothesis or overlooking data that undermine it, and focusing on a single favored hypothesis while forgetting to consider whether data consistent with that hypothesis might be as or more supportive of a rival hypothesis. A common recommendation for precluding such biases entails identifying observable implications of rivals as well as the main working hypothesis before gathering data.^{Footnote 37} However, this advice can be problematic for two reasons.

First, deducing observable implications beforehand may be infeasible, because any hypothesis may be compatible with a huge number of possible evidentiary findings—just with varying probabilities of occurrence. For qualitative research on complex socio-political phenomena, there is essentially no limit to the different kinds of evidence we might encounter, and there is no way to exhaustively catalogue these infinite possibilities in advance.

Second, anticipating observable implications may foster even greater bias. If we have already elaborated hypotheses to be considered and evidence expected under each, we are now better situated to seek out the sorts of evidence that will support our pet theory, compared to a situation where we collect evidence without necessarily anticipating what will support which hypothesis. This caveat embodies classic advice from Doyle’s Sherlock Holmes: “It is a capital mistake to theorize before one has data. Insensibly one begins to twist facts to suit theories, instead of theories to suit facts” (A Scandal in Bohemia).

Risks of confirmation bias can be better controlled by conscientiously endeavoring to follow Bayesian reasoning. Tendencies to seek evidence that supports a favored hypothesis, interpret evidence as overly favorable to that hypothesis, and underweight evidence that runs against that hypothesis are counteracted by following Bayesian prescriptions to condition probabilities on all relevant information available, without presuming anything beyond what is in fact known, or bringing mere opinions or desires into the evaluation. Furthermore, remembering that the critical inferential step in Bayesian inference entails assessing likelihood ratios, P(E|H _jI)⁄P(E|H _kI), precludes the pitfall of restricting attention to a single hypothesis—we must always ask whether a given explanation makes the evidence more or less likely compared to rivals.

In contrast to confirmation bias, the dual problem of ad-hoc hypothesizing involves over-tailoring an explanation to fit a particular, contingent set of observations. This danger underpins calls for distinguishing exploration from confirmation and testing hypotheses with new data. Within logical Bayesianism, however, an ad-hoc hypothesis that is too closely tailored to fit arbitrary details of the data incurs a low prior probability, which protects us from favoring it over a simpler hypothesis that adequately explains the data. If an explanation is ad-hoc, careful consideration should reveal that it is just one member of a large family of more or less equally ad-hoc hypotheses, characterized by multiple parameters or arbitrary choices that must be fine-tuned to the data. Each of these related hypotheses might explain a different set of contingent facts, yet none of them would seem any more credible than the others in the absence of the particular body of observations obtained. Even if the overall prior probability of the family of hypotheses {H ₁ or H ₂ or . . . H _N} is appreciable, this prior probability must be spread over all of the constituent possibilities, such that the prior for any particular H _i will be small.

Consider an example adapted from astrophysicist Bill Jefferys.^{Footnote 38} A stranger at a party shuffles a deck of cards, and you draw the six of spades. We might reasonably hypothesize that this card was arbitrarily selected from a randomly-shuffled deck (H _R). A rival hypothesis proposes that the stranger is a professional magician relying on a trick deck that forces you to draw the six of spades (H _6♠). While the likelihood of selecting this particular card is 1/52 under H _R, it is far larger under H _6♠. However, H _6♠ must be penalized by a factor of 1/52 relative to H _R, because without observing your draw, there would be no reason to predict the six of spades as the magician’s forced card. H _6♠ should be treated as one of 52 equally plausible related hypotheses whereby the magician forces some other card. Accordingly, our single draw provides insufficient evidence to boost the credibility of H _6♠ above H _R.

Logical Bayesianism thus penalizes complex hypotheses if they do not provide enough additional explanatory power relative to simpler rivals, in line with Occam’s razor and Einstein’s dictum that things should be as simple as possible, but no simpler. In quantitative analysis, this task is accomplished via Occam factors that are automatically built into Bayesian probability.^{Footnote 39} Online appendix C discusses Occam factors in more detail and illustrates how the penalty of 1/52 in our card-draw example emerges when we formally apply Bayesian analysis.

In qualitative social science, the role of Occam factors in penalizing overly-complex explanations was first highlighted by Western.^{Footnote 40} Although many authors identify complexity of theories with the number of assumptions or causal factors, there are no universal prescriptions for assessing how ad-hoc a hypothesis is in qualitative research. Our recommended stratagem entails scrutinizing new hypotheses to evaluate how much additional complexity they introduce compared to rivals. If the hypothesis invokes many more causal factors or very specific or elaborate conjunctions of causal factors, good practice would entail penalizing its prior relative to the rivals, although it can be difficult to judge precisely the tradeoff between simplicity and explanatory power—a point Western also stresses.^{Footnote 41}

In sum, within logical Bayesianism, likelihood ratios help guard against confirmation bias, while priors help guard against ad-hoc hypothesizing. These safeguards are absent within frequentism, where hypothesis testing usually focuses on the probability of the data only under the null hypothesis, rather than relative likelihoods under rival hypotheses, and where the concept of probability applies only to data obtained through a stochastic sampling procedure, not to hypotheses. Frequentist inference therefore requires pre-specifying sampling and analysis procedures in detail to avoid confirmation bias, and strictly separating data used in theory-building from data used for theory-testing to prevent ad-hoc hypothesizing, whereas such strictures are unnecessary for Bayesian inference.

Biased Priors

Concerns: (a) Given psychological difficulties in “getting something out of our mind,” we may be unable to assign priors that are not influenced by what we already know about our data.

(b) Given vulnerabilities to cognitive biases, we may over-fit inductively-devised hypotheses to the evidence without adequately penalizing their priors.

Pre-specifying priors is not a sensible solution. We cannot assess a prior before devising the hypothesis, and once we formulate the hypothesis, all relevant information—both background knowledge and evidence E _pre acquired thus far—must inform P(H|E _preI), which serves as the “prior” moving forward. Moreover, whether we evaluate P(H|I) and then the likelihood for the total evidence E _T =E _preE _post ultimately collected, P(E _T|I), or whether we update along the way, evaluating P(H|E _preI) and then P(E _post|E _preHI)×P(H|E _preI), the final inference should be the same—consistency checks can help ensure equivalence. The timing of when we assess or record priors is irrelevant.

To guard against subconsciously-biased priors (concern a), best practices should include the following. First, describe the most salient background information and explain why it motivates a particular choice of priors. If priors are obviously biased in favor of an inductively-derived hypothesis, beyond what is justified by the background information discussed, readers should notice the discrepancy. For instance, in our state-building example, readers might balk if our prior odds strongly favored H _LRA over the well-established resource-curse and warfare hypotheses. Likewise, if a well-known study or salient literature is overlooked, readers will request reconsideration of priors in light of that further background information.

Second, consider conducting the analysis with equal prior odds, which avoids biasing initial assessments in favor of any hypothesis. This approach shifts focus to likelihood ratios, with the aspiration that even if scholars disagree about priors—which will be almost inevitable given that everyone has different background information—we may still concur on the direction in which our odds on the hypotheses should shift in light of the evidence. Third, consider using several different priors to assess how sensitive conclusions are to these initial choices, along the lines of our analysis in the section on Iteration in Practice.

For qualitative research that follows Bayesian logic heuristically, the first guideline entails carefully discussing the strengths and weaknesses of rival explanations based on existing literature, which is common practice. The second guideline entails recognizing that readers may initially view a hypothesis with much more skepticism than the author, such that all parties in the scholarly dialogue should pay close attention to scrutinizing the evidence and the inferential weight it provides. Authors should be conservative with their inferential claims until the weight of evidence becomes strong.

Regarding concern b, scholarly dialogue again serves as a corrective to sloppy analysis. If an inductive hypothesis manifesting multiple fine-tuned variables or inordinate complexity is granted too much initial credence, readers should notice and demand additional evidence to overcome an unacknowledged or underestimated Occam penalty. Beyond the simple advice to treat inductively-devised hypotheses with healthy skepticism, three suggestions can help curtail ad-hoc hypothesizing: start with reasonably simple theories and add complexity incrementally as needed; critically assess whether all casual factors in the theory actually improve explanatory leverage; and ask whether the explanation might apply more broadly.

In contrast, reporting the temporal sequencing of the research process in and of itself does not help ascertain how severe an Occam penalty a hypothesis should suffer. The critical point is that a hypothesis that is post-hoc—devised after the evidence—is not necessarily ad-hoc—arbitrary or overly complex. These are distinct concepts. As argued in the Iteration in Practice section, H _LRA is post-hoc (relative to E ₂), but not ad-hoc, because it is no more arbitrary or complex than its rivals.

Biased Likelihoods

Concern: We may succumb to confirmation bias in overstating how strongly evidence favors an inductively-derived hypothesis.

Suggestions for pre-registration and time-stamping in qualitative research^{Footnote 47} aim to address these concerns, on the premise that differentiating exploratory from confirmatory analysis allows us to more credibly evaluate inductively-inspired hypotheses. Importing this prescription into a Bayesian framework would entail assigning likelihoods to clues we might encounter before gathering data.

Even in light of human cognitive limitations, we find this approach unhelpful. Although a scholar’s prospective assessment of likelihoods for “new evidence” might be less prone to confirmation bias than retrospective analysis of “old evidence,” confirmation bias could just as easily intrude when gathering additional evidence—by subconsciously looking harder for clues that favor the working hypothesis or overlooking those that do not (refer to Curtailing Confirmation Bias and Ad-Hoc Theorizing).

Moreover, we reiterate the impossibility of foreseeing all potential evidentiary observations in the complex world of social science. Anticipating coarse-grained categories of observations is not adequate for specifying likelihoods for any actual, concrete evidence that might fit within that class, because specific details of evidence obtained can matter immensely to likelihoods under different hypotheses. Consider the example Bowers et al. present in their discussion of pre-analysis plans for qualitative research: a government has cut taxes, and we wish to assess hypothesis H _K = Tax cuts were motivated by an interest in Keynesian demand management.^{Footnote 48} The authors delineate evidence E=Records of deliberations among cabinet officials about the tax cut show “prominent mention of . . . Keynesian stimulus,” and they judge the probability of finding such evidence if H _K is true to be very high. However, E as stated above is too vague to assign a meaningful likelihood in advance. Here are two different clues we might encounter in the records:

Suppose further that the time and attention devoted to these mentions of Keynesiansim are similar for E′ and E′′, such that both qualify as instances of E as articulated above, even though they carry very different import. Whereas the likelihood of E′ might well be high if H _K is true, the likelihood of E′′ certainly is not—E′′ would be extremely surprising in a world where H _K is correct.

Bowers et al. recognize this “problem of precision,” noting that E as defined earlier “still leaves some things open. Just how prominent do mentions of Keynesian logic have to be … ? How many actors have to mention it? What forms of words will count as the use of Keynesian logic?”^{Footnote 49} However, they underestimate the problem. The issue is not just how many mentions or how many actors or what terms we associate with Keynesianism, but an endless array of other possibilities and nuances that depend on the context and manner in which Keynesianism is discussed. However much additional detail we specify before gathering data, we can always invent—and the real world may well produce—another twist or tweak that matters nontrivially. Despite efforts to anticipate what might surprise us ahead of time, science advances most when evidence surprises us in unforeseen ways.

Jaynes, an outspoken advocate of logical Bayesianism in the physical sciences, reinforces these key points:

What matters is how sound the inferences are in light of the arguments and evidence presented, not in comparison to every twist and turn of analysis before the author arrived at the final conclusions, or what the author would have thought had the data turned out differently.

Returning to the core concern of mitigating bias when assessing likelihoods, first, recall that inference always requires evaluating likelihood ratios, which forces us to ask how well the evidence fits with rival explanations. Second, we reiterate our central point regarding scholarly scrutiny: if despite efforts to follow logical Bayesianism, a scholar nevertheless over-estimates how much the evidence favors an inductively-inspired hypothesis, readers can independently weigh that evidence and critically assess the author’s judgments. Subsequent debate may encourage the author to bring more background information to light that was previously used implicitly, or acknowledge that the evidence is not as strong as previously maintained. In our state-building example, readers might contest our assessment that E ₁E ₂ very strongly favors H _LRA over H _R, perhaps suggesting that this evidence only moderately favors the inductively-inspired hypothesis. Open discussion would then result in greater consensus or at least greater clarity on why scholars interpret the evidence differently.

Integrity

Concern: We need mechanisms to discourage scholars from choosing procedures after the fact to get the results they want, or manipulating evidence to strengthen results.

The first malpractice—post-hoc choice of analytical procedures—is a bigger concern for frequentist inference, which requires predefined stochastic data-generation models. Within a Bayesian framework for case-study research, we must make judgments about which hypotheses to consider, how to acquire evidence, and how to interpret that evidence. However, the underlying inferential procedure remains the same: apply probabilistic reasoning to update beliefs regarding the plausibility of rival hypotheses in light of relevant evidence. Analysis always involves assessing priors, assessing likelihoods, and updating probabilities via Bayes’ rule. Unlike frequentist statistics, there is no need to choose among sampling procedures, stopping rules, estimators, tests statistics, or significance levels.

The second malpractice, e.g., deliberately cherry-picking evidence, can certainly occur in qualitative research. However, time-stamping does little to deter such abuses. Any scholar intent on exaggerating results or willing to commit fraud can find ways to do so regardless. Ansell and Samuels make similar observations regarding the related issue of results-blind review—it is always possible to “sweep dirt an author wants no one to see under a different corner of the publishing carpet.”^{Footnote 51} As a device for signaling integrity, mechanisms like pre-registration or time-stamping risk imposing a substantial burden of time and effort on honest scholars without preventing dishonest scholars from sending the same credibility signals. Recall that the retracted LaCour-Green study was pre-registered, yet evidence of fraud was uncovered not by comparing the published article to the pre-registration plan, but by scrutinizing the article and accompanying dataset.^{Footnote 52}

The only viable strategy in our view involves disciplinary norms. First, we need a commitment to truth-seeking and scientific integrity. As Van Evera observed long before DA-RT: “Infusing social science professionals with high standards of honesty is the best solution.”^{Footnote 53} Second, adjusting publication norms regarding requisite levels of confidence in findings would mitigate incentives for falsely bolstering results. For qualitative research, embracing Bennett and Checkel’s dictum, “conclusive process tracing is good, but not all good process tracing is conclusive,”^{Footnote 54} would be a major step toward reducing temptations to overstate the case in favor of the author’s hypothesis. An associated best practice entails explicitly addressing the pieces of evidence that on their own run most counter to the overall inference; transparency of this type could both encourage critical thinking and signal integrity in a more meaningful way. We recognize that these suggestions are neither panaceas nor quick fixes. But in the long term, rethinking disciplinary norms and practices along these lines and adopting a more Bayesian perspective could help us better acknowledge and communicate the uncertainty that surrounds our inferences, which is critical for scientific inquiry.