2. From methods to evidence

It is often assumed that MMR is about combining data obtained and analyzed by different means (typically qualitative and quantitative) for the purpose of better supporting a causal hypothesis compared to data obtained from a single method. This idea is behind what we call simple methodological pluralism, according to which different methods produce different kinds of data that, when combined in the right way, provide better support for causal inferences. The problem with simple methodological pluralism is that it remains unclear why combining data produced by different methods should in itself improve causal inference, over and above there being more data in support of the hypothesis (with the obvious proviso that the data agree).

Let us think of a method as the causal process whereby the data are generated, together with the inferential practices applied to them, and then further distinguish data from evidence. Data comprise the causally produced, concrete, and public artifacts that carry information from the phenomenon of interest (Bogen and Woodward, Reference Bogen and Woodward1988; Leonelli Reference Leonelli2015). As such they are always of, or from, something, their identity and type determined by the process through which they were created. Evidence, in turn, is a claim or a fact (or some other propositionally structured entity) about the data, usually grounded on a systematic pattern of some sort, which speaks in favor of, or provides a reason to believe in, some specific claim (e.g., Achinstein Reference Achinstein2001; Cartwright Reference Cartwright2013). As such, evidence is always for something: Its identity as evidence is relational to a claim or a hypothesis.

The distinction between data and evidence is not mere philosophical sophistry. The epistemic benefits conferred by variety reflect the relevant differences in the evidence, not in the data or their production as such. The point of bringing together diverse evidence is to substantiate the reasons for believing a hypothesis (in this case a causal claim or a claim about a mechanism). Data in themselves do not speak in favor of a claim, nor do they count as reasons to believe; they do so only when some of their features are interpreted as evidence for a specific claim. As material artifacts, data simply exist, whereas evidence lives in the space of reason. As Leonelli’s (Reference Leonelli2015, 2019) work convincingly shows, data can also be reused and cited as evidence for claims other than the one for which they were originally produced. The distinction between data as material artifacts and evidence as a reason to believe means that it is a category mistake to equate kinds of methods with kinds of evidence, and methodological variety with evidential variety. We should point out that this is not to claim that the methods whereby data are produced are irrelevant to evidential relevance: For example, as we show later on, features of the causal processes whereby the data are produced are relevant to achieving the variety of evidence required for triangulation (Kuorikoski and Marchionni Reference Kuorikoski and Marchionni2016).

We should also point out that if different methods always produce relevantly different evidence, then the epistemic justification of MMR could be straightforwardly explicated in terms of exploiting their complementary strengths and weaknesses, therefore simple methodological pluralism would suffice as a foundation for MMR. For example, it is a widespread belief among social scientists that knowledge of mechanisms can only be generated by case-based methods. This argument is also well rehearsed in the MMR literature, in which it is presupposed that case-based methods and statistical analyses produce essentially different kinds of evidence. For example, most MMR frameworks simply assume that whereas quantitative data yield evidence for population-level claims about average causal effects, qualitative data provide evidence of the process that connects the cause to the effect in each particular case (e.g., Lieberman Reference Lieberman2005). Derek Beach, an influential methodologist in the field of political science, insists (e.g., 2020) that both statistical studies, including randomized controlled trials (RCTs) and cross-country comparisons, and case studies produce different kinds of evidence: difference-making evidence and mechanistic evidence, respectively. ^{Footnote 1}

However, differences in methods and evidence do not always go hand in hand. Evidence about mechanisms may be obtained in different ways, including RCTs and statistical analyses (Illari Reference Illari2011; Marchionni and Reijula Reference Marchionni and Reijula2019). Conversely, the same method may generate different kinds of evidence. For example, as discussed in the fields of development economics and criminology, RCTs may generate evidence for the efficacy of an intervention as well as evidence about the mediating mechanism (so-called policy and mechanism experiments, respectively; see Ludwig et al. Reference Ludwig, Kling and Mullainathan2011). Methodological diversity does not guarantee the kind of difference in evidence that is necessary for exploiting the epistemic benefits of evidential variety.

3. Varieties of evidential variety

If simple methodological pluralism is not a sufficient epistemic foundation for MMR, then what is? MMR rationale is sometimes articulated in terms of methodological triangulation (Johnson et al. Reference Johnson, Onwuegbuzie and Turner2007): the idea that bringing different methods each with its own characteristic weaknesses and strengths to bear on the same causal claim helps to remedy those weaknesses (see also Crasnow Reference Crasnow2010). This idea has been questioned, however (Seawright Reference Seawright2016, 4–10), and other MMR researchers seem to hold different conceptions of the purpose of mixing methods (see Johnson et al. 2007).

As Ahmed and Sil (Reference Ahmed and Sil2012) point out, motivating MMR in terms of triangulation is problematic because in many cases such as nested analysis the different methods clearly address different questions. Rather than a matter of triangulation, Plano Clark and Ivankova (Reference Plano Clark, Ivankova, Clark and Ivanova2016) describe MMR as “the process of integrating quantitative and qualitative research to more completely address a study’s purpose or research questions” (emphasis added). Accordingly, the pieces of evidence integrated into MMR address different questions, not the same question as in the case of triangulation. Shan and Williamson (Reference Shan and Williamson2021) argue that evidential pluralism, which they understand as the claim that causal inference typically requires evidence of both difference making and mechanisms, provides a suitable epistemic foundation for MMR.

As we have already argued elsewhere, there are multiple distinct sources of this value, and distinct epistemic rationales for combining evidence, namely triangulation, security, and integration (Figure 1) (Kuorikoski and Marchionni Reference Kuorikoski and Marchionni2022).

Figure 1.

A schematic representation of kinds of evidential variety. P is a proposition about a phenomenon, E_n is evidence, and M is a mechanism; arrows represent causal relations whereas lines represent evidential relations.

These forms of reasoning are sometimes presented as mutually exclusive alternative rationalizations for the variety-of-evidence thesis, but this need not be case. Because the epistemic rationales for these different forms of the value of variety appear to make sense in their own right, we propose that, depending on what is the inferential problem at hand and/or what characteristics a certain piece of evidence has in relation to a given causal claim, there are simply different ways in which evidence can be combined and different nonaggregative benefits that can accrue. Furthermore, different self-described MMR procedures appear to realize different principles. Hence, dividing the value of variety in this way has analytic value (Kuorikoski and Marchionni Reference Kuorikoski and Marchionni2022).

Triangulation refers to the use of multiple independent lines of evidence for a single claim about a phenomenon such as the existence of a causal relation (e.g., Heesen et al. Reference Heesen, Bright and Zucker2019; Kuorikoski and Marchionni Reference Kuorikoski and Marchionni2016; Schupbach Reference Schupbach2018). Accordingly, the added value of evidential variety derives from the fact that concordance between multiple independent lines of evidence indicates that the common result is unlikely to be an artifact of errors and biases characteristic of any of the individual methods.

Successful triangulation therefore requires that the different pieces of evidence are evidence for the same causal claim, and that they are suitably independent. Philosophers of science have debated how this independence should be understood (e.g., Fitelson Reference Fitelson2001; Stegenga Reference Stegenga2009; Claveau, Reference Claveau2013),^{Footnote 2} but two things are clear. First, the diversity of methods by which data are produced is not, in itself, sufficient for obtaining evidence that is independent in the right way. Second, the independence condition may be hard to meet in practice and other epistemic considerations, such as the reliability or informativeness of individual sources of evidence, may outweigh the value of its variety (Claveau and Grenier Reference Claveau and Grenier2019). This matters, because at least some of the skepticism related to MMR arises from its being regarded as realizing triangulation. For example, multimethod large-N qualitative analysis is popular in political science and international relations, its aim being to combine statistical analysis with case studies to provide independent evidence for population-level claims about average causal effects. As Rosa Runhardt (Reference Runhardt2022) argues, however, the statistical evidence does not add anything to the evidence about mechanisms obtained from case studies. If this is the case, then it is clearly not a suitable design for realizing triangulation.

Security constitutes another source of value related to evidential variety. Here, one piece of evidence remedies the deficiencies of another piece of evidence relative to a given causal claim (Staley Reference Staley2004; Zahle Reference Zahle2019). The main difference with the case of triangulation is that the different pieces of evidence are not direct evidence supporting the same claim. Variety brings security when there is a causal claim, first-order evidence in its support, and then second-order evidence in favor of the first-order evidence being reliable and relevant for the causal claim in question. To clarify the difference between triangulation and security, we suggest thinking of triangulation as the support afforded by two independent eyewitnesses each testifying to having seen the suspect at the crime scene. Independence here could mean that the eyewitnesses did not know each another and were in different spatial locations with respect to the suspect, for example. According to this analogy, security is involved when the testimony of an expert placing the eyewitness in a position to see the suspect clearly enough counts as second-order evidence in support of the reliability of the eyewitness as first-order evidence (for the claim about the culprit). As in triangulation, second-order evidence can only strengthen the inference if it is not subject to the same errors and biases as the first-order evidence.

Zahle (Reference Zahle2019) suggests that the combination of observational evidence and interview evidence within a case study sometimes follows the logic of security: Under proper conditions, evidence obtained through participant observation provides second-order evidence that interview evidence is reliable first-order evidence for the phenomenon. Evidence obtained through interviews, for example, may confirm that the researchers’ observations are in fact relevant evidence. Extending this idea to MMR is relatively straightforward, even though skeptics might claim that the challenge of incommensurability that besets it does not arise in this case. For example, in arguing for the use of MMR in impact evaluations of development projects (a kind of causal inference), Michael Woolcock (Reference Woolcock2019) lists multiple uses for qualitative methods, most of which are about producing evidence that the primary quantitative (experimental or quasi-experimental) evidence is indeed reliable.

The third source of added value arises when evidence for distinct claims about the same phenomenon together produces a fuller picture of it such that the epistemic whole is better supported by the totality of evidence than the sum of the individual hypotheses (Lloyd Reference Lloyd2009; Cartwright Reference Cartwright2021). Integration differs from triangulation and security because the evidence need not be independent, nor does it need to be about the same claim. To continue with the trial analogy, two experts produce integrating evidence when they testify on different aspects of the case, one being a ballistics expert testifying about the way the gun was fired and the other being a criminal psychologist testifying about the suspect’s motives.

A useful way of understanding how integration works is to use causal graphs. One could think of the mechanism as the full causal graph and the different pieces of evidence as being about different components of the same graph. Again, diverse evidence in this sense is often produced using different methods, but it need not be. For example, evidence for a mechanism mediating an effect in a field experiment could be produced by a mechanism experiment.

Integration may be realized in a number of ways in various epistemic contexts. Cartwright’s (Reference Cartwright2021) proposal for MMR comes closest to our notion of integration. Cartwright’s argument is that a mix of methods may well be needed if the aim is to support different kinds of subsidiary claims (such as claims about the presence or absence of support factors) that must hold true for the overall claim to be true. Likewise, Shan and Williamson (Reference Shan and Williamson2021) point out that “to establish a causal claim one normally needs to establish the existence of an appropriate conditional correlation and the existence of an appropriate mechanism,” proposing this as a foundation for MMR. Combining evidence for the existence of a correlation and evidence for the existence of a mechanism fits both the logic of integration and some instances of MMR. For example, with regard to nested analysis, statistical studies provide evidence for average causal effects, and case studies evidence for case-specific causal mechanisms. However, the nested analysis framework connects these two types of analysis in a relatively restricted way: case studies either provide (dis)confirmatory instances of the mechanistic hypotheses behind the statistical analysis, or serve as a heuristic source of new hypotheses for subsequent statistical analysis.

4. Causal pluralism and mutually supporting evidence

Some defenders of MMR support the idea that different methods of data production and analysis embody different notions or conceptions of causality, which in turn correspond to alternative philosophical theories of causation (e.g., Cartwright Reference Cartwright2007; Reiss Reference Reiss2009; Illari and Russo Reference Illari and Russo2014). Whereas quantitative methods are usually assumed to presuppose a dependence/difference-making concept of causation, qualitative methods are more in tune with production and process conceptions. In turn, it is sometimes claimed, differences in conceptions of causation imply differences in the rules of causal inference. Johnson et al. (Reference Johnson, Russo and Schoonenboom2019, 155) write, for example, that evidential pluralism, which holds that “many sources of evidence are needed and used in our making of causal claims,” entails conceptual pluralism, namely that “there are different kinds of causes at the conceptual level.” The implication is that there are also different kinds of causes on the level of reality, in other words ontological pluralism. However, if evidential pluralism entailed conceptual and especially ontological pluralism, it would create more problems for MMR than it solves.

First, we have already questioned the idea that having methods of different kinds automatically results in stronger evidence.^{Footnote 3} Second, as Beach and Kaas (Reference Beach and Kaas2020) have argued, strong conceptual pluralism entails that the causal claims supported by different methods are really incommensurable, and therefore that MMR is not feasible to begin with. Our position that it is the variety of evidence, not of methods or data, that is epistemically important further strengthens Beach and Kaas’s argument: If the causal claims for which the distinct methods produced evidence were built on incommensurable concepts of causation, there could not be any well-defined inferential relations between them, and thus no epistemic gains from a variety of evidence.

Finally, conceptual causal pluralism has undesirable practical consequences. If different methods produced evidence for claims that embody different concepts of cause, they could also legitimately support apparently contradictory causal conclusions. For example, in qualitative studies it is claimed that active labor-market policies have a negative impact on employment (through demotivation and exit from the market), whereas quantitative studies indicate that their impact is positive. For the end user of this information (e.g., policy makers) this is deeply unsatisfactory, but for the conceptual pluralist there is not necessarily anything particularly puzzling about it. The apparent contradiction could be attributed to the different methods implying different concepts of “impact” (presumably process and difference making).^{Footnote 4}

As we cannot accept that all MMR research is doomed to either failure or practical irrelevance because of conceptual incommensurability, we have to deny genuine pluralism on either the conceptual or the ontological level. For MMR to work, different kinds of causal claims built from different causal concepts have to be inferentially relatable to one another. This requires at least some degree of conceptual unity and coherence in causal reasoning and inference, which conceptual pluralism does not give. Fortunately, it is one thing to point out that social scientists make use of different causal concepts (e.g., average treatment effect, causal relevance, causal capacity, mechanism, process) with distinct inferential properties, and another to claim that these causal concepts correspond to fundamentally different concepts of causation. Conceptual, let alone ontological, pluralism cannot be motivated by the plurality of causal concepts alone (Reiss Reference Reiss2009).

Even though we cannot conclusively argue in favor of any specific causal account that unifies the causal concepts relevant to MMR, we agree with Rohlfing and Zuber (Reference Rohlfing and Zuber2021) that the interventionist theory of causation utilizing causal graphs is promising in terms of analytic and expressive power. The interventionist account of causation could arguably be used to clarify and regiment inferential relations between unit- and population-level claims, between type and token claims, as well as between causal-effect and mechanism claims. The interventionist framework is not fully neutral with respect to the epistemology of causation, however. Most importantly, it denies the implicit assumption that causal-process observations obtained using case-based research methods could somehow sidestep the fundamental problem of causal inference, namely that the relevant counterfactual values of the variables cannot be directly observed. Beach (Reference Beach2020, 167–68), for example, regards counterfactuals as one possible test for causality that could be achieved by experimental means rather than through analyses of the meaning of the claim being tested. The idea seems to be that if your concept of causation is singularist and process-oriented, you can simply observe causation if you look closely enough.^{Footnote 5}

This is not to say that purely observational case-based causal inference is impossible. What makes it possible, however, is not a distinct concept of causation together with some kind of causal sixth sense. It is rather that the detailed documentation of a series of interactions could ground causal claims when, and insofar as, background causal knowledge is sufficiently reliable. It is quite common in social science for this background knowledge to be based on everyday reasoning and common folk-psychological interpretational practices, and not on explicit social or psychological theories. This may make it easy to forget that reading causation into everyday social interactions does require causal assumptions.

5. Concluding remarks

We have proposed an understanding of MMR in terms of the variety-of-evidence thesis in support of causal inferences, arguing that different MMR procedures can produce and benefit from three distinct kinds of variety of evidence. We also addressed the concern that MMR is not feasible because some methods of data production and analysis embody different, and incommensurable, conceptions of causality. The upshot is that it has the potential to be a powerful methodology for studying causal inference in social science, but its added value can only be secured given the right kind of variety of evidence and the right procedure for bringing it together. The focus on the differences between methods in current methodological discussions on MMR risk distracting scholars from substantial questions concerning the claims for which the diverse evidence is evidence of, how it is diverse over and above being produced by different methods, and in what way different kinds of diversity work in support of causal inference.