The proliferation of multi-method research (MMR), research that employs two or more distinct methods to advance a claim, has become increasingly prominent in a variety of disciplines and fields, from sociology and psychology to education, nursing, and management studies.Footnote 1 As one interdisciplinary text puts it, “the multimethod strategy is simple but powerful. For if our various methods have weaknesses that are truly different, then their convergent findings can be accepted with far greater confidence than any single method's findings would warrant.”Footnote 2 In political science, the debates spurred by King, Keohane, and Verba's (KKV's) Designing Social Inquiry Footnote 3 have evolved into a growing consensus over the virtues of MMR. This is especially true in comparative politics and international relations, fields where MMR seems to have proliferated most rapidly.
MMR can take a number of forms, reflecting the many possible permutations of methods drawn from the broad categories of qualitative, quantitative, and formal approaches.Footnote 4 MMR can combine discrete techniques within one of these families of methods as, for example, when a study employs two formal models predicated on fundamentally different assumptions and parameters (say, a static equilibrium model that assumes perfect information alongside a dynamic model that features continuous Bayesian updating following principles borrowed from cognitive psychology). Similarly, given the wide range of approaches labeled “qualitative,” a multi-method approach can feature two or more types of qualitative analysis (say, a structured-focused small-N comparison alongside an ethnographic study of a single community). On a grander scale, it is possible to imagine MMR that incorporates models or techniques from different disciplines, from economics and social psychology to evolutionary biology and cognitive neuroscience.Footnote 5 The most common multi-method designs in the social sciences, however, tend to combine some type of qualitative research with a statistical analysis or a formal model. It is these forms of MMR that we are most concerned with.
Multi-method scholarship has contributed to methodological pluralism insofar as it implies that formal, quantitative, and qualitative methods are all valuable tools. Many in the discipline, however, are going further. For them, MMR represents a coherent, unified strategy for overcoming the trade-offs of different methods and for generating more valid inferences than any one method can generate on its own.Footnote 6 Viewed in these terms, MMR is not merely a pragmatic option for dealing with different elements of a research program or with practical challenges that arise in the course of research. Instead, it is emerging as a “best practice,” accompanied by the expectation that a single scholar will produce better research by using two or more methods in executing a single project. The tacit corollary is that scholarship based on single-method research (SMR) is handicapped by its built-in limitations and its vulnerability to method-specific error.
We ask whether MMR consistently generates “better” scholarship than SMR and consider the long-term costs of elevating the former to disciplinary best practice. We appreciate that MMR has led to some excellent scholarship and fostered greater appreciation of the value of different methods. We are concerned, however, that treating MMR as an emerging disciplinary best practice would require accepting the underlying premise that it is inherently and consistently capable of reducing error and increasing the validity of findings. Without a more open-ended conversation about when and where this premise actually holds, the uncritical acceptance of MMR as best practice may lead to a new form of method-driven research while marginalizing certain forms of SMR and forfeiting the distinctive benefits these may have to offer. We contribute to such a conversation by advancing two related arguments.
First, any claim that the use of multiple methods reduces error or increases the validity of a finding is defensible only to the extent that the methods used proceed from proximate foundational assumptions. This may be possible, for example, where large-N and small-N studies proceed from a similar empiricist perspective to examine the covariance of particular variables within a common set of boundary conditions. In such cases, there is a possibility that key concepts, variables, and mechanisms can be carefully translated and compared. But this is not possible where one method advances a nomothetic proposition intended to function as a “covering law” while another proceeds from a phenomenological view of the world and offers a context-specific idiographic narrative. Because these approaches are predicated on fundamentally distinct ontologies and conceptions of causality, the findings they generate are ultimately incommensurable and do not serve to strengthen each other. Even in such cases, a multi-method study may yet produce related findings or spur new conversations or inquiries, but this would not constitute a distinctive advantage over a collection of separate studies each based on a single method.
Second, if this is the case, then it is an open question whether the discipline stands to benefit from the unchecked proliferation of multi-method skill sets. Recalling several instances of well-executed SMR, we argue that promoting MMR at the expense of SMR requires trading away the benefits that emerge from the iterated execution of a single method. These benefits include the accumulated experience among those invested in a given method, the expanded possibilities for more innovative applications of that method over time, and the setting of standards for training and evaluation of research employing that method (whether in a single- or multi-method project). In spite of these advantages and in spite of the fact that single-method researchers retain the option of collaborating on multi-method projects, the space available for SMR is likely to decline as professional rewards and opportunities increasingly favor those with multi-method skill sets. This would especially hurt idiographically-oriented qualitative research, which is less easily combined with quantitative and formal approaches. None of this implies that MMR should be discouraged, only that it is not a guarantor of methodological diversity and comes with some costs that have been hidden from view.
We first offer a brief history of MMR, from its emergence in psychology to its current articulation in political science. We then argue that claims emphasizing MMR's ability to reduce error and deliver cross-validated findings are viable only for methods predicated on sufficiently similar ontologies and sufficiently similar conceptions of causality. This argument is developed in the next two sections through an examination of efforts to combine some type of qualitative research first with statistical regressions and then with formal models. Next we examine the potential costs to the discipline in the event that the benefits of specialization, as evident in some exemplary studies employing SMR, are lost in the process of channeling professional rewards to individuals committed to multi-method skill sets. Our conclusion is that if MMR cannot be shown to be consistently coherent and superior to SMR, then the ideal of methodological pluralism is better served not by promoting MMR at the expense of SMR, but by encouraging greater awareness of the distinctive payoffs and tradeoffs of various forms of single- and multi-method research.
The Ascendance of Multi-Method Research
The idea that multiple methods may be deployed simultaneously in a single study—and that such a study would be less prone to built-in errors, and thus superior to research employing a single method—has its origins in the behavioral revolution of the 1950s. A seminal article in the field of psychology introduced a “multitrait-multimethod matrix,”Footnote 7 in which fundamentally different methods of observation are relied upon to counter method-specific variance and facilitate greater cross-validation. The related notion of methodological “triangulation” gained prominence in the 1960s and 1970s in such disciplines as anthropology, sociology, education, and organizational research.Footnote 8 Although very little was done to actually facilitate the use of multiple methods in research practice, leading methodology texts in these various disciplines increasingly emphasized “the desirability of mixing methods given the strengths and weaknesses found in single method designs.”Footnote 9
In political science, the first explicit moves in this direction were evident by the 1970s, when scholars under the influence of the behavioral revolution came to view case studies and small-N comparisons as viable complements to quantitative approaches in their efforts to generate or test hypotheses.Footnote 10 However, the former were still thought of as second best, to be deployed to compensate for the lack of reliable data and accurate measurement. Moreover, methodological pluralism at the time tended to imply that different kinds of approaches could be mobilized to investigate hypotheses derived from a single general theory. For example, modernization theory in comparative politics and structural-realism in international relations both allowed for the application of different methods to compare or test propositions derived from the general theory. But this presumption depended on a prior commitment to a theoretical paradigm.
It is only since the mid-1990s that we see a growing commitment to the notion that a research product can and should employ multiple methods whenever possible to limit method-specific errors and cross-validate findings. KKV, while they did not endorse MMR, set the stage for it by seeking to establish uniform principles for executing and assessing qualitative and quantitative research. Downplaying epistemology, KKV argued that the process of identifying, ordering, and representing “significant” facts involved essentially the same inferential process in both quantitative approaches as well as different types of qualitative work. While acknowledging that their rules of inference were simpler to represent in the language of quantitative researchers, KKV emphasized that much social scientific research “does not fit neatly into one category or the other,” and suggested that the best research “often combined features of each.”Footnote 11 With this move, KKV not only drew qualitative researchers into a broader debate over methodological issues, but also encouraged the discipline to embrace the ideal of methodological pluralism.Footnote 12
While the resulting discussions paved the way for a more nuanced understanding of “qualitative” research, most relevant to the later proliferation of multi-method designs was Sidney Tarrow's discussion of triangulation between quantitative and qualitative strategies.Footnote 13 Tarrow resisted KKV's efforts to assimilate concepts and logics in qualitative studies into the standard language of quantitative analysis. For example, whereas KKV equated process-tracing with increasing the number of relevant observations, Tarrow treated it as a fundamentally different type of observation—an insight that would inform later efforts to identify the distinctive challenges of defining and analyzing causal process.Footnote 14 Tarrow went on to identify practical strategies for fruitfully combining the distinctive strengths of quantitative and qualitative research. These included the use of qualitative research to identify non-systematic sources of variation and to adjust the value of systematic variables in a statistical analysis; this is the case, for example, where “tipping points” identified through historical research are used to mark shifts in the value attached to systematic variables over time. Tarrow also noted that quantitative studies can help to more clearly demonstrate the representativeness of cases being analyzed through qualitative methods.
It is worth emphasizing that each of the examples of triangulation Tarrow discussed brought in additional methods to cope with practical issues in the course of research.Footnote 15 In each case, the deployment of separate methods was accompanied by the respecification of hypotheses in ways that reflect the parameters and profiles of those methods while keeping the focus on the same empirical phenomenon. For example, in Valerie Bunce's study of leadership rotation in Western and socialist systems, it was the partial nature of quantitative data and the uneven possibilities for qualitative investigation that created a need to use both to make the most robust case possible.Footnote 16 And in Tarrow's own book, a qualitative study focused on a single community (Florence) led to findings that seemed worth corroborating through quantitative analysis designed to reveal patterns of protest across similar communities throughout Italy.Footnote 17 In all these instances, the use of multiple methods was not an end in itself but rather a response to challenges and opportunities that emerged in the process of conducting research.
Subsequent discussions would pave the way for a bolder conception of MMR. Brady and Collier's Rethinking Social Inquiry Footnote 18 compiled some of the most sophisticated responses to KKV to distill insights that would guide subsequent efforts to combine quantitative and qualitative research. One key insight the editors emphasized was that MMR need not mean a full-blown unification of methodological systems but rather the selective deployment of specific tools of data collection and interpretation. They did not see the diversity of methodological tools as an impediment to finding common principles for advancing and assessing scholarship. In fact, an emphasis on tools effectively disaggregated methods into their component parts; for example, a case study did not have to be wedded to a historical narrative, and a statistical analysis had applications that went beyond generating inferences. Yet the net effect of this move was to encourage research designs that would systematically incorporate quantitative and qualitative strategies in order to distinguish and link “data-set observations” and “causal process observations.”Footnote 19
Another move that hastened the proliferation of multi-methodism was the elevation of “problem-driven” over “method-driven” research, as evident in a volume edited by Ian Shapiro, Rogers Smith, and Tarek Masoud.Footnote 20 While most contributors declined to draw a stark distinction between problem-driven and method-driven research, they rejected adherence to a specific method in favor of approaches adapted to address substantive problems. Elsewhere, Shapiro and Wendt warned against a fixation on the most technical aspects of scientific inquiry and argued that the nature of the question should determine the most effective approach: “Sometimes, quantitative, cross-sectional analysis will provide strong material for good abductive inferences. In other cases, qualitative or historical analysis will be more appropriate.”Footnote 21
Others have gone further in seeking to embrace and promote MMR. Even as scholars have moved away from KKV's uniform rules of inference, they have sought to create a stronger rationale for MMR, viewing it as the best or only way to operationalize methodological pluralism while increasing the likelihood of more valid findings. For David Laitin, so long as there is a generalized commitment to a “scientific frame,” a tripartite methodology combining formal models, statistical analysis, and qualitative narratives “is the best defense we have against error and the surest hope for valid inference.”Footnote 22 From a different angle, Evan Lieberman's “nested analysis”—discussed in greater detail later—purports to integrate large-N and small-N techniques in order to identify appropriate cases, generate historical explanations for specific outcomes, and advance general hypotheses that are then tested across a larger population of cases.Footnote 23 And John Gerring has offered a unified framework for assessing social-scientific analysis on the basis of fundamental criteria that apply in varying degrees to any strategy intended to establish causation, regardless of the types of data and method used.Footnote 24
To be sure, these positions are not all identical. Compared to Laitin's emphasis on error-reduction through multiple methods, for example, Gerring adopts a more pragmatic stance that recognizes the inescapability of trade-offs. Even more stark differences have been evident in contentious debates that have erupted over the assumptions, objectives and payoffs of different multi-method designs.Footnote 25 Nevertheless, most proponents of MMR have gravitated toward a common view that despite the challenges of incommensurability, certain uniform principles of “scientific” analysis (such as parsimony, falsifiability, coherence, etc.) permit the integration of findings generated through diverse methods; and that such integration is desirable whenever possible because the complementary strengths of different approaches allow them to collectively offset the weaknesses of each and to yield more valid inferences than any one method allows for.Footnote 26
The growing acceptance of this premise has led to disciplinary practices and professional incentives that increasingly favor MMR at the expense of SMR.Footnote 27 This is evident in the institutional support for MMR (for example, in the form of the renamed American Political Science Association's Organized Section for Qualitative and Multi-Method Research and the Institute for Qualitative and Multi-Method Research); in the launching of journals such as The Journal of Mixed Method Research that exclusively publish work employing more than one method; and in the growing interest of major scholarly presses in books that employ MMR on the assumption that they will appeal to a wider audience.Footnote 28 Even more significant is that new cohorts of graduate students and junior scholars—particularly in comparative politics and international relations—are being rewarded for employing multiple methods, as evident in the awarding of dissertation prizes, the hiring and promotion of faculty at top research universities, the prospects for publication at top journals and presses, and opportunities to secure the most prestigious grants. In view of these trends, the time is ripe for a critical examination of the presumed benefits and unacknowledged costs of a further proliferation of MMR.
The Problem of Foundations
The intuitive appeal of MMR is understandable. As one methods text contends, “the flaws of one method are often the strengths of another; and by combining methods, observers can achieve the best of each while overcoming their unique deficiencies.”Footnote 29 On the surface, this appears to be a compelling statement. It effectively echoes the common-sense view that SMR faces methodological trade-offs that MMR can bypass by reducing method-specific errors and facilitating cross-method validation. We argue here that these premises hold for two or more methods only when they happen to proceed on the basis of ontologies that are compatible if not similar, and only when these ontologies incorporate similar conceptions of causality cast at the same level of abstraction. Where these conditions do not hold, the use of multiple methods neither eliminates method-specific error, nor serves to cross-validate the same set of propositions. In this case, MMR may yet generate complementary insights and related findings, but it would be neither inherently superior to SMR nor fundamentally different from a set of discrete research products that employ different approaches to investigate a given substantive problem.
Our argument builds on Peter Hall's contention that “the appropriateness of a particular set of methods for a given problem turns on the nature of the causal relationships they are meant to discover.”Footnote 30 While one may dispute Hall's characterization of typical research programs or the novelty of the challenges posed by complexity, his broader argument affirms the importance of being self-conscious with respect to the ontologies implied in the investigation of a research question and the adoption of a given method.Footnote 31 Merely adopting a “problem-driven” stance does not suffice since problems can be formulated to privilege certain methods predicated on certain ontologies.Footnote 32 For example, a problem can be constructed so that endogeneity is limited enough to permit probabilistic inferences on the basis of a frequency distribution across a set of observations. Alternatively, when a problem encompasses extensive endogeneity, it lends itself to a processual approach that can highlight complex interactions within a narrowly delimited set of contexts. And when a problem is formulated on the basis of an ontology that accords the same status to unobservable identities or cognitive dispositions as it does to observable actions and practices, it is more likely to require a hermeneutic approach.
Certainly it is possible to overstate the problem of incommensurability. Foundational assumptions can get reified to the point where they obscure connections between substantive findings and block off communication across research communities dealing with aspects of the same problems. Indeed, there are occasions when it is possible to relax some of these assumptions so as to facilitate the articulation of a puzzle from different theoretical vantage points.Footnote 33 This is especially true in the case of “analytic eclecticism,” which aims to develop flexible, middle-range frameworks to tackle substantive problems while linking together analytic constructs originally formulated within research traditions thought to be incommensurable.Footnote 34
Yet not all foundational assumptions can be relaxed easily, particularly those cast at the level of ontology and directly relevant to the execution of specific methods. In principle, ontology may be distinct from epistemology and methodology, but ontological assumptions can place significant constraints on one's range of epistemic commitments and methodological choices. Ontological assumptions imply certain boundary conditions for the investigation of particular questions, assign priority to certain types of observations and certain aspects of social reality, specify certain understandings of whether and when some set of observations constitute causation, and suggest certain strategies for presenting and assessing research products. Consequently, modes of inquiry premised on one set of ontologies are not automatically interchangeable with those predicated on a different set, which implies that not all multi-method combinations are equally capable of generating a coherent set of findings, let alone provide cross-method validation for these findings.
Our point is not that there is a one-to-one correspondence between a method and a research problem, or that claims advanced through different methods are always incommensurable. It is that there are limits to the range of methods that can be triangulated to validate a given substantive claim, and that these limits derive from certain foundational assumptions about the nature of the social world and about the possibilities and goals of social inquiry. And even within this range of methods, significant effort would have to be devoted to carefully translating the observations, variables, mechanisms and causal principles associated with different methods to minimize the dangers of unrecognized conceptual problems or excessive conceptual stretching.Footnote 35
To illuminate the varying degrees of commensurability between different methods, it is necessary to dispense with the standard tripartite distinction between formal, quantitative, and qualitative studies. In part, this is because the category of qualitative methods is a residual one, encompassing a wide range of approaches predicated on quite different foundations. As Bevir and Kedar have argued, the divide between “naturalist” and “anti-naturalist” ontologies separates not only quantitative from qualitative methods, but often represents a significant chasm within the family of qualitative methods itself.Footnote 36 In effect, two qualitative approaches may be at a significantly greater distance from each other in terms of their ontologies than one of these approaches is from a regression analysis or a game-theoretic model. A crucial-case analysis and a study based on participant-observation, for example, proceed from radically different foundations and have quite different purposes. The former is designed to test a general proposition in a “must fit” case and may be usefully triangulated with other approaches aimed at generating causal inferences. The latter, by contrast, eschews generalization altogether and requires the investigator to directly experience a set of social relations and practices in order to uncover context-specific meanings held by actors within a specific community. Indeed, even the category of quantitative analysis is not as uniform as generally assumed. This is evident, for example, when we contrast the ontologies undergirding field experiments (which involve data that can only be measured in a given context) and lab experiments (which involve controls intended to minimize contextual effects).
Hence, the problem of foundations in MMR is best viewed not in terms of methodological approaches but in relation to the extent to which they conform to ideal-typical forms of nomothetic or idiographic analysis. In practice, the two terms essentially represent poles of a continuum along which there are gradations as we move from context-bound interpretations to general axioms or law-like principles.Footnote 37 At the poles, the distinction may partially overlap with, but is not identical to, other familiar distinctions in the philosophy of science, such as positivist-relativist, objectivist-subjectivist, or materialist-idealist. And, in principle, research generated at different points along the continuum can all be framed as interpretive acts intended to generate “causal stories” of varying levels of abstraction.Footnote 38 Nevertheless, differentiating social-scientific research products along a continuum between idealized notions of nomothetic and idiographic research can be useful for capturing the varying degrees to which incommensurability poses a problem for research involving different methods.
As we move from the nomothetic end of the continuum to the idiographic end, fundamental differences emerge in the aims of researchers, and these aims are related to fundamental differences in what counts as causality and what counts as evidence for a particular kind of claim.Footnote 39 Near the nomothetic end of the continuum, some forms of qualitative small-N or single-case studies may proceed from a foundation that is not fundamentally different from that informing the conventional quantitative worldview.Footnote 40 In fact, the very act of treating one's object of empirical analysis as a “case” and one's observations as “data points” suggests a commitment to an endeavor ultimately geared towards identifying or confirming general laws or law-like regularities. Most formal models, although they can be adapted to shed light on contextually situated behavior, are also closer to the nomothetic pole in view of their emphasis on internal logical consistency in axiomatically deriving causal propositions from universal first principles. As we move toward the idiographic end of the spectrum, however, the aims of researchers move away from the uncovering of general law-like propositions towards progressively “thicker” narratives intended to produce historicized understandings of contextually situated social action. These can take the form of descriptive accounts that eschew explicit causal claims and instead seek to represent behaviors, social relations, and sequences of events as precisely as possible. For those even closer to the idiographic pole, however, the fundamental inter-subjectivity of both social life and social inquiry places a premium on uncovering and understanding the often unobservable processes of “meaning making” among actors engaged in shared practices and symbolic interactions.Footnote 41
Where different methods are situated in relatively close proximity to one another along the nomothetic-idiographic continuum, the problems of incommensurability are reduced and the possibility of cross-validated inferences correspondingly increased. However, where the methods are located at a greater distance from each other, MMR can do no more than generate a set of discrete findings that have something to do with one another but do not serve to cross-validate each other. This suggests that the unreflective pursuit of MMR without due attention to ontology effectively does little more than juxtapose single-method studies that, at best, produce findings that share a family resemblance as a result of significant “conceptual stretching,”Footnote 42 and, at worst, employ one method as “mere window dressing” while relying primarily on another to substantiate the core causal argument.Footnote 43 The next two sections elaborate on this point by examining efforts to combine some type of qualitative research with, respectively, statistical analysis and formal models.
When Cases Meet Regressions: Complementarity or Cross-Validation?
While quantitative and qualitative researchers are often thought to belong to two distinct cultures,Footnote 44 certain strands of qualitative research proceed from the same basic empiricism that informs standard quantitative analysis.Footnote 45 Whereas formal modelers take for granted the primacy of deductive logic and internal logical consistency, empiricists are generally hostile to unobservables and seek to ground inferences in directly observable social phenomena.Footnote 46 Empiricism effectively privileges an inductive orientation as well as a probabilistic view of causality in which mean causal effects are inferred from a finite set of observations. Empiricism can thus provide a foundation for both inferential statistics as well as several well-known types of qualitative analysis including “crucial case” analyses, small-N comparisons of “least similar” or “most similar” cases, and more recently, comparative analysis based on set theory.Footnote 47 These approaches may vary in the number of cases analyzed and in the extent to which they draw upon data set observations or causal process observations.Footnote 48 Nevertheless, they converge on the broader principle that observable empirical phenomena constitute the primary basis for portable inferences about likely causal relationships.
In fact, James Mahoney has sought to articulate a “unified theory of causality” that permits translation of variables and causal logics across the quantitative-qualitative divide. He argues that the understanding of causal effects employed in inferential large-N analyses, while distributed across a full population, is ultimately derivative of the understanding of causation underlying case-based studies that seek to identify necessary, sufficient, and INUS causes (causes that are insufficient but necessary components of unnecessary but sufficient causal factors).Footnote 49 It is worth stressing, however, that this conception of causality is not intended to reach all types of qualitative research. This is true of studies employing a more historicized conception of causality that is not attached to discrete necessary and sufficient conditions but rather embedded in long-term evolutionary processes featuring complex interactions, threshold effects, and feedback loops.Footnote 50 It is even more true in the case of idiographically oriented qualitative studies proceeding from philosophical foundations such as hermeneutics, phenomenology, or phronesis. Such studies tend to focus on complexes of meanings assigned to experiences and practices within singular contexts and thus have no functional equivalents for general necessary and sufficient conditions.Footnote 51
In research practice, combinations of regressions and case studies tend to be justified primarily in terms of the complementarity of findings rather than the commonality of foundations. Each approach offers something that theoretically compensates for the limitations of the other. Statistical analysis offers a large number of observations so that appropriate cases can be identified from the range of possible outcomes and so that causal hypotheses can be inferred from robust correlations; however, it is not able to trace the operation of causal mechanisms. Case studies and small-N comparisons (even when explicitly designed to generate or test hypotheses) cannot precisely capture the frequency distribution of effects, but are thought to be more useful in tracing processes and identifying mechanisms that link certain initial conditions to outcomes. The combination of the two is presumed to yield complementary sets of advantages and limitations that are so well fitted that the result is seen as a thoroughly unified, integrated approach that is more valuable than either of the individual methods. This is precisely the rationale for the “nested” approach advanced by Evan Lieberman: regression analysis facilitates unbiased case selection (by establishing the frequency distribution of outcomes) and helps identify the most relevant mechanisms (through statistical estimates of the relative strength of variables), with the case studies subsequently revealing how these mechanisms contribute to the unfolding of causal processes within the relevant contexts.Footnote 52
While this view makes perfect sense for certain research questions, it is important not to conflate the benefits of complementarity with error-reduction and cross-validation. It may be intuitively appealing to assume that methods that complement each other also work to neutralize each other's method-specific errors and increase the validation of each other's findings. But whether this assumption holds for all combinations of methods in all research endeavors is very much an open question. As even proponents of MMR recognize, method-specific errors do not always cancel each other out, and undetected sources of error can compromise quantitative and qualitative methods.Footnote 53 Even in a well-constructed nested approach, it is possible that some errors get compounded as, for example, when a variable is excluded from the within-case analysis because a slight misspecification of the quantitative model makes it appear to be statistically insignificant.Footnote 54
More importantly, as Mahoney notes, case studies and regressions work differently to achieve different aims: “Case studies seek to tell us why particular outcomes happened in specific cases; statistical studies try to estimate the average effects of variables of interest. Both are important … but for some topics one cannot pursue them at the same time.”Footnote 55 The strength of case studies lies in their ability to trace causal processes, and yet the very notion of process implies an ontology characterized by “extensive endogeneity and the ubiquity of complex interaction effects.”Footnote 56 Among other things, such an ontology requires the endogenization of temporal elements as duration, tempo, sequence, and timing, which can be critical to determining the size of the effects generated by particular mechanisms under distinctive sets of initial conditions.Footnote 57 Establishing the relevance of such elements requires careful attention to the background conditions operating in each case and thus to the manner in which case-specific contextual factors influence the operation of general causal mechanisms.Footnote 58 Also, to operationalize variables that can be measured across a large number of observations, quantitative studies tend to compress or simplify complex pieces of historical information, but this process may require making sweeping assumptions that conflict with case-specific historical evidence that has held up over repeated examinations of qualitative researchers with expertise on the relevant cases.Footnote 59 This is not simply a matter of generalizability or careful research; it is a matter of recognizing that statistical inferences and case studies, even when investigating problems that are closely related, do not necessarily generate same-level causal propositions and thus do not automatically offer external validation to each other.Footnote 60
This point is evident in Lieberman's substantive applications of nested analysis. Race and Regionalism and Boundaries of Contagion are both prize-winning books touted for the exemplary blending of quantitative and qualitative analysis.Footnote 61 Both books represent excellent pieces of scholarship that speak well of Lieberman's impressive ability to design and carry out quantitative and qualitative analyses. What we want to emphasize here, however, is that neither book is offered as, nor can be treated as, evidence that MMR is uniquely capable of overcoming methodological trade-offs and generating cross-validated findings. We view the value-added of the two books through the lens of the more modest case for methodological triangulation advanced in Tarrow's aforementioned response to KKV. What is noteworthy is how different kinds of tools are sequentially employed to discharge several related tasks: the construction of his research question, the establishment of variation, the selection of cases, the identification of the most significant mechanisms, the tracing of causal processes that generate the outcomes, and the testing of the causal explanation. In Race and Regionalism, Lieberman offers a paired comparison of the historical evolution of the tax state in Brazil and South Africa alongside a large-N analysis intended to establish the frequency distribution of various types of tax systems and the relative strengths of factors hypothesized to account for that variation. In Boundaries of Contagion, quantitative studies help to relate the divisiveness of ethnic competition to variation in the effectiveness and scale of government responses to the HIV/AIDS epidemic in developing areas; a paired comparison of Brazil and South Africa is deployed to build an explanatory model showing how the institutionalization of ethnic boundaries can generate different behaviors among the most and least affected groups; and a third case study of India and a statistical analysis of variations among Indian states are used to test the model.
One may quibble with Lieberman's model-specification and case-selection,Footnote 62 but there is no denying that it is truly an achievement for one scholar to be able to do everything that Lieberman has done in the two books. This achievement does not, however, herald a new era of cross-validation. For this to happen, it would be necessary to not only deploy different methods for discrete research objectives but to use them to affirm the same truth claims at the same level of generality. This does not appear to be the case in Lieberman's work. In Race and Regionalism, for example, the case studies rely on archival research and interviews to reconstruct the path of institutional development across various critical junctures. The emergent historical-institutionalist explanation for the particular outcome in each case depends on tracing the evolving motivations of upper-class actors in the choice of tax policies within the context of a given conception of the national political community. The paired comparison generates a complex account of the variation across Brazil and South Africa, one that demonstrates which factors mattered most, and when and how. The statistical analysis shows that certain measurable components of the case-specific narratives appear to have stronger effects than variables representing rival hypotheses. But it cannot possibly offer external validation for claims related to the shifting motivations of various groups across critical junctures that are defined in relation to country-specific historical processes. Thus, the complementary findings Lieberman offers, while original and useful, do not provide evidence that nested analysis can bypass fundamental methodological trade-offs and increase the validity of a particular finding.
The difficulties of squaring the findings from quantitative analysis and qualitative case studies may be one reason why, in some cases of MMR, the “heavy lifting” sometimes ends up being done by one of these approaches. Two other excellent books published in the same year illustrate this point perfectly. Yoshiko Herrera's Imagined Economies is an impressive study of regional activism in post-Soviet Russia that relies primarily on a first-rate content analysis of local reports and leaders' statements in the regions of Samara and Sverdlosk. It is this qualitative treatment, supported by rich historical background, that provides the main evidence for her propositions concerning regional perceptions of relative economic status. The statistical analysis is deployed to expose the weak effects of variables posited in pre-existing explanations, but it is not designed to offer the same kind of test for the more complex constructivist argument that emerges from her case studies.Footnote 63 Conversely, Jon Pevehouse's Democracy from Above offers a highly original account of how interactions between domestic elites and regional organizations influence democratic transitions in post-authoritarian settings. What makes the argument convincing is a series of high-caliber statistical analyses presented in more than two dozen tables distributed across three separate chapters. The case-based evidence is drawn from six countries that are carefully selected and that produce stories that match the statistical findings. However, none of the case studies goes much beyond a dozen pages of continuous discussion, and none is designed to engage contending narratives that area specialists may consider relevant in accounting for a country-specific outcome.Footnote 64 It seems fair to suggest that Herrera has expended much more time and effort in tracing case-specific processes than in developing quantitative models, just as it is clear that Pevehouse has expended more time and effort in developing his quantitative models than in constructing original narratives for each of his cases. This does not constitute a criticism of either book. It does suggest, however, that the value-added of many studies that combine quantitative and qualitative analysis often depends on leveraging one or the other rather than in the balanced integration of the two.
Finally, it bears emphasizing that interpretive research that is more idiographically oriented is even more difficult to combine with statistical analysis than is a case study designed to test causal hypotheses or trace the effects of causal mechanisms. Studies that proceed from a hermeneutic or phenomenological foundation, for example, are less concerned with causal generalization than with understanding the shared meanings that govern discursive practices and social relations situated within a particular time- and space-bound context.Footnote 65 This is evident in ethnographic explorations based on the observer's protracted immersion in a single community, studies of hidden transcripts or performative practices among members of local communities, and comparisons designed to accentuate the uniqueness of cultural practices in separate contexts.Footnote 66 Researchers who execute these kinds of studies are no less concerned with evidence or rigor than other social scientists; they do, however, adopt a quite different understanding of what counts as evidence, and how it is to be identified and articulated.Footnote 67 The focus is on the inter-subjective processes through which actors draw upon specific templates or worldviews to develop symbolic practices and ascribe meaning to their shared experiences within a particular context. Efforts to interpret these processes cannot be easily reformulated as, or subsumed within, general propositions that incorporate quantifiable variables or causal mechanisms. This is perhaps why MMR, in practice, does not often feature combinations of statistical and interpretive work, but this is also a reason why multi-method scholarship cannot function as a guarantor of methodological pluralism.
Formal Models and Narratives: Illustration without Cross-Validation
Prior to the emergence of MMR as an ideal approach to the study of politics, efforts to provide empirical support for formal models focused primarily upon quantitative data. Bruce Bueno de Mesquita's The War Trap, for example, followed a formula whereby empirical propositions derived deductively from formal utility-maximization models would be subjected to rigorous statistical tests.Footnote 68 Reversing the sequence, John Goldthorpe called for the deployment of formal rational-choice models to uncover the causal logics behind inferences generated through the quantitative analysis of large-N data sets.Footnote 69 Despite the foundational differences between the empiricism underlying statistical analysis and the logicism underlying formal models,Footnote 70 findings generated by deploying the two approaches were deemed to be combinable and compelling in part because both approaches are nomothetically oriented—offering general concepts and law-like propositions intended to apply to a full population of cases.
Yet the past decade has witnessed a growing discomfort in the discipline with studies that rely solely upon quantitative analysis for the purpose of model-testing. Some have pointed to crucial differences between formal and statistical approaches when it comes to the treatment of empirical regularities, particularly in establishing the significance of outliers.Footnote 71 Others have expressed skepticism that any quantitative analysis specifically designed to test propositions derived from a formal model is likely to be compromised by the restrictive assumptions of the model and by insufficient attention to alternative causal factors.Footnote 72 Still others go further, insisting that given the lack of explicit attention to causal mechanisms in statistical models, the most compelling forms of MMR need to incorporate detailed case studies to both inform and test formal models.Footnote 73 In some formulations, the sequential deployment of formal modeling and case study is displaced by a more integrated process featuring a rapid and continuous iteration between deductive model-building and inductive analysis of cases.Footnote 74
In practice, however, there are some fundamental limits to the reliance on case-based empirics for the purpose of developing or refining a proper formal model. It may be possible to test for the plausibility of the assumptions of a model given the background conditions present in a particular case. It may even be possible to revise expectations about how a given mechanism interacts with case-specific contextual attributes. However, the notion that case-specific observations can play a significant role in shaping or reformulating a model requires relaxing the very conception of a formal model. While not all models are predicated on general covering laws or universal axioms, a formal model by definition proceeds from a set of pure concepts and first principles within which causally significant elements remain hermetically embedded. If such elements were to be open to reformulation on the basis of a qualitative case study, it would no longer be a formal model in the proper sense of the term.
There is certainly value in using cases to illustrate the relevance of a general causal principle with rich detail or to think through whether a model is putting excessive weight on specific assumptions or principles. There is also value in relying on models to tease out the operation of uniform logics and general mechanisms that exert similar influences across cases that correspond to different historical outcomes. But such benefits do not constitute a basis for ensuring error-reduction or cross-validation. Moreover, in the case of models that are specifically designed to explain a particular outcome, there is the danger that model and case are so closely aligned to one another that the former will become idiosyncratic and thus lose one of its main benefits—the articulation of the essential logics that account for uniformities in individual behavior and influence outcomes across varied contexts. In such cases, the rationale provided for doing a case study is so compelling that one wonders whether the attached model is anything more than a purely academic exercise. Consider the following lucid exposition of the functions of case studies in MMR that also includes formal models:
case studies can trace not only which choices were considered and actions were taken, they can also show that some other actions were deliberately avoided in anticipation of the choices and actions of the other player(s). Moreover, case studies are not yoked to the assumption that any unavoidably simplified formal model represents the true data-generating process…. Case studies can trace and establish causal mechanisms in the midst of a potentially overwhelming number of otherwise confounding factors. Even if the empirical process does not exactly match the formal model, case studies can often still offer a judgment of the relative fit and relevance of the proposed mechanism.Footnote 75
Certainly case studies that can do all these things are likely to offer valuable insights in any project that includes them. What is less obvious is just how much is left for a formal model to do, especially where the model is designed to highlight the operation of mechanisms in the context of a specific case or outcome. In such projects, if a case study can by itself “trace and establish causal mechanisms,” the modeling exercise almost seems superfluous at least for the purpose of analyzing causal processes and explaining outcomes in particular cases.
The flip side of this argument—that qualitative case studies cannot substantiate the distinctive insights that formal models are designed to offer—is apparent in Kenneth Schultz's Democracy and Coercive Diplomacy, regarded as an exemplary application of MMR in international relations. Schultz offers a candid assessment of just what a case study brings to the table once a formal model is introduced. He acknowledges the value of studies in capturing the richness of detail, but then insists that the value of formal models lies precisely in their ability to abstract away this richness in order to expose the “essential strategic dynamic that lurks underneath complex interactions.”Footnote 76 Moreover, because the predictions of formal models depend on stipulating the effects of one variable while holding all others constant, Schultz explicitly states that qualitative case studies are not as effective as statistical analyses in introducing the controls required to test such predictions. This effectively leaves qualitative studies as clearly the least essential of the three approaches in relation to the building and testing of theoretical propositions.
There are other methodological challenges as well when formal models are combined with qualitative case studies. Often, the cases chosen for qualitative analysis are the “most likely” cases that fit well with the model rather than the “least likely” cases needed to perform a crucial-case analysis.Footnote 77 Moreover, there is a danger that an epistemic commitment to general individual-level causal mechanisms effectively biases the kinds of data and sources to be utilized. Even historically-minded qualitative scholars specializing in specific countries and time periods run the risk of selection bias as they contend with the multiplicity of sources and diverse traditions of historiography.Footnote 78 The problem is magnified when case studies are designed to track the operation of general causal logics stipulated in a model and, in so doing, fail to engage empirical observations and competing narratives furnished by other qualitative researchers with expertise in the relevant case.
The tension between context-specific observations and formal modeling is most pronounced in the case of “analytic narratives,” a particular form of MMR that purports to integrate “thick” idiographic narratives and extensive-form game-theoretic models.Footnote 79 Proponents of this approach claim that it is designed to capture the “benefits to be gained from the systematic use of theory” while also valuing a “close dialogue with case materials” in the development of theory. The latter reveal context-specific historical processes, behaviors, and interaction patterns, which is why the task of articulating the operation of general mechanisms is left to extensive-form game-theoretic models that feature “explicit and formal lines of reasoning which facilitate both exposition and explanation.”Footnote 80 More flexible conceptions of analytic narratives allow for models that are not exclusively wedded to game theory or rational-choice theory.Footnote 81 Even so, the models primarily depend on the operation of general individual-level mechanisms that are assumed to exert similar effects, all other things being equal. Thus, while an analytic narrative demonstrates an impressive range of skills, the narrative ultimately performs an illustrative function, not an explanatory one; it is the analytic model that spotlights the operative causal mechanisms in general explanations of outcomes.
What stands in the way of a more meaningful integration of “thick” narratives with “thin” formal models is the vast chasm between the ontologies undergirding a nomothetic model and idiographic narrative when it comes to the interpretation of social action. Even if a generic methodological individualism might be said to provide a metatheoretical link between a game-theoretic model and certain forms of idiographic analysis—for example, Harold Garfinkel's ethnomethodologyFootnote 82—the latter are still concerned with how a particular gestalt of beliefs, norms, and habits influence how individuals define their social situations and construct their identities, interests and roles within a given setting. Thus, in a typical analytic narrative, while a few auxiliary assumptions of a game-theoretic model may be adjusted to accommodate specific cases, the general axioms and assumptions at the core of an actor-centered model cannot be validated through narratives designed to reveal their plausibility within a given context. Such narratives, however “thick,” would necessarily have to privilege the logic of consequences in order to illustrate the relevance of the model, thereby neglecting patterns of social action that appear to conform more to the logic of appropriateness.Footnote 83
In sum, a qualitative case study, wherever it is situated on the nomothetic-idiographic continuum, is not in a position to refute or validate causal propositions axiomatically derived from untestable and indisputable first principles. Even where a case study and formal model highlight the effects of a particular mechanism, they operate at different levels of generality; at best, the former can be treated as a single instance of a general outcome that is explained through a deductively-derived general proposition characterized by internal logical consistency. In studies where a model is designed around a specific outcome and continually altered in response to findings from a single case-study, there is the question of what distinctive value the modeling would have if it would have to be reformulated for the analysis new cases and outcomes. In either case, while qualitative case studies are valuable for illustrating the plausibility of explanatory principles in a formal model, illustration cannot be treated as cross-validation.
The Hidden Costs of MMR? Recalling What SMR Brings to the Table
While the discipline surely stands to benefit from adding MMR to the repertoire of methods available to researchers, we now consider the expectation, most visible in the fields of comparative politics and international relations, that individual scholars need to invest in multi-method skill sets in order to leverage whatever benefits MMR has to offer. This expectation may seem reasonable to established practitioners of MMR, and it is being internalized by students in doctoral programs that require extensive training in more than one method. We fear, however, that it places an undue burden on, and limits the potential contributions of, those who might otherwise prefer to focus their energies on certain forms of SMR. Here we consider what SMR has been bringing to the table and what the discipline might lose by viewing the putative benefits of MMR as dependent upon multi-method skill sets for individuals. A preference for SMR does not imply method-driven research or dogmatic belief in the superiority of any one method; and it can be accompanied by a commitment to methodological pluralism and a willingness to explore other approaches. What distinguishes single-method scholarship is, first, that its main claims are produced through the skillful use of a particular method, and second, that it follows the epistemic norms and evidentiary standards set by others who are also proficient in that method.
We begin by taking note of three highly regarded books that constitute examples of well-executed SMR in three different subfields—one focused on quantitative analysis, one on textual interpretation, and one on complex and innovative applications of game theory.Footnote 84 Although the authors of these works embrace methodological pluralism and, in some cases, have worked with other methods, each of the works primarily relies on the masterful application of a single method to advance its core claims. Each work evinces a high level of innovation and refinement that is made possible by the experience and expertise accumulated through working extensively (though not necessarily exclusively) with a particular method over the course of several projects. And each offers a set of compelling substantive findings that depend primarily on the sophisticated deployment of one method but that also provide insights and foils for scholars who may bring other methods to bear in addressing the same research questions.
Larry Bartels' Unequal Democracy presents the results of six years of empirical research into the political causes and consequences of economic inequality in the United States.Footnote 85 Like other practitioners of single method research, Bartels also brings to his work diverse methodological sensibilities, having amassed a significant amount of historical knowledge over the course of his career. While this knowledge enables him to engage a variety of theoretical perspectives in the process of developing and employing different measures of inequality in the United States, the study itself relies primarily on the masterful application of different types of statistical techniques to analyze vast amounts of data. The result is an empirically compelling and theoretically complex explanation that integrates elements of political behavior and political institutions while challenging standard accounts based on purely economic models. His book does not claim to end all debate on the subject, but it does offer a powerful argument, as well as a new baseline, for further research by scholars who may employ different methodologies and analytic lenses.
Lisa Wedeen's Peripheral Visions: Publics, Power and Performance in Yemen employs a textual interpretivist approach to deciphering the performative practices through which people forge national attachments in the absence of strong state institutions.Footnote 86 Drawing upon her extensive fieldwork as well as her experience from researching her previous book on Syria, Wedeen is able to identify previously obscured connections between events as disparate as a presidential election, a poetry reading, and the trial of a serial killer, all of which she treats as potential sites of national engagement. Her interpretation of the significance of “qat-chews” stands out as a particularly compelling exposition of a form of political engagement that is neither captured in standard analyses of electoral democracy nor reducible to Habermasian notions of deliberation in the public sphere. Wedeen is fully aware of the limits of context-sensitive interpretive analysis; at the same time, her analysis points to new arenas and questions for research into political participation and democratic practice.
If Bartels' and Wedeen' books are relatively straightforward illustrations of the value of honing a single method, this is no less true of Elinor Ostrom's Understanding Institutional Diversity, Footnote 87 despite her public endorsement of multi-method strategies.Footnote 88 Indeed, Ostrom's “institutional analysis and development” framework is one of the more complex and eclectic strands within the new institutional economics, and its development has benefited greatly from insights garnered from large-N quantitative analysis as well as case-specific qualitative analysis. Yet there is little doubt that the signal contribution of Understanding Institutional Diversity—the identification of discrete sets of rules and choice structures that inhere in discrete “action situations”—lies in her creative adaptation of formal models to characterize and overcome collective action dilemmas. That is, what enables Ostrom to generate novel insights about the challenges of managing common pool resources is not her ability to triangulate different methods, but her ability to reconfigure and apply an array of deductive models—from various forms of game theory to agent-based modeling and complex simulations—on the basis of the specific action situations that emerge in distinct institutional settings. Ostrom's book benefits from her impressive grasp of a wide range of real-world common pool resources, but what makes it such a tour de force of institutional analysis is her long-standing expertise in adapting formal rational choice analysis to capture the situational logic inherent in particular collective action problems and to formulate institutionalized incentives for overcoming these problems. None of this suggests that Ostrom is a methodological purist or that she privileges models over empirics; far from it. Her book does, however, highlight the value of her vast experience in working with formal rational-choice analysis. This experience plays a critical role in guiding Ostrom's efforts to combine expansive empirical knowledge and varied theoretical perspectives in ways that are innovative and rigorous but which also invite collaboration and dialogue with those who wish to employ other approaches to study collective action.Footnote 89
All three works serve as a powerful reminder of the many benefits that stem from the accumulated experience and growing proficiency of scholars who invest in iterated applications of a given method. These benefits include a degree of innovation and sophistication in the application of a particular method that a single scholar would be hard-pressed to duplicate in a single multi-method study since she or he must necessarily distribute a limited amount of time and effort across several components of a multi-method project. Also, the particular kinds of theoretical and empirical insights that might emerge from well-executed pieces of SMR, while often reflecting the influence of varied methodological perspectives, can elude those who are explicitly committed to multi-method research designs and thus focused on integrating and cross-validating findings generated through multiple methods. Moreover, scholars who are at the forefront in the application of a specific method are also in a position to set standards and help ensure the quality of training and evaluation offered to others who use the same approach, whether as part of single-method or multi-method projects. These benefits redound to the discipline as a whole and warrant preserving space for those committed to various forms of SMR.
This argument should not be construed as a defense of method-driven research or a call to dispense with multi-method approaches altogether. Indeed, the discipline's history prior to the advent of MMR makes all too clear the dangers of the “tunnel vision” that emerges when scholarly pursuits and debates revolve solely around the relative merits of methods rather than the investigation of substantive problems.Footnote 90 Practitioners of MMR have a vital role to play in the discipline in guarding against such outcomes by exposing both the fault lines and interconnectedness of insights generated through different methods.Footnote 91 Moreover, even if MMR does not serve such ambitious goals as error-reduction and cross-validation, it still offers numerous other benefits—from helping to refine strategies for selecting cases and coding variables, to facilitating the use of different analytic lenses to explore anomalies and recalibrating the significance of certain concepts, mechanisms or data points. Such benefits more than justify maintaining space for MMR as well as opportunities for individual scholars inclined to acquire multi-method skill sets. What is less clear is whether these benefits warrant the standardization of multi-method skill sets for individual researchers.
Scholars specializing in separate methods but interested in similar problems retain the option of collaborating with each other to design and execute the same types of multi-method studies that individuals with varying multi-method skill sets might produce. This begs the question of whether the latter skill sets are necessarily indispensible even where MMR is deemed to hold especially great promise. The competent execution of a multi-method project requires more than familiarity with a set of different methods; it requires a tremendous amount of time and effort in attaining and maintaining a level of proficiency in each of the methods, and then leveraging each method to the extent possible to produce findings that are not only coherent but also original and compelling on their own. Such findings, we submit, may be more easily and efficiently generated through collaboration among two or more scholars, each of whom has mastered a given method and can assess the quality of research produced using that method. It is possible that the single-handed pursuit of MMR may deliver some efficiency gains in the start-up phase of research since one person can design compatible studies using multiple methods without having to constantly communicate with a collaborator. But these gains can be offset in the course of the actual research when two or more scholars work simultaneously to apply their respective methodological skills with greater ease and confidence.Footnote 92
Moreover, over time and in subsequent research products the collaborative execution of MMR is likely to become even more efficient as each of the collaborators becomes more adept at applying a given method, especially in view of the difficulties a single individual faces in maintaining proficiency in the use of several methods. Consider, for instance, the challenge of maintaining the language skills required to carry out archival research or ordinary language interviewing in a given locale. It is no less of a challenge to have to continually update one's technical skills to make use of the latest statistical models or the most sophisticated variants of game theory. These challenges are not insurmountable, but they do create pressures on individuals pursuing MMR to work with the less complicated variants of selected methods in the interest of being able to “do it all.” Alternatively, it may be tempting to bypass certain related questions, privilege certain levels of analysis, or omit certain kinds of data for fear that these might detract from the coherence of the findings or require additional investments of time and effort. At least as problematic is the possibility that inadequate attention will be devoted to harmonizing the defining attributes and causal properties of concepts deployed in different methodological systems.Footnote 93 Such difficulties may be more easy to manage when MMR is pursued by two or more collaborators rather than by a single individual.
There is also a distinct possibility that the standardization of multi-method skill sets will ultimately reduce the range of methodological diversity. As scholars become more conscious of, and seek to evade, the conceptual and practical problems in combining different methods, their research designs are more likely to incorporate qualitative approaches that are closer to the nomothetic end of the spectrum than those that are more idiographically oriented. This is because the former—which include hypothesis-generating small-N comparisons or case studies designed to trace the effects of general mechanisms—will be relatively more congruous with formal and quantitative approaches. This means that more-idiographically oriented variants of qualitative research will be progressively more difficult to justify, since the narratives they generate are likely to be context-bound and hence more difficult to triangulate with causal generalizations produced by formal or large-N approaches. Thus further proliferation of MMR not only threatens to reduce space for all SMR, but may squeeze out those committed to more idiographic forms of SMR.
None of the above suggests that we should discard MMR or that individuals who are committed to leveraging multi-method skill sets should instead specialize in one of the methods. We only claim that those who invest in a given method have distinctive contributions to make and should not be pressured into deploying additional methods in order for their research to be taken seriously. As the above discussion of Bartells, Ostrom, and Wedeen suggests, scholars can leverage their experience with various forms of SMR to generate certain distinctive insights that are valuable in their own right and that are often uncovered through a focused search for certain kinds of evidence to support certain kinds of claims using certain kinds of methods. Such scholars also effectively set the standards for the skillful execution of research using their respective methods, without forfeiting the option of pursuing multi-method designs in collaboration with colleagues with advanced training in different methods. Moreover, to the extent that job satisfaction matters for one's productivity and professional commitment, it is worth considering the impact of treating MMR as disciplinary best practice on scholars who tend to be more energized and creative when working with a particular method. For them, further proliferation of MMR will entail either professional costs as SMR is progressively marginalized, or, alternatively, emotional costs from having to devote time and energy to other methods solely to meet disciplinary norms and expectations. These conditions could very well lead to a net decline in the productivity and dynamism of the discipline over the longer term.
Conclusion: MMR, SMR and the Quest for Pluralism
The initial proliferation of MMR was largely a blessing for the discipline, clearing up misunderstandings that plagued past methodological debates and prompting greater appreciation of a wider array of methods. What has motivated us in this article, however, is a fear that the discipline is fast approaching a tipping point. MMR is no longer simply an option for pragmatically coping with emergent challenges or opportunities in advancing a research agenda. It is increasingly viewed as disciplinary best practice and as a rationale for promoting standardized multi-method skill sets. This might be warranted if multi-method approaches were to consistently generate “better” (that is, more valid) findings than any one method, or if single-method researchers were to offer little that multi-method researchers could not easily duplicate. We have argued that neither condition holds.
The putative benefits of MMR with respect to cross-validation and error-reduction are more limited than assumed once we consider the challenges of harmonizing concepts and findings across methods predicated on incongruous foundations. These challenges do not rule out MMR, but they do restrict the range of questions and combinations where cross-validation is even feasible. More often than not, MMR is likely to yield a set of related insights and findings that could be generated through a collection of separate studies. Yet such benefits could be realized through collaboration between two or more scholars, each of whom has achieved mastery in a particular method and each of whom is in a position to assess the quality of research employing that method. The distinctive intellectual gains such scholars potentially offer are not very likely to be matched by individuals who must distribute their time and effort across the many tasks required to maintain proficiency across several methods.
Yet as an ever-increasing number of graduate students and younger scholars continue to invest in multi-method skill sets, there is bound to be a concomitant decline in the number of scholars who invest in mastering a single method. As the proliferation of MMR leads to a restructuring of professional opportunities and rewards in the disciplinary mainstream—in the form of prospects for hiring and tenure, allocation of grants and prizes, and opportunities for publication in top-ranked journals—the visibility and standing of single-method researchers, especially qualitative ones, are likely to diminish over time. Moreover, because only certain combinations of methods, featuring a select number of qualitative methods, proceed from ontologies that are congruous enough to generate coherent findings, there is a very good chance that other qualitative methods—particularly more idiographically-oriented ones—will be gradually squeezed out.
In addition to professional considerations, such a trend would have important substantive implications for all fields of inquiry. Given the inherent bias of MMR toward certain ontological perspectives, treating it as disciplinary best practice would detract from the overall pool of knowledge we have available to us as a scholarly community and will likely lead to a narrower range of theoretical perspectives. In turn, this would limit the kinds of approaches taken and, perhaps more importantly, the kinds of questions asked in a given area of research. If such a trend continues unabated, the discipline will have missed out on ensuring the diverse intellectual gains that different forms of MMR and SMR have to offer. In so doing, it will have also subverted the methodological pluralism that once gave impetus to MMR. Genuine pluralism lies in the preservation of an atmosphere conducive to the exploration of different questions using different approaches, whether single- or multi-method, and this in turn maximizes the kinds of benefits that different kinds of scholars can potentially offer the discipline.
Thus, if there is a disciplinary “best practice,” it is likely to be found not in the uniform adoption of MMR or of any given methodological approach, but in ongoing dialogue among scholars employing different methods and being able to translate concepts to make them intelligible to individuals approaching the same question from different perspectives. In this process, MMR is best thought of not as a magic bullet for overcoming trade-offs across methods, but rather as a valuable addition within a diverse repertoire of methodological approaches, one that comes with its own trade-offs. The value of MMR lies not in its supposed ability to neutralize the weaknesses of individual methods and give us a closer approximation to the “truth” than any SMR can offer. It lies in the role multi-method studies can play in expanding the scope for “cross-cultural communication”Footnote 94 among researchers trained in different methods, each of which has distinctive payoffs and limitations. For this purpose, the discipline does not require further proliferation of multi-method skill sets, only opportunities for collaboration among scholars, each proficient in a given method and fully cognizant of the foundational assumptions it is predicated on.Footnote 95 Critical to this effort (and to preempt excessive compartmentalization along methodological lines) will be the role of disciplinary practices that do not unduly penalize either single-method scholarship or cross-method collaboration; professional associations that encourage methodological cross-fertilization and create opportunities for scholars whose approaches are at risk of being “selected out”; and journals and book series that explicitly seek to foster dialogue among scholars investigating wide-ranging substantive problems using different sorts of research designs, whether single- or multi-method. Ongoing communication—and sometimes collaboration—among those who pursue MMR and those who engage in diverse forms of SMR, rather than the elevation of MMR as best practice, is the surest way to ensure the quality of method-specific research and training, expand the font of insights collectively held by the discipline, and preserve genuine methodological pluralism.Footnote 96
