Evan Lieberman argues that Political Science should mimic biomedical research, and recognize that science progresses in cycles, each consisting of a sequence from exploratory research (often in the form of case studies), through correlational studies, and on to randomized controlled trials. Because biomedical researchers understand that the early phases of the cycles are foundational predicates for the later stages, Lieberman contends that descriptive, associational, and early experimental work is highly valued. This recognition is reflected foremost in the willingness of high-impact biomedical journals to publish exploratory research. In contrast, he says, Political Science undervalues the key contributions that exploratory research makes, and hence does not reward it appropriately. In this brief commentary we consider Lieberman’s argument, and suggest some areas that might benefit from a more extensive treatment when considering his recommendations for the discipline of Political Science.
Lieberman’s argument involves three separate but related claims. First, that biomedical research really progresses in cycles, with exploratory research being valued and rewarded as he describes. Second, that this trajectory produces better science (and potentially better social science) than research conducted in the absence of a cycle—in other words, there is a principled epistemic basis for the conduct.Footnote 1 Finally, that the circumstances that make research cycles feasible in the biomedical sciences are also present in Political Science. While proponents of multi-method research will likely welcome Lieberman’s suggestion to increase the value attributed to descriptive and exploratory research in Political Science, we suggest that his proposal is somewhat underdeveloped on each of its three primary arguments.
Description and Evidence of Research Cycles
Lieberman’s central claim is that biomedical research proceeds, and is understood to proceed, in the sequences he describes, yet the essay lacks either an explicit account of how biomedical research cycles function in practice, or convincing evidence of their operation. We would welcome a more fine-grained description that explores, for example, what mechanisms delineate the boundary of each research cycle, or how conduct is guided within and between its phases.Footnote 2 Moreover, we are not convinced that Lieberman has yet offered sufficient evidence to support his argument that biomedical research really is characterized by the self-conscious operation of cycles. Very little literature is cited, and he draws mainly on a single issue of the New England Journal of Medicine. We suspect that a more comprehensive examination might not find a consensus about the role and value of exploratory research, and its resulting publication in leading venues. While exploratory research obviously includes more than just case studies and cognate approaches, how qualitative research is treated provides useful clues for how welcoming the journals are likely to be to descriptive work more generally. Shuval et al. looked at the proportion of qualitative research in 67 medical journals between 1998 and 2007.Footnote 3 Although there was a more than three-fold increase over that time period, the proportion remained low, increasing from 1.2 percent in 1998 to 4.1 percent in 2007. While these numbers are a little dated, a recent controversy regarding the BMJ: British Medical Journal suggests that things may not have changed too much in the intervening years.Footnote 4 An open letter to the journal, signed by dozens of qualitative biomedical researchers, responded to a September 2015 article rejection letter that included the following explanation:
Thank you for sending us your paper. We read it with interest but I am sorry to say that qualitative studies are an extremely low priority for The BMJ. Our research shows that they are not as widely accessed, downloaded, or cited as other research.Footnote 5
If anything, the editors’ reply to the open letter was even more explicit:
… we do not prioritise qualitative research because, as mentioned in our information for authors, qualitative studies are usually exploratory by their very nature and do not provide generalisable answers.Footnote 6
Indeed, a review of the reactions to the exchange between the open letter writersFootnote 7 and the editors of the BMJ Footnote 8 suggest that some of the concerns of qualitative biomedical researchers would be readily recognizable by political scientists.Footnote 9 We also suspect that these anxieties would not be limited to the BMJ. One of the commentaries on the ruckus included an observation by a former BMJ editor that:
The journals with the highest impact factors—Nature, Science, the New England Journal of Medicine, and the Lancet—hardly ever publish qualitative research, but The BMJ was a place where [qualitative researchers] could hope to publish. Now even it has been largely shut out. This upsets them.Footnote 10
While we do not have recent data to show that exploratory research is not as widely valued or as well-published as Lieberman suggests, we think a closer look is required before concluding that he has this right.
Epistemic Basis of Research Cycles
A second set of issues involves the epistemic foundations of the sequential approach recommended by Lieberman. What is the basis for believing that combining multiple studies using different methods produces better science and social science, and in particular, how does the temporal sequencing of those studies arrive at stronger conclusions? At its core, Lieberman’s strategy is a claim for the value of multi-method research, and that later phases in the cycle are contingent on the successful completion of earlier stages.Footnote 11 However, there are at least two different understandings of how multiple studies employing different methods relate to each other:
The central idea of integrative multi-method research is to use each method for what it is especially good at, and to minimize inferential weaknesses by using other methods to test, revise, or justify assumptions. This is in contrast to triangulation designs, in which whole inferences—fully dependent on both the strengths and the weaknesses of each method—are carried out and then somehow compared and combined. Integrative designs thus use multiple modes of inference to substitute strengths for weaknesses, whereas triangulation designs compare whole causal inferences whose individual strengths and weaknesses are the same as those of single-method designs.Footnote 12
It is not entirely clear from Lieberman’s essay whether cycles involve triangulation, integration, or a combination of the two. It is possible to read the sequences in the cycle as consisting of successive but separate applications that arrive at distinct understandings of the phenomenon under investigation. In this linear approach, the sequences are like the legs in a relay race, where each runner hands the baton to the next. Each runner runs her own leg of the relay independently, but starts from where the previous runner ended, and finishes where the next takes over. However, it is also possible to adopt a more complex and nuanced view of the sequences, where the legs share more than a beginning/end point. For example, in this integrative view qualitative methods might be used in a natural experiment to determine whether a particular assignment mechanism was really as-if random.Footnote 13 It would be helpful to know whether the integrative or triangulation approach is a better fit for the sequences that Lieberman advocates.
Conditions for Research Cycles
The final set of questions concern whether biomedical research is sufficiently similar to political science such that a cycle approach would fit with the latter. Limited space precludes a full discussion of the strongly monist view of natural and social sciences which is implicit in Lieberman’s recommendation. But even granting that view, as Lieberman himself recognizes in his essay, the sociology of the two fields differs in important ways. For example, political science has far fewer researchers, much less funding, and nothing analogous to the central planning facilitated by the National Institutes of Health (NIH), the Centers for Disease Control and Prevention (CDC), and the pharmaceutical industry. Accordingly, even if there are research cycles in biomedicine, and even if they can be shown to be beneficial in producing better research, we would like to hear more about whether these differences might make the implementation of the approach challenging in political science, at least for the foreseeable future. We think it would be helpful to see some more evidence that the factors present in biomedicine, but absent in political science, are not necessary conditions for the operation of research cycles.
Concluding Thoughts
Lieberman is right that exploratory research is undervalued in political science, and that disciplinary (including editorial) incentives should change. Nevertheless, it is not yet clear that biomedical research provides a model that political science would do well to mimic. We would welcome more evidence supporting, and a more fine-grained description of, the operation of sequences in biomedical research; more information about the epistemic grounds for combining complementary work conducted at different stages using different methods; and an explanation for why the arrangements for funding and administering biomedical science are not necessary conditions for research cycles to work.
While these lacunae are addressed, there are concrete and practical steps that could be taken in the meantime to facilitate publication of exploratory research. One possible approach is to seek convergence on standards for evaluating exploratory research, and then to include those standards in editorial guidance for authors.Footnote 14 Similarly, journals could encourage submission of exploratory research by signaling that they value different kinds of articles, in much the same way that the once common “research note” explicitly recognized that not all journal content was poured from the same mold.Footnote 15 In short, political science can begin to redress the publishing imbalance that Lieberman recognizes without waiting for the cycles view to be validated.
