Is Biomedical Science a Reasonable Model for Political Science?

A first challenge, raised by McClure as well as by Elman and Burton, concerns the validity of the comparison between political science and the biomedical sciences, particularly because of substantial differences in scale. For example, McClure highlights the vast size of the flagship New England Journal of Medicine (NEJM), both in terms of staff and readership. And although she applauds their “long-standing record of editorial responsibility to both the medical profession and the public,” in a manner consistent with what I describe in my essay, she asks whether we should temper our expectations in a far smaller discipline? Elman and Burton point out that the funding and administration of biomedical sciences are structured in a manner and at a scale so distinct from political science that one cannot help but wonder if those are not necessary conditions for success?

I certainly acknowledge that given such differences in the respective enterprises, one could not possibly argue that “it works for them, so it will work for us.” Moreover, probably few biomedical researchers would describe their own larger enterprise as entirely coherent, self-conscious, or fully successful! None of these caveats should prevent us from considering the apparent use of research cycles as a heuristic for re-thinking how we interact as a scientific community.

First, and this is really my key point: because clinicians and public health officials implement scientifically-validated treatments and protocols every day, often with life and death consequences, biomedical researchers really must care about causal effects and sound research designs for establishing causation. Nonetheless, that community also values a great deal of non-causal research. At the very least, this ought to make political scientists take pause and consider if we also should value explicitly non-causal research.

To respond more specifically to the possible “disanalogy,” I don’t see why disciplinary scale should be a limiting factor for the development of research cycles. Surely we have enough political scientists, even segmented into sub-fields, to engage in some division of labor? Perhaps one area that would require additional resources would be in the administration of our scholarly journals. Right now, our mostly “volunteer” workforce of peer reviewers and academic editors is highly constrained. In order to realize the vision I articulate in the original essay, we would require much more rapid turnaround in the editorial review process than is currently the norm. I don’t have an easy solution, but I also think this is a “fixable” problem.

Another concern is whether I am fairly characterizing the biomedical research process—at least the clinical research process, as McClure highlights. Elman and Burton wonder whether my characterization of self-conscious cycles within the biomedical sciences would be evident if I had probed more extensively. Here, McClure comes to the rescue, as she independently reviewed a few decades of NEJM editions, and concludes that “on Prof. Lieberman’s view of editorial virtue, NEJM thus does well with regard to both the clinical research cycle and the biomedical research continuum.” Indeed, it was my own reading of many editions of various biomedical journals, as well as attendance at various biomedical conferences that led me to the conclusions that prompted me to write the essay in the first place. Virtually every issue of the leading clinical journals I cite provides clear evidence of research findings at very different stages of what might be called a “cycle.” While of course, many papers cross boundaries and carry out multiple functions, the point is that it is very easy to find these different types of discrete contributions.

Are Cycles Really Cyclical?

Although my original essay emphasizes the role of research cycles, two of the response essays correctly highlight that in my discussion of potential best practices, I seem to emphasize more linear progressions and endpoints. Hibbing disagrees with my assertion that we might regard description as an “end in itself.” Falletti challenges my hierarchical ordering, that research “progresses from pure description, through correlational analyses and natural experiments, to phased randomized controlled trials (RCTs).” She goes on to argue that experimental research may “feed into … other stages of research.” However, Falletti takes the opposite view of Hibbing and argues “that the research process may productively end at the observational stage.”

What are we to make of this exchange? Debates about whether descriptive research can satisfy our scholarly appetites are perhaps matters of taste, but I think there are also other issues at play. On the one hand, I agree with Hibbing that ultimately, we always want to understand the root causes of interesting descriptive findings. On the other hand, I suppose my intention—and here perhaps Falletti would be sympathetic—is that even if we do not have empirical evidence or a well-grounded theory to account for an “interesting” outcome, simply describing outcomes and patterns may still constitute an important and valuable research finding. We could argue further that such observations are generally interesting because they might speak to prior causal research by challenging such accounts. Even if there were not an immediately clear path forward for providing an explanation or causal account, certain types of descriptive findings elucidate the political world in which we live.

What Role for Qualitative and Multi-Method Research?

Elman and Burton challenge my assertion that biomedical journals appreciate exploratory and descriptive work by highlighting some notable rebuffs to qualitative submissions to biomedical journals. But in all fairness, I never claimed that those journals valued qualitative research, and I would highlight that much exploratory and descriptive work is not qualitative! As they cited the British Medical Journal, I immediately went to that journal’s most recent edition, and the first article I read was a study of blood pressure in people with type 2 diabetes. The conclusion?

This is my point exactly! The authors asked an important question, analyzed some relevant high-quality datasets, did not pretend they could make strong causal claims, but provided some important directions for future research by accurately describing the observed patterns. The leading biomedical journals are replete with such articles and I would describe this as descriptive and exploratory. They recognize that much of scientific progress is tentative. Of course, it is reasonable to fear that casual readers will interpret the findings as causal, but the authors and the editors have clearly done their job in explaining the nature of the finding.

Moreover, the leading clinical journals are filled with individual case reports and narrative accounts of various kinds (which I would take to be a type of qualitative research). I would acknowledge that there is surely a hierarchy in how different types of contributions within a leading journal are valued, as they should be: the very interesting but brief clinical report as well as the long-term double-blind clinical trial may get published in the same issue, but the latter will be viewed as a more substantial scientific effort. The relevant point for political science is that the biomedical research community does not simply expect the New York Times health reporters to document all clinically interesting findings; rather, the academic journals determine the sets of findings that ought to appear in their pages, in turn setting the agenda for what journalists will subsequently cover in the news.

Units, Effect Sizes and Causal Complexity

Falletti raises other important points about the relationship between “causal complexity” and research practice, and challenges my attempts to find some equivalence in the value of research outputs irrespective of effect size. Rather, she argues that many null- and small-effect findings are the product of weak theorizing or research designs. She is certainly at least partially correct in that diagnosis. On the other hand, if we accept that social-scientific inquiry is uncertain and much about the political world is random, we need to acknowledge that even some very well-designed studies with promising theories will fail to reject the null hypothesis. We need to find a good place in the research cycle for publishing and valuing such studies, with appropriate criteria.

Building on the biomedical analogy, Falletti offers the important example of increasingly individualized cancer treatment, illustrating that some researchers and doctors are using detailed knowledge and understanding of particular processes to formulate very individualized conclusions about likely treatment effects that can only be “tested” in very small samples. My guess is that biomedical researchers are far from unified in their assessments of this emerging practice. But I agree with Falletti that this is an interesting and important cautionary tale, especially as political scientists tend to estimate average treatment effects over large samples, and emerging best practices are weighing against post hoc estimation of heterogeneous effects in RCTs. The example serves as a reminder of the potential problems of labeling any particular method as an unqualified “gold standard” for attaining knowledge.

A Way Forward?

Let me also highlight a few of the excellent, productive suggestions made by the various authors. Elman and Burton suggest as a step forward more explicit journal recognition of different types of contributions before we can demonstrate the value of research cycles.^{Footnote 2} I agree. And to that, I would add that other gatekeepers in the discipline need to highlight and to value the different types of contributions scholars can make.

Hibbing offers the view that we might benefit from journals that contain excellent syntheses of what has come before.^{Footnote 3} I agree that awareness of and consensus about the state of knowledge in a research area is certainly critical for an effective research cycle.

I think this discussion comes at a pivotal moment for our discipline: information is flowing faster than ever before in history; we have many more tools for analyzing very large quantities of data; and along with all these developments, the nature of political life seems to be mutating quite rapidly, with enormous consequences for the human condition. Political scientists can and should play a big role in helping to understand these myriad changes. But we probably need to rethink some of our own institutions for producing knowledge if we hope to make important contributions and to take full advantage of our collective skills, knowledge, and talent pool.

I view this exchange over the possible relevance of the biomedical research cycle as one of many possible discussions we might be having about whether the surprisingly rigid institutions we have built up to produce social-scientific research are really the best ones for the current context in which we work.