Peer review is the essential gatekeeping process that determines what counts as high quality political science research, yet almost every practicing scholar has had legitimate occasion to complain about one or more aspects of peer review. Scholars, editors, and disciplinary associations including APSA are currently discussing ways to improve the peer review process; including Diana Mutz’s (2015) article in this journal, which proposes a novel mechanism to incentivize the production of quality reviews, and Brendan Nyhan’s (2015a) contribution largely focused on reversing the tendency of journals only to publish studies that produce narrowly statistically significant results. But there is another persistent pattern of bias in the peer review selection process: that is the tendency to reject work that is grounded in postpositivist or nonpositivist epistemologies, including particularly, constructivist work in International Relations (see Zarakol, this issue). That happens, at least in part, because reviewers with a positivist orientation ^{Footnote 1} evaluate postpositivist and nonpositivist work by standards that are inappropriate for the intellectual projects that their authors are engaged in.

Some of the proposals to improve the peer review process have the potential to exacerbate the bias in favor of work grounded in a positivist epistemological framework. For example Brendan Nyhan (Reference Nyhan2015b) recently laid out recommendations to improve the peer review process by creating a checklist of items that reviewers should consider when reviewing published work, in order to professionalize and standardize the practice. While I applaud the effort to attempt to increase the professionalization of the work of reviewers and editors, Nyhan’s proposal seems to assume that political science will be based nearly exclusively on statistical analyses. Indeed, no fewer than 8 of his 21 checklist points assume the reviewer will be checking the logical implications of the author’s use of statistical data. The problems with peer review are broader than debates about the meaning of statistical significance.

Miscommunication in peer review across epistemological divides limits what gets published. If constructivists only read constructivist work, and positivists only read other positivist work, everyone’s understanding of the political phenomenon of interest remains limited. Worse, such narrow understandings of what “counts” as political science ultimately undermine our discipline’s collective effort to improve understanding of important political phenomena and our collective ability to produce knowledge that is relevant and interesting to the public (Isaac Reference Isaac2015b). One solution is to improve the way peer reviewers read and consider work produced by scholars with different starting points on positivist, nonpositivist, and postpositivist issues. To do that effectively, reviewers, authors, and editors all must do more to improve communication on epistemological issues.

In recent decades, positivists have largely controlled the leading journals of political science in the United States (Zarokol, this issue). ^{Footnote 2} But how does that matter for our substantive understanding of political issues? In my own research area of international law and international organizations, the consequences of the problem are evident. It has been 30 years since Kratochwil and Ruggie (Reference Kratochwil and Ruggie1986) pointed out that laws and norms are social facts and that scholars should recognize that they have an intersubjective ontological nature (Onuf Reference Onuf1989, 66–78). Moreover, norms do not cause behavior, they “guide,” “inspire,” “motivate,” “rationalize,” “justify,” and give intersubjective meaning to behavior (Ruggie Reference Ruggie1998). For positivists a key question is whether or not rules cause outcomes, but in practice it is difficult to build methods grounded in a positivist epistemology that can prove a causal relationship. Norms can be counterfactually valid: just because one or more important actors violate a social norm, a rule, or an institutional requirement, it does not immediately follow that the norm in question no longer has any social influence whatsoever (Ruggie Reference Ruggie1998, 97). Indeed, the evidence for the strength of a social norm is often clearest when we watch how actors react to violations of laws or norms. Unsurprisingly, social science organized in a positivist vein finds it difficult to show that norms of any sort cause behavior. This is particularly true when the positivist approach in question understands norms (or really any role for ideas) from an approach of methodological individualism. The flow of communication between social scientists with a positivist orientation and those with a constructivist/interpretivist approach remains stifled, and scholarship is rare that builds on insights from multiple epistemological perspectives. Of course, in the United States there is an influential popular school of thought that holds international law is nonexistent, or irrelevant, or epiphenomenal. Political science is handicapped in its ability to address that debate by the ontological and epistemological wars that divide the discipline.

This challenge to communication amongst scholars with differing ontological and epistemological assumptions, but similar substantive interests, is manifest particularly in the peer review process at top American political science journals. ^{Footnote 3} Positivists largely conclude that international rules and norms are of little or no consequence, but interpretivist analyses that show the political and social significance of international law have difficulty gaining traction both in some top political science journals and in the broader popular discourse.

Constructivists begin from the observation that the world is composed of intersubjective social facts, and that one way of studying those phenomena is through the Weberian interpretivist tradition. From the perspective of third-generation constructivism, it hardly seems like a radical proposition. You might expect that political scientists studying regimes from any epistemological and ontological position would at least be familiar with the constructivist approach, and the basic assumptions of research that proceeds in that vein. Yet, the experience of constructivists submitting work to journals with a predominantly positivist orientation suggests that many reviewers in the discipline either do not understand constructivist epistemological, ontological, and methodological approaches, or they refuse to accept them as legitimate. In the 2014 TRIP surveys, 23% of international relations scholars worldwide described their own work as falling in a constructivist paradigm—the highest total for any paradigmatic approach—with realism following at 19% and liberalism at 12%. For the United States, the ratio is roughly similar, with 19% describing themselves as using a constructivist approach, while 18% report using realism, and 15% liberalism. Taken at face value, one might expect that about a quarter of published research would take a constructivist approach, but that is not the case in major American journals. Worldwide, just under 50% of scholars describe their work is positivist, with the remainder roughly evenly split between scholars who see their research work as either nonpositivist or postpositivist (Maliniak, Petersen, and Tierney 2014). In the United States, a higher percentage of scholars identify as positivist at 61%, but that still leaves nearly 40% of scholars in the postpositivist or nonpositivist camps. And yet, constructivist work—and particularly constructivist work with interpretivist methodological approaches grounded in nonpositivist or postpositivist epistemologies—rarely graces the pages of top US-based political science journals (Zarakol, this issue).

No one expects that positivists will stop writing about the world of international law and global governance from their own paradigmatic commitments. But positivist reviewers should learn to read constructivist/interpretivist work and suspend their own philosophical commitments in reviewing that work. Postpositivist and interpretivist work should be evaluated on criteria internal to those epistemological traditions. The overall quality of scholarship is likely to be enhanced, and not weakened, if scholars working on similar issues from different epistemological and methodological approaches remain engaged with each other’s work. For this reason, having only postpositivist reviewers on postpositivist projects, positivist reviewers on positivist projects, and so on, is undesirable.

The first half of the communication problem between positivists and interpretivists occurs when the reviewer has positivist commitments and the author is writing from a postpositivist/ constructivist type of perspective. ^{Footnote 4} This is particularly important, as it is the most common sort of reviewer/author epistemological/ontological mismatch that we are likely to see. Since most of what is published (at least in top American political science journals) is from a positivist perspective, the natural pool of reviewers is likely to be skewed in that direction. Some reviewers with positivist commitments may think their professional responsibility precisely is to prevent other types of political science research from being published. But if those folks rule the discipline, we are left with a depressingly narrow political science. As Jeffrey Isaac argues (Reference Isaac2015a), interpretivist work holds potential to help make political science relevant to a larger public, potentially increasing the policy relevance of the discipline. One hopes for a more catholic approach to the disciplining of political science in America to gain hold.

What can be done to improve communication in such a case? I have some recommendations based largely on my own experience as a constructivist scholar publishing and reviewing in a field where positivism is the dominant paradigm. Authors have a duty to make clear the paradigmatic, ontological, and epistemological commitments of their own work. Since they are likely not engaged in hypothesis testing, they should clearly explain the intellectual contribution of their research. If the purpose of the scholarship is an interpretation, intended to show why things are so, and not otherwise, authors have a duty to alert their readers that this is the goal of the project.

Positivist-oriented reviewers familiar with the substantive area of the contribution should not decline to review such work. Such reviewers with a positivist orientation must be careful though not to recommend against publication on the grounds that the contribution fails to engage in hypothesis testing or some other activity expected only of positivist projects. Mismatched paradigmatic thinkers may have useful suggestions for each other’s work, and the thesis here is that research from different paradigms can be mutually supportive if scholars work to read across methodological and epistemological boundaries.

Finally, editors need to play a constructive role in bridging the communication divide between positivists and postpositivists or nonpositivists. This means finding well-qualified reviewers from multiple epistemological backgrounds to review a piece. It also means forming particularly strong relationships with reviewers who show a willingness to recommend publication for work that is outside of their own paradigm.

Each of these are lessons from my own experiences; I discuss a few examples to illustrate the lessons I think authors, reviewers, and editors all should learn if we want to have a vibrant discipline that shares scholarship from multiple epistemological paradigms.

Like many constructivists, more than once I have had the experience of having two favorable reviews of an article disposed toward publication, both of which largely accepted the social constructivist and interpretivist ontological approach of the work, only to have the paper rejected because a third reviewer evaluated the paper by criteria that would only be appropriate for a positivist project. A typical example is a reviewer report that criticizes a paper for failing to engage in hypothesis testing. I recall in particular one paper that contained claims regarding the contingency of agent decisions about how to interpret conflicting norms that was criticized for failing to provide “falsifiable” or “testable” propositions. The goal of the author was to provide a description of particular important cases to show how contingent decisions by actors at key points led to a particular outcome. To the extent the author made specific claims about why particular outcomes were obtained, those claims were case-specific and dependent on the historical context and so could not be reformulated as general propositions that could be tested under other circumstances.

Another common misreading of constructivist work by positivist reviewers occurs with papers that deal with agent–structure interaction. Positivists often insist that one has to break apart the relationship of constant constitution of agents and structures, and instead tell stories about how agents “cause” structures to come into being at time one, followed by structures “causing” agents to do things at time two. These misunderstandings are too common when positivists are asked to review social science work that is grounded in a more interpretive tradition of the social sciences. Undoubtedly, constructivist authors share some of the blame for not making our own epistemological and ontological claims explicitly clear; but must we always rewrite the lessons from Kratochwil and Ruggie (Reference Kratochwil and Ruggie1986) in every paper in order to be understood?

There are lots of legitimate reasons why editors may have decided that constructivist papers written by me or others were not a good choice for a particular journal. The point here is that criteria for good research in a positivist paradigm cannot be required of research whose intellectual contribution is grounded in postpositivist or nonpositivist epistemologies. Editors should make a habit of making clear whether or not they were persuaded by particular major critiques that reviewers offer in decision letters. Such letters can help reviewers to learn which kinds of feedback were most helpful to editors. When a reviewer’s criticisms are not well founded, editors should be willing to reverse the reviewer’s recommendations on publication. This proposal undoubtedly would increase the burdens on busy editors, but it seems a necessary price to pay if we are going to have disciplinary journals that publish high quality research from multiple paradigmatic approaches.

My own experience cannot demonstrate to a skeptic that reviewers and editors consistently fail to evaluate postpositivist, nonpositivist, and interpretivist works by their own epistemological logic. For the reader who is skeptical that one instance is indicative of a larger pattern, you can easily find supporting or disconfirming evidence for the trend by asking an interpretivist or constructivist political scientist near you. Of course it is harder to find such a person at a top-30 PhD-granting institution in the United States (see Subotic, this issue), but according the TRIP survey, they are all around.

What is the answer? It is not an easy question, but barring positivists from reviewing interpretivist work is not the answer. Good social scientists should learn how to evaluate research reports that begin from ontological and epistemological premises that are very different from their own. After all, most good constructivists know how to evaluate good positivist work. Is it too much to ask for reciprocation? The alternative is of course similar to the actual world that we live in, where interpretivist work is published in European journals or in books, and positivist methods are required in American political science journals; and if they want to, various groups of social scientists can ignore each other entirely. But this, I think, tends to decrease the effectiveness of our overall collective project to better understand the political world. Pressure to publish in prestigious journals that narrowly limit the methods and epistemologies they are willing to publish also makes it difficult for scholars to do research and report on it in ways that are accessible to policy makers, media outlets, and the public (Goldgeier and Jentleson Reference Goldgeier and Jentleson2015).

As a constructivist I have also been asked to review work with a positivist approach to studying the role of international law, sometimes for top American journals. In some cases, I have even recommended publishing work with positivist evidence, including large-N statistical evidence. From these experiences, I am convinced that constructivist reviewers often do contribute to improving papers written from a positivist perspective. For example, positivists writing about international law often write as though they are the first social scientists ever to show that international law and norms shape actor behavior. Positivists often assert that the notion that actors care about intersubjective rules is a radical new finding at odds with what “political science” (aka, positivist political science) shows: namely, that actors are only capable of acting in self-interested ways without regard for social rules that lack external enforcement mechanisms. These grandiose claims likely arise from a desire to impress reviewers and editors with the importance of an author’s contribution. But they also reflect a stunning ignorance of the constructivist literature. Constructivist scholarship with titles like “Multilateralism Matters” and “Anarchy is What States Make of It” and decades of work since has consistently demonstrated that international law plays a critical role in creating the social context in which international politics occurs (Ruggie Reference Ruggie1998; Wendt Reference Wendt1992). I remember a particularly striking example of a literature review for a positivist paper that claimed “political science and International Relations in particular lack a good grasp of how precedent operates in international law.” Claims like that demonstrate a striking unawareness of the core constructivist literature, including works like Rules, Norms, and Decisions (Kratochwil Reference Kratochwil1989) and World of Our Making (Onuf Reference Onuf1989), which address exactly that question. Perhaps many positivist scholars don’t necessarily read that sort of thing. But that likely is why there is a huge blind spot in American political science about the role of international law in world affairs.

So what can we learn about international law by reading articles in top, postitivist led American journals from the last decade or so that deal with the subject? An informal review of a dozen relevant articles published by the American Political Science Review shows a few recurring themes. Mainstream positivist work generally struggles to demonstrate conclusively that international law impacts behavior. In his analysis of state compliance with international law of war, James Morrow proposes to test whether or not realist, liberal, or constructivist explanations for state compliance with international law are best supported by extensive statistical analysis of when states comply with or violate particular rules of law regulating the conduct of war. But he goes on to say: “In practice, it is difficult to pull explicit hypotheses about the patterns of compliance from each view” (Morrow Reference Morrow2007, 561). This is not just a minor frustration. Of course, constructivist approaches to international politics are not intended to provide generalizable hypotheses about how international law achieves compliance. In the end, Morrow scores 0 on his chart for his “constructivist hypotheses,” where 0 reflects a null result, and + and – indicate that a hypothesis was confirmed or contradicted. Another important debate in positivist-oriented, APSR analyses of international law focuses on whether or not states’ compliance with international law happens because states “self-select” into treaties that they already comply with, rather than treaty commitment “causing” their subsequent behavior (Von Stein Reference Von Stein2005). Constructivists would note that this whole debate misses the important point that a major function of treaty-based commitments is to create new identity communities. States do this not because of low compliance costs, nor because they already plan to comply with the relevant norms, but because they fear it will be difficult to maintain standards without social support, particularly in areas like upholding human rights norms or protecting the environment.

So the subject–object ontological understanding of international rules and norms persists in major American political science journals. That view limits the political science community’s ability to understand how social norms work. Constructivists in International Relations proposed solutions to the problem in the 1980s. The time has come for positivists to listen. The key is for authors to be clear about the research goals of their presentations, reviewers to suspend the tendency to apply criteria to what counts as good research that may not apply to the project before them, and for editors to be sure that judgments about publication decisions are made based on a critical appraisal of reviewers comments and not a simple counting of votes for or against a particular paper. If we can learn to communicate across the epistemological divide, perhaps we can ensure that future articles from a constructivist approach will not be marginalized in the special issue ghetto of top American political science journals.