HOW DO JOURNALS SELECT REFEREES? WHAT SHOULD EDITORS KNOW ABOUT THEIR REFEREES?

Editorial staff at academic journals comb graduate program websites, other published work, and conference registration lists to maintain an ongoing database of potential referees. Each journal has its own policies and guidelines regarding how many reviews they require for each submission as well as whom they will invite to be referees; some journals, such as International Organization or the American Political Science Review, do not reach out to anyone without a PhD.

Within those parameters, when deciding whom they should ask to review a manuscript, journal editors generally try to cover their bases. In other words, a typical review panel will include experts in the field, scholars with general knowledge of the field, and scholars with specific methodological and substantive expertise. If, when an author resubmits a manuscript after making suggested revisions, editors decide the revisions need to be reviewed, they generally ask the same set of reviewers. In a few cases, however, editors do reach out to reviewers beyond the original panel (for instance, when the paper includes a methodological innovation that requires specific expertise, or when previous reviewers have noted their own lack of expert knowledge in some area).

As important as knowing who is asked to serve as a referee is knowing who may not review a manuscript. Most editors try to exclude discussants and chairs of panels where the paper previously has been presented. If their suggestions have been followed, they may be biased in favor; if their suggestions have been disregarded, they may react negatively. Similarly, scholars acknowledged by the author are generally excluded, as are those the author feels have a bias that would prevent an impartial review.

In theory, the review process is “double blind”; none of the reviewers should know who the authors of the manuscript are and the authors should not know who is reviewing their paper. In practice, this is a difficult standard to maintain, and some leading economics and psychology journals have turned to a “single-blind” process. Reviewers who want to identify the author(s) of the manuscript can easily turn to search engines (this is particularly true within more specialized fields). Nonetheless, it is critical that referees approach their review without any bias, or, if they feel they have a slight bias, to acknowledge this to the editor. In those instances where the bias is serious, potential referees should turn down the invitation to review the manuscript. Lastly, reviewers should explicitly note (whether as a part of their review or, separately, in a note to the editor) those cases/issues where they feel they do not have deep expertise (for example, procedural issues in papers about Congress) so that editors can assess their reviews fairly, and where necessary invite additional referees.

WHAT GOES INTO A REVIEW?

In this section we discuss the different components of a “typical” review. Of course, reviewers have their own style and prioritize things differently. That said, making sure that several, if not most, of these elements are included in your review is the best way to insure that it is as helpful to the author(s) as it can be.

HOW SHOULD YOU WRITE A GOOD REVIEW?

A good review—one that helps journal editors make an informed decision about whether to publish a paper—is not merely one that covers the various components from the previous section. Certainly being comprehensive is important, but it is not what separates a good review from a passable one. In what follows, we discuss ways that will help you write an effective and helpful review and earn the respect of editors, authors, and just might earn you a “Super Reviewer” badge at the next academic conference you attend.Footnote ⁴

WHAT ELSE SHOULD YOU KNOW ABOUT REFEREEING MANUSCRIPTS FOR ACADEMIC JOURNALS?

Academic journals frequently require referees to supplement their written review with a quantitative evaluation of the manuscript. This quantitative section might ask you to assess, for instance, the importance of the topic, the clarity of the argument, or the persuasiveness of the evidence. Each journal has a slightly different set of questions, and a different (often arbitrary) scale on which they should be assessed. In other words, if the responses seem subjective to you, it is because they are. Keep in mind that what is most important about this section is not the specific number you assign to each answer. Rather, editors ask these questions because they underscore the parameters—relevance, clarity, persuasiveness, originality, writing—that should frame your review. Therefore, rather than agonizing over whether the paper you are reviewing is a 3 or a 4 on any given question, make sure you cover all the questions in your written review.

When refereeing a manuscript, your main responsibility is to give the journal's editorial staff advice about whether it should be published in that specific journal, and why. In practice, what this implies is that you need to make a substantiated argument about the scholarly merits of the paper while keeping in mind the journal's purpose and audience. Also, to the extent that there are any tradeoffs between arguing the (de)merits of the paper versus general comments to improve it, remember that the paper is not yours to fix. The specific decision—accept, reject, or revise and resubmit—is also not yours to make, so whereas you should feel free to suggest a course of action, do not feel compelled to.

Along the same lines, it is always useful to visit the journal's website and educate yourself about its purpose, audience, and manuscript acceptance rate. This will clarify the journal's publication criteria, which in turn allows you, as a reviewer, to offer a more considered and effective opinion. Comparing your reviews to the journal's acceptance rate will also help you calibrate your own standards as a reviewer; for instance, if you have rejected all five articles you were asked to review for a journal that typically publishes 50% of its submissions, you are probably being too critical in your assessment. If you have recommended acceptance or minor revisions on all five articles sent to you by a journal that typically publishes only 10% of submissions, you are being too lenient (and will probably be invited less often).

Visiting journal websites will also help you assess more accurately which journals would be most receptive to publishing your own research, thereby saving you the anxiety of waiting several months only to find that the journal thinks that your paper does not fit or is not of high enough quality for its readership. In fact, of the 15% to 30% (depending on journal) of manuscripts that get “desk rejected” (in other words, the journal's editor does not even send them out to reviewers), many are simply deemed inappropriate for the journal's target audience. Manuscripts whose topics do not fall within the scope of the journal will never be published there, no matter how good they are per se.

Finally, to reiterate an earlier point—namely, to take the review process seriously—we would remind would-be referees of the following. Although you are not, strictly speaking, under review yourself, you have every incentive to do at least due diligence. On the one hand, a thorough, persuasive review will not only earn you respect and a reputation, you might even be invited to publish a formal response to the original article in the same journal. On the other hand, a particularly shoddy review will often elicit an apology on your behalf from the journal's editor to the manuscript's author, acknowledging that yours was a particularly superficial and unhelpful review. Further, the memo that is sent by the editor to the author is also sent to you and the other referees. The confidential nature of the review process means that you escape with your dignity somewhat intact, but only just!

NOW WHAT?

With those guidelines out of the way, that leaves one last question: “How do I sign up?” Some journals, especially subfield journals, encourage would-be reviewers to e-mail the journal's managing editor to volunteer. Other journals, like AJPS and APSR, will approach you (subject to their specific policies), based on the work you have published or presented at conferences. The longer answer is that, although in general the journals will approach you (subject to their specific policies), you should always feel free to contact them directly if you are interested.

ACKNOWLEDGMENTS

Our thanks to Vinay Jawahar for editorial assistance and Sean Twombly for organizational assistance. This article came out of a 2011 APSA panel on refereeing organized by Dustin Tingley.