RESPONSES TO GIBSON’S CRITIQUE

Gibson critiques editors’ power in two ways. First, he suggests that “Desk rejecting violates the hallowed principle of peer review” that colleagues should review journal submissions because editors cannot be experts on everything. This misunderstands the nature of academic journals. Academic journals publish peer-reviewed research but are not in the business of providing peer review to anyone who submits. Gibson concedes this point, noting that some proportion of papers do not merit peer review. Of course, this raises the question of who should decide which papers should be desk rejected. Gibson implies that peer reviewers should make these choices, but this is illogical—it suggests that editors should send papers out for review to decide which papers to send out for review. In any case, it is the editors’ responsibility to make such decisions based on criteria stated on their journal’s submissions page, which reflect the editors’ vision about a journal’s content. Reviewers do not and cannot share the broad perspective that editors use to identify which papers might and might not fit because reviewers do not read the full slate of submissions that a journal receives.

Second, Gibson suggests that “Desk rejecting is procedurally unfair” for two reasons. First, because editors’ desk decisions take too long. This could be true, and editors should strive to make desk decisions quickly. However, even if editors decided faster, Gibson still would object based on the previous principle. (In 2019 at CPS, papers spent an average of 19 days in internal review.) Second, because editors often provide no substantive justification for a desk reject, their decisions appear arbitrary. To the extent that this is true, Gibson still would object even if editors offered substantive reasons for desk rejects—again, because editors are not “peers.”

Gibson’s argument also is inconsistent. He agrees at one point that editors should not be mere “bean counters,” but then he reverts to his argument that the power to desk reject goes too far. It is never clear where editors should reside on this continuum between “bean counters” and “arbitrary dictators.” Instead, Gibson offers three suggestions to reduce desk-rejection rates. However, all of these skirt around his core complaint about editorial power and instead seek to reduce submissions so that editors can rely on peer reviews for all decisions.

This focus implies that Gibson’s real objection is not to an excess of editorial power but rather to an excess of weaker papers in the pipeline, which incentivizes editors to desk reject papers that otherwise might go out for review. Of course, it is not clear how an excess of weaker papers forces editors to desk reject stronger papers. Furthermore, reducing the number of submissions would not reduce editors’ power at all.

In any case, Gibson’s suggestions to reduce submissions are either impractical or elitist. First, he suggests that journals could charge a submission fee and pay reviewers. He states that some mechanism “might be established to subsidize those who claim they cannot afford the submission fee,” but this is impractical. Who would manage the system, given the decentralized nature of journal ownership? Who would pay this subsidy? The answer cannot be “members of the association affiliated with that journal” because membership in an association is not and should not be a prerequisite for submission—and many journals (e.g., CPS) are not affiliated with an association. Gibson recognizes that many scholars may not be able to afford such fees. Productive scholars have multiple papers under review simultaneously, they often receive multiple rejections per year, and they may have to resubmit the same papers multiple times. It is true that this model has been adopted by economics—but, in many cases, the fee is paid and the article still is desk rejected, which we presume would be even less desirable for Gibson.

Second, Gibson suggests that journals could charge a publication fee in addition to a submission fee. Yet, many or even most scholars could not afford these fees, particularly graduate students and scholars outside of wealthy countries. Our discipline is increasingly global. Pressure to publish in top English-language journals is growing in every region, and scholars working outside of wealthy countries often have fewer resources. In addition to the growing demand for open-access fees, more general publication fees are likely to prove unpopular.

Third, Gibson suggests that journals should discourage graduate students from submitting. Most graduate students, he argues, do not “produce publishable work” and allowing “unworthy graduate student papers to muck up the review process is much too high a price to pay.” This is simply elitist. We could equally suggest that most older scholars are long past their prime and should submit fewer papers. Both suggestions are equally offensive and counterproductive for the field. In any case, the equilibrium outcome is easy to imagine: faculty would decide that their students’ papers are worthy of submission but their colleagues’ students’ papers are not. Asking for fewer graduate submissions has all the hallmarks of the collective-action problems we study as political scientists—it is unlikely that any individual graduate student would choose to refrain from submitting unless all graduate students were prevented from doing so.

In summary, adopting Gibson’s approach would impose monetary burdens on productive scholars and make the discipline more elitist, but it would not resolve this deeper issue of “salami slice” or “minimum publishable unit” papers being submitted. Asking editors to increase the desk-reject rate would not necessarily solve these problems either—but it would not be elitist, as long as editors conduct their internal reviews blindly.Footnote ¹

Gibson does identify the real challenge that editors face: the unceasing growth in submissions. Yet, given this trend and like many editors, we have chosen to judiciously conserve our most valuable resources—reviewers’ time and goodwill—by not sending out for review papers that we regard as unsuitable. Desk rejecting also is a better use of an author’s time than sending a paper out for review that has no chance. In some cases, it is difficult both to find reviewers and to obtain promised reviews—months may pass that authors might not have wasted had they sent the paper to a more appropriate journal or spent more time working on it in the first place. Even with our desk-reject policy in place, we need to ask five reviewers to get three reviews for the average manuscript, and it typically takes two weeks to secure them. Many papers require more than 10 reviewer requests, which substantially delays the process.

RESULTS OF AN INTERNAL AUDIT

Are desk rejections an efficient use of editorial resources? All editors make both Type I and Type II errors—that is, we accept some papers that will go unread and uncited and reject others that will gain wide readership and numerous citations. How frequent are these errors for desk-rejected papers as well as those rejected after peer review? Our job is to make judgment calls about which papers are already strong and likely to be highly impactful if they are nudged in a particular direction. How successful are we at making these judgment calls?

To answer these questions, we analyzed the fates of all submissions to CPS from the day we took over (i.e., September 1, 2013) through the end of 2014 (N = 874). This timeframe allowed us to track what happens to papers that we accepted, desk rejected, and rejected after review. As part of its standard business practice, the CPS publisher (i.e., SAGE Publications) tracks “lost citations” of submissions that CPS rejected that other journals eventually published. However, SAGE can track a paper that CPS rejects only if the title remains unchanged when it subsequently appears in a different journal. Therefore, SAGE could track only about one third of the articles that CPS rejected. We searched manually for the remaining papers using Google, Google Scholar, and authors’ personal and/or professional web pages.Footnote ² In most cases, the title was only slightly altered and the paper rejected from CPS was easy to find. In cases in which it was obvious that titles did not match, we compared the abstract of the CPS submission to the abstract of the eventually published paper. Papers that subsequently appeared in a peer-reviewed journal with the same author(s) and fundamentally similar arguments, variables, and cases were regarded as “the same” as the CPS submission.

After we compiled information on the fates of papers rejected from CPS, the question was: How can we judge whether we had made a Type I or Type II error in rejecting a submission? It is not obvious what should count as a Type I error on our part, but “receiving zero citations” is a reasonable metric.

What should count as a Type II error, given the possibility that authors can revise after rejection? Answers to three questions provide a reasonable basis for this judgment. First: What proportion of rejected submissions was eventually published? Second: What proportion of rejected submissions was eventually published in higher-ranking journals? Of course, it is possible that a paper that we reject that subsequently appears in a higher-ranking journal may never get cited, making it a Type I error for that journal and arguably not a Type II error for CPS. Thus, the third question is: What proportion of submissions we rejected eventually receives a higher number of citations than the average paper that we accepted?

We used the two-year impact factor from the Journal Citation Reports to make cross-journal comparisons. This number is used widely to “rank” journals. To compare citations, we used the number of citations that each paper received in the year it was published (in a scheduled volume and issue, not online) plus two full calendar years, using the Social Sciences Citation Index. We would have preferred to count citations after four or five years, but 69% of all submissions to CPS in 2013–2014 that we rejected were published in 2016 or later. We wanted to compare the papers that we accepted against as wide a sample of those we rejected that subsequently appeared elsewhere. Even with a two-year window for citations, we still had to exclude all papers that CPS rejected that were published in 2018 or later—about 20% of the total (the research was completed in September 2020).

To be clear, only papers submitted to CPS in late 2013 through the end of 2014 are included in our analysis. Papers that we rejected that subsequently appeared elsewhere but only in 2018 or later were excluded entirely from the analysis because they did not have the full two years to accumulate citations to make an “apples-to-apples” comparison with the set of papers (both accepted and rejected) that were in print for at least two years. Papers that were not published anywhere were given zero citations and are included in the analysis. It certainly is possible that some of these papers eventually will be published; therefore, the following results should be considered a comparison of the relative success of papers submitted to CPS in 2013–2014 as of late 2020.

Table 1 provides results of our internal audit. First, we did make some Type I errors—a few of the papers that we accepted received no citations after two years. However, this is true of fully 70% of the papers that we desk rejected—and it even is the case for more than half of the papers we rejected after review. This is partly because, as the second column reveals, a substantial proportion of papers that we desk rejected and rejected after review did not appear in any peer-reviewed journal almost seven years after our editorial decision.

Table 1

CPS Internal Audit Results

Regarding Type II errors, papers published in CPS receive, on average, almost five times the number of citations as papers that we desk reject and almost three times the number of citations as papers that we rejected after review. Similar differences exist in terms of average impact factor. Again, these results are largely a function of the fact that so many rejected papers are never published. Nevertheless, we found that only 5.1% of papers that we desk rejected and 11.1% of those we rejected after review received more citations than the average paper we accepted. To emphasize the point: even after receiving full peer review, authors of only 11.1% of submissions we rejected subsequently revised their work so that the paper did as well as or better than the average paper we accept—and only about one in 20 papers that we desk rejected reached that mark. Likewise, only 2.7% of papers that we desk rejected were eventually published in journals with a higher impact factor than CPS; the corresponding percentage for papers rejected after review is 6.2%.Footnote ³

Some might wonder if there are other systematic differences in the papers we desk reject compared to those that we review and either accept or reject, not least the international origin of papers. However, breaking down submissions by country of origin of the corresponding author, we see that limiting submissions to those from the United States (46% of all submissions), accepted papers had 6.1 average citations and our CPS impact factor of 3.2. Review-rejected papers had 2.2 average citations and an average journal impact factor of 1.8 and desk-rejected papers had 0.95 citation and an impact factor of 1.3.

Rejection should not be the end of a paper’s life. The results of this internal audit suggest that sometimes we do make Type II errors—that we should not have rejected a specific paper. For desk-rejected papers, however, this outcome is rare. Of course, the rejection may have been what spurred the authors to improve their paper. Overall, our decision process appears to be fairly efficient, undermining Gibson’s claim that desk rejections are arbitrary and illegitimate. The fate of papers at CPS largely reflects their initial appropriateness and/or quality, given our mission to curate what the journal publishes. Overall, the majority of papers that we desk reject are not published after more than six years, appear in journals with a lower impact factor than CPS, and/or obtain fewer citations than the average paper published in CPS. We encourage editors of other journals to conduct similar audits as a way to gauge editorial effectiveness.