Survey of Journal Editors

To assess the transparency of political science journals in maintaining and releasing submission data, we first searched the websites of the 30 highest ranked journals, as rated on their impact by Garand et al. (Reference Garand, Giles, Blais and McLean2009). We looked for whether these websites relayed simple record-keeping information, such as average turnaround time, as well as more detailed submission data. The search turned up little in the way of the release of detailed submission data, and we rarely found that even simpler summary statistics were being disseminated. Only 10 of the 30 journal websites provided basic information, and in most of these 10 cases, the websites provided only a rough estimate of average turnaround times.

Editors unwilling to publish this information on their journal's website may provide it in print or by request. To more formally assess journals' transparency in releasing submission data, in July 2009,Footnote ⁹ we sent an e-mail questionnaire (see figure A1 in the appendix) to the editors (or editorial staff) of the top 30 political science journals, receiving at least a partial response from 20 of the 30 journals surveyed.Footnote ¹⁰ About half of the editors of the 30 journals responded to the entire survey or directed us to print or web material to answer our questions. Table 1 summarizes the responses to the questionnaire, ranks the journals on their transparency in releasing submission data, and provides general information on and comparisons of the record-keeping practices of the top 30 journals. Figures 1 and 2 rank the responding journals by acceptance rates and turnaround time from initial submission to first decision, respectively.

Table 1

Summary of Political Science Journal Questionnaire Responses

Notes.

^a Web, print, and by request = 1; web and by request = 2; print and by request = 3; by request only = 4; no contact made with journal or no information found denoted by bar.

^b Number includes resubmissions.

^c 82% of manuscripts turned around in 60 days or fewer.

Figure 1

Responding Journals' Acceptance Rates, Most Recent Year

Figure 2

Responding Journals' Average Turnaround Time in Days from Manuscript Submission to First Decision, Most Recent Year

The journal transparency rankings depend on the availability of journal information (e.g., acceptance rates, turnaround rates, number of submissions per year) via three different mediums: on the web, in print, and by request. Journals that provided information via all three mediums received a ranking of one. Journals delivering the information via two mediums received a ranking of either two or three. A journal that offered the information both on the web and by request was ranked higher than a journal that provided the information in print and by request, because visiting a journal's website generally imposes fewer opportunity costs than does obtaining a journal's print copy.Footnote ¹¹ Finally, journals that provided such information only by request received a ranking of four.

While the measure is a bit crude, much can be learned from this first attempt at assessing political science journals' transparency in releasing submission data. First, Garand et al.'s (Reference Garand, Giles, Blais and McLean2009) rankings indicating impact do not necessarily correlate with the transparency rankings. Although the APSR and the Journal of Politics (ranked first and third, respectively, under Garand et al.'s system) both received a transparency ranking of one, Political Research Quarterly (ranked 16th with Garand et al.'s system) also received a top transparency score. Five journals received a ranking of two, and three received a ranking of three. Almost half of the journals (11 of 23)Footnote ¹² indicated that they would be willing to provide such information only upon request. Many journals clearly keep but do not openly share their submission data.

In the spirit of transparency, table 1 also provides some of the additional survey information provided by the journal editors. These data showcase the ample variation that exists in the submission and review processes of the profession's top journals. The top political science journals vary widely in the number of submissions they receive, their acceptance rates, and their average turnaround time until first decision.

Transparency Case Study

Our examination of submission data for American Politics Research (APR)Footnote ¹³ from January 2006 to December 2008 highlights the many benefits of more extensive journal record-keeping. While past work has examined submission and publication data from the more general political science journals, we could find no scholarship that focused on a field-specialized journal like APR. The following assessment of the articles published in relation to the articles submitted offers a more complete view of the journal's specialization and biases.

Our examination focused on a few understudied relationships in the field of political science journal publishing, including the effect of the lead author's region, university type, and professional status on a manuscript's likelihood of acceptance. A lead author's geographic location has been shown to influence article acceptance in academic journals in the medical field (Boulos Reference Boulos2005; Tutarel Reference Tutarel2002), so we expected that such a geographic bias might also favor APR's acceptance of manuscripts submitted by lead authors from the Mid-Atlantic region. Authors from these locales are likely to have a greater familiarity with the editor and editorial staff as the result of shared attendance at regional conferences or service on regional boards.

We also expected that authors who work at institutions that place a greater emphasis on research would have more success publishing their work in APR. Research institutions typically give their faculty lighter teaching loads so that they will have more time to research and publish. In academia at large, articles by authors from academic settings are accepted at scholarly peer-reviewed journals at significantly higher rates than articles by authors from nonacademic settings (Tenopir and King Reference Tenopir and King1997). We expected to find a similar difference in acceptance rates of submitted manuscripts from authors in academic settings with different foci and resources.

Finally, we posited that an author's professional status would impact the likelihood of his or her article being accepted. Seasoned academics (associate and full professors) are more likely than less experienced scholars to know the norms of publishable scholarship and be better able to correctly match their manuscript to an appropriate venue (Pasco Reference Pasco2002). Additionally, submitting authors who have attained their doctorate but have not yet gained tenure (usually assistant professors) would likely have a higher probability of acceptance than would graduate students. These authors have the advantage of some research and publishing experience and are driven to produce high-quality work by the incessant ticking of the tenure clock. In examining these hypotheses, we controlled for manuscript turnaround time, the number of authors on a submission, the subject area of the manuscript, and whether the manuscript had a female lead author (see table A2 in the appendix for information on variable measurement).

The reasons why editorial bias may creep into editorial decision-making are easy to explain, even if an editor is trying to be conscientious and fair. Take, for example, two submissions of approximately equal quality. Reviewers return two equally critical sets of reviews, but the editor knows the author of the first manuscript and not the author of the second. The editor may be willing to hazard that the author of the first manuscript is capable of managing the revisions and give that author the benefit of the doubt, sending him or her a decision of revise-and-resubmit. But, being completely in the dark about the author of the second manuscript, the editor may not extend him or her the same benefit of the doubt and may instead send a rejection letter.

It is important to note that by “knowing” the author of the first manuscript, we do not mean that this author has to have been a student, much less an advisee of the editor. Sometimes just having been on a conference panel together or having met a time or two is enough. And, of course, editors are not always even conscious of what biases might be operating, which is why conscientious editors should be interested in serious data collection and analysis.

Case Study Findings

Over the 2006–08 period, the APR journal staff of one full-time editor and one half-time graduate student processed 491 manuscripts.Footnote ¹⁴ During this time period, 111 manuscripts were accepted. Twenty-four of the manuscripts that received a decision of revise-and-resubmit were never returned (see table A1 in the appendix for additional data on submissions).Footnote ¹⁵ As hypothesized, articles submitted by lead authors from the Mid-Atlantic region have a slight advantage over articles submitted from other regions: just over 30% of manuscripts submitted by lead authors from the Mid-Atlantic region were accepted to APR (see table A3 in the appendix for the descriptive statistics). The probit model's excluded category is the Mid-Atlantic region (see table 2). The signs for nearly every region (except international) are negative, indicating that papers submitted by lead authors from Mid-Atlantic institutions are more likely to be accepted than papers from other regions. Although these relationships are not statistically significant,Footnote ¹⁶ the accompanying predicted probabilitiesFootnote ¹⁷ (see table 3) provide a better sense of their substantive significance. Lead authors from the Mid-Atlantic region generally have a 15 to 20 percentage point advantage over lead authors from other regions in terms of article acceptance at APR. International papers appear to have the next best chance of acceptance, but this finding could be an artifact of the small number of manuscripts (n = 11) that were submitted by authors working abroad.

Table 2

Predicting Final Acceptance at APR

Notes. N = 464. Log likelihood = −229.254. Pseudo-R ² = .089. Coefficients and standard errors calculated using probit regression. p values are two-tailed. Graduate student is the excluded variable for author status. Bachelor's degree is the excluded category for institution type. Mid-Atlantic is the excluded variable for region. The model includes the three subject categories with the most submissions; a number of other categories are excluded (American political development, interest groups, media, other, parties, policy, presidency, public opinion, and subnational).

Table 3

Predicted Probability of Acceptance at APR

Note. Predicted probabilities calculated using the observed values approach.

As hypothesized, articles submitted by lead authors from research institutionsFootnote ¹⁸ performed better at APR than articles submitted by scholars from other institution types. The excluded category in the model is the bachelor's degree institution type. The sign for master's degree institutions is negative, indicating that papers submitted by lead authors from these institutions are less likely to be accepted than papers submitted by authors from bachelor's degree institutions. The signs for the three research institution rankings are positive, suggesting that papers submitted by lead authors from these institution types are more likely to be accepted than papers submitted by authors from bachelor's degree institutions. Although these relationships are not statistically significant, the predicted probability of acceptance at APR is generally higher for authors from institutions with a research focus (see table 3).

Somewhat surprisingly, lead authors who are graduate students, assistant professors, and associate professors all have similar likelihoods of manuscript acceptance at APR, while full professors' chances of manuscript acceptance lag slightly behind. The excluded category in the model is the status of graduate student. The manuscripts of assistant and associate professors are more likely to be accepted at APR than the work of graduate students, but full professors seem to have less success. None of these relationships reach accepted levels of statistical significance. Substantively, the most surprising finding is that graduate students have a 19% predicted probability of having their manuscript accepted at APR, while full professors have an 18% likelihood of acceptance (see table 3). Two types of selectivity bias may well be behind this finding. First, APR has a notable reputation for publishing the work of younger scholars who are just launching their careers. These scholars may well send their best work to the journal. Second, senior faculty, who are also cognizant of this reputation, may prefer to send their best work elsewhere.

The nonsignificance of many of these relationships led us to look at an additional relationship—whether authors who serve as APR board members have a higher probability of acceptance than non-board members. Indeed, manuscripts with an author who is an APR board member are 6 percentage points more likely to be accepted than manuscripts without a board member author (see table 3). Although not statistically significant, this increased rate may be explained by some factors other than editorial bias. First, board members may have a greater familiarity with the types of articles that APR publishes than the average American politics-focused political scientist. Second, board members are chosen not at random, but predominantly because of their past publishing successes.

In summary, many of the relationships explored here are neither statistically nor substantively significant. Here, null findings prove a relief. While some patterns exist between the personal characteristics of APR authors and the likelihood of manuscript acceptance, none show particularly strong evidence of editorial bias. Nevertheless, awareness of even slight tendencies for bias can inform the editorial staff in future decision-making.