Preaccepted Articles

One response to concerns about publication bias and replication failures is the preregistration of experimental trials—a practice that is mandated in some areas of medicine and is beginning to happen voluntarily by social science researchers conducting field experiments (particularly in development economics) or studying unobserved or uncollected observational data (e.g., Grossman and Pierskalla n.d.; Monogan Reference Monogan2013). An emerging cross-disciplinary movement of researchers argues for preregistration because it forces authors to publicly disclose their hypotheses and analysis plans before data have been collected, which should reduce the risk of spurious results (e.g., Gerber and Malhotra Reference Gerber and Malhotra2008a; Humphreys, de la Sierra, and van der Windt Reference Humphreys, Sierra and Windt2013; Miguel et al. Reference Miguel, Camerer, Casey, Cohen, Esterling, Gerber and Glennerster2014; Monogan Reference Monogan2013; Wagenmakers et al. Reference Wagenmakers, Wetzels, Borsboom, van der Maas and Kievit2012). One of the best examples of this practice to date is the Oregon Health Insurance Experiment. The authors publicly archived their analysis plan before data were available (Finkelstein and Baicker n.d.) and explicitly labeled all unplanned analyses in published studies resulting from the experiment (Baicker et al. Reference Baicker, Taubman, Allen, Bernstein, Gruber, Newhouse and Schneider2013; Finkelstein et al. Reference Finkelstein, Taubman, Wright, Bernstein, Gruber, Newhouse, Allen and Baicker2012; Taubman et al. Reference Taubman, Allen, Wright, Baicker and Finkelstein2014; see also Casey, Glennerster, and Miguel Reference Casey, Glennerster and Miguel2012). In political science, Humphreys, de la Sierra, and van der Windt (Reference Humphreys, Sierra and Windt2013) likewise proposed comprehensive but nonbinding registration for experiments.

Unfortunately, preregistration alone will not solve the problem of publication bias. The format limits authors’ ability to produce the statistically significant findings that reviewers and editors demand, which may lead authors to opt out of registration or to shelve nonsignificant findings. As a result, authors have little incentive to engage in the practice unless it offers an appealing route to publication in prestigious journals. ^{Footnote 3} In addition, scholars may be tempted to deviate from preregistered analysis plans when those plans are not linked to journal publication practices. Most fundamental, however, if studies are more apt to be published when they report statistically significant results, then publication bias is still likely to ensue even when preregistration is commonly practiced, as studies frequently have found in examining results from clinical trials (see, e.g., Dwan et al. Reference Dwan, Gamble, Williamson and Kirkham2013). ^{Footnote 4}

A better practice would be for journals to offer authors an option in which articles with prespecified analysis plans would be accepted in principle before the study was conducted. By offering this voluntary option of planned confirmatory studies, journals would create a positive incentive for preregistration that would minimize file-drawer bias and avoid the mandates and constraints to which some scholars object (e.g., Laitin Reference Laitin2013; Publications Planning Ad Hoc Committee 2014). Although this approach, which is known as Registered Reports, may seem radical, the format is currently being offered by a number of journals, including AIMS Neuroscience (Chambers et al. Reference Chambers, Feredoes, Muthukumaraswamy and Etchells2014), Cortex (Chambers Reference Chambers2013), Perspectives on Psychological Science (n.d.), and Social Psychology (Nosek and Lakens Reference Nosek and Lakens2014). ^{Footnote 5} Most notable, a planned special issue of Comparative Political Studies will offer preaccepted articles for the first time in the discipline (Findley et al. Reference Findley, Jensen, Malesky and Pepinsky2014).

Using this format, an article’s introduction, theory, and methods sections would be written in advance. Articles would be accepted in principle after a first-stage review to prevent editors and reviewers from backing out of publishing an article due to a null or mixed finding. The authors then would conduct the study and populate the results section based on a prespecified analysis plan. Post-hoc analysis and interpretation would be allowed—scientific discovery is important!—but must be explicitly labeled as exploratory. The article then would be reviewed in a second round to ensure that the theory and hypotheses had not been changed, that any results that deviate from the prespecified plan are identified appropriately, and that the analysis of results was conducted professionally. ^{Footnote 6} Rejecting the article on the basis of a null or mixed result would be prohibited in principle. ^{Footnote 7}

Although the preaccepted article format clearly is most useful for surveys and experiments, the studies in question could include other types of observational data. The problem, however, is that preacceptance of observational studies would require editors to trust authors’ self-reports that the data had not been collected or analyzed previously—which creates an incentive problem of the type that preacceptance is intended to alleviate. ^{Footnote 8} An alternate approach would be to encourage authors of observational studies to submit articles that offer out-of-sample predictions (e.g., Monogan Reference Monogan2013). For instance, scholars who study democratic stability could offer forecasts of future irregular regime changes or coups throughout a prespecified period that would be evaluated after a specified interval of perhaps two to four years (e.g., Beger Reference Beger2014; Ulfelder Reference Ulfelder2014).

Despite these benefits, it is still possible that relatively few authors would use a preaccepted article format. To address this concern, the American Political Science Review (APSR) and other political science journals should consider offering incentives for authors to use the format by publishing preaccepted submissions as lead articles in each issue and—to the extent possible—fast-tracking them in the editorial and peer-review process. It may be possible to raise funds to offer authors whose designs are accepted in this format, as currently done by Perspectives on Psychological Science (n.d.), or for the APSR to partner with Time-Sharing Experiments in the Social Sciences (TESS) to preaccept articles that then would be fielded for free as part of a TESS omnibus survey.

Finally, it is worth noting the benefits that a shift toward a preaccepted-article format could have for research and publishing practices in the discipline as a whole. The format would create healthy pressure on authors, editors, and reviewers to (1) identify hypotheses that are substantively important and developed carefully in relation to the previous literature; (2) answer research questions for which results would be worth publishing even if they are not surprising or counterintuitive; (3) keep articles short given the likelihood of null or mixed results ^{Footnote 9} ; and (4) ensure that studies have sufficient statistical power to detect the hypothesized effect most of the time. In this way, the format could have positive spillover effects even if it is used infrequently. ^{Footnote 10}

Results-Blind Peer Review

An alternative approach to countering publication bias that is less closely linked to the experimental method is to offer “result-blind peer review” for quantitative studies of any type (e.g., Glymour and Kawachi Reference Glymour and Kawachi2005; Green, Humphreys, and Smith n.d.; Greve, Bröder, and Erdfelder Reference Greve, Bröder and Erdfelder2013; Hanson Reference Hanson2010; Smulders Reference Smulders2013). In this approach, reviewers would assess the theory and research design of a manuscript and make an initial decision without access to the statistical findings, which initially would be withheld. It is hoped that such a decision would constrain (at least partly) the post-hoc reasoning that produces publication bias against null results. Editors and reviewers then would assess the merits of a manuscript in a second round when the results were unblinded.

This approach faces significant implementation challenges. First, a small pilot study of this approach by Sridharan and Greenland (Reference Sridharan and Greenland2009) at Archives of Internal Medicine found that there is reason for concern about publication bias even with a two-step review process. “[O]ver 7% of positive articles benefited from editors changing their minds between steps 1 and 2 of the alternate review,” they found, but “this never occurred with the negative studies” (Sridharan and Greenland Reference Sridharan and Greenland2009). Journals would need to carefully avoid allowing publication bias to gradually return after the results were known to reviewers and editors. Second, some evaluations may not be truly results-blind if editors and reviewers previously read the paper or heard it presented at a conference (a particular problem in social science, which has long publication lags). Third, offering this format as an option rather than as a requirement could create an adverse selection problem in which results-blind submissions would consist mostly of articles with null or mixed results, thereby allowing reviewers to infer that authors’ hypotheses likely were not supported. This inference could create potential biases against authors’ theory and design in first-stage reviews, allowing publication bias to gradually return as well as reducing the perceived value or status of articles published in the format. To avoid this adverse selection problem, journals might consider randomizing a subset of manuscripts in this process or making it mandatory for certain special issues or sections (e.g., Findley et al. Reference Findley, Jensen, Malesky and Pepinsky2014).

Verifying Replication Data and Code

Every APSA journal should follow the practice of journals including the American Journal of Political Science (n.d.) and American Economic Review (AER) (n.d.), which require authors of quantitative studies to provide a full replication archive before publication. This change would be consistent with the standards of data access that are currently part of the APSA’s code of ethics (APSA Committee on Professional Ethics, Rights, and Freedoms 2012; see also Lupia and Elman Reference Lupia and Elman2014). As King (Reference King1995) and many others argued, replication increases the likelihood of detecting errors and building cumulative scientific knowledge, which improves the rigor and value of published results in our field. Journals therefore should publish more replications of major results—although how to do so is a contested and difficult issue (for a discussion of several possible approaches, see Ishiyama Reference Ishiyama2014). However, APSR and other journals could further encourage careful research practice by hiring qualified graduate students to ensure that the results of accepted articles can be reproduced from submitted replication files before publication, as done at the Quarterly Journal of Political Science (n.d.). By institutionalizing a practice that typically is used only on an ad-hoc basis in class assignments (King Reference King2006) and demonstration projects (Dewald, Thursby, and Anderson Reference Dewald, Thursby and Anderson1986), a mandatory journal-replication policy would improve the incentive for scholars to engage in careful and systematic research practices. For instance, one in five articles examined in 2006–2008 in AER did not fully satisfy the requirement that results be reproducible from submitted data and code, which led the journal to require review by contracted graduate students (Glandon Reference Glandon2010).

A Frequent-Flier System for Journals

Journals depend on the free labor provided by academics in the peer-review process. Reviewing is a generally thankless task that falls disproportionately on prominent and public-minded scholars who receive little credit for their work. Therefore, academics often prioritize other tasks before the reviews that they have committed to provide. As a result, manuscripts often are backlogged for months in “review limbo,” which slows the publication process and stalls the production of knowledge that our field can share with the public.

One idea is to develop a point system for APSR or for all of the APSA journals (as well as participating section journals) that is analogous to a frequent-flier program. Each review would earn a scholar a specific number of points, with bonuses awarded by editors for especially timely or high-quality reviews. To request a rapid review of their own manuscript, authors then could redeem those points when they submit to that particular journal. ^{Footnote 11}

The journal, in turn, would offer points to those who review the manuscript quickly, thereby expediting the process. This system would not be useful for reviewers who do not make frequent submissions to a particular journal. However, for reviewers and authors who frequently interact with a journal such as APSR during a period of decades, the system could provide greater incentives for rapid and thoughtful reviewing. ^{Footnote 12}

Referee Report Roll-Downs to Section Journals

One impediment to the rapid dissemination of knowledge from political science is the serial nature of the journal-submission process. In many cases, it can take six to 12 months or more to receive reviews from an initial submission to a journal. If that submission is declined, the author must resubmit it to a new journal and restart the process, which frequently can result in duplication of effort by reviewers and unnecessary delays for authors. One way to improve the efficiency of this process is for the APSA to offer authors the option of forwarding their APSR submissions and the resulting reviews to other APSA or affiliated section journals. Many submissions are declined at highly selective journals such as the APSR with reviews indicating that the manuscripts should move forward toward publication at field journals. By allowing authors the option to redirect the manuscript and reviews to such a journal (a process that should be possible in online publishing systems), they could move toward publication more quickly while reducing the reviewing burden on other scholars during the submission process. ^{Footnote 13} The American Economic Association implemented such a system for the AER and its affiliated American Economic Journal (AEJ) field journals. Authors are given the option to forward referee reports and correspondence from the AER to the AEJ editors (see, e.g., American Economic Journal: Applied Economics n.d.). Although the APSA does not publish field journals, many of its affiliated sections do (e.g., Legislative Studies Quarterly, Political Analysis, and Political Behavior). The association should consider proposing such a roll-down policy for those journals and explore partnerships with other field journals. ^{Footnote 14}

Triple-Blind Reviewing

Several studies have documented potential biases in evaluating scientific articles and abstracts as well as grant, fellowship, and job applications. For instance, scientific submissions were found to be evaluated more favorably when they originate in the United States or elite universities than those from people with a different background (Ross et al. Reference Ross, Gross, Desai, Hong, Grant, Daniels and Hachinski2006). In addition, several studies found that submissions and applications from women are viewed less favorably than those from men (Moss-Racusin et al. Reference Moss-Racusin, Dovidio, Brescoll, Graham and Handelsman2012; Steinpreis, Anders, and Ritzke Reference Steinpreis, Anders and Ritzke1999; Wennerås and Wold Reference Wennerås and Wold1997), although these results generally have not been replicated in larger samples (Ceci and Williams Reference Ceci and Williams2011). Concerns about these potential biases are one reason for the dominance of double-blind review procedures among journals in our field, but they apply no less to editors than to referees. Why would an author’s identity be material to a decision about the merits of a manuscript?

Given these concerns, all APSA journals should conceal the identity of the author(s) from journal editors just as they do for referees—a so-called triple-blind review process. Of course, editors may have previously encountered a manuscript or can use Google to determine an author’s identity (which also is true for reviewers). Some have suggested bowing to these pressures by possibly moving to a single-blind review system (e.g., Publications Planning Ad Hoc Committee 2014). I argue that we instead should seek to minimize potential biases in editorial and review decisions. The triple-blind procedure already is used by the philosophy journals Mind (n.d.) and Ethics (n.d.) and the American Business Law Journal (Cahoy Reference Cahoy2010) and easily could be implemented in online manuscript-review systems. ^{Footnote 15}

By minimizing any inadvertent or unconscious biases in editorial decision making, triple-blind review could improve the quality of publication decisions and increase confidence in the scientific integrity of the process among APSA members and disciplinary stakeholders.