JOURNALS AND REPLICATION POLICIES

Political Analysis—the journal of the Society for Political Methodology (SPM)—was among the first journals in social science to develop a replication policy for papers published in the journal, beginning after the publication of King’s Reference King1995 paper. For example, in 1996, some papers published in Political Analysis stored their replication materials in the ICPSR archive (Box-Steffensmeier and Lin Reference Box-Steffensmeier and Lin1996). However, because the policy was largely voluntary, other articles published in the journal at that time make no mention of replication materials. By 2000, the journal began a more systematic collection of replication materials, storing them on the SPM website.^{Footnote 3}

Beginning in 2012, Political Analysis developed a new replication policy: all papers reporting data analyses (including simulation modeling) would be required to store the materials necessary to replicate the results reported in each published paper in the journal’s Dataverse, prior to publication.^{Footnote 4} This made Political Analysis one of only a few journals before 2015 requiring the provision of replication materials prior to publication.^{Footnote 5} Currently, replication materials are requested before final acceptance of a manuscript for publication, which are reviewed by both the editor overseeing the manuscript and one of the journal’s graduate editorial assistants. Only after the replication materials have been reviewed are they released to the public on the journal’s Dataverse and the paper sent to the publisher for production. Although many issues arise during our review of replication materials, we have not had authors who refused to meet the journal’s current replication requirement.

MEETING REPLICATION REQUIREMENTS

In April 2016, one of us published a study in PS: Political Science & Politics titled “How Are We Doing? Data Access and Replication in Political Science” (Key Reference Key2016a). This study examined the replication polices of six major journals, including Political Analysis, and determined how many recently published papers in those top journals had replication materials. The research found that during the study period, there was a significant bifurcation in the percentages of papers published with available replication materials. Three journals had a high percentage of studies that did not have them: 67.6% of the papers published in American Political Science Review, followed by Journal of Politics (51.1%) and British Journal of Political Science (50.0%). At the other end of the distribution were International Organization (10.2%), American Journal of Political Science (9.3%), and Political Analysis (1.9%).

Ordinarily, the fact that 98.1% of papers published in this study period in Political Analysis had available replication materials might be cause for celebration, especially given that the journal had the highest compliance rate among the major political science journals. However, from the journal’s perspective, Political Analysis has universal compliance with the replication requirement.^{Footnote 6} This discrepancy highlights the importance of journal replication archives and raises the issue of accessibility.

Because Key (Reference Key2016b) published a replication archive, it was possible to identify the Political Analysis article by Bowers, Fredrickson, and Panagopoulos (Reference Bowers, Fredrickson and Panagopoulos2013) that had been coded as unavailable. Although Bowers et al. had provided a replication archive on the journal’s Dataverse, additional software was needed to extract the file. In other words, the issue was not with the availability of the replication material; rather, it was that the single file containing the materials was in a format not easily accessible using standard software available on Macs and PCs.^{Footnote 7} The more steps involved in retrieving replication materials, the less useful these materials become.

Like a ramp that is too steep, journal replication archives that require users to download not only the data they want to access but also specialty software to open the replication file make the archives inaccessible to a wider audience. Although many researchers at R1 institutions have access to a variety of statistical packages and powerful machines, graduate students and those at smaller institutions may not be as fortunate. This resource discrepancy highlights the tension between the admirable goals of DA-RT and the realities faced by authors, editors, and replication-archive users. This led us to understand that there are important, pressing problems regarding replication materials—in particular, ensuring that the materials are provided in usable and accessible formats.

COMMON ISSUES THAT ARISE WITH JOURNAL REPLICATION MATERIALS

By now, we have much experience with replication policies and materials. Recently, Political Analysis began to devote more time and resources to the process of reviewing and using the replication materials before final acceptance of a paper for publication. It is during the verification stage that we often find many common issues that arise with replication materials. This section discusses many of those issues, followed by a presentation of several recommendations for authors and other journals to improve the practice of replication and minimize accessibility issues that arose with the Bowers et al. replication materials. The most common issues found can be grouped broadly in three categories: organization, clarity, and usability.

RECOMMENDATIONS

Based on our experiences with journal replication archives, we offer the following recommendations to authors and other journal editors to improve organization, clarity, and usability. Whereas the widespread practice of providing replication materials is relatively recent, we hope that journals, professional societies, and publishers will coordinate in the near future to develop common standards for replication materials.^{Footnote 10}

1. 1. Archive

   1. a) Journal replication materials should be stored in permanent archives that ensure their availability after long periods. Although there are many data-storage options available, some are more durable than others. Personal websites are susceptible to “link rot” caused by website reconfiguration, changing institutions, and other maintenance issues (Key Reference Key2016a). At present, the best archives are those with an institutional guarantee, such as Dataverse and other university- or consortium-backed archives.

   2. b) Authors should store their materials using file formats that are likely to be accessible in the future (e.g., comma-delimited or flat text files for data rather than specific proprietary software files).^{Footnote 11} In addition to the preservation advantage of using these types of files, they can be easily read on currently available statistical software, simplifying their use across different platforms. If it is important that files be compressed or stored in a particular format, authors should provide documentation for future users who may not be familiar with how to uncompress or access these files.

2. 2. Readme Files . The creation of a clear and sufficiently (but not overly) detailed readme file is a key element of every replication package. Several items should be included in a readme file, as follows:

   1. a) A reference to the associated paper or publication.

   2. b) A short description of the files and file types included in the replication package: for example, raw data, processed data, scripts to manage the data, and scripts to produce estimates.

   3. c) An indication of the order in which the scripts are to be run, as well as noting where the different tables and figures found in the associated publication are generated and stored. Authors should indicate whether there are intermediate outputs generated by one script and used by another so that it is clear to users how to proceed (e.g., if a script handles the raw data to produce a processed dataset to be used in another script, this should be noted in the readme file).

   4. d) A list of software and software packages (as well as the dependent packages on which they rely) and the operating system used to produce results in the paper. Technical information on the hardware used to produce the results also is helpful (e.g., the number of cores), especially if computationally demanding techniques were applied. Authors should indicate which versions were used because packages frequently are altered to fix bugs and resolve other issues. These version changes can produce different results when running the same code with a different version of the package or software. Some models also require additional software such as JAGS or a C++ compiler to be available in the computer. Indicating this in the readme file is useful for a less-familiar or inexperienced user.

   5. e) If unusual file extensions are used, authors should clearly indicate—to the extent possible—how to proceed with these files.

3. 3. File Names. File names should be easy to understand and provide information about the file content. This is particularly important for replication materials that include multiple scripts and data files. If multiple scripts are to be run in a certain order, then including the order in the file name is useful (e.g., “1_DataProcessing,” “2_Estimation”). Alternatively, authors can create a master file that calls on the different scripts.

4. 4. User-Created Software and Packages. Sometimes researchers create their own statistical packages to produce the results in the paper. These packages ideally should be archived in a stable location, such as CRAN for R. If possible, authors should include a copy of the package or software in the replication package.

5. 5. Operating-System Compatibility. Ideally, authors should ensure that their replication materials function under common operating systems. If this is not possible, they should indicate on which operating system(s) they do function.

6. 6. Script Documentation

   1. a) The scripts should include a short description of what the code does (e.g., “data recoding”), what dependencies it has, which package(s) it requires, and what the outputs are.

   2. b) Authors should avoid scripts that include several pages of commands with little or no indication of what each line (or group of lines) is doing. Code should be annotated where necessary to indicate the purpose of each line or group of lines. This facilitates identification of specific parts of the data-management or estimation process and also helps to identify potential issues or mistakes.

   3. c) Authors should strive to organize their scripts well by avoiding interspersing data management and recoding with estimations, unless absolutely necessary for the analysis.

7. 7. Outputs. Replication scripts should have tables and figures as a clearly identified output (i.e., either a file or an object in the statistical software) exactly as they appear in the paper (to the greatest extent possible). There are many ways that authors can achieve this. For example, for tables with several models estimated in Stata, they can use a combination of eststo and esttab, among many other commands available. In R, authors can use stargazer. For tables that are more ad hoc, they can store the outputs of multiple commands in an object (typically a matrix) in the statistical software that can be printed to a file.

8. 8. Excluded Outputs. Many replication materials include scripts (or parts of scripts) and data for outputs that are not included in the final paper (which typically are excluded at some point in the review process) or are relegated to an online appendix. In this case, authors should clearly differentiate between what is and is not included in the paper. This is particularly important for excluded robustness checks that tend to appear similar to the main results included in the paper, thereby potentially causing confusion.

9. 9. Intermediate Outputs. In many cases, scripts generate intermediate outputs that are then further processed to produce the final outputs included in the paper. In certain circumstances, these intermediate outputs require a significant amount of time to be generated. Among many others, the outputs include the generation of simulated datasets in simulation studies and the Markov Chains when estimating via MCMC. In this case, it is useful to include these intermediate outputs in the replication materials.

10. 10. Random Sequences. For simulation studies or estimation strategies that use randomization, authors should always include the random sequence of numbers used to produce the results in the paper (i.e., the seed) and the pseudorandom-number algorithm. Although the exact seed used should not be critical in terms of results obtained, the availability of the seed simplifies replication of the exact numbers in the paper. The location of the seed should be clearly identified at the beginning of the code in case the user wants to change it.

11. 11. Directories and Folders. Directory paths should be easily identifiable in the scripts so that users can change them accordingly. Ideally, the directory setting should need to be specified only at the beginning of the code; replacing multiple directories in different parts of the code can become cumbersome. If the replication materials are organized in multiple folders, authors should indicate at the beginning of each script where to set the main directory. All directory changes within the scripts should be automatic.

12. 12. Computing Time. Authors should indicate (in both the code and the readme file) whether a particular script or part of a script requires a considerable amount of time to compute. This is important because unaware users experiencing a lengthy estimation might incorrectly assume that there is an issue with the materials.

13. 13. Parallel Computing. Authors are increasingly making use of parallel computing in their scripts because it can accelerate computations significantly. Unfortunately, not all parallelized scripts work on all computers and operating systems. Therefore, authors should take care in highlighting the part of the script that is being parallelized and suggest how to proceed in case of incompatibilities with the user’s computer or operating system.

14. 14. Warnings and Errors. It is not unusual to find scripts that correctly reproduce the output in the paper but for some reason return warnings or errors (usually related to inappropriate use of a package or differences between operating systems). In this case, authors should note the reasons why the warnings and errors are not a concern.

All replication materials are different and creating a set of recommendations that applies to each one is a monumental task. A simple rule for authors to follow when creating their replication materials is to put themselves in the place of another person whose only knowledge about the materials is what the author provides. In the future, the construction of well-organized, clear, and usable replication materials will depend on the training of scholars so that research replication is effectively incorporated into their workflows. Researchers can find further advice, particularly relative to coding practices, in articles by Nagler (Reference Nagler1995) and Bowers and Voors (Reference Bowers and Voors2016).