SHORTCOMINGS IN JOURNAL REPRODUCTION MATERIALS

This section discusses the most common problems that occur during the reproduction review process at Political Analysis—problems that we suspect are typical for political science journals with research-transparency requirements. We use the example of Political Analysis because it has one of the longest standing policies of research transparency among journals in political science. We also focus on Political Analysis because both authors have had hands-on experience with development of the journal’s policies and their implementation.Footnote ¹

Editors of journals with research-transparency requirements have the goal that authors provide all of the materials necessary to reproduce (precisely) the quantitative claims made in their soon-to-be published manuscript. This is consistent with the policies on research transparency and data access of the American Political Science Association and the Society for Political Methodology. To meet this goal, Political Analysis requests that authors provide the following:

1. 1. A README file that describes the materials that the author provided for reproduction and the computing environment used for analysis.

2. 2. Well-documented, well-named, and user-friendly code that reproduces (precisely) and saves the tables and figures in the manuscript.

3. 3. Software packages and other materials that are necessary to reproduce the results reported in the manuscript.

4. 4. The data needed to reproduce the results reported in the manuscript.

5. 5. Good documentation that other researchers can use to understand how to run the code to obtain the results reported in the manuscript.

These requirements are relatively simple and not controversial.Footnote ² However, many if not most authors fail to meet these minimal requirements when they provide initial reproduction materials for their manuscript. By not initially providing the appropriate materials, authors often cause the failure of the journal editorial team’s efforts to use those materials to produce the results reported in the manuscript. This necessitates additional communication with the authors and the subsequent revision(s) of their reproduction materials, with the process reiterating until a point at which editors can release well-documented and well-organized reproduction materials on Dataverse. Authors often face delays in their manuscript’s production process because Political Analysis will not send a paper into production until the reproducibility requirement has been met. It is concerning that authors often fail to produce usable materials when they are required to publish their paper, which implies that the principle of research transparency likely has not been incorporated into the study from its inception.

This also raises a normative question: Should the author or the journal bear the costs associated with the production of research-transparency materials that are portable and that a journal’s research-transparency team can use successfully? We argue that both should bear some costs. On the journal’s side, the research-transparency team (i.e., the replicators) must have a generalist’s understanding of the primary software languages in use in their field (i.e., for political science, primarily Stata, R, and Python); they must have current versions of these software languages and operating systems; and they need access to computational resources that can run most larger-scale processes in a reasonable amount of time. However, we do not believe that replicators need to undertake line-by-line code review or to debug why some provided code only runs using outdated libraries or packages.

We argue that authors should bear most of the costs associated with the production of well-documented, usable code. In fact, we encourage authors to build into their workflow the type of practices that will produce good code and documentation; as more authors engage with this, we believe that these issues will largely disappear. More important, however, is that writing code for many studies is a significant component of the research endeavor. Because the research conducted is the author’s responsibility, we do not believe that it is inappropriate to expect authors to pay the same close attention to their code as they do to collecting accurate data, to provide appropriate citation of previous work, and to follow best practices that generally govern high-quality social science research.

Although it is beyond the scope of this article to describe in detail how authors can build these workflows, we note that funding agencies require detailed data-management plans, which often is an important part of a research workflow. Moreover, it is increasingly common for social scientists to use version-control and code-collaboration tools such as Git, Bitbucket, and GitHub. Finally, introductory graduate methods courses are beginning to include research transparency and other ethical issues in the curriculum, thereby training the next generation of scholars in these best practices. Also, professional societies can provide short courses and other educational materials for those who want to learn how to build research transparency into their workflows.

DOCUMENTATION: THE IMPORTANCE OF THE README FILE

To be usable by other researchers, reproduction materials require documentation, which means that all reproduction materials must provide a simple README file. The README is the first file that a user opens after downloading the data material; therefore, it must contain all of the information that a user needs to run the code. It is obvious that this depends to some degree on the material in question; however, any README file must contain five basic sections: (1) a list of all folders, subfolders, and data files contained in the material; (2) hardware specifications used to run the code; (3) software used, including all packages or libraries (e.g., for R or Python) and their respective versions; (4) a list of all code files to produce the output used in the paper; and (5) the approximate running time of each code file based on hardware specifications. Figure 1 is an anonymized example of an insufficient README submitted to Political Analysis. It omits all five sections and provides no information about what to expect or how to proceed with the data analysis.

Figure 1

Example of Insufficient README File

This is an anonymized example of an insufficient README file submitted to Political Analysis.

Code and Output

Code and data files should be set up in a self-contained project. In R, this should be an R Project while using the here package (Bryan Reference Bryan2018). This sets the working directory to the R Project folder for all script files in the material. Files then can be loaded and saved with relative path files starting at the main replication folder. Local working directories with setwd(), as shown in figure 2, do not represent a practical workflow because they work for only one user on one local machine. Self-contained project working directories, conversely, work on all machines without any manual user input.

Figure 2

Working Directories Should Not Be Set Locally

Local working directories with setwd().

It is imperative to name data sources, R objects, and output objects clearly and consistently. This makes code easily traceable and objects identifiable, and it avoids unnecessary confusion. Figure 3 is an example of two .csv source files and three R objects that are all based on the word “data.” Not only does this confuse users who are unfamiliar with the material, it also overwrites the R base function data(), which is potentially problematic.

Figure 3

Confusingly Named Files and R Objects

This is an example in which two. csv source files and three R objects are all based on the word “data.”

The code must save output files for every figure and table in the main text and the manuscript appendix. Figures should be saved in .pdf format and tables in .csv, .tex, or .html. Each figure and table should be named according to its number in the manuscript (e.g., Figure1.pdf, Table3.tex) to render the output clearly and easily identifiable. Crucially, the saved output must show identical content as the resulting respective figure or table. Figure 4 is an example of saved R output in .csv form that bears no resemblance to the corresponding manuscript table. Whereas the information presented in the manuscript table may be part of the produced .csv file, it is not possible to interpret this information in the data’s current form.

Figure 4

Code Output (Left), Manuscript Table (Right)

The saved output bears no resemblance to the corresponding manuscript table.

Finally, we have seen numerous examples in which the authors of papers with simulations or methods that involve sampling or resampling fail to set random-number seeds. Failure to use the same random-number seed when trying to reproduce manuscript results is problematic because the reproduction will not generate the exact results reported. Thus, authors should always set the random-number seed when conducting simulations or using sampling methods, and this should be well documented in their code.

LOOKING AHEAD: DEVELOPING STANDARDS AND BEST PRACTICES FOR SOCIAL SCIENCE RESEARCH TRANSPARENCY

Improving research transparency is becoming a higher priority for political science scholars, journals, and professional associations, but much work remains. This article identifies two practices that social science should adopt to help resolve the crisis: (1) journals must provide detailed replication guidance and run provided material; and (2) authors must begin their work with replication in mind.

Many journals have been building stronger research-transparency requirements into their standards. Earlier in the evolution of these standards, the goal was simply to ensure that all manuscripts making empirical claims provided some code, data, and documentation, without paying much attention to standardization and the quality of those materials. Today, the top quantitative journals in political science all have strong research-transparency requirements and require that data and code be provided before a paper’s publication. However, some of the journals do not provide or use a permanent and public archive for these materials, and few at this point actually confirm that the data and code reproduce the claims reported in a paper. Therefore, we urge all political science journals to shift their focus from the mere implementation of research-transparency requirements to a rigorous evaluation of the quality, executability, and user-friendliness of the research materials. A promising technological development for reproducibility is the use of Docker containers (see the supplementary materials). Journals such as Political Analysis are moving in this direction, for example, with the use of Code Ocean.

We also encourage journals to provide detailed replication guidance to authors to assist reproduction efforts. The guidance can range from elaborated bullet points to templates that showcase what is required (demonstrated in the supplementary materials). Additionally, we urge journals to establish a research-transparency team in which all members are trained to efficiently run Stata, R, and Python code and to diagnose common problems. With these training structures in place, graduate students are sufficiently equipped to conduct cost-effective journal reproductions. Journals also must ensure that their reproducers have access to computational resources that will reliably run complex simulations, machine- and deep learning, and that can handle larger-scale datasets.

On the author’s side, highly disorganized and virtually unusable reproduction material is still consistently the norm. Virtually none of the reproduction datasets submitted to Political Analysis during the past 18 months ran without producing errors. Scholars must pay closer attention to the documentation and usability of the research materials that they make available to journals and other scholars. They should begin a quantitative study with reproducibility in mind to avoid a frantic rush to collect and document their research late in the publication process. This will ensure that their material meets transparency requirements when it is submitted to journals.