DATA ACCESS AND TRANSPARENCY: THE STATE OF AFFAIRS

Calls for data transparency are not new, but few scholars voiced these calls 20 or 30 years ago. A prominent example of a proponent of data transparency during the past 20 years is Harvard professor Gary King. In several articles (King Reference King1995; Reference King2003; Reference King2011), he admonished researchers to think about guidelines and rules to increase data access and transparency in the social sciences. However, until the 2010s, these calls remained unanswered. With few exceptions, political science associations, academic journals, and researchers did not discuss the challenges of sharing and replicating published work in a sustained and serious manner. Yet, spearheaded by early initiatives and attempts to establish guidelines on data transparency—which are associated with Political Analysis, American Journal of Political Science, and State Politics & Policy Quarterly—the discipline during the past five years suddenly has become interested in questions of how to scientifically prepare, present, and share public work and data. Several developments have fostered this interest.

First, the logistical and monetary costs to share data decreased tremendously in the 2010s. At no cost, researchers can make their data publicly available on their personal website, in an online appendix of the journal in which they publish their article, or in an online depository (e.g., Harvard Dataverse). Second, replication studies in neighboring disciplines highlight serious issues with the state of scientific conduct. For example, in the field of education studies, Freese (Reference Freese2007) and Evanschitzky et al. (Reference Evanschitzky, Baumgarth, Hubbard and Scott Armstrong2007) illustrated two worrisome developments: (1) there is a disturbing trend featuring a lack of replication studies; and (2) many studies do not adhere to the standards of rigorous scientific works, rendering replication even more important. To underline these developments, Evanschitzky (Reference Evanschitzky, Baumgarth, Hubbard and Scott Armstrong2007) claimed that “[t]eachers are advised to ignore the findings until they have been replicated, and researchers should put little stock in the outcomes of one-shot studies.” (For a similar and more recent study, see Open Science Collaboration 2015.) Third, and partly as a result of these first two developments, researchers are commonly aware that the lack of access to data is a major impediment to progress in science. Combined with this awareness is a growing willingness among researchers to share their data. To illustrate, Tenopir et al. (Reference Tenopir, Allard, Douglass, Umur Aydinoglu, Wu, Read, Manoff and Frame2011) highlighted that nearly 75% of 1,361 scientists polled were willing to share their data. However, few did so due to “insufficient time” (54%), “lack of funding” (40%), “no rights to make data public” (24%), “no place to put data” (24%), and “a lack of standards” (20%).

Together, these developments made researchers and journal editors in the political science discipline (as well as others) realize that journals and political science associations must take the lead in the quest for more rigorous data-transparency norms (Ishiyama Reference Ishiyama2014). Normatively, this pressure has increased with the publication of several manifestos expressing the need for clear guidelines in favor of an open research culture and transparency in social science research (Miguel et al. Reference Miguel, Camerer, Casey, Cohen, Esterling, Gerber and Glennerster2014). APSA responded to these claims more vigorously than other associations. The APSA Council developed the so-called Data Access and Research Transparency Statement (DA-RT), which is based on the premise that researchers have an ethical obligation to “facilitate the evaluation of their evidence-based knowledge claims through data access, production transparency, and analytic transparency so that their work can be tested or replicated” (Lupia and Elman Reference Lupia and Elman2014, 20). Two further initiatives emanated from DA-RT: (1) the Journal Editors’ Transparency Statement (JETS), and (2) the Transparency and Openness Promotion Statement (TOPS). First, in the JETS, 27 leading political science journals committed to greater data access and research transparency and to implementing policies requiring authors to make as accessible as possible the empirical foundation and logic of inquiry of evidence-based research.^{Footnote 2} Second, the TOPS guideline listed eight transparency standards that journal editors might want to adopt: citation standards, data transparency, analytic-methods (i.e., code) transparency, research-materials transparency, design and analysis transparency, study preregistration, analysis-plan registration, and replication.^{Footnote 3}

These developments are changing what we generally label as “good research practice.” To illustrate, Gherghina and Katsanidou (Reference Gherghina and Katsanidou2013) identified 120 political science journals that publish quantitative research and reported that only 18 journals had any type of data-sharing policy listed on their website. Only three years later, we found that more than 80% of political science journals publishing quantitative work had a transparency statement on their website. More importantly, the 27 JETS signatories practice the JETS guidelines reasonably well. A recent study by Key (Reference Key2016) revealed that International Organizations, American Journal of Political Research, and Political Analysis—all of which are JETS signatories—have replication materials available for more than 80% of their articles. However, the same article reported that the replication material is available in only about 35% of the studies published in the British Journal of Political Science, the Journal of Politics, and the American Political Science Review—none of which have a binding data-sharing policy.

This recent evidence points toward the fact that the adoption of a stringent data-sharing policy “forces” authors to make their data available. However, what happens to authors who publish their research in a journal that does not require researchers to make their data publicly available? Do they post it on the journal’s online depository anyway? If not, do they share it with colleagues, if asked? If they do share it, is it sufficiently well prepared to allow for replication? Finally, do the replications provide results identical to those published in the respective journal article? This article attempts to answer these questions. To do so, we selected the 2015 edition of three prominent peer-reviewed journals in the field of political behavior: Electoral Studies,^{Footnote 4} Party Politics,^{Footnote 5} and Journal of Elections, Public Opinion & Parties (JEPOP).^{Footnote 6} We deemed these three referents a good fit for our study because they mostly publish quantitative pieces, and none of the journals had a binding data-sharing policy as of 2015.^{Footnote 7} Although based in Europe, all three journals are international and mostly publish articles by authors from institutions in Europe as well as the United States and Canada. They provide a good cross section of the political behavior discipline as a whole. This also is reflected by the composition of the journals’ editorial boards, which include prominent experts in the field.

RESEARCH DESIGN

To determine whether researchers make data from their published articles publicly available—and, if so, whether it is possible to replicate these results—we engaged in a multi-step research process. First, we selected all articles that used quantitative methods from all those published in the three journals of reference. This resulted in 145 articles (i.e., 73 published in Electoral Studies, 52 in Party Politics, and 20 in JEPOP). Second, we checked whether the journals publish replication data on their website or whether the articles contain a link that allows other researchers and the interested public to access these data. We also consulted the authors’ personal websites to ascertain whether the data and/or code are published there. Third, in all cases in which we did not find the data and/or the code readily available, we asked the authors via email to share their data, codebook, and code. The email stated clearly that we wanted to do a replication study of their published article, and we assured them that we would not distribute the data to any third party. When authors did not answer, we sent three reminders in two-week intervals. Fourth, for those articles on which we obtained the data, we attempted to replicate the results.^{Footnote 8}

Regarding data availability, our results suggest that slightly more than half of the authors were willing to share their data (i.e., data, code, or both) (table 1

Table 1

Frequency/Percentage of Articles for Which We Could Access Data

). However, few shared it when not specifically asked to do so.