Zigerell (this issue) is concerned that “reporting flexibility” allows researchers to make post hoc changes to research designs in order to find “support for the policy preferences of a particular political party or ideology.” ^{Footnote 1} To justify his concerns, Zigerell cites the findings of his recent reanalysis (Zigerell Reference Zigerell2015) of the data used in our 2013 article on the role of gender in citation patterns in the international relations literature (Maliniak et al. Reference Maliniak, Powers and Walter2013) and then asserts that our study showed “a preference for statistically-significant results.” However, Zigerell focuses on how his reanalysis affects a single model from our original article and ignores the fact that we reported statistically insignificant results when they arose in our original article. We report below how Zigerell’s (Reference Zigerell2015) changes to our sample have little effect on the substantive implications of most of the models that we reported in our article.

Our study demonstrated that articles written by women, and untenured women in particular, are cited fewer times and are less central in the IR literature citation network compared to similar articles written by men alone or coauthored with men. Our sample included roughly half of the IR articles published in 12 top peer-reviewed journals over the period 1980–2006. Zigerell (Reference Zigerell2015) notes that our dataset, but not our published analysis, included data on articles published in 2007 and reports on how the inclusion of those observations affects a single model from our article. We thank Zigerell for so closely reviewing our work and highlighting our omission of the 2007 observations. This omission resulted from a miscommunication between coauthors and we take full responsibility for it. After adding the observations from 2007 to our “Kitchen Sink” citation count model, the p-value on the variable of interest, All female, inflated from .093 to .223. Despite this, the substantive implications of nearly every other specification of the citation count models in our original article are similar after including observations from 2007 (table 1). The coefficient on All Female remains negative across all models, is of similar magnitude to those in our original analyses, and is statistically significant across all but the most saturated models (the last two). ^{Footnote 2} Given his present concern with reporting flexibility, we find it odd that Zigerell does not make this clear to readers in either the present essay or in his 2015 sensitivity analysis.

Table 1

Estimates from Maliniak et al. (Reference Maliniak, Powers and Walter2013) with 2007 observations. Model titles represent the addition of different sets of article or author attributes

Standard errors in parentheses. *p<0.10, **p<0.05, ***p<0.01.

Additionally, we were not shy about reporting statistically insignificant results when they arose in our original analysis. We reported a statistically insignificant effect of author gender on citation counts in our decade sub-sample analysis of the 1990s and 2000s. ^{Footnote 3} Similarly, we reported that the gender effect appears to be smaller for more recently published articles and is not statistically detectable for articles less than a decade old at the time of our data collection. Finally, the p-value of .093 on the All Female coefficient in the “Kitchen Sink” model that we reported in the original article is not, by some standards, statistically significant. We think it is clear from these examples that we were not selectively reporting only statistically significant results.

We are encouraged to see that Zigerell’s (Reference Zigerell2015) effort is joined by others who continue to study gender bias in citations. For example, Fogerty (Reference Fogarty2015) argues that the distribution of our citation count variable is more consistent with distributional assumptions of a Poisson inverse Gaussian (PIG) regression model than those of a negative binomial (NB) regression model. Fogerty re-estimates the “Kitchen Sink” citation count model and includes the 2007 observations. The coefficient is negative and statistically significant, but estimated effect of gender is substantially smaller in magnitude. While we view Fogerty’s reanalysis of the models that we reported in our original paper as broadly consistent with our findings, his method does produce less dramatic and more mixed results than ours. Separately, Roberts, Stewart, and Nielsen (Reference Roberts, Stewart and Nielsen2016) reanalyze our data (including articles from 2007) using their newly-developed matching technique for high-dimensional data, like the text of journal articles, to balance the sample and find results very consistent with those in our original study: articles written by women are cited less than articles written by men or coauthored by men and women. In our view, the best response to such a pattern of results is to develop more robust theories of how gender bias might manifest in academia and then test those theories using of variety of theoretically-motivated research designs or newly collected data.

We conclude with a general comment. Given Zigerell’s particular concerns about ideological bias, we expected him to propose mechanisms uniquely suited to addressing such biases. He does not do this. Instead, he calls for preregistration of research designs and pre-acceptance of journal articles. We are not opposed to either of these efforts, but neither of them addresses the underlying issue of ideological diversity. Indeed, if a lack of ideological diversity biases the questions we ask and conclusions we reach in political science, it is likely that a lack of diversity on other dimensions that affect how individuals interpret the world around them are problematic as well. One tractable solution is to strengthen departmental diversity policies to help ensure recruitment of faculty and students that are more diverse in terms of age, physical ability, ethnicity, gender, race, sexual orientation, socioeconomic status, and potentially, political ideology. ^{Footnote 4} As such, we view Zigerell’s concerns as a subset of broader concerns about the lack of diversity in our discipline. In that light, we join the author in calling for greater diversity in political science.