EXAMPLES OF REPORTING FLEXIBILITY IN THE RACE AND SEX DISCRIMINATION LITERATURE

Reporting flexibility includes which outcome variables to report. For example, the questionnaire for the survey experiment reported on in Rattan et al. (Reference Rattan, Levine, Dweck and Eberhardt2012) about the effect of a treatment manipulating the race of a juvenile offender contained four items that could be used as outcome variables, but Rattan et al. (Reference Rattan, Levine, Dweck and Eberhardt2012) mentioned only two of these items, both of which could be used to detect a statistically significant difference between experimental conditions; standardized estimates of antiblack discrimination for these items were 0.17 and 0.19 ^{Footnote 2} ; however, using the same research design and the Rattan et al. (Reference Rattan, Levine, Dweck and Eberhardt2010) data, neither of the unreported items detected a statistically significant difference between experimental conditions, with standardized estimates of –0.01 and 0.09. Such selective reporting thus created an overestimate of antiblack discrimination detected in the experiment.

Reporting flexibility includes coding of outcome variables. The survey experiment reported on in Banks (Reference Banks2014) measured attitudes about President Barack Obama and the Democrats’ health care reform proposal with multiple items: one item measured approval for the way that Obama is handling health care, on a scale from 1 (approve strongly) to 5 (disapprove strongly); another set of items measured support for the health care bill with a dichotomous favor-or-oppose item and with a follow-up item about the strength of favoring or opposing. Banks (Reference Banks2014) reported results for only the dichotomous favor-or-oppose item, which permitted an inference that “anger uniquely increases the impact of racial attitudes on health care opinions” (p. 508). However, based on the Banks (Reference Banks2010) data, the inference about the unique influence of anger is not supported when predicting the approval item or when predicting a favor-or-oppose item coded to include the follow-up measure of strength of favoring or opposing (see Zigerell Reference Zigerell2016).

Reporting flexibility for cross-time data includes selection of endpoints for the analysis. Maliniak, Powers, and Walter (Reference Maliniak, Powers and Walter2013) reported that female-authored international relations articles have received fewer citations than equivalent male-authored international relations articles, based on an analysis restricted to selected articles published between 1980 and 2006; for this selection, the two-tailed p value from the model with the full set of control variables was 0.093 for the citation difference between female-authored articles and male-authored articles. However, data on articles from 2007 were available in the dataset, and reproduction code for the article indicated that models for the main table contained a control for articles published in 2007. When articles from 2007 were included in the analysis, the two-tailed p value inflated to 0.223 for the citation difference between female-authored articles and male-authored articles (Zigerell Reference Zigerell2015b). ^{Footnote 3}

Previously discussed examples are consistent with a preference for statistically significant results, but reporting flexibility also permits statistically significant results to be unreported. Discussing an Associated Press report of an implicit association test estimating that 56% of Americans had antiblack sentiments (Agiesta and Ross Reference Agiesta and Ross2012), Moore (Reference Moore2012) disclosed that the report failed to mention that the same implicit association test also produced an estimate of 33% of Americans with antiwhite sentiments. This criticism about a failure to report estimated discrimination against whites when reporting estimated discrimination against blacks can be extended to the manuscript on which the Associated Press report was based (Pasek, Krosnick, and Tompson Reference Pasek, Krosnick and Tompson2012) and to other political science publications (see Zigerell Reference Zigerell2015a).

The Benard (Reference Benard2005) survey experiment is an example of an entire study whose results have not yet been reported on in the literature, even though the study permitted detection of race and sex discrimination. The survey experiment tested for race and sex discrimination in ratings of target workers whose names signaled race and sex: Brad, Kareem, Kristen, and Tamika. Straightforward analysis of the data did not permit detection of racial discrimination in overall ratings among white respondents: whites rated black targets 0.09 standard deviations higher than white targets (two-tailed p value of 0.204). However, straightforward analysis of the data did permit detection of racial discrimination among black respondents: blacks rated black targets 0.61 standard deviations higher than white targets (two-tailed p value of 0.018). Moreover, straightforward analysis of the data permitted an inference of sex discrimination that favored women: across all respondents, female targets were rated 0.22 standard deviations higher than male targets (two-tailed p value less than 0.001). The absence of this study from the literature produces an underestimation of detected discrimination against whites and men.

Certain fields have relatively unique types of reporting flexibility, such as the measurement of racial discrimination. Researchers testing for racial discrimination using the 2012 American National Election Studies (ANES) Time Series study can measure antiblack discrimination with symbolic racism items, feeling thermometer ratings for blacks and/or for whites, stereotype scales for blacks and/or whites being hardworking and/or intelligent, an item about whether blacks have too much influence in American politics, and an item about how much discrimination blacks face in the United States today. Researchers failing to find racial discrimination in the ANES dataset can then check racial discrimination items in the General Social Survey or other surveys.

Reporting flexibility in the use of multiple measures of racial discrimination can be illustrated with the survey experiment reported on in Krupnikov and Piston (Reference Krupnikov and Piston2015), which included items measuring blacks’ and whites’ perceived intelligence, laziness, and amount of influence in American politics. Results predicting a difference in candidate thermometer ratings were reported in that article with the “black influence” measure used as the measure of racial attitudes in model 1 of table 2 of that article, which permitted detection of racial discrimination at a conventional level of statistical significance in a triple interaction of racial attitudes, black ad sponsor, and negative ad. But reanalysis of the Krupnikov (Reference Krupnikov2012) data indicated that use of the stereotype items as the measure of racial attitudes did not permit detection of racial discrimination in that triple interaction term at a conventional level of statistical significance.

Another area of relatively unique reporting flexibility in the study of racial discrimination is sample restriction. Researchers testing for antiblack discrimination in a survey can analyze data for all respondents or can restrict the sample to groups such as nonblack respondents, white respondents, or non-Hispanic white respondents; it is also possible to disaggregate samples by sex (e.g., Hutchings, Walton, and Benjamin Reference Hutchings, Walton and Benjamin2010) and to mix sample restrictions within a study. Baker (Reference Baker2015) used a sample of non-Hispanic whites for one survey experiment and a sample of nonblacks for another survey experiment reported on in the same article. Using the Baker (Reference Baker2012) data and code posted for the analysis with the nonblack sample, restricting the sample to white respondents raised the two-tailed p value from 0.080 to 0.285 for the test of whether respondents supported more aid to persons in Guyana or Armenia. In this case, the choice to use a nonblack sample or a white sample determined whether an inference of discrimination in favor of Africans was supported. ^{Footnote 4}

THE POTENTIAL FOR POLITICAL BIAS IN POLITICAL SCIENCE ESTIMATES, AND HOW TO REDUCE IT

The aforementioned studies permitted flexibility to report a range of estimates for race or sex discrimination or to not report an estimate at all. Reporting choices could have been influenced by many factors, such as sincere researcher theoretical preference for one research design over another or requests from journal peer reviewers or journal editors to report certain results or to analyze the data in certain ways. The reasons for these reporting choices are less important than the presence of these choices and the fact that political science currently has no mechanism to guarantee that reported results represent the full range of estimates from reasonable research designs. This reporting flexibility might not be much of a problem if selective reporting of estimates canceled out in the aggregate, but it is unreasonable to assume such balancing, given the political imbalance in political science, with surveys indicating that Democrats outnumber Republicans by a ratio of 5.6:1 (Klein and Stern Reference Klein and Stern2005) or 7.25:1 (Rothman, Lichter, and Nevitte Reference Rothman, Robert Lichter and Nevitte2005) and liberals outnumbering conservatives 81:2 (Rothman et al. Reference Rothman, Robert Lichter and Nevitte2005).

Given the combination of reporting flexibility and the substantial political imbalance within political science, it would not be unreasonable to suspect that political science estimates have been and are biased toward the preferred inferences of the political Left. Duarte et al. (Reference Duarte, Crawford, Stern, Haidt, Jussim and Tetlock2015) noted that greater ideological diversity in social psychology can produce benefits for the field, such as combatting confirmation bias. Greater ideological diversity in the political science peer reviewer pool might help reduce reporting flexibility if peer reviewers skeptical of a claim require reporting of additional models or require declarations about the presence of unreported analyses.

But more straightforward—and easier to accomplish—mechanisms for eliminating reporting flexibility are preregistration (see Monogan Reference Monogan2015) and preacceptance (see Nyhan Reference Nyhan2015). In a preregistered study, researchers announce their planned research design choices before completing data collection, and then report results from this preannounced research design; readers can thus have more confidence that reported results have not been influenced by reporting bias. Preregistration has an advantage that researchers can announce planned research design choices unilaterally, but a shortcoming of preregistration is that there is no guarantee that preregistered studies will enter the literature. This shortcoming can be eliminated through preacceptance of articles in which journals agree to publish an article based on a preregistered research design no matter what the results are, as was done in a special issue of Comparative Political Studies (Findley et al. Reference Findley, Jensen, Malesky and Pepinsky2016). ^{Footnote 5}

Monogan (Reference Monogan2013) is an exemplar of a preregistered study. The preregistered Monogan (Reference Monogan2010) file describes the planned research design to assess whether a Republican House candidate’s percentage of the major-party vote in 2010 was influenced by the tone of the candidate’s website statement on immigration. The preregistered file identified the planned outcome variable, explained how the explanatory variable would be measured, identified which control variables would be included in the model and how these control variables would be measured, and explained how the data would be analyzed. Analysis of the vote data collected after the research design had been made public did not provide support for the expectation that Republican candidates’ stance on immigration influenced constituent vote choice. ^{Footnote 6}