1. Introduction

There is a troubling state of doublespeak in contemporary science. At least, that is what a stylized reading of the peer review bias literature suggests. In public, truth and innovation are valorized by individual scientists (Neuliep and Crandall Reference Neuliep and Crandall1990, Reference Neuliep and Crandall1993; Simmons, Nelson, and Simonsohn Reference Simmons, Nelson and Simonsohn2011; Nosek, Spies, and Motyl Reference Nosek, Spies and Motyl2012), journals (Fontanarosa and DeAngelis Reference Fontanarosa and DeAngelis2004; Lancet 2013), granting agencies (National Science Foundation 2011a; National Institutes of Health 2014b), decrees by Congress (America COMPETES Act, H.R. 2272, 110th Cong. [2007]), and the executive office of the president (Orszag and Holdren Reference Orszag and Holdren2010). Behind closed doors, individual scientists and scientific institutions thwart the achievement of these goals through biases in peer review.

Science’s primary aim—truth—has taken a hit. In the aggregate, reviewers and editors prefer to publish new, statistically significant effects (Dickersin Reference Dickersin1990; Neuliep and Crandall Reference Neuliep and Crandall1990, Reference Neuliep and Crandall1993; Olson et al. Reference Olson, Rennie, Cook, Dickersin, Flanagin, Hogan, Zhu, Reiling and Pace2002). The disproportionate publication (Fanelli Reference Fanelli2010b) of statistically significant results has led to false and exaggerated meta-analysis measures (Simes Reference Simes1986; Palmer Reference Palmer2000). Furthermore, authors, in chase of statistical significance, have increasingly turned to scientifically questionable (Fanelli Reference Fanelli2009; John, Loewenstein, and Prelec Reference John, Loewenstein and Prelec2012) and fraudulent (Fang, Steen, and Casadevall Reference Fang, Steen and Casadevall2012) research practices, proliferating the publication of false positives (Simmons et al. Reference Simmons, Nelson and Simonsohn2011).

Innovation has also taken a hit. The perception that peer review is intellectually conservative is shared by grant applicants (Gillespie, Chubin, and Kurzon Reference Gillespie, Chubin and Kurzon1985; McCullough Reference McCullough1989; National Research Council 2007, 149), grant agency directors (Carter Reference Carter1979; Kolata Reference Kolata2009), and granting institutions (National Science Foundation 2007a, 2). In response, applicants to the National Institutes of Health (NIH) and the National Science Foundation (NSF) downplay the more transformative aspects of their research proposals (McCullough Reference McCullough1989; Travis and Collins Reference Travis and Collins1991) or (some worry) forgo submitting paradigm-challenging ideas altogether (National Science Foundation 2007a, 7).

Science, the stylized reading goes, is in a situation of doublespeak. In public, scientists and scientific institutions celebrate truth and innovation. In private, they perpetuate peer review biases that thwart these goals. How has science gotten into this situation? And what can be done about it?

In this article, I will introduce a new kind of peer review bias—namely, commensuration bias—to explain how this state of affairs has been permitted and facilitated by the undercharacterization of peer review criteria taken as a heterogeneous collective. In peer review, reviewers, editors, and grant program officers must make interpretive decisions about how to weight the relative importance of qualitatively different peer review criteria—such as novelty, significance, and methodological soundness—in their assessments of a submission’s final/overall value. Not all peer review criteria get equal weight (Rockey Reference Rockey2011); further, weightings can vary across reviewers and contexts even when reviewers are given identical instructions (Langfeldt Reference Langfeldt2001, 828; Lamont Reference Lamont2009, 166). I identify this process of commensuration—of converting scores along heterogeneous criteria into a single metric or scale of value (Espeland and Stevens Reference Espeland and Stevens1998)—as the locus for commensuration bias.

Commensuration in peer review can be characterized as being “biased” in a number of ways. Traditionally, peer review “bias” has been understood as reviewer deviation from the impartial application of peer review criteria (Lee et al. Reference Lee, Sugimoto, Zhang and Cronin2013, 4–5). Extending this line of thought, commensuration bias can be conceptualized as reviewer deviation from the impartial weighting of peer review criteria in determinations of a submission’s final value. Sources of partiality in commensuration can include reviewer idiosyncrasy (Lamont Reference Lamont2009) and social bias—such as subconscious racism—against authors/applicants (Lee and Erosheva Reference Lee and Erosheva2014).

In this article, I will explore yet another way of conceptualizing how commensuration may be biased in peer review: in particular, I will explore how reviewer judgments deviate from whatever commensuration practices would, in the aggregate, effectively and efficiently achieve science’s goals. This way of conceptualizing commensuration bias evaluates the instrumental value of peer review criteria and commensuration practices toward broader epistemic ends.Footnote ¹ Commensuration bias, understood in this way, illuminates how intellectual priorities in individual peer review judgments can collectively subvert the attainment of community-wide goals and foregrounds a shared possibility space of structural strategies—applicable to heterogeneous peer review domains—to ameliorate and counterbalance it. This analysis also animates a less Orwellian interpretation of the current state of peer review in science by revealing how commensuration bias can result from an uncoordinated confluence of individual decisions and structural conditions, with multiple loci for enacting change.

2. Commensuration in Peer Review

Reviewers for manuscripts and grant proposals are asked to make evaluations that have normative and structural features in common. In both cases, reviewers are asked to evaluate submissions along multiple normative dimensions. Journal editors at the top 100 science journals ask reviewers to assess the novelty, significance, and methodological soundness of submitted manuscripts (Frank Reference Frank1996). NIH asks reviewers to provide separate scores for a proposal’s significance, innovation, and approach (a methodology component), as well as for the investigator’s qualifications and institutional resources to carry out the proposed research (National Institutes of Health 2014b). NSF asks ad hoc reviewers to assess proposals for “intellectual merit” and “broader impacts,” which collectively include subcriteria for a proposal’s potential for transformative research (an innovation component), rationale (a methodology component), whether the PI/team have sufficient qualifications and institutional resources to carry out the project successfully, and social impact/benefits (National Science Foundation 2012).

However, in the end, reviewers for manuscripts and grant proposals are asked to provide a recommendation about where a submission lies along a single dimension of evaluation. In the case of manuscript review, the ordinal scale of value includes ranked options like “accept,” “accept with minor revisions,” “revise and resubmit,” and “reject” (Hargens and Herting Reference Hargens and Herting1990). At NIH, reviewers are asked to provide an overall impact score on a 9-point rating scale, where “reviewers are instructed to weigh the different criteria as they see fit in deriving their overall scores” (National Institutes of Health 2014c). And, at NSF, ad hoc reviewers and panelists are asked to rate proposals as “excellent,” “very good,” “good,” “fair,” or “poor.” To make these final, global recommendations, reviewers in all of these cases must transform strengths and weaknesses for heterogeneous review criteria into a single scale of value.

This practice of commensuration—of converting distinct qualities into a single metric—is a ubiquitous practice in our social lives (Espeland and Stevens Reference Espeland and Stevens1998). We assign prices to dissimilar objects/services and calculate disability-adjusted life years lost to radically different kinds of afflictions (Murray and Acharya Reference Murray and Acharya1997). By quantifying along a single scale of value, commensuration creates relationships that are easy to represent and compare across fundamentally different things. Its ability to simplify is powerful but “inherently interpretive, deeply political, and too important to be left implicit” (Espeland and Stevens Reference Espeland and Stevens1998, 315).

In the philosophy of science, commensuration plays a tacit role in central questions about theory choice. The practice of quantifying along a single scale of value to compare fundamentally different scientific theories is most overt in Bayesian models of science, which provide a calculus for measuring and comparing evidential support for competing scientific theories along a single scale of credence. Commensuration may, on the face of it, seem less obvious when theory choice is driven by epistemic values such as accuracy, scope, and fruitfulness; however, even these cases implicitly involve ranking scientific theories along a scale of epistemic and/or pragmatic “acceptability.”

As Kuhn (Reference Kuhn1977, 325–26) observed, in the absence of a single, objective weight function determining the joint application of distinct epistemic values, one can raise historically, sociologically, and philosophically rich questions about how epistemic values are weighted across individuals, disciplines, and epochs.Footnote ² Analogously, there are important historical, sociological, and philosophical questions to ask about how peer review criteria are weighted across individuals, disciplines, and institutional contexts in science. How do scientists weight heterogeneous peer review criteria in manuscript and grant review? Are there better and worse ways of weighting these criteria to promote truth and innovation in science? If so, how can we change current review processes, institutions, and communities to promote and sustain such improvements?

3. Commensuration Bias in Peer Review

To begin, consider manuscript review at top science journals, which involves evaluations of novelty, significance, and methodological soundness (Frank Reference Frank1996): novelty promotes the discovery of new truths, methodological soundness assesses the likely truth of study conclusions by evaluating the reliability of data collection and analysis strategies, and determinations of significance tell us which novel truths are most interesting or important (Lamont Reference Lamont2009).

To protect the truth-seeking aims of science, two classes of research should be understood as intellectually significant. Replications are needed to correct and increase confidence in already-published results (Nosek et al. Reference Nosek, Spies and Motyl2012, 617). Further, null results are needed to prevent publication bias—the disproportionate publication of studies reporting statistically significant rather than null results—which, in the aggregate, leads to false and exaggerated meta-analysis measures (Simes Reference Simes1986; Palmer Reference Palmer2000).Footnote ³

However, null results are vanishing wholesale from the publication record. A systematic study of 4,600 papers across the physical, social, and biomedical sciences found that the percentage of published papers reporting statistically significant rather than null results ranges from 70.2% in space science journals all the way up to 91.5% in psychology and psychiatry journals (Fanelli Reference Fanelli2010b). Publication bias rates increased between 1990 and 2007 by 22% (Fanelli Reference Fanelli2012).

Publication bias appears to be caused primarily by authors who fail to submit null results in preemptive anticipation of their likely rejection (Dickersin, Min, and Meinert Reference Dickersin, Min and Meinert1992; Ioannidis Reference Ioannidis1998) and perceived unimportance (Easterbrook et al. Reference Easterbrook, Berlin, Gopalan and Matthews1991). To make matters worse, authors increasingly game publication standards (Lee Reference Lee2013) by adopting methodologically questionable (Fanelli Reference Fanelli2009; John et al. Reference John, Loewenstein and Prelec2012) and fraudulent (Fang et al. Reference Fang, Steen and Casadevall2012) research practices that boost the likelihood of achieving statistically significant but false effects (Simmons et al. Reference Simmons, Nelson and Simonsohn2011).

Unfortunately, replications are also held in low esteem. Among social science editors, 72% view new effects as “more important for the advancement of the field and thus inclusion in their journal” than successful replications, while 58% viewed new effects as more important than failed replications (Neuliep and Crandall Reference Neuliep and Crandall1990, 88). Social science reviewers, while having a slightly higher opinion of replication results, overwhelmingly thought that replications should take up 10% or less of journal space (Neuliep and Crandall Reference Neuliep and Crandall1993, 25). Natural science journal editors publish replications at lower rates than social science editors (Madden, Easley, and Dunn Reference Madden, Easley and Dunn1995, 78).

Overall, the systematic overweighting of intellectual significance qua statistical significance in manuscript review should be understood as a kind of commensuration bias: in the aggregate, it distorts the publication record in ways that interfere with reporting true conclusions and archiving a broad, unbiased evidence base for accurate meta-analysis measures.

Commensuration bias in the evaluation of grant proposals involves a different kind of weighting problem. If we take a primary goal of grant agencies to be “to advance the frontiers of knowledge” (National Science Foundation 2013) and “foster fundamental creative discoveries, innovative research strategies, and their applications” (National Institutes of Health 2013), then novelty must be given sufficient weight to avoid commensuration bias in grant review. However, regression analyses of 32,546 NIH applications for fiscal year 2010 estimated that the independent contribution of innovation toward overall impact scores was relatively low (Rockey Reference Rockey2011): overall impact scores changed, on average, by only 1.4 points for each 1 point change in innovation score (holding scores for other review criteria constant); in contrast, overall impact scores changed, on average, by 6.7 points for each 1 point change in approach score (holding scores for other review criteria constant).

Especially innovative projects face special challenges. A survey of 288 NIH reviewers for proposals to the Director’s Pioneer Award Program found that, for reviewers, the most innovative projects involved methodological risk, including especially “the use of equipment or techniques that have not been proven or [are] considered difficult” (Lal et al. Reference Lal, Wilson, Jonas, Lee, Richards and Peña2012, 82–83). Similarly, in-depth interviews of reviewers at the European Research Council found that “frontier” proposals were conceptualized in ways that made it more difficult for them to appear methodologically rigorous (Luukkonen Reference Luukkonen2012, 54). These results suggest that high innovation is conceptually tied to weaker approach/methodology evaluations.

To counteract conservativism and commensuration bias against innovative projects, program officers could fund highly innovative projects with lower overall scores. However, at NSF, program officers report committing only 1%–3% of their resources to funding these kinds of proposals, despite having the authority to fund more (National Research Council 2007, 152).

To make matters worse, if applicants come to believe that novelty is too lightly weighted by reviewers and program officers, we would expect applicants to downplay the novel aspects of their proposed projects or fail to submit potentially transformative projects altogether (McCullough Reference McCullough1989; Travis and Collins Reference Travis and Collins1991). Along these lines, a report found that the majority of surveyed NSF reviewers (>60%) evaluated submissions of which fewer than 10% were transformative in nature (National Science Foundation 2007b, 14).

Assessing the future impact and likely success of innovative projects is difficult to do prospectively (National Science Foundation 2007a, 1) and from the limited perspective of established scientific theory (Shatz Reference Shatz2004; Haufe Reference Haufe2013). Furthermore, scientists may resist prioritizing highly innovative work over incremental science (National Science Foundation 2011b). However, giving sufficient weight to innovation is essential for achieving science’s goals. More broadly, the notion of commensuration bias helps us think directly about how such goals are or are not effectively and efficiently achieved by current peer review criteria, norms, and practices.

4. Debiasing Commensuration

Because commensuration practices are considered biased as a downstream consequence of evaluations made by individuals over multiple stages and levels of hierarchy, the notion of commensuration bias opens the door to a multilayered range of strategies for ameliorating and counterbalancing it. Even though journals and grant agencies serve different functions in science, the shared normative and structural features of their peer review processes foreground a common possibility space of strategies for debiasing commensuration. In what follows, I will articulate a number of these strategies. I will not be able to evaluate them here (for lack of space); however, I will close by articulating the fruitful questions and future projects that this analysis animates.

5. Conclusion

In this article, I located commensuration practices in peer review as a locus for a new kind of peer review bias, namely, commensuration bias. Commensuration bias illuminates how the practice of prioritizing some peer review criteria over others can, in the aggregate, obstruct scientific aims. To address commensuration bias, I identified progressive strategies and institutions designed to decrease the looseness with which peer review criteria are jointly applied by moving toward more finely tuned criteria, scoring instructions, and/or distributional/institutional policies. Future research will need to explore the normative desirability and practical feasibility of these strategies for realigning commensuration practices with the aims of science, with a special sensitivity to questions about how to reengineer the reward structures and culture of science.