1. Introduction

Within the scientific community, there is a common understanding that its hypercompetetive reward system fuels the replication crisis and drives demoralized researchers out of the academic pipeline. Conversely, there is also a shared sense that, in order to change these cultures and behaviors in ways that would improve science, the scientific community must coordinate across institutions to change how credit is assigned to individual scientists (Alberts et al. Reference Alberts, Kirschner, Tilghman and Varmus2014; National Science Foundation 2015; Nosek et al. Reference Nosek2015; Aalbersberg et al. Reference Aalbersberg2017; Blank et al. Reference Blank, Daniels, Gilliland, Gutmann, Hawgood, Hrabowski, Pollack, Price, Reif and Schlissel2017; NASEM 2018). The hope is that changes to individual researchers’ incentives will improve the integrity and completeness of the published record and secure the health of our future scientific workforce.Footnote ¹

Analogously, philosophers working in the “credit economy” tradition adopt the working assumption that there is some amount of credit that agents can accrue for different acts under different states of nature. This assumption allows them to use decision and game theoretic tools to model how credit seeking among individual scientists can give rise to behavior and norms that support or thwart the achievement of community-wide goals. When, in the aggregate, individual credit seeking cuts against collective ends, their approach can explore how changes to individuals’ incentive structures can nudge and redirect individual behavior (Kitcher Reference Kitcher1990; Strevens Reference Strevens2003; Bright Reference Bright2017; Bruner and O’Connor Reference Bruner, O’Connor, Boyer-Kassem, Mayo-Wilson and Weisberg2017; Heesen Reference Heesen2017; Rubin and O’Connor Reference Rubin and O’Connor2018; Zollman Reference Zollman2018). Different philosophers make different assumptions about the norms by which credit gets allotted—for example, whether credit is all or nothing (Strevens Reference Strevens2003; Bright Reference Bright2017; Heesen Reference Heesen2017) or comes in degrees (Bruner and O’Connor Reference Bruner, O’Connor, Boyer-Kassem, Mayo-Wilson and Weisberg2017; Rubin and O’Connor Reference Rubin and O’Connor2018; Zollman Reference Zollman2018). However, the general approach assumes that there is some precise way to assign credit to different acts under different states of nature—an assumption that allows these philosophers to model credit-seeking behavior and the emergence of scientific norms in formally tractable ways.

But, how much credit gets assigned to any given act under any given state of nature? Just as each of us simultaneously belongs to multiple social categories, each of which is tied to implied social hierarchies (Crenshaw Reference Crenshaw1989; Macrae, Bodenhausen, and Milne Reference Macrae, Bodenhausen and Milne1995), each scholar simultaneously belongs to multiple communities of value with implied social hierarchies for assigning credit. To which of an agent’s communities—which reference class—should credit determinations be indexed and why?

In this article, I use examples from the current context of science’s complex and dynamic culture to motivate and illuminate what I will call the reference class problem for credit valuation in science. Those familiar with the generality problem for reliabilist epistemologies (Feldman Reference Feldman1985) will recognize some structural commonalities. In both, the valuation of a token case depends on the type to which it is assigned; however, because there is no determinate way to identify how narrowly or broadly those types should be characterized and because different characterizations lead to different valuations, we are left with indeterminacy in the valuation of token cases. To close, I identify desiderata, strategies, and challenges for solving ambiguity in credit assignments.

2. The Reference Class Problem for Credit Valuation in Science

The contours of this puzzle about the “coin of recognition” (Merton Reference Merton1968, 56) become visible when one moves beyond thinking about credit in generic abstractions of scientific communities toward the heterogeneous communities we find today. I start from this more concrete perspective because prestige requires recognition by individuals and forums that are themselves valued by credit-seeking scholars (Zuckerman and Merton Reference Zuckerman and Merton1971; Lee Reference Lee2013): credit worthiness in science is a function of the individuals and systems designed to assess, allocate, dispute, and enforce it. Although some aspects of Harriet Zuckerman and Robert Merton’s narrative about the origins of the normative structure of science have been contested by historians (Biagioli Reference Biagioli2002; Csiszar Reference Csiszar, Padovani, Richardson and Tsou2015), we see the social dynamics they proposed clearly at play in contemporary science. For example, Nature Publishing Group recently found that—for the 18,354 authors in science, engineering, and medicine surveyed—the reputation of a journal is the primary factor driving choices about where to submit their work, when reputation is primarily determined by the journal’s impact factor and its standing “as the place to publish the best research” (Nature Publishing Group 2015).Footnote ² Factors associated with a journal’s ability to archive and disseminate research—things like a journal’s time from acceptance to publication, indexing services, or open access options—are much less important.Footnote ³

Within academia, each of us simultaneously belongs to multiple communities of value. The reference class problem arises when these different communities of value disagree about the amount of credit an agent accrues for an act under some state of nature. Although I take this problem to be general, for the sake of clarity and simplicity in presentation, I focus my examples on communities that can be described as having a nesting structure: for example, individual scholars belong to specific subdisciplines, which are nested within disciplines, which are nested within a more general population of scholars. A subpopulation that is nested within a population can have a credit subculture whose valuations differ from those of the population, whose valuations can differ from those of the superpopulation. In these cases, changing how narrowly or broadly one draws the boundaries of an agent’s community of valuation can change the amount of credit assigned to a scholarly accomplishment, just as changing how one gerrymanders the boundaries of a voting district can change its election outcomes. This gives rise to the reference class problem for credit valuation in science: to which of the agent’s communities—which reference class—should credit valuations be indexed when determining the amount of credit the agent accrues for an act under some state of nature?

There are many examples across academia in which nesting community structures can give rise to paradoxes and pathologies in credit assignments. For example, a scholar’s individual sense of what counts as quality may deviate from what is endorsed in the subdiscipline’s or discipline’s status hierarchy (Centola, Willer, and Macy Reference Centola, Willer and Macy2005; Willer, Kuwabara, and Macy Reference Willer, Kuwabara and Macy2009; Correll et al. Reference Correll, Ridgeway, Zuckerman, Jank, Jordan-Bloch and Nakagawa2017).Footnote ⁴ A question, technique, or approach that is thought to have high impact across fields may have less prominence within each of those fields. For example, consider a hypothetical scenario involving an interdisciplinary project whose authors and content represent a set of nonoverlapping disciplines: scholars in each of these disciplines prefer purely disciplinary projects over the interdisciplinary project; however, when these scholars’ preferences are aggregated, their collective preference is for the interdisciplinary project over any single purely disciplinary project (because they prefer interdisciplinary projects over purely disciplinary projects that originate from outside their own fields). Imagine now that this project gets published in a journal—a journal valued by those disciplines—that seeks papers of interest across and beyond disciplines (not just within disciplines), akin to the mission statements for Science and Nature.Footnote ⁵ Which reference class would be most relevant in evaluating the value of the interdisciplinary project (and why)?

There are other ways of dividing scholarly communities into nesting structures that create tensions in credit assignments. The pressures a scholar may feel from the incentive structure affecting her department/school may be slightly different from the incentive structure affecting her university. A coarse but concrete way to see this is to think about the prestige structure reified and reinforced by ranking systems (Sauder and Espeland Reference Sauder and Espeland2006; Espeland and Sauder Reference Espeland, Sauder, Davis, Fisher, Kingsbury and Merry2012, Reference Espeland and Sauder2016), which transform “the ways professional opportunities are distributed” within organizations (Espeland and Sauder Reference Espeland and Sauder2016, 7). Imagine that an untenured business school professor with a potentially high impact manuscript needs to burnish her prestige in the eyes of both her dean and her provost, since both will evaluate her tenure case. If her provost is working to gain stature on the Academic Ranking of World Universities (ARWU), the professor should submit her manuscript to Science or Nature, since the ARWU ranks universities by their publications in these journals (Academic Ranking of World Universities 2018). However, if her dean is trying to gain stature on the Financial Times international ranking of MBA programs (Ormans Reference Ormans2016), she should submit to one of the 50 business, economics, or psychology journals by which the Financial Times ranking system evaluates business school prestige—notably, the journal list does not include Science or Nature. What should the business school professor do?

Finally, credit assignments can vary depending on how long a time window a scholar keeps in view. A coarse but concrete way to think about this is by looking at how metrics for evaluating scholarship change over time. Journal impact factors are becoming less useful measures for evaluating an individual’s scholarly contribution: since the advent of the digital age, the most elite journals (including Science and Nature) are publishing a decreasing percentage of the top-cited papers (Larivière, Lozano, and Gingras Reference Larivière, Lozano and Gingras2013); the relationship between journal impact factor and paper citations has declined over time (Lozano, Larivière, and Gingras Reference Lozano, Larivière and Gingras2012); and, the citation distributions between journals “overlap extensively” (Larivière et al. Reference Larivière, Kiermar, MacCallum, McNutt, Patterson, Pulverer, Swaminathan, Taylor and Curry2016). The current wisdom is that if quantitative indicators are to be used to evaluate research, it is more useful to use article-level metrics such as citations as well as alternative metrics such as downloads and views (DORA 2013; Hicks and Wouters Reference Hicks and Wouters2015; Wilsdon et al. Reference Wilsdon, Bar-Ilan, Frodeman, Lex, Peters and Wouters2017). On the horizon, there are now calls for creating new metrics that can encourage researchers and journals to be transparent and open in their reporting practices (Aalbersberg et al. Reference Aalbersberg2017; Wilsdon et al. Reference Wilsdon, Bar-Ilan, Frodeman, Lex, Peters and Wouters2017; NASEM 2018), where the rise of such metrics—as well as the growing metaresearch literature that ranks journals by the replicability (Schimmack Reference Schimmack2015) or sample size and statistical power of their published results (Fraley and Vazire Reference Fraley and Vazire2014)—makes it possible for a journal’s impact factor and epistemic credibility to come apart (Fang and Casadevall Reference Fang and Casadevall2011). Analogously, these new metrics, if assigned to individual researchers, may not only reward transparent and open research (Moher et al. Reference Moher2019) but also reveal ways in which traditional markers of prestige (e.g., journal impact factor, citations, institutional rank) and epistemic credibility can come apart. Other dynamic considerations can also give rise to the reference class problem: for example, the audience to which junior scholars aim their accomplishments (e.g., related to hiring within a disciplinary department or professional school) may differ from the audience they wish to command as senior scholars.

Decision theorists and game theorists capture the risky nature of individual choices by allowing for uncertainty about which states of the world will come to be, and, when the probabilities attached to different outcomes are understood subjectively, these models permit a kind of subjectivity in estimates of expected credit for different acts. However, the examples throughout this section animate genuine ambiguity in credit due to the reference class problem for credit valuation in science.

3. Desiderata, Strategies, and Challenges for Solving the Reference Class Problem

How might decision theorists and game theorists try to solve the reference class problem for credit valuation in science? In order to solve the underlying conceptual problem, one must provide theories of community and credit that address two fundamental but vexing questions. How should one define and gerrymander the boundaries of the relevant communities invoked in the proposed solution? And, how does one determine the amount of credit those communities would assign to an act under some state of nature in a way that would be normatively and descriptively apt?

These questions may not be independently answerable. The boundaries of a community may need to be defined in terms of patterns of shared lore among its members about how credit is accrued—shared beliefs that coordinate credit seeking, credit allotment, and enforcement behavior in cases in which status beliefs are internalized as norms and in cases in which they are not (Merton Reference Merton and Storer1973; Ridgeway and Correll Reference Ridgeway and Correll2006; Willer et al. Reference Willer, Kuwabara and Macy2009). Conversely, in recognition that some community members can have more influence than others on the content of reigning status beliefs, a community’s credit assignments may need to be defined with some reference to the causal patterns of interaction among specific individuals and clusters of individuals—including status judges who wield “social control through their evaluation of role performance and their allocation of rewards for that performance” (Zuckerman and Merton Reference Zuckerman and Merton1971, 66). However, answers to these questions should not exclusively inform each other: in particular, we must be careful not to allow the size of a scholarly population or the power of its status judges to fully determine the intellectual value of the questions pursued by any particular partition of the scholarly universe.

Strategies that try to address the reference class problem by tackling one question without reference to the other simply underscore their mutual dependence. For example, let’s imagine an approach that begins by arguing for the “correctness” of one community as opposed to others.Footnote ⁶ Justifying and defending the centrality of the chosen community is difficult to do without reference to substantive causal claims about how credit gets allotted. For example, to justify using disciplines as the primary credit-assigning community, it would make good sense to observe that scholarly prizes are distributed for excellence in particular disciplines (e.g., Nobel, Fields, and academic society prizes) and that judgments about disciplinary excellence drive evaluations of quality and merit even in interdisciplinary contexts (Lamont Reference Lamont2009). However, it is not difficult to see how others could challenge the idea that disciplines should be the sole arbiter of credit: after all, Nobel Prize–winning work can originate and have higher impact in disciplines outside its awarded field (Szell, Ma, and Sinatra Reference Szell, Ma and Sinatra2018), and evaluations of disciplinary excellence can themselves be driven by evaluations of subdisciplinary excellence (Lee Reference Lee2012). However one feels about the claim that we should use disciplines as the ultimate arbiters of prestige, it is clear that its justification and defense should advert to normative and descriptive claims about how credit assignments work.

Likewise, we can see how the justification and defense for particular credit assignments should depend on normative and descriptive claims about community structure. For example, let’s imagine a strategy for credit assignment that calculates the credit value of a scholarly contribution by summing the credit valuation of multiple communities. This approach would need to identify exactly how much to weight each community’s valuation, with a rationale for why, since different weightings could lead to different overall credit valuations.Footnote ⁷ Some scholars take this style of approach when trying to measure the relative prestige of journals: in particular, the Eigenfactor score rates journals according to the number of incoming citations, where the “relative importance” of each incoming citation is contextualized by the frequency with which the citing journal is itself cited (West, Bergstrom, and Bergstrom Reference West, Bergstrom and Bergstrom2010). Justifying and defending this strategy requires invoking judgments about how best to demarcate the specific communities whose credit valuations are aggregated—in the case of Eigenfactor, populations are demarcated by individual journals. At a more basic level, note that even credit assignments derived from a single community require adverting to claims about how to demarcate that particular partition of the scholarly universe.

Finally, any theory of community and credit must countenance apparent heterogeneity in community and credit types. A number of recent policy papers call for moving toward broader conceptions of research excellence that recognize the diversity of research missions among individual scholars, programs, and academic institutions (Hicks and Wouters Reference Hicks and Wouters2015; Wilsdon et al. Reference Wilsdon2015). Some call for valuing researchers whose work promotes rigorous science by using open practices (i.e., sharing code, materials, and data) or by replicating and synthesizing existing research (Moher et al. Reference Moher2019). Others call for recognizing the intellectual value of community-engaged scholarship that creates, disseminates, and implements knowledge in coordination with the public to identify social interventions, change social practice, and influence policy (Boyer Reference Boyer1990; DORA 2013; Escrigas et al. Reference Escrigas, Granados Sánchez, Hall, Tandon, Escrigas, Granados Sánchez, Hall and Tandon2014; Hicks and Wouters Reference Hicks and Wouters2015). Both of these movements cut across disciplinary and subdisciplinary lines, suggesting heterogeneity in credit assignments within fields and subfields.

Note that any approach to addressing heterogeneity must decide how much it can be tolerated within gerrymandered communities before those communities must be fractured or otherwise redrawn. Too much tolerance and the model becomes descriptively and normatively ungrounded. Too little tolerance and the model’s scope is radically decreased: for example, a model capturing an individual agent’s personal community and credit functions—or the community and credit functions of an “intersectionally” categorized group of agentsFootnote ⁸—would generalize over a much smaller population than typical models that assume credit is distributed in the same way, using the same credit function, across a more broadly drawn community of agents.

Note too that any approach to addressing apparent heterogeneity in community and credit types must be careful about ways that reference class problems can recur. For example, if we address heterogeneity in credit types by adopting a multiattribute utility model involving qualitatively different types of credit, then we would not know how much credit to assign—this time, to multiple credit types—in cases in which different communities assign different amounts for each. In this case, increased technical sophistication does not address the conceptual problem.

4. Conclusion

Scientific credit—the “coin of recognition” (Merton Reference Merton1968, 56)—is assessed, allocated, disputed, and enforced by many different communities and institutions within science that support and sustain a multiplicity of status hierarchies. This gives rise to what I have called the reference class problem for credit valuation in science.

Those who wish to model the implications of different approaches for solving the reference class problem may try to do so by setting up hypothetical communities that assign community boundaries and credit assignments in de facto ways to see what kinds of behaviors and norms emerge. This work could reveal interesting insights into how different ways of gerrymandering intellectual populations—by shifting subdisciplinary and disciplinary lines, journal scope, and grant agency program areas/panels—could change the kinds of projects and areas that “win.”

However, solving the underlying conceptual problem requires developing rich, mutually informed theories of community and credit that are based on fine-grained information about the structure and status systems of complex, heterogeneous scholarly networks. Richer theories of this sort would help us better understand the scope, descriptive power, and normativity of decision and game theoretic models of credit seeking in science.