Incentives for Quality and Visibility

Academics are under never-ending pressure to “publish or perish.” In the United States, the tenure system puts a constant strain on faculty, particularly junior faculty. In the United Kingdom, this pressure was institutionalized through the Research Excellence Framework system, which makes public funding to universities contingent on publications and “impact.”^{Footnote 4} The European Research Council requires all grant applicants to provide information about their “track record,” including the number of publications and citations. In this context, easy access to GS may provide incentives for scholars to emphasize quality over quantity. They may want to produce better publications that are cited more instead of more publications with fewer citations. Focusing on citation counts also may inspire authors to publicize their work more, such as through social media.

Academic Coordination and Open Access to Research

GS may facilitate the dissemination of ideas and intellectual networking. A GS search on a keyword can expose a reader to a new scholar, and the reader can easily see the scholar’s other work through their GS profile. The citation count shows which of the author’s articles and books are most popular. In addition, GS’s automatic email notifications can encourage intellectual networking. Interested readers can sign up to receive automatic emails when scholars post new work, and scholars can be notified when somebody cites them—often at the working-paper stage.

Our own experiences suggest that GS nudges scholars toward open-access practices (Hendrix Reference Hendrix2016). When people see the convenient links to open-source versions of others’ work, they may make their own work more publicly available via their personal websites or on academic networking sites (e.g., SSRN, Researchgate, and Academia.edu).

Consistency in Research Evaluation

The GS profile, with its full list of academic production, citation counts, and H-index, provides straightforward measures of scholarly quality and impact. These measures are highly reliable (i.e., anyone looking them up gets the same value) and consistent (i.e., the same coding scheme applies to all scholars). Given the prevalence of biases against women and people of color in the academy (Ginther et al. Reference Ginther, Schaffer, Schnell, Masimore, Liu, Haak and Kington2011; Milkman, Akinola, and Chugh Reference Milkman, Akinola and Chugh2012; Moss-Racusin et al. Reference Moss-Racusin, Dovidio, Brescoll, Graham and Handelsman2012), these measures offer grounds to challenge unfounded perceptions and prejudices.

Finally, the easy availability of citation counts can help scholars obtain credit for books and articles that are not published in the most famous outlets but nevertheless are of intellectual importance. Citation counts can inform evaluators how the work has affected the field, independent of the publisher’s prestige.

Originality and Innovation

GS counts are biased toward incremental work and away from boldness and innovation. Highly original work that does not fit neatly into an existing literature might establish a new research agenda and expand interest in the topic, but its impact will not be visible in citation counts for many years. According to GS, John Nash’s foundational paper defining Nash equilibrium received only 16 citations in the first five years after publication.

Another example is Stathis Kalyvas’s Reference Kalyvas1999 article, “Wanton and Senseless? The Logic of Massacres in Algeria,” which was cited only 20 times (excluding author citations and citations from drafts of the same article) in the four years after its publication—despite winning the Luebbert Award in 2000. However, its early citations came from a small group of colleagues who established a new agenda in the field of civil war and ethnic conflict, including Elisabeth Wood, John Mueller, David Laitin, and James Fearon. Kalyvas’s (Reference Kalyvas2006) book on civil wars currently has more than 3,000 citations, an impact impossible to predict from the 20 citations of his 1999 article four years after its publication.

In general, the number of citations that an article or book receives in the five or so years after publication reveals little about its long-term impact. Wang, Song, and Barabasi (Reference Wang, Song and Barabasi2013) reviewed a sample of physics papers and found that having 50 citations in the first five years after publication was not associated with more citations after 20 years. In fact, papers with the most citations in 30 years tended to have relatively few citations early on. Stephan, Veugelers, and Wang (Reference Stephan, Veugelers and Wang2017) examined 660,000 research articles in the Web of Science database and found that highly original papers were less likely to be highly cited within three years of publication but more likely to be highly cited three or more years after publication.

Early-Career Scholars

Because hiring and promotion decisions occur fairly early in a scholar’s career, citation counts may be more consequential precisely when they are least informative. Papers and books that are cited many years after publication are arguably more important than papers and books cited only shortly after publication. The time-lag problem thus poses particular challenges in evaluating younger scholars.

Using GS to evaluate early-career scholars creates perverse incentives. From a numerical standpoint, it is better to publish incremental work on topics in which there is a large, active subgroup of scholars who cite one another than it is to open up a new field of research. Short-term, citation-centered evaluations discourage boldness and innovation, especially among early-career political scientists. Many important scholarly works initially defy easy contextualization and fit poorly into existing literatures, as the example of John Nash’s paper shows. Tenure and promotion decisions that are based only or largely on citations garnered within five years of publication reward competence over lasting significance.

Research-Community Size

Scholars in larger research communities have an advantage over scholars in smaller fields of study: a larger pool of scholars who can cite them.^{Footnote 6} A paper cited in 16 of 100 articles published in a given year on the US Congress probably has less “impact” than a paper cited in eight of the 10 articles published that year on Pakistan, but it has double the count. Scholars producing incremental improvements in well-tilled fields thus tend to have larger citation counts than scholars producing novel insights in small but important or growing fields.

Gender

Maliniak, Powers, and Walter (Reference Maliniak, Powers and Walter2013) analyzed more than two decades of IR publications and found that—controlling for various factors such as publication venue, methodology, and tenure status—an article written by a woman receives 80% as many citations as a similar article written by a man. Women are less likely to be cited by the most influential articles and less likely to self-cite, possibly because they are penalized for self-promotion (Moss-Racusin and Rudman Reference Moss-Racusin and Rudman2010).

Dion, Sumner, and Mitchell’s (forthcoming) study shows that women are more likely than men to cite works by women in multiple social science journals. In general, men and mixed-author teams tend to under-cite women’s work, although the size of the gender-citation gap varies according to the number of women in the field. As numbers of women scholars increase, work by women tends to be cited more by both men and women authors. They conclude that “citation practices are influenced by gender diversity” (Dion, Sumner, and Mitchell, forthcoming).

Biases in favor of men arise even if scholars genuinely cite those who influence them most. Colgan (Reference Colgan2017) found that male IR instructors are less likely than female instructors to assign work by women scholars, whereas women also are less likely than men to assign their own work. Although there have been few systematic studies in political science on whether other underrepresented scholars experience similar biases, it is highly plausible that such biases exist.

Coauthorship

Scholars who tend to coauthor can generate citations more easily than those who tend to solo author. Citation counts are not divided by the number of coauthors; that is, if a paper with five authors is cited once, each author receives one citation, not 0.2 citations. Higher citation counts for coauthoring scholars can exacerbate other biases. For example, Teele and Thelen (Reference Teele and Thelen2017) demonstrated that all-male teams authored most of the collaborative work in 10 of the most prominent political science journals.

Strategic Citation

Scholars face an array of professional imperatives: they want their work to appeal to reviewers, be published, and garner citations. Thus, they may engage in several forms of strategic citation (Aizenman and Kletzer Reference Aizenman and Kletzer2011).

Some books and articles are cited almost entirely for their flaws, not their importance. Authors often cite poorly executed studies, easy targets, and “straw-man” pieces to explain what they are arguing against (Nexon and Jackson Reference Nexon and Jackson2015). Journalists and policy analysts, in particular, may cite scholarship that generates splashy headlines and overlook less provocative work of higher quality (Colgan Reference Colgan2016).

Citation choices also may be guided by expectations of likely peer reviewers. Strategic citations include fellow members of academic networks likely to be favorably predisposed, producing a bias against citing scholars who are critical or outside of those networks.

Moreover, strategic citations may be driven by calculations about which works editors and reviewers will expect to be listed in the bibliography. Journal articles are increasingly subject to fewer word limits. Omitting citations helps, but authors do not want to be criticized for missing key works. This fear encourages “drive-by citations”—that is, citing papers merely because similar papers cite those papers, regardless of their actual relevance. Anecdotally, scholars who are cited in this way often are those whose work is repeatedly cited erroneously for arguments they did not actually make.

Strategic citation leads scholars to cite authors of works already deemed important, not because of their relevance but rather because they are perceived as gatekeepers, hold key editorial positions, or reside in powerful departments. Junior scholars have expressed concerns that their submissions to journals will face rejection if they anger prominent scholars by criticizing or not mentioning their work. Like the biases discussed previously, strategic citation reproduces existing inequalities and disfavors underrepresented minorities, scholars from lower-ranked institutions, and those doing innovative work that does not fit neatly into existing literatures.

The disadvantages of GS do not foreclose its potential to serve as one indicator of scholarly impact, but we should understand the factors that affect decisions about whom to cite. If departments, deans, and granting institutions look at GS counts and external letter writers use them as a guide, then citation numbers effectively are being counted twice or more, thereby reducing the independence of external scholars’ judgment. This double-counting is particularly problematic when the data-generating process that leads to higher and lower citation counts has not yet been fully understood. Moreover, because a GS profile is a public signal, it can have a disproportionate effect on opinions because a person seeing it knows that others also see it (Chwe Reference Chwe2016).