Data Limitations in the SSCI Data

While some problems earlier identified with SSCI (see e.g., Cnudde 1986) have been fixed, there remain a number of limitations in using the SSCI data available online from the Web of Science web site. The most important of these is that of coping with identical last names, further compounded by the following: (a) the Web of Science does not give author's first names but only initials; (b) some authors are not consistent in how they list themselves (for example, one of the present authors used his middle initial for publications early in his career, but subsequently dropped it); (c) marriage or divorce (or simply choices about reclaiming heritage) may lead to changes in professional name; and (d) inconsistencies in how SSCI handles compound and hyphenated names make it especially hard to do reliable citation counts for scholars with multiple or hyphenated names (for example, citations to Bruce Bueno de Mesquita are listed under four different variants of his name). In order to cope with such difficulties we physically reviewed the SSCI listings for each of the 4,000+ scholars who taught at U.S. Ph.D.-granting institutions ca. 2002 and tried to use our knowledge of the profession to distinguish work by the scholar we were interested in from work by his or her namesakes. We recognize that mistakes have inevitably been made. We ask the forgiveness of any scholar whom we may have inadvertently slighted.

A second major problem with the SSCI data is how limited the information we have on co-authorships is. The SSCI has been fully updated to now identify co-authors for journal publications. These updates have not been applied to books; therefore first-authors continue to receive sole credit for co-authored (or co-edited) books. However, although the SSCI officially documents co-authors for journal articles, we have found that not every journal entry has been updated. Thus, while scholars now receive some credit for co-authored work, citations to co-authored work are still undoubtedly undercounted. (This point was made by a reviewer, and by others who read a draft of our paper.) This problem of course, has its greatest impact on authors who frequently co-author, and among those, on ones which use the alphabetical order convention and whose last names begin with letters later in the alphabet. Unfortunately, given the size of the data set we are dealing with, there is simply nothing we can do to remedy this problem. (For co-authors aggrieved by what they have just read, we might simply note that they are not alone. For example, one of the authors of this paper co-edited a 1994 book which received some 80 citations on which he was the second-listed co-editor. The first-listed scholar's last name begins with the letter D, and all citations went to that scholar.)

In sum, using the SSCI requires human ingenuity. We have tried very hard to clean the data, but we ask the reader's understanding of the limits of what is possible.

Using the SSCI Data to Create Rankings

One issue with use of the SSCI data is that, even when a full set of co-authors does get citation credit, an argument can be made that this overweights co-authored work, since multiple authors are getting credit for the same work. In this view, work which is co-authored should not yield full citation counts to each of the co-authors, and credit should be normalized so that the sum of the author citation counts is one for each co-authored article. Moreover, it has also been suggested to us that the relative credit awarded to each might depend in some fashion on the order of authorship. However, because of the form in which citation data comes on the Web of Science it is essentially impossible to use any rule for fractional weighting of co-authored work (even if one wanted to do so) without recoding each and every citation by hand to separate out singly-authored and co-authored work, and to distinguish second, third, fourth, etc. authors. We have thus, of necessity, treated a citation as a citation as a citation, despite the fact that this leads to inconsistencies in how earlier and later citations involving co-authorship are being treated.¹⁵

There are other issues related to assigning weights to citations in addition to the issue of whether to weight co-authored articles differently than singly authored articles.

One reviewer suggested that we ought to weight citations in “major” journals more heavily than citations in “minor” journals. In principle, we agree and, as noted in the text, there have been a number of ranking schemes that, at least in part, do limit themselves to citations (or publications) in “major” journals or seek to combine information about citations with information about where an author is published or cited (see e.g., Miller, Tien, and Peebler 1996). But, once we get past the APSR and perhaps the top two regional U.S. journals, we see great difficulties in getting agreement across subfields (and, especially across different methodological approaches) about what are the “top” journals. While there are rankings of political science journals that identify factors such as visibility in the political science community, reputation among those familiar with the journal, and various measures of journal impact, (a) these measures tap different dimensions, and it is not obvious how to combine and weight them for ranking purposes, and (b) if we limit ourselves to a citations in a relative handful of journals, this makes it harder to assess the overall impact of a scholar—especially since that may extend beyond citations in just political science journals. The bottom line is that the SSCI list of journals is a very extensive one (1,000+ in the social sciences as a whole) and it seemed a reasonably “neutral”—although far from perfect—way to identify the set of journals in which citations would “count.”¹⁷

A second additional issue in using citations to rank individual scholars is raised by advocates of the “h index” (Hirsch 2005) who propose to rank scholars not by their total citations but by the number of papers they have that are highly cited, where “highly cited” gets defined in a precise way reminiscent of the Lorenz curve. To use the “h index,” we take an author's publications and list them in order of their citation count. Then, counting down from the top, if the kth most cited publication has a citation count of h_k, then the h score equals k for the largest value of k such that h_k is greater than or equal to k. For example, if an author had written 30 papers or books and the first 20 most cited of these each received anywhere from 30 citations to 21 citations (i.e., the author's most cited paper or book had 30 citations while the 20th most cited paper or book had only 21 citations), and also the 21^st most frequently cited article had only 19 citations, then the h score would be 20. Indeed, the h score would remain 20 if the author's most cited item had received 300 citations; or 3,000 citations; or 30,000 citations. We strongly prefer simple summation to using h scores. These scores have very strange properties, e.g., someone who published only three books, each of which had citations in the thousands, but who had done nothing else, would receive an h score of 3 whereas someone who had published, say, exactly 20 articles, each of which got 20 citations, would receive an h score of 20. Such a ranking rule would tend to underrate the Karl Marx's of the world.

A third additional issue in using citations for ranking is whether to use cumulative citation data over the course of a scholar's lifetime, or to confine oneself to citations received in some more recent period. In preparing this paper, we generated data on cumulative citation counts broken down by decades,¹⁸

There are other potential problems with using SSCI citations for rankings. For example, the SSCI makes use of the citations that appear in academic journals, and does not include citations that appear in books or edited volumes. It is possible that this restriction might lead to some bias against particular styles of work which tend to be published as books rather than as journal articles, but that is a question that goes beyond the scope of the present work. (We suspect, however, that the books or articles which are heavily cited in books are roughly the same books or articles which are heavily cited in articles.) Additionally, some citations are to textbooks or computer manuals or to computer packages (or articles explaining them) while others are to more substantive research monographs and, in principle, we might want to distinguish those different types of references. But that is essentially impossible given our database limitations without attempting an impossibly huge task of classification. Thus, citations to each of these types of works are included in our counts. Finally, since not all citations are positive, and some may involve harsh criticism of an author's work, it might be nice to distinguish positive from negative citations. Still, the fact that some author bothers to criticize a piece is an indication that it is being taken seriously. In any case, whatever we might like to be able to do, given the size of the database we are dealing with and the limitations of the automated features of the SSCI online database, we are not in a position to make such distinctions even if we wished to do so. Thus, in sum, and of pragmatic necessity, is why we stick to the one citation equals one citation rule.

Data Issues in Identifying Who is to be Included

We take our data of who is eligible to be in the Political Science 400 from Graduate Faculty and Programs in Political Science (APSA 2000), supplemented by the 2002–2004 Directory of Political Science Faculty (APSA 2002) and the APSA Centennial Biographical Directory of Members (APSA 2000). As of 2002, there were 4,103 regular faculty members in the 132 Ph.D.-granting departments in political science in the U.S. For analyses in terms of departmental Ph.D. production, there were 362 (8%) faculty members for whom we did not have complete information regarding their educational background. Among those we identify as members of the Political Science 400 we are missing data on educational background for only one scholar.

The most important thing to note about our selection choice is who it excludes. While the faculty in the list of 400 include a few political science faculty trained outside the U.S., and even a few whose degrees are in a discipline other than political science, for reasons of manageability and consistency, the only faculty who were eligible for inclusion were those who have their primary affiliation in a U.S. Ph.D.-granting political science department (ca. 2002).

These restrictions can be very important ones.

They exclude all faculty who teach at a department which is not one of the 132 Ph.D. political science-granting departments in the U.S. (e.g., those who teach at a purely undergraduate-institution or whose highest degree is a masters), and those with jobs outside of academia (e.g., president of the World Bank), or jobs outside the U.S. Indeed, even, political science faculty teaching exclusively in professional schools are not included in our list. Thus, we know that some political science faculty teaching in the U.S. with whose citation visibility we are otherwise familiar, such as Pippa Norris (Kennedy School) or Michael Cohen (School of Information, University of Michigan), are not included in the current Political Science 400 list even though their citation counts would entitle them to this honor. Similarly, there are no doubt political scientists teaching at European universities (or in Canada or elsewhere) whose citation counts would put them into the top 400 ranks, but who are excluded because they are not teaching in graduate programs in the U.S.

The eligibility rules for the Political Science 400 also excluded those with adjunct or lectureship appointments, and faculty who are emeritus if they are no longer listed on faculty rosters. However, as several of those who have been kind enough to read an earlier draft of this essay have reminded us, the way departments treat emeriti is not at all consistent. Some departments continue to list faculty on their masthead long after they are no longer participating members of the department; indeed, perhaps long after they have left the state or even moved to teach at other institutions. We have used the simple rule that emeriti still treated as faculty members on the list provided by departments to APSA are eligible to be in the Political Science 400 regardless of when they might or might not have officially retired or where they might have retired to. In the third essay of this series, where we rank departments by the citations to their members, the decision to include emeriti still listed on departmental faculty lists provided to the APSA has some further implications for department rankings which we will discuss.

Perhaps the next most important problem in trying to decide eligibility has to do with scholars with joint appointments. Since we are interested in later using the citation counts to evaluate the success of Ph.D.-producing institutions in turning out Ph.D.s who go on to be highly cited, in cases of joint appointments, we have usually not counted as eligible someone whose Ph.D. was in a discipline other than political science (or whose other appointment is in a law school). In particular, we have tried to exclude courtesy appointments of faculty who are not really part of the host department. On the other hand, if there are faculty trained in other disciplines whose principal home appears now to be political science, we have tried to leave them eligible for membership in the Political Science 400.

We have tried to be consistent in our standards for inclusion and exclusion but, in many instances, deciding who should be in and who out has required a judgment call (sometimes based on looking at departmental or personal home pages and sometimes based on the personal knowledge of one or more of the present authors). For example, a faculty member at our home institution with a Ph.D. in political science and whose work is exclusively in political science who holds an appointment in our department that gives her departmental voting rights but whose primary appointment is in another interdisciplinary department was considered eligible for the list. But we also wish to acknowledge that our decisions do have major consequences for who was excluded. For example, after having done a preliminary version of the Political Science 400, we looked in more detail at those on this preliminary list, and faculty with J.D.s whose primary appointment appears to be in a discipline other than political science (e.g., law school faculty with public law interests such as Bruce Ackerman, Sanford Levinson, and Cass Sunstein; or economists such as Gordon Tullock) were excluded from eligibility, even though their citation count would otherwise entitle them to inclusion in the list.